2024-03-18T14:29:01.853 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-18T14:29:01.854 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} --owner scheduled_pdonnell@teuthology --name pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808 -- /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808/orig.config.yaml 2024-03-18T14:29:01.908 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T14:29:02.012 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808 branch: wip-batrick-testing-20240314.183243 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} email: pdonnell@redhat.com first_in_suite: false job_id: '7608808' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240314.183243 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - mons down, quorum - out of quorum - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240314.183243 sha1: 4e19f704afd9957ea657ab858f9ecf76401fef6f owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4770 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af sleep_before_teardown: 0 subset: 498/512 suite: fs suite_branch: wip-batrick-testing-20240314.183243 suite_path: /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: 4e19f704afd9957ea657ab858f9ecf76401fef6f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi076.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCWpALrBoQvuUiaJDQhKUlohaMGQ0wsdizGxtvPlhUYw5HE0o4PC0u6aro8nZNuevXVmRfSdO1afFoXllzPOKig= smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMNyDAZKMLog6PLFE+IwCS25pUFWSWLWrdAhKeZLMgjr53isI77fU6mxCqgGFl6o772X1hJly5rhSC+D7AAQl2E= smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDVzy4F7uac53A8VBi3YySOjaxKUtZbqhCTXnVvxMQelv1unfyzt+iUl5kXWs+w6dCcItRoOhni439BvlCWFM/Y= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-18_14:19:46 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-18T14:29:02.013 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa; will attempt to use it 2024-03-18T14:29:02.013 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks 2024-03-18T14:29:02.013 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-18T14:29:02.015 INFO:teuthology.task.internal:Checking packages... 2024-03-18T14:29:02.036 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af' 2024-03-18T14:29:02.036 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T14:29:02.036 INFO:teuthology.packaging:ref: None 2024-03-18T14:29:02.036 INFO:teuthology.packaging:tag: None 2024-03-18T14:29:02.036 INFO:teuthology.packaging:branch: wip-batrick-testing-20240314.183243 2024-03-18T14:29:02.036 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:29:02.037 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240314.183243 2024-03-18T14:29:02.179 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2162.g3d78f5ba 2024-03-18T14:29:02.181 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-18T14:29:02.188 INFO:teuthology.task.internal:no buildpackages task found 2024-03-18T14:29:02.188 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-18T14:29:02.208 INFO:teuthology.task.internal:Saving configuration 2024-03-18T14:29:02.228 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-18T14:29:02.234 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T14:29:02.259 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:21:09.421660', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCWpALrBoQvuUiaJDQhKUlohaMGQ0wsdizGxtvPlhUYw5HE0o4PC0u6aro8nZNuevXVmRfSdO1afFoXllzPOKig='} 2024-03-18T14:29:02.281 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:21:09.420710', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMNyDAZKMLog6PLFE+IwCS25pUFWSWLWrdAhKeZLMgjr53isI77fU6mxCqgGFl6o772X1hJly5rhSC+D7AAQl2E='} 2024-03-18T14:29:02.304 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:21:09.419531', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDVzy4F7uac53A8VBi3YySOjaxKUtZbqhCTXnVvxMQelv1unfyzt+iUl5kXWs+w6dCcItRoOhni439BvlCWFM/Y='} 2024-03-18T14:29:02.304 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-18T14:29:02.311 INFO:teuthology.task.internal:roles: ubuntu@smithi076.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-18T14:29:02.311 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-18T14:29:02.311 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-18T14:29:02.311 INFO:teuthology.run_tasks:Running task console_log... 2024-03-18T14:29:02.407 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fda3fcc2160>, signals=[15]) 2024-03-18T14:29:02.407 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-18T14:29:02.413 INFO:teuthology.task.internal:Opening connections... 2024-03-18T14:29:02.413 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi076.front.sepia.ceph.com 2024-03-18T14:29:02.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:29:02.505 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2024-03-18T14:29:02.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:29:02.594 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-03-18T14:29:02.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:29:02.680 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-18T14:29:02.687 DEBUG:teuthology.orchestra.run.smithi076:> uname -m 2024-03-18T14:29:02.704 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2024-03-18T14:29:02.704 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2024-03-18T14:29:02.761 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Stream" 2024-03-18T14:29:02.761 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="9" 2024-03-18T14:29:02.761 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos" 2024-03-18T14:29:02.761 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora" 2024-03-18T14:29:02.761 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="9" 2024-03-18T14:29:02.761 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el9" 2024-03-18T14:29:02.761 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T14:29:02.761 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31" 2024-03-18T14:29:02.762 INFO:teuthology.orchestra.run.smithi076.stdout:LOGO="fedora-logo-icon" 2024-03-18T14:29:02.762 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T14:29:02.762 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://centos.org/" 2024-03-18T14:29:02.762 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T14:29:02.762 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T14:29:02.762 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T14:29:02.762 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2024-03-18T14:29:02.788 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2024-03-18T14:29:02.805 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2024-03-18T14:29:02.806 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2024-03-18T14:29:02.862 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="CentOS Stream" 2024-03-18T14:29:02.862 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="9" 2024-03-18T14:29:02.863 INFO:teuthology.orchestra.run.smithi077.stdout:ID="centos" 2024-03-18T14:29:02.863 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="rhel fedora" 2024-03-18T14:29:02.863 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="9" 2024-03-18T14:29:02.863 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el9" 2024-03-18T14:29:02.863 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T14:29:02.863 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2024-03-18T14:29:02.863 INFO:teuthology.orchestra.run.smithi077.stdout:LOGO="fedora-logo-icon" 2024-03-18T14:29:02.863 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T14:29:02.863 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://centos.org/" 2024-03-18T14:29:02.863 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T14:29:02.863 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T14:29:02.863 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T14:29:02.863 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-03-18T14:29:02.888 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-03-18T14:29:02.906 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-03-18T14:29:02.906 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-03-18T14:29:02.963 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream" 2024-03-18T14:29:02.963 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="9" 2024-03-18T14:29:02.963 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos" 2024-03-18T14:29:02.964 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora" 2024-03-18T14:29:02.964 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="9" 2024-03-18T14:29:02.964 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el9" 2024-03-18T14:29:02.964 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T14:29:02.964 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31" 2024-03-18T14:29:02.964 INFO:teuthology.orchestra.run.smithi174.stdout:LOGO="fedora-logo-icon" 2024-03-18T14:29:02.964 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T14:29:02.964 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/" 2024-03-18T14:29:02.964 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T14:29:02.964 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T14:29:02.964 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T14:29:02.964 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-03-18T14:29:02.988 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-18T14:29:02.998 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-18T14:29:03.004 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-18T14:29:03.004 DEBUG:teuthology.orchestra.run.smithi076:> test '!' -e /home/ubuntu/cephtest 2024-03-18T14:29:03.006 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2024-03-18T14:29:03.009 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-03-18T14:29:03.025 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-18T14:29:03.031 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-18T14:29:03.031 DEBUG:teuthology.orchestra.run.smithi076:> test -z $(ls -A /var/lib/ceph) 2024-03-18T14:29:03.063 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2024-03-18T14:29:03.066 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-03-18T14:29:03.177 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-18T14:29:03.256 INFO:teuthology.run_tasks:Running task kernel... 2024-03-18T14:29:03.271 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-03-18T14:29:03.272 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-18T14:29:03.272 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-18T14:29:03.272 DEBUG:teuthology.orchestra.run.smithi076:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T14:29:03.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:29:03.288 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2024-03-18T14:29:03.345 INFO:teuthology.orchestra.run.smithi076.stdout:5.14.0-361.el9.x86_64 2024-03-18T14:29:03.345 INFO:teuthology.task.kernel:Running kernel on smithi076: 5.14.0-361.el9.x86_64 2024-03-18T14:29:03.346 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum install -y kernel 2024-03-18T14:29:05.519 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-18T14:29:07.153 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-18T14:29:08.335 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - BaseOS 9.6 MB/s | 8.0 MB 00:00 2024-03-18T14:29:21.652 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - AppStream 1.5 MB/s | 19 MB 00:12 2024-03-18T14:29:26.146 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - CRB 6.4 MB/s | 6.1 MB 00:00 2024-03-18T14:29:27.670 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-03-18T14:29:34.366 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 3.1 MB/s | 21 MB 00:06 2024-03-18T14:29:39.430 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 8.5 kB/s | 1.7 kB 00:00 2024-03-18T14:29:41.611 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-18T14:29:41.612 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-18T14:29:41.721 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-18T14:29:41.727 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-18T14:29:41.727 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2024-03-18T14:29:41.727 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-18T14:29:41.727 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-18T14:29:41.727 INFO:teuthology.orchestra.run.smithi076.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-18T14:29:41.727 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-03-18T14:29:41.728 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-18T14:29:41.728 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-18T14:29:41.728 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-18T14:29:41.728 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:29:41.728 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-18T14:29:41.728 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-18T14:29:41.728 INFO:teuthology.orchestra.run.smithi076.stdout:Install 4 Packages 2024-03-18T14:29:41.728 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:29:41.728 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 100 M 2024-03-18T14:29:41.728 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 123 M 2024-03-18T14:29:41.728 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-18T14:29:44.107 INFO:teuthology.orchestra.run.smithi076.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 2.6 MB/s | 6.0 MB 00:02 2024-03-18T14:29:53.247 INFO:teuthology.orchestra.run.smithi076.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 1.8 MB/s | 21 MB 00:11 2024-03-18T14:30:08.055 INFO:teuthology.orchestra.run.smithi076.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 1.5 MB/s | 39 MB 00:26 2024-03-18T14:30:09.247 INFO:teuthology.orchestra.run.smithi076.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 1.3 MB/s | 33 MB 00:25 2024-03-18T14:30:09.247 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-18T14:30:09.247 INFO:teuthology.orchestra.run.smithi076.stdout:Total 3.6 MB/s | 100 MB 00:27 2024-03-18T14:30:10.135 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-18T14:30:10.533 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-18T14:30:10.533 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-18T14:30:11.912 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-18T14:30:11.913 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-18T14:30:14.892 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-18T14:30:15.555 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-18T14:30:15.686 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T14:30:18.493 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T14:30:18.684 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T14:30:23.389 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T14:30:23.594 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:30:28.354 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:30:57.699 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:30:57.699 INFO:teuthology.orchestra.run.smithi076.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-18T14:30:57.699 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:30:57.784 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:30:58.283 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:30:58.283 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-18T14:30:58.283 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T14:30:58.283 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T14:30:58.724 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:30:58.724 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:30:58.724 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-18T14:30:58.724 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-18T14:30:58.724 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-18T14:30:58.724 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-18T14:30:58.724 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-18T14:30:58.724 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:30:58.724 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-18T14:30:59.236 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T14:30:59.377 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-18T14:30:59.378 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-18T14:30:59.378 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-18T14:30:59.378 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-18T14:30:59.378 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-18T14:30:59.378 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-18T14:30:59.378 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi076.front.sepia.ceph.com, path=None, version=distro) 2024-03-18T14:30:59.379 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T14:30:59.498 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-18T14:30:59.498 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-18T14:30:59.498 DEBUG:teuthology.orchestra.run.smithi076:> sudo rpm -qi grub2-tools 2024-03-18T14:30:59.540 INFO:teuthology.orchestra.run.smithi076.stdout:Name : grub2-tools 2024-03-18T14:30:59.540 INFO:teuthology.orchestra.run.smithi076.stdout:Epoch : 1 2024-03-18T14:30:59.540 INFO:teuthology.orchestra.run.smithi076.stdout:Version : 2.06 2024-03-18T14:30:59.540 INFO:teuthology.orchestra.run.smithi076.stdout:Release : 68.el9 2024-03-18T14:30:59.540 INFO:teuthology.orchestra.run.smithi076.stdout:Architecture: x86_64 2024-03-18T14:30:59.540 INFO:teuthology.orchestra.run.smithi076.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-18T14:30:59.540 INFO:teuthology.orchestra.run.smithi076.stdout:Group : Unspecified 2024-03-18T14:30:59.541 INFO:teuthology.orchestra.run.smithi076.stdout:Size : 8264254 2024-03-18T14:30:59.541 INFO:teuthology.orchestra.run.smithi076.stdout:License : GPLv3+ 2024-03-18T14:30:59.541 INFO:teuthology.orchestra.run.smithi076.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-18T14:30:59.541 INFO:teuthology.orchestra.run.smithi076.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-18T14:30:59.541 INFO:teuthology.orchestra.run.smithi076.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-18T14:30:59.541 INFO:teuthology.orchestra.run.smithi076.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-18T14:30:59.541 INFO:teuthology.orchestra.run.smithi076.stdout:Packager : builder@centos.org 2024-03-18T14:30:59.541 INFO:teuthology.orchestra.run.smithi076.stdout:Vendor : CentOS 2024-03-18T14:30:59.541 INFO:teuthology.orchestra.run.smithi076.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-18T14:30:59.541 INFO:teuthology.orchestra.run.smithi076.stdout:Summary : Support tools for GRUB. 2024-03-18T14:30:59.541 INFO:teuthology.orchestra.run.smithi076.stdout:Description : 2024-03-18T14:30:59.541 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:30:59.541 INFO:teuthology.orchestra.run.smithi076.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-18T14:30:59.541 INFO:teuthology.orchestra.run.smithi076.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-18T14:30:59.541 INFO:teuthology.orchestra.run.smithi076.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-18T14:30:59.542 INFO:teuthology.orchestra.run.smithi076.stdout:hardware devices. 2024-03-18T14:30:59.542 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:30:59.542 INFO:teuthology.orchestra.run.smithi076.stdout:This subpackage provides tools for support of all platforms. 2024-03-18T14:30:59.543 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-18T14:30:59.543 INFO:teuthology.task.kernel:Updating grub on smithi076 to boot 5.14.0-427.el9.x86_64 2024-03-18T14:30:59.543 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-18T14:31:00.201 INFO:teuthology.orchestra.run.smithi076.stderr:Generating grub configuration file ... 2024-03-18T14:31:01.582 INFO:teuthology.orchestra.run.smithi076.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-18T14:31:01.628 INFO:teuthology.orchestra.run.smithi076.stderr:done 2024-03-18T14:31:01.630 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2024-03-18T14:31:01.645 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.2vS3BGP7jX 2024-03-18T14:31:01.645 DEBUG:teuthology.orchestra.run.smithi076:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2vS3BGP7jX 2024-03-18T14:31:01.714 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.2vS3BGP7jX 2024-03-18T14:31:01.852 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.2vS3BGP7jX is 6KB 2024-03-18T14:31:01.900 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.2vS3BGP7jX 2024-03-18T14:31:01.914 DEBUG:teuthology.orchestra.run.smithi076:> sudo /bin/ls /boot/loader/entries || true 2024-03-18T14:31:01.978 INFO:teuthology.orchestra.run.smithi076.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-18T14:31:01.978 INFO:teuthology.orchestra.run.smithi076.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-18T14:31:01.979 INFO:teuthology.orchestra.run.smithi076.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-18T14:31:01.979 INFO:teuthology.orchestra.run.smithi076.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-18T14:31:01.980 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-18T14:31:02.139 DEBUG:teuthology.orchestra.run.smithi076:> sudo shutdown -r now 2024-03-18T14:31:32.142 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-18T14:31:32.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-18T14:31:32.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:31:50.562 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-03-18T14:31:59.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-18T14:31:59.565 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:32:02.622 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-03-18T14:32:14.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-18T14:32:14.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:32:17.694 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-03-18T14:32:32.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-18T14:32:32.696 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:33:04.637 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-03-18T14:33:05.001 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-18T14:33:05.001 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-427.el9.x86_64"... 2024-03-18T14:33:05.001 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2024-03-18T14:33:05.017 INFO:teuthology.orchestra.run.smithi076.stdout:5.14.0-427.el9.x86_64 2024-03-18T14:33:05.017 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-18T14:33:05.017 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-18T14:33:05.018 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-18T14:33:06.019 DEBUG:teuthology.parallel:result is None 2024-03-18T14:33:06.019 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-18T14:33:06.026 INFO:teuthology.task.internal:Creating test directory... 2024-03-18T14:33:06.027 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T14:33:06.029 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T14:33:06.032 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T14:33:06.050 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-18T14:33:06.056 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-18T14:33:06.069 INFO:teuthology.task.internal:Creating archive directory... 2024-03-18T14:33:06.070 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T14:33:06.087 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T14:33:06.090 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T14:33:06.143 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-18T14:33:06.151 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-18T14:33:06.152 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T14:33:06.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:33:06.189 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T14:33:06.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:33:06.208 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T14:33:06.210 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T14:33:06.233 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T14:33:06.271 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T14:33:06.272 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T14:33:06.288 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T14:33:06.289 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T14:33:06.305 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T14:33:06.347 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T14:33:06.349 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-18T14:33:06.356 INFO:teuthology.task.internal:Configuring sudo... 2024-03-18T14:33:06.356 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T14:33:06.359 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T14:33:06.361 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T14:33:06.391 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-18T14:33:06.405 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-18T14:33:06.405 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T14:33:06.432 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T14:33:06.435 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T14:33:06.451 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T14:33:06.531 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T14:33:06.592 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T14:33:06.636 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T14:33:06.679 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T14:33:06.679 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T14:33:06.747 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T14:33:06.794 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T14:33:06.847 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T14:33:06.936 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T14:33:06.979 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-18T14:33:06.979 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T14:33:07.048 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T14:33:07.093 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T14:33:07.148 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T14:33:07.234 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T14:33:07.278 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-18T14:33:07.278 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T14:33:07.347 DEBUG:teuthology.orchestra.run.smithi076:> sudo service rsyslog restart 2024-03-18T14:33:07.350 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2024-03-18T14:33:07.352 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-03-18T14:33:07.398 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T14:33:07.405 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T14:33:07.429 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T14:33:07.909 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-18T14:33:07.918 INFO:teuthology.task.internal:Starting timer... 2024-03-18T14:33:07.918 INFO:teuthology.run_tasks:Running task pcp... 2024-03-18T14:33:07.946 INFO:teuthology.run_tasks:Running task selinux... 2024-03-18T14:33:07.957 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-18T14:33:07.957 DEBUG:teuthology.orchestra.run.smithi076:> sudo service auditd rotate 2024-03-18T14:33:08.026 INFO:teuthology.orchestra.run.smithi076.stdout:Rotating logs: 2024-03-18T14:33:08.028 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2024-03-18T14:33:08.078 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2024-03-18T14:33:08.080 DEBUG:teuthology.orchestra.run.smithi174:> sudo service auditd rotate 2024-03-18T14:33:08.129 INFO:teuthology.orchestra.run.smithi174.stdout:Rotating logs: 2024-03-18T14:33:08.131 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-18T14:33:08.131 DEBUG:teuthology.orchestra.run.smithi076:> /usr/sbin/getenforce 2024-03-18T14:33:08.154 INFO:teuthology.orchestra.run.smithi076.stdout:Permissive 2024-03-18T14:33:08.155 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2024-03-18T14:33:08.179 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2024-03-18T14:33:08.179 DEBUG:teuthology.orchestra.run.smithi174:> /usr/sbin/getenforce 2024-03-18T14:33:08.199 INFO:teuthology.orchestra.run.smithi174.stdout:Permissive 2024-03-18T14:33:08.199 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi076.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi077.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi174.front.sepia.ceph.com': 'permissive'} 2024-03-18T14:33:08.199 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T14:33:08.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:33:08.229 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T14:33:08.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:33:08.258 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T14:33:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:33:08.287 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-18T14:33:08.287 DEBUG:teuthology.orchestra.run.smithi076:> sudo /usr/sbin/setenforce permissive 2024-03-18T14:33:08.323 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2024-03-18T14:33:08.356 DEBUG:teuthology.orchestra.run.smithi174:> sudo /usr/sbin/setenforce permissive 2024-03-18T14:33:08.393 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-18T14:33:08.403 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-18T14:33:08.423 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-18T14:33:08.424 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi076.front.sepia.ceph.com,smithi077.front.sepia.ceph.com,smithi174.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-18T14:38:21.725 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi076.front.sepia.ceph.com'), Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi174.front.sepia.ceph.com')] 2024-03-18T14:38:21.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-18T14:38:21.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:38:21.819 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-03-18T14:38:21.914 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-18T14:38:21.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-18T14:38:21.915 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:38:22.010 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-03-18T14:38:22.081 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-18T14:38:22.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-18T14:38:22.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:38:22.173 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-03-18T14:38:22.240 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-18T14:38:22.241 INFO:teuthology.run_tasks:Running task clock... 2024-03-18T14:38:22.251 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-18T14:38:22.251 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T14:38:22.252 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T14:38:22.254 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T14:38:22.254 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T14:38:22.256 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T14:38:22.256 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T14:38:22.284 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T14:38:22.285 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T14:38:22.301 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T14:38:22.302 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T14:38:22.311 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T14:38:22.328 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T14:38:22.330 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: ntpd: command not found 2024-03-18T14:38:22.331 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2024-03-18T14:38:22.344 INFO:teuthology.orchestra.run.smithi076.stdout:506 Cannot talk to daemon 2024-03-18T14:38:22.345 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2024-03-18T14:38:22.357 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: ntpd: command not found 2024-03-18T14:38:22.361 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T14:38:22.361 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T14:38:22.371 INFO:teuthology.orchestra.run.smithi174.stdout:506 Cannot talk to daemon 2024-03-18T14:38:22.377 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T14:38:22.377 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T14:38:22.387 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T14:38:22.403 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T14:38:22.421 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-03-18T14:38:22.424 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T14:38:22.424 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-03-18T14:38:22.424 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:22.424 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:22.425 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:22.425 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:22.427 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-03-18T14:38:22.431 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T14:38:22.431 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-03-18T14:38:22.431 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:22.431 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:22.431 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:22.431 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:22.447 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-03-18T14:38:22.451 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T14:38:22.451 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-03-18T14:38:22.451 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:22.451 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:22.451 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:22.451 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:22.452 INFO:teuthology.run_tasks:Running task install... 2024-03-18T14:38:22.461 DEBUG:teuthology.task.install:project ceph 2024-03-18T14:38:22.461 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}} 2024-03-18T14:38:22.461 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'} 2024-03-18T14:38:22.462 INFO:teuthology.task.install:Using flavor: default 2024-03-18T14:38:22.470 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-18T14:38:22.470 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-18T14:38:22.470 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T14:38:22.471 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:38:22.473 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T14:38:22.473 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:38:22.474 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T14:38:22.474 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:38:22.620 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T14:38:22.620 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T14:38:22.623 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T14:38:22.623 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T14:38:22.731 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T14:38:22.732 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T14:38:22.736 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T14:38:22.737 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-18T14:38:22.737 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T14:38:22.764 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T14:38:22.764 DEBUG:teuthology.orchestra.run.smithi174:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T14:38:22.833 DEBUG:teuthology.orchestra.run.smithi174:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T14:38:22.859 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T14:38:22.859 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-18T14:38:22.859 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T14:38:22.888 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T14:38:22.888 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T14:38:22.912 DEBUG:teuthology.orchestra.run.smithi174:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T14:38:22.957 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T14:38:22.977 INFO:teuthology.orchestra.run.smithi174.stdout:check_obsoletes = 1 2024-03-18T14:38:22.983 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum clean all 2024-03-18T14:38:22.989 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T14:38:22.989 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T14:38:22.989 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T14:38:23.020 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T14:38:23.020 DEBUG:teuthology.orchestra.run.smithi076:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T14:38:23.038 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T14:38:23.090 DEBUG:teuthology.orchestra.run.smithi076:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T14:38:23.104 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2024-03-18T14:38:23.106 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2024-03-18T14:38:23.169 DEBUG:teuthology.orchestra.run.smithi076:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T14:38:23.217 INFO:teuthology.orchestra.run.smithi174.stdout:74 files removed 2024-03-18T14:38:23.237 INFO:teuthology.orchestra.run.smithi076.stdout:check_obsoletes = 1 2024-03-18T14:38:23.239 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum clean all 2024-03-18T14:38:23.243 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T14:38:23.339 INFO:teuthology.orchestra.run.smithi077.stdout:74 files removed 2024-03-18T14:38:23.366 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T14:38:23.477 INFO:teuthology.orchestra.run.smithi076.stdout:74 files removed 2024-03-18T14:38:23.504 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T14:38:24.552 INFO:teuthology.orchestra.run.smithi174.stdout:ceph packages for x86_64 81 kB/s | 83 kB 00:01 2024-03-18T14:38:24.653 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 83 kB/s | 83 kB 00:00 2024-03-18T14:38:24.885 INFO:teuthology.orchestra.run.smithi076.stdout:ceph packages for x86_64 78 kB/s | 83 kB 00:01 2024-03-18T14:38:25.060 INFO:teuthology.orchestra.run.smithi174.stdout:ceph noarch packages 21 kB/s | 10 kB 00:00 2024-03-18T14:38:25.121 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 24 kB/s | 10 kB 00:00 2024-03-18T14:38:25.452 INFO:teuthology.orchestra.run.smithi076.stdout:ceph noarch packages 19 kB/s | 10 kB 00:00 2024-03-18T14:38:25.594 INFO:teuthology.orchestra.run.smithi174.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-03-18T14:38:25.763 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 3.1 kB/s | 1.9 kB 00:00 2024-03-18T14:38:26.027 INFO:teuthology.orchestra.run.smithi076.stdout:ceph source packages 3.5 kB/s | 1.9 kB 00:00 2024-03-18T14:38:26.113 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-18T14:38:26.348 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-18T14:38:26.705 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-18T14:38:27.488 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-03-18T14:38:27.699 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - BaseOS 7.7 MB/s | 2.0 MB 00:00 2024-03-18T14:38:28.096 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-18T14:38:28.271 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-18T14:38:28.689 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-18T14:38:28.912 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-18T14:38:34.039 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 4.0 MB/s | 19 MB 00:04 2024-03-18T14:38:38.132 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 19 MB 00:08 2024-03-18T14:38:38.879 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - AppStream 2.1 MB/s | 19 MB 00:08 2024-03-18T14:38:42.730 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-18T14:38:44.164 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 938 kB/s | 6.1 MB 00:06 2024-03-18T14:38:45.322 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-18T14:38:45.573 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-03-18T14:38:46.315 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - CRB 1.3 MB/s | 6.1 MB 00:04 2024-03-18T14:38:47.324 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-18T14:38:47.632 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - Extras packages 86 kB/s | 16 kB 00:00 2024-03-18T14:38:47.977 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 9.0 MB/s | 21 MB 00:02 2024-03-18T14:38:48.767 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-18T14:38:52.374 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 7.5 kB/s | 1.7 kB 00:00 2024-03-18T14:38:52.923 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-18T14:38:53.892 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 9.3 kB/s | 1.7 kB 00:00 2024-03-18T14:38:54.852 INFO:teuthology.orchestra.run.smithi076.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T14:38:54.852 INFO:teuthology.orchestra.run.smithi076.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T14:38:54.865 INFO:teuthology.orchestra.run.smithi076.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T14:38:54.866 INFO:teuthology.orchestra.run.smithi076.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T14:38:54.867 INFO:teuthology.orchestra.run.smithi076.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T14:38:54.868 INFO:teuthology.orchestra.run.smithi076.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T14:38:54.869 INFO:teuthology.orchestra.run.smithi076.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T14:38:54.872 INFO:teuthology.orchestra.run.smithi076.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T14:38:54.873 INFO:teuthology.orchestra.run.smithi076.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T14:38:54.954 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-18T14:38:54.974 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================================ 2024-03-18T14:38:54.975 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-03-18T14:38:54.975 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================================ 2024-03-18T14:38:54.975 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-18T14:38:54.975 INFO:teuthology.orchestra.run.smithi076.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T14:38:54.975 INFO:teuthology.orchestra.run.smithi076.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T14:38:54.975 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T14:38:54.975 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T14:38:54.975 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T14:38:54.975 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T14:38:54.975 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T14:38:54.975 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T14:38:54.975 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T14:38:54.976 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T14:38:54.976 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T14:38:54.976 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T14:38:54.976 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T14:38:54.976 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T14:38:54.976 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T14:38:54.976 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T14:38:54.976 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T14:38:54.976 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T14:38:54.976 INFO:teuthology.orchestra.run.smithi076.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T14:38:54.976 INFO:teuthology.orchestra.run.smithi076.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T14:38:54.976 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T14:38:54.976 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T14:38:54.977 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T14:38:54.977 INFO:teuthology.orchestra.run.smithi076.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T14:38:54.977 INFO:teuthology.orchestra.run.smithi076.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T14:38:54.977 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T14:38:54.977 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T14:38:54.977 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T14:38:54.977 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T14:38:54.977 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T14:38:54.977 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T14:38:54.977 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T14:38:54.977 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T14:38:54.977 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T14:38:54.977 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T14:38:54.977 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T14:38:54.978 INFO:teuthology.orchestra.run.smithi076.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T14:38:54.978 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrading: 2024-03-18T14:38:54.978 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T14:38:54.978 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T14:38:54.978 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T14:38:54.978 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T14:38:54.978 INFO:teuthology.orchestra.run.smithi076.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T14:38:54.978 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T14:38:54.978 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T14:38:54.978 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T14:38:54.978 INFO:teuthology.orchestra.run.smithi076.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T14:38:54.978 INFO:teuthology.orchestra.run.smithi076.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T14:38:54.978 INFO:teuthology.orchestra.run.smithi076.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T14:38:54.979 INFO:teuthology.orchestra.run.smithi076.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T14:38:54.979 INFO:teuthology.orchestra.run.smithi076.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T14:38:54.979 INFO:teuthology.orchestra.run.smithi076.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T14:38:54.979 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T14:38:54.979 INFO:teuthology.orchestra.run.smithi076.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T14:38:54.979 INFO:teuthology.orchestra.run.smithi076.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T14:38:54.979 INFO:teuthology.orchestra.run.smithi076.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T14:38:54.979 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T14:38:54.979 INFO:teuthology.orchestra.run.smithi076.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T14:38:54.979 INFO:teuthology.orchestra.run.smithi076.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T14:38:54.979 INFO:teuthology.orchestra.run.smithi076.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T14:38:54.979 INFO:teuthology.orchestra.run.smithi076.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T14:38:54.979 INFO:teuthology.orchestra.run.smithi076.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T14:38:54.980 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-03-18T14:38:54.980 INFO:teuthology.orchestra.run.smithi076.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T14:38:54.980 INFO:teuthology.orchestra.run.smithi076.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T14:38:54.980 INFO:teuthology.orchestra.run.smithi076.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T14:38:54.980 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T14:38:54.980 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T14:38:54.980 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T14:38:54.980 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T14:38:54.980 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T14:38:54.980 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T14:38:54.980 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T14:38:54.980 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T14:38:54.980 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T14:38:54.980 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T14:38:54.980 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T14:38:54.981 INFO:teuthology.orchestra.run.smithi076.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T14:38:54.981 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T14:38:54.981 INFO:teuthology.orchestra.run.smithi076.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T14:38:54.981 INFO:teuthology.orchestra.run.smithi076.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T14:38:54.981 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T14:38:54.981 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T14:38:54.981 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T14:38:54.981 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T14:38:54.981 INFO:teuthology.orchestra.run.smithi076.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T14:38:54.981 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T14:38:54.981 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T14:38:54.981 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T14:38:54.981 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T14:38:54.981 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T14:38:54.982 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T14:38:54.982 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T14:38:54.982 INFO:teuthology.orchestra.run.smithi076.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T14:38:54.982 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T14:38:54.982 INFO:teuthology.orchestra.run.smithi076.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T14:38:54.982 INFO:teuthology.orchestra.run.smithi076.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T14:38:54.982 INFO:teuthology.orchestra.run.smithi076.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T14:38:54.982 INFO:teuthology.orchestra.run.smithi076.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T14:38:54.982 INFO:teuthology.orchestra.run.smithi076.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T14:38:54.982 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T14:38:54.982 INFO:teuthology.orchestra.run.smithi076.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T14:38:54.982 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T14:38:54.982 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T14:38:54.982 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T14:38:54.983 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T14:38:54.983 INFO:teuthology.orchestra.run.smithi076.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T14:38:54.983 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T14:38:54.983 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T14:38:54.983 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T14:38:54.983 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T14:38:54.983 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T14:38:54.983 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T14:38:54.983 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T14:38:54.983 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T14:38:54.983 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T14:38:54.983 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T14:38:54.983 INFO:teuthology.orchestra.run.smithi076.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T14:38:54.983 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T14:38:54.983 INFO:teuthology.orchestra.run.smithi076.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T14:38:54.984 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T14:38:54.984 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T14:38:54.984 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T14:38:54.984 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T14:38:54.984 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T14:38:54.984 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T14:38:54.984 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T14:38:54.984 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T14:38:54.984 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T14:38:54.984 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T14:38:54.984 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T14:38:54.984 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T14:38:54.984 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T14:38:54.984 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T14:38:54.984 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T14:38:54.985 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T14:38:54.985 INFO:teuthology.orchestra.run.smithi076.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T14:38:54.985 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T14:38:54.985 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T14:38:54.985 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T14:38:54.985 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T14:38:54.985 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T14:38:54.985 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T14:38:54.985 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T14:38:54.985 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T14:38:54.985 INFO:teuthology.orchestra.run.smithi076.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T14:38:54.985 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T14:38:54.985 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T14:38:54.985 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T14:38:54.985 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T14:38:54.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T14:38:54.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T14:38:54.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T14:38:54.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T14:38:54.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T14:38:54.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T14:38:54.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T14:38:54.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T14:38:54.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T14:38:54.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T14:38:54.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T14:38:54.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T14:38:54.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T14:38:54.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T14:38:54.987 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T14:38:54.987 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T14:38:54.987 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T14:38:54.987 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T14:38:54.987 INFO:teuthology.orchestra.run.smithi076.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T14:38:54.987 INFO:teuthology.orchestra.run.smithi076.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T14:38:54.987 INFO:teuthology.orchestra.run.smithi076.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T14:38:54.987 INFO:teuthology.orchestra.run.smithi076.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T14:38:54.987 INFO:teuthology.orchestra.run.smithi076.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T14:38:54.987 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T14:38:54.987 INFO:teuthology.orchestra.run.smithi076.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T14:38:54.987 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2024-03-18T14:38:54.987 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T14:38:54.987 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:38:54.988 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-18T14:38:54.988 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================================ 2024-03-18T14:38:54.988 INFO:teuthology.orchestra.run.smithi076.stdout:Install 150 Packages 2024-03-18T14:38:54.988 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrade 24 Packages 2024-03-18T14:38:54.988 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:38:54.988 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 224 M 2024-03-18T14:38:54.988 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-18T14:38:55.479 INFO:teuthology.orchestra.run.smithi174.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T14:38:55.480 INFO:teuthology.orchestra.run.smithi174.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T14:38:55.494 INFO:teuthology.orchestra.run.smithi174.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T14:38:55.495 INFO:teuthology.orchestra.run.smithi174.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T14:38:55.497 INFO:teuthology.orchestra.run.smithi174.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T14:38:55.497 INFO:teuthology.orchestra.run.smithi174.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T14:38:55.499 INFO:teuthology.orchestra.run.smithi174.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T14:38:55.502 INFO:teuthology.orchestra.run.smithi174.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T14:38:55.503 INFO:teuthology.orchestra.run.smithi174.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T14:38:55.584 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-18T14:38:55.607 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================================ 2024-03-18T14:38:55.607 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-03-18T14:38:55.607 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================================ 2024-03-18T14:38:55.607 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-18T14:38:55.607 INFO:teuthology.orchestra.run.smithi174.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T14:38:55.608 INFO:teuthology.orchestra.run.smithi174.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T14:38:55.608 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T14:38:55.608 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T14:38:55.608 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T14:38:55.608 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T14:38:55.608 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T14:38:55.608 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T14:38:55.608 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T14:38:55.608 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T14:38:55.608 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T14:38:55.608 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T14:38:55.608 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T14:38:55.608 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T14:38:55.609 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T14:38:55.609 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T14:38:55.609 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T14:38:55.609 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T14:38:55.609 INFO:teuthology.orchestra.run.smithi174.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T14:38:55.609 INFO:teuthology.orchestra.run.smithi174.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T14:38:55.609 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T14:38:55.609 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T14:38:55.609 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T14:38:55.609 INFO:teuthology.orchestra.run.smithi174.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T14:38:55.609 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T14:38:55.609 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T14:38:55.609 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T14:38:55.610 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T14:38:55.610 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T14:38:55.610 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T14:38:55.610 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T14:38:55.610 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T14:38:55.610 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T14:38:55.610 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T14:38:55.610 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T14:38:55.610 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T14:38:55.610 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T14:38:55.610 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrading: 2024-03-18T14:38:55.610 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T14:38:55.611 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T14:38:55.611 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T14:38:55.611 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T14:38:55.611 INFO:teuthology.orchestra.run.smithi174.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T14:38:55.611 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T14:38:55.611 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T14:38:55.611 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T14:38:55.611 INFO:teuthology.orchestra.run.smithi174.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T14:38:55.611 INFO:teuthology.orchestra.run.smithi174.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T14:38:55.611 INFO:teuthology.orchestra.run.smithi174.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T14:38:55.611 INFO:teuthology.orchestra.run.smithi174.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T14:38:55.611 INFO:teuthology.orchestra.run.smithi174.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T14:38:55.611 INFO:teuthology.orchestra.run.smithi174.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T14:38:55.611 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T14:38:55.611 INFO:teuthology.orchestra.run.smithi174.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T14:38:55.612 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T14:38:55.612 INFO:teuthology.orchestra.run.smithi174.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T14:38:55.612 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T14:38:55.612 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T14:38:55.612 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T14:38:55.612 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T14:38:55.612 INFO:teuthology.orchestra.run.smithi174.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T14:38:55.612 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T14:38:55.612 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-03-18T14:38:55.612 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T14:38:55.612 INFO:teuthology.orchestra.run.smithi174.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T14:38:55.612 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T14:38:55.612 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T14:38:55.613 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T14:38:55.613 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T14:38:55.613 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T14:38:55.613 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T14:38:55.613 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T14:38:55.613 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T14:38:55.613 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T14:38:55.613 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T14:38:55.613 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T14:38:55.613 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T14:38:55.613 INFO:teuthology.orchestra.run.smithi174.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T14:38:55.613 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T14:38:55.613 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T14:38:55.613 INFO:teuthology.orchestra.run.smithi174.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T14:38:55.614 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T14:38:55.614 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T14:38:55.614 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T14:38:55.614 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T14:38:55.614 INFO:teuthology.orchestra.run.smithi174.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T14:38:55.614 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T14:38:55.614 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T14:38:55.614 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T14:38:55.614 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T14:38:55.614 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T14:38:55.614 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T14:38:55.614 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T14:38:55.614 INFO:teuthology.orchestra.run.smithi174.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T14:38:55.614 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T14:38:55.615 INFO:teuthology.orchestra.run.smithi174.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T14:38:55.615 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T14:38:55.615 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T14:38:55.615 INFO:teuthology.orchestra.run.smithi174.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T14:38:55.615 INFO:teuthology.orchestra.run.smithi174.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T14:38:55.615 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T14:38:55.615 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T14:38:55.615 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T14:38:55.615 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T14:38:55.615 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T14:38:55.615 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T14:38:55.615 INFO:teuthology.orchestra.run.smithi174.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T14:38:55.615 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T14:38:55.615 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T14:38:55.615 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T14:38:55.616 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T14:38:55.616 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T14:38:55.616 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T14:38:55.616 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T14:38:55.616 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T14:38:55.616 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T14:38:55.616 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T14:38:55.616 INFO:teuthology.orchestra.run.smithi174.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T14:38:55.616 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T14:38:55.616 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T14:38:55.616 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T14:38:55.616 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T14:38:55.616 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T14:38:55.616 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T14:38:55.617 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T14:38:55.617 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T14:38:55.617 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T14:38:55.617 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T14:38:55.617 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T14:38:55.617 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T14:38:55.617 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T14:38:55.617 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T14:38:55.617 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T14:38:55.617 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T14:38:55.617 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T14:38:55.617 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T14:38:55.617 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T14:38:55.617 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T14:38:55.617 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T14:38:55.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T14:38:55.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T14:38:55.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T14:38:55.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T14:38:55.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T14:38:55.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T14:38:55.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T14:38:55.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T14:38:55.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T14:38:55.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T14:38:55.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T14:38:55.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T14:38:55.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T14:38:55.619 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T14:38:55.619 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T14:38:55.619 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T14:38:55.619 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T14:38:55.619 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T14:38:55.619 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T14:38:55.619 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T14:38:55.619 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T14:38:55.619 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T14:38:55.619 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T14:38:55.619 INFO:teuthology.orchestra.run.smithi174.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T14:38:55.619 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T14:38:55.619 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T14:38:55.620 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T14:38:55.620 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T14:38:55.620 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T14:38:55.620 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T14:38:55.620 INFO:teuthology.orchestra.run.smithi174.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T14:38:55.620 INFO:teuthology.orchestra.run.smithi174.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T14:38:55.620 INFO:teuthology.orchestra.run.smithi174.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T14:38:55.620 INFO:teuthology.orchestra.run.smithi174.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T14:38:55.620 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T14:38:55.620 INFO:teuthology.orchestra.run.smithi174.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T14:38:55.620 INFO:teuthology.orchestra.run.smithi174.stdout:Installing weak dependencies: 2024-03-18T14:38:55.620 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T14:38:55.620 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:38:55.620 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-18T14:38:55.621 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================================ 2024-03-18T14:38:55.621 INFO:teuthology.orchestra.run.smithi174.stdout:Install 150 Packages 2024-03-18T14:38:55.621 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrade 24 Packages 2024-03-18T14:38:55.621 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:38:55.621 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 224 M 2024-03-18T14:38:55.621 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-18T14:38:56.346 INFO:teuthology.orchestra.run.smithi076.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 10 kB/s | 6.6 kB 00:00 2024-03-18T14:38:56.353 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T14:38:56.354 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T14:38:56.366 INFO:teuthology.orchestra.run.smithi077.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T14:38:56.367 INFO:teuthology.orchestra.run.smithi077.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T14:38:56.367 INFO:teuthology.orchestra.run.smithi077.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T14:38:56.368 INFO:teuthology.orchestra.run.smithi077.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T14:38:56.370 INFO:teuthology.orchestra.run.smithi077.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T14:38:56.372 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T14:38:56.373 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T14:38:56.455 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-03-18T14:38:56.464 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================ 2024-03-18T14:38:56.464 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-03-18T14:38:56.464 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================ 2024-03-18T14:38:56.464 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-03-18T14:38:56.465 INFO:teuthology.orchestra.run.smithi077.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T14:38:56.465 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T14:38:56.465 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T14:38:56.465 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T14:38:56.465 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T14:38:56.465 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T14:38:56.465 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T14:38:56.465 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T14:38:56.465 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T14:38:56.465 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T14:38:56.465 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T14:38:56.465 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T14:38:56.466 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T14:38:56.466 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T14:38:56.466 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T14:38:56.466 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T14:38:56.466 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T14:38:56.466 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T14:38:56.466 INFO:teuthology.orchestra.run.smithi077.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T14:38:56.466 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T14:38:56.466 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T14:38:56.466 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T14:38:56.466 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T14:38:56.466 INFO:teuthology.orchestra.run.smithi077.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T14:38:56.466 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T14:38:56.466 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T14:38:56.467 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T14:38:56.467 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T14:38:56.467 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T14:38:56.467 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T14:38:56.467 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T14:38:56.467 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T14:38:56.467 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T14:38:56.467 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T14:38:56.467 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T14:38:56.467 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T14:38:56.467 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T14:38:56.467 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2024-03-18T14:38:56.467 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T14:38:56.468 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T14:38:56.468 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T14:38:56.468 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T14:38:56.468 INFO:teuthology.orchestra.run.smithi077.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T14:38:56.468 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T14:38:56.468 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T14:38:56.468 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T14:38:56.468 INFO:teuthology.orchestra.run.smithi077.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T14:38:56.468 INFO:teuthology.orchestra.run.smithi077.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T14:38:56.468 INFO:teuthology.orchestra.run.smithi077.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T14:38:56.468 INFO:teuthology.orchestra.run.smithi077.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T14:38:56.468 INFO:teuthology.orchestra.run.smithi077.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T14:38:56.468 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T14:38:56.469 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T14:38:56.469 INFO:teuthology.orchestra.run.smithi077.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T14:38:56.469 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T14:38:56.469 INFO:teuthology.orchestra.run.smithi077.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T14:38:56.469 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T14:38:56.469 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T14:38:56.469 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T14:38:56.469 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T14:38:56.469 INFO:teuthology.orchestra.run.smithi077.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T14:38:56.469 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T14:38:56.469 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-03-18T14:38:56.469 INFO:teuthology.orchestra.run.smithi077.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T14:38:56.469 INFO:teuthology.orchestra.run.smithi077.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T14:38:56.469 INFO:teuthology.orchestra.run.smithi077.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T14:38:56.470 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T14:38:56.470 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T14:38:56.470 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T14:38:56.470 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T14:38:56.470 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T14:38:56.470 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T14:38:56.470 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T14:38:56.470 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T14:38:56.470 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T14:38:56.470 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T14:38:56.470 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T14:38:56.470 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T14:38:56.471 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T14:38:56.471 INFO:teuthology.orchestra.run.smithi077.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T14:38:56.471 INFO:teuthology.orchestra.run.smithi077.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T14:38:56.471 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T14:38:56.471 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T14:38:56.471 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T14:38:56.471 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T14:38:56.471 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T14:38:56.471 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T14:38:56.471 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T14:38:56.471 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T14:38:56.471 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T14:38:56.471 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T14:38:56.471 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T14:38:56.472 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T14:38:56.472 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T14:38:56.472 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T14:38:56.472 INFO:teuthology.orchestra.run.smithi077.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T14:38:56.472 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T14:38:56.472 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T14:38:56.472 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T14:38:56.472 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T14:38:56.472 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T14:38:56.472 INFO:teuthology.orchestra.run.smithi077.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T14:38:56.472 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T14:38:56.472 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T14:38:56.472 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T14:38:56.472 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T14:38:56.473 INFO:teuthology.orchestra.run.smithi077.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T14:38:56.473 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T14:38:56.473 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T14:38:56.473 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T14:38:56.473 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T14:38:56.473 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T14:38:56.473 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T14:38:56.473 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T14:38:56.473 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T14:38:56.473 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T14:38:56.473 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T14:38:56.473 INFO:teuthology.orchestra.run.smithi077.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T14:38:56.473 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T14:38:56.473 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T14:38:56.473 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T14:38:56.474 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T14:38:56.474 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T14:38:56.474 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T14:38:56.474 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T14:38:56.474 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T14:38:56.474 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T14:38:56.474 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T14:38:56.474 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T14:38:56.474 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T14:38:56.474 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T14:38:56.474 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T14:38:56.474 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T14:38:56.474 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T14:38:56.475 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T14:38:56.475 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T14:38:56.475 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T14:38:56.475 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T14:38:56.475 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T14:38:56.475 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T14:38:56.475 INFO:teuthology.orchestra.run.smithi077.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T14:38:56.475 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T14:38:56.475 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T14:38:56.475 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T14:38:56.475 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T14:38:56.475 INFO:teuthology.orchestra.run.smithi077.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T14:38:56.475 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T14:38:56.475 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T14:38:56.475 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T14:38:56.476 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T14:38:56.476 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T14:38:56.476 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T14:38:56.476 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T14:38:56.476 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T14:38:56.476 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T14:38:56.476 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T14:38:56.476 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T14:38:56.476 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T14:38:56.476 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T14:38:56.476 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T14:38:56.476 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T14:38:56.476 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T14:38:56.476 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T14:38:56.477 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T14:38:56.477 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T14:38:56.477 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T14:38:56.477 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T14:38:56.477 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T14:38:56.477 INFO:teuthology.orchestra.run.smithi077.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T14:38:56.477 INFO:teuthology.orchestra.run.smithi077.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T14:38:56.477 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T14:38:56.477 INFO:teuthology.orchestra.run.smithi077.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T14:38:56.477 INFO:teuthology.orchestra.run.smithi077.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T14:38:56.477 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T14:38:56.477 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T14:38:56.477 INFO:teuthology.orchestra.run.smithi077.stdout:Installing weak dependencies: 2024-03-18T14:38:56.477 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T14:38:56.478 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-18T14:38:56.478 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-03-18T14:38:56.478 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================ 2024-03-18T14:38:56.478 INFO:teuthology.orchestra.run.smithi077.stdout:Install 150 Packages 2024-03-18T14:38:56.478 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 24 Packages 2024-03-18T14:38:56.478 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-18T14:38:56.478 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 224 M 2024-03-18T14:38:56.479 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-03-18T14:38:56.725 INFO:teuthology.orchestra.run.smithi174.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-03-18T14:38:57.843 INFO:teuthology.orchestra.run.smithi077.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 11 kB/s | 6.6 kB 00:00 2024-03-18T14:38:58.517 INFO:teuthology.orchestra.run.smithi174.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 497 kB/s | 890 kB 00:01 2024-03-18T14:38:58.691 INFO:teuthology.orchestra.run.smithi076.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 379 kB/s | 890 kB 00:02 2024-03-18T14:38:58.833 INFO:teuthology.orchestra.run.smithi174.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 457 kB/s | 145 kB 00:00 2024-03-18T14:38:59.083 INFO:teuthology.orchestra.run.smithi076.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 370 kB/s | 145 kB 00:00 2024-03-18T14:38:59.535 INFO:teuthology.orchestra.run.smithi077.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 526 kB/s | 890 kB 00:01 2024-03-18T14:38:59.746 INFO:teuthology.orchestra.run.smithi077.stdout:(3/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 2.2 MB/s | 5.4 MB 00:02 2024-03-18T14:38:59.823 INFO:teuthology.orchestra.run.smithi077.stdout:(4/174): ceph-immutable-object-cache-19.0.0-216 504 kB/s | 145 kB 00:00 2024-03-18T14:39:00.528 INFO:teuthology.orchestra.run.smithi077.stdout:(5/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 3.0 MB/s | 2.4 MB 00:00 2024-03-18T14:39:00.817 INFO:teuthology.orchestra.run.smithi076.stdout:(4/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 1.1 MB/s | 5.4 MB 00:05 2024-03-18T14:39:01.250 INFO:teuthology.orchestra.run.smithi174.stdout:(4/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 1.0 MB/s | 2.4 MB 00:02 2024-03-18T14:39:01.903 INFO:teuthology.orchestra.run.smithi077.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 735 kB/s | 1.5 MB 00:02 2024-03-18T14:39:02.202 INFO:teuthology.orchestra.run.smithi076.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 1.1 MB/s | 1.5 MB 00:01 2024-03-18T14:39:02.312 INFO:teuthology.orchestra.run.smithi077.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 2.6 MB/s | 4.7 MB 00:01 2024-03-18T14:39:02.850 INFO:teuthology.orchestra.run.smithi076.stdout:(6/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 642 kB/s | 2.4 MB 00:03 2024-03-18T14:39:04.642 INFO:teuthology.orchestra.run.smithi174.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 451 kB/s | 1.5 MB 00:03 2024-03-18T14:39:06.350 INFO:teuthology.orchestra.run.smithi076.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 1.1 MB/s | 4.7 MB 00:04 2024-03-18T14:39:08.128 INFO:teuthology.orchestra.run.smithi077.stdout:(8/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el9 1.7 MB/s | 9.8 MB 00:05 2024-03-18T14:39:08.220 INFO:teuthology.orchestra.run.smithi077.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 276 kB/s | 25 kB 00:00 2024-03-18T14:39:11.441 INFO:teuthology.orchestra.run.smithi174.stdout:(6/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 363 kB/s | 5.4 MB 00:15 2024-03-18T14:39:13.233 INFO:teuthology.orchestra.run.smithi174.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 560 kB/s | 4.7 MB 00:08 2024-03-18T14:39:14.866 INFO:teuthology.orchestra.run.smithi076.stdout:(8/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el9 1.2 MB/s | 9.8 MB 00:08 2024-03-18T14:39:14.966 INFO:teuthology.orchestra.run.smithi076.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 252 kB/s | 25 kB 00:00 2024-03-18T14:39:15.291 INFO:teuthology.orchestra.run.smithi174.stdout:(8/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 1.1 MB/s | 21 MB 00:18 2024-03-18T14:39:15.400 INFO:teuthology.orchestra.run.smithi174.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 233 kB/s | 25 kB 00:00 2024-03-18T14:39:20.618 INFO:teuthology.orchestra.run.smithi076.stdout:(10/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 970 kB/s | 17 MB 00:17 2024-03-18T14:39:20.958 INFO:teuthology.orchestra.run.smithi076.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 648 kB/s | 221 kB 00:00 2024-03-18T14:39:21.050 INFO:teuthology.orchestra.run.smithi076.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 295 kB/s | 27 kB 00:00 2024-03-18T14:39:21.150 INFO:teuthology.orchestra.run.smithi076.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 316 kB/s | 32 kB 00:00 2024-03-18T14:39:22.576 INFO:teuthology.orchestra.run.smithi076.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 524 kB/s | 747 kB 00:01 2024-03-18T14:39:22.677 INFO:teuthology.orchestra.run.smithi077.stdout:(10/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 830 kB/s | 17 MB 00:20 2024-03-18T14:39:22.909 INFO:teuthology.orchestra.run.smithi076.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 490 kB/s | 163 kB 00:00 2024-03-18T14:39:23.159 INFO:teuthology.orchestra.run.smithi076.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 507 kB/s | 127 kB 00:00 2024-03-18T14:39:23.311 INFO:teuthology.orchestra.run.smithi077.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 349 kB/s | 221 kB 00:00 2024-03-18T14:39:23.411 INFO:teuthology.orchestra.run.smithi077.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 270 kB/s | 27 kB 00:00 2024-03-18T14:39:23.513 INFO:teuthology.orchestra.run.smithi077.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 317 kB/s | 32 kB 00:00 2024-03-18T14:39:23.882 INFO:teuthology.orchestra.run.smithi174.stdout:(10/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 945 kB/s | 9.8 MB 00:10 2024-03-18T14:39:23.918 INFO:teuthology.orchestra.run.smithi076.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 661 kB/s | 501 kB 00:00 2024-03-18T14:39:24.141 INFO:teuthology.orchestra.run.smithi174.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 858 kB/s | 221 kB 00:00 2024-03-18T14:39:24.241 INFO:teuthology.orchestra.run.smithi174.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 271 kB/s | 27 kB 00:00 2024-03-18T14:39:24.408 INFO:teuthology.orchestra.run.smithi174.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 190 kB/s | 32 kB 00:00 2024-03-18T14:39:24.520 INFO:teuthology.orchestra.run.smithi077.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 741 kB/s | 747 kB 00:01 2024-03-18T14:39:24.762 INFO:teuthology.orchestra.run.smithi077.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 676 kB/s | 163 kB 00:00 2024-03-18T14:39:24.979 INFO:teuthology.orchestra.run.smithi077.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 585 kB/s | 127 kB 00:00 2024-03-18T14:39:25.175 INFO:teuthology.orchestra.run.smithi174.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 974 kB/s | 747 kB 00:00 2024-03-18T14:39:25.333 INFO:teuthology.orchestra.run.smithi174.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.0 MB/s | 163 kB 00:00 2024-03-18T14:39:25.500 INFO:teuthology.orchestra.run.smithi174.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 761 kB/s | 127 kB 00:00 2024-03-18T14:39:25.579 INFO:teuthology.orchestra.run.smithi077.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 835 kB/s | 501 kB 00:00 2024-03-18T14:39:25.992 INFO:teuthology.orchestra.run.smithi174.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 1.0 MB/s | 501 kB 00:00 2024-03-18T14:39:28.813 INFO:teuthology.orchestra.run.smithi077.stdout:(18/174): ceph-common-19.0.0-2162.g3d78f5ba.el9 670 kB/s | 21 MB 00:31 2024-03-18T14:39:28.929 INFO:teuthology.orchestra.run.smithi077.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 388 kB/s | 45 kB 00:00 2024-03-18T14:39:29.238 INFO:teuthology.orchestra.run.smithi077.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 426 kB/s | 131 kB 00:00 2024-03-18T14:39:29.613 INFO:teuthology.orchestra.run.smithi077.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 432 kB/s | 162 kB 00:00 2024-03-18T14:39:30.205 INFO:teuthology.orchestra.run.smithi077.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 541 kB/s | 320 kB 00:00 2024-03-18T14:39:30.517 INFO:teuthology.orchestra.run.smithi174.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 1.1 MB/s | 5.1 MB 00:04 2024-03-18T14:39:30.625 INFO:teuthology.orchestra.run.smithi174.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 417 kB/s | 45 kB 00:00 2024-03-18T14:39:30.789 INFO:teuthology.orchestra.run.smithi077.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 511 kB/s | 298 kB 00:00 2024-03-18T14:39:30.801 INFO:teuthology.orchestra.run.smithi174.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 751 kB/s | 131 kB 00:00 2024-03-18T14:39:30.984 INFO:teuthology.orchestra.run.smithi174.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 883 kB/s | 162 kB 00:00 2024-03-18T14:39:30.997 INFO:teuthology.orchestra.run.smithi077.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 477 kB/s | 99 kB 00:00 2024-03-18T14:39:31.206 INFO:teuthology.orchestra.run.smithi077.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 410 kB/s | 85 kB 00:00 2024-03-18T14:39:31.318 INFO:teuthology.orchestra.run.smithi174.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 961 kB/s | 320 kB 00:00 2024-03-18T14:39:31.643 INFO:teuthology.orchestra.run.smithi174.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 916 kB/s | 298 kB 00:00 2024-03-18T14:39:31.785 INFO:teuthology.orchestra.run.smithi076.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 669 kB/s | 5.1 MB 00:07 2024-03-18T14:39:31.818 INFO:teuthology.orchestra.run.smithi174.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 569 kB/s | 99 kB 00:00 2024-03-18T14:39:31.927 INFO:teuthology.orchestra.run.smithi174.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 790 kB/s | 85 kB 00:00 2024-03-18T14:39:31.959 INFO:teuthology.orchestra.run.smithi076.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 258 kB/s | 45 kB 00:00 2024-03-18T14:39:32.202 INFO:teuthology.orchestra.run.smithi076.stdout:(20/174): ceph-common-19.0.0-2162.g3d78f5ba.el9 579 kB/s | 21 MB 00:36 2024-03-18T14:39:32.293 INFO:teuthology.orchestra.run.smithi076.stdout:(21/174): python3-ceph-common-19.0.0-2162.g3d78 393 kB/s | 131 kB 00:00 2024-03-18T14:39:32.585 INFO:teuthology.orchestra.run.smithi076.stdout:(22/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 423 kB/s | 162 kB 00:00 2024-03-18T14:39:32.938 INFO:teuthology.orchestra.run.smithi076.stdout:(23/174): python3-rados-19.0.0-2162.g3d78f5ba.e 499 kB/s | 320 kB 00:00 2024-03-18T14:39:33.161 INFO:teuthology.orchestra.run.smithi076.stdout:(24/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 518 kB/s | 298 kB 00:00 2024-03-18T14:39:33.186 INFO:teuthology.orchestra.run.smithi076.stdout:(25/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 397 kB/s | 99 kB 00:00 2024-03-18T14:39:33.370 INFO:teuthology.orchestra.run.smithi076.stdout:(26/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 409 kB/s | 85 kB 00:00 2024-03-18T14:39:33.786 INFO:teuthology.orchestra.run.smithi076.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 409 kB/s | 170 kB 00:00 2024-03-18T14:39:33.881 INFO:teuthology.orchestra.run.smithi077.stdout:(26/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 634 kB/s | 5.1 MB 00:08 2024-03-18T14:39:33.903 INFO:teuthology.orchestra.run.smithi076.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 291 kB/s | 34 kB 00:00 2024-03-18T14:39:34.254 INFO:teuthology.orchestra.run.smithi076.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 396 kB/s | 139 kB 00:00 2024-03-18T14:39:34.256 INFO:teuthology.orchestra.run.smithi077.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 455 kB/s | 170 kB 00:00 2024-03-18T14:39:34.356 INFO:teuthology.orchestra.run.smithi077.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 340 kB/s | 34 kB 00:00 2024-03-18T14:39:34.427 INFO:teuthology.orchestra.run.smithi174.stdout:(26/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 1.2 MB/s | 3.1 MB 00:02 2024-03-18T14:39:34.703 INFO:teuthology.orchestra.run.smithi174.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 620 kB/s | 170 kB 00:00 2024-03-18T14:39:34.773 INFO:teuthology.orchestra.run.smithi077.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 333 kB/s | 139 kB 00:00 2024-03-18T14:39:34.794 INFO:teuthology.orchestra.run.smithi174.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 371 kB/s | 34 kB 00:00 2024-03-18T14:39:34.961 INFO:teuthology.orchestra.run.smithi174.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 832 kB/s | 139 kB 00:00 2024-03-18T14:39:36.598 INFO:teuthology.orchestra.run.smithi077.stdout:(30/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 587 kB/s | 3.1 MB 00:05 2024-03-18T14:39:37.337 INFO:teuthology.orchestra.run.smithi076.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 596 kB/s | 1.8 MB 00:03 2024-03-18T14:39:37.545 INFO:teuthology.orchestra.run.smithi174.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 711 kB/s | 1.8 MB 00:02 2024-03-18T14:39:38.987 INFO:teuthology.orchestra.run.smithi076.stdout:(31/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 545 kB/s | 3.1 MB 00:05 2024-03-18T14:39:39.282 INFO:teuthology.orchestra.run.smithi077.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 408 kB/s | 1.8 MB 00:04 2024-03-18T14:39:39.587 INFO:teuthology.orchestra.run.smithi076.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 414 kB/s | 248 kB 00:00 2024-03-18T14:39:39.754 INFO:teuthology.orchestra.run.smithi076.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 296 kB/s | 49 kB 00:00 2024-03-18T14:39:39.854 INFO:teuthology.orchestra.run.smithi076.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 165 kB/s | 16 kB 00:00 2024-03-18T14:39:39.874 INFO:teuthology.orchestra.run.smithi077.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 420 kB/s | 248 kB 00:00 2024-03-18T14:39:40.049 INFO:teuthology.orchestra.run.smithi077.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 282 kB/s | 49 kB 00:00 2024-03-18T14:39:40.150 INFO:teuthology.orchestra.run.smithi077.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 164 kB/s | 16 kB 00:00 2024-03-18T14:39:40.413 INFO:teuthology.orchestra.run.smithi076.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 472 kB/s | 264 kB 00:00 2024-03-18T14:39:40.724 INFO:teuthology.orchestra.run.smithi077.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 459 kB/s | 264 kB 00:00 2024-03-18T14:39:41.788 INFO:teuthology.orchestra.run.smithi076.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 544 kB/s | 748 kB 00:01 2024-03-18T14:39:41.888 INFO:teuthology.orchestra.run.smithi076.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 182 kB/s | 18 kB 00:00 2024-03-18T14:39:41.999 INFO:teuthology.orchestra.run.smithi077.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 587 kB/s | 748 kB 00:01 2024-03-18T14:39:42.038 INFO:teuthology.orchestra.run.smithi076.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.3 MB/s | 662 kB 00:00 2024-03-18T14:39:42.100 INFO:teuthology.orchestra.run.smithi077.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 183 kB/s | 18 kB 00:00 2024-03-18T14:39:42.172 INFO:teuthology.orchestra.run.smithi076.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 6.8 MB/s | 920 kB 00:00 2024-03-18T14:39:42.239 INFO:teuthology.orchestra.run.smithi076.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 1.5 MB/s | 106 kB 00:00 2024-03-18T14:39:42.314 INFO:teuthology.orchestra.run.smithi076.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 4.0 MB/s | 307 kB 00:00 2024-03-18T14:39:42.340 INFO:teuthology.orchestra.run.smithi076.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 1.2 MB/s | 32 kB 00:00 2024-03-18T14:39:42.384 INFO:teuthology.orchestra.run.smithi077.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 2.3 MB/s | 662 kB 00:00 2024-03-18T14:39:42.434 INFO:teuthology.orchestra.run.smithi077.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-03-18T14:39:42.448 INFO:teuthology.orchestra.run.smithi076.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-18T14:39:42.467 INFO:teuthology.orchestra.run.smithi077.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-18T14:39:42.482 INFO:teuthology.orchestra.run.smithi076.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 512 kB/s | 17 kB 00:00 2024-03-18T14:39:42.507 INFO:teuthology.orchestra.run.smithi076.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 403 kB/s | 10 kB 00:00 2024-03-18T14:39:42.509 INFO:teuthology.orchestra.run.smithi077.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.3 MB/s | 307 kB 00:00 2024-03-18T14:39:42.541 INFO:teuthology.orchestra.run.smithi076.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-18T14:39:42.542 INFO:teuthology.orchestra.run.smithi077.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 954 kB/s | 32 kB 00:00 2024-03-18T14:39:42.566 INFO:teuthology.orchestra.run.smithi076.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-18T14:39:42.616 INFO:teuthology.orchestra.run.smithi076.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-18T14:39:42.634 INFO:teuthology.orchestra.run.smithi077.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-03-18T14:39:42.658 INFO:teuthology.orchestra.run.smithi076.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-03-18T14:39:42.668 INFO:teuthology.orchestra.run.smithi077.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 513 kB/s | 17 kB 00:00 2024-03-18T14:39:42.683 INFO:teuthology.orchestra.run.smithi076.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.8 MB/s | 46 kB 00:00 2024-03-18T14:39:42.702 INFO:teuthology.orchestra.run.smithi077.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 301 kB/s | 10 kB 00:00 2024-03-18T14:39:42.725 INFO:teuthology.orchestra.run.smithi076.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-18T14:39:42.744 INFO:teuthology.orchestra.run.smithi077.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-18T14:39:42.778 INFO:teuthology.orchestra.run.smithi077.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-18T14:39:42.792 INFO:teuthology.orchestra.run.smithi076.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 187 kB 00:00 2024-03-18T14:39:42.820 INFO:teuthology.orchestra.run.smithi077.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 16 MB/s | 661 kB 00:00 2024-03-18T14:39:42.834 INFO:teuthology.orchestra.run.smithi076.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-18T14:39:42.859 INFO:teuthology.orchestra.run.smithi076.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-18T14:39:42.870 INFO:teuthology.orchestra.run.smithi077.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-18T14:39:42.920 INFO:teuthology.orchestra.run.smithi077.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 929 kB/s | 46 kB 00:00 2024-03-18T14:39:42.962 INFO:teuthology.orchestra.run.smithi077.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-18T14:39:42.985 INFO:teuthology.orchestra.run.smithi076.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-18T14:39:43.004 INFO:teuthology.orchestra.run.smithi077.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-18T14:39:43.052 INFO:teuthology.orchestra.run.smithi076.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 23 MB/s | 1.5 MB 00:00 2024-03-18T14:39:43.054 INFO:teuthology.orchestra.run.smithi077.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-18T14:39:43.088 INFO:teuthology.orchestra.run.smithi077.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-18T14:39:43.110 INFO:teuthology.orchestra.run.smithi076.stdout:(57/174): postgresql-contrib-13.11-1.el9.x86_64 14 MB/s | 811 kB 00:00 2024-03-18T14:39:43.177 INFO:teuthology.orchestra.run.smithi076.stdout:(58/174): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-18T14:39:43.213 INFO:teuthology.orchestra.run.smithi077.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-18T14:39:43.297 INFO:teuthology.orchestra.run.smithi077.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 18 MB/s | 1.5 MB 00:00 2024-03-18T14:39:43.327 INFO:teuthology.orchestra.run.smithi076.stdout:(59/174): postgresql-server-13.11-1.el9.x86_64. 38 MB/s | 5.7 MB 00:00 2024-03-18T14:39:43.356 INFO:teuthology.orchestra.run.smithi077.stdout:(57/174): postgresql-contrib-13.11-1.el9.x86_64 14 MB/s | 811 kB 00:00 2024-03-18T14:39:43.378 INFO:teuthology.orchestra.run.smithi076.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 20 MB/s | 1.0 MB 00:00 2024-03-18T14:39:43.390 INFO:teuthology.orchestra.run.smithi077.stdout:(58/174): postgresql-private-libs-13.11-1.el9.x 3.9 MB/s | 134 kB 00:00 2024-03-18T14:39:43.420 INFO:teuthology.orchestra.run.smithi076.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.8 MB/s | 78 kB 00:00 2024-03-18T14:39:43.603 INFO:teuthology.orchestra.run.smithi076.stdout:(62/174): python3-babel-2.9.1-2.el9.noarch.rpm 32 MB/s | 5.8 MB 00:00 2024-03-18T14:39:43.614 INFO:teuthology.orchestra.run.smithi077.stdout:(59/174): postgresql-server-13.11-1.el9.x86_64. 25 MB/s | 5.7 MB 00:00 2024-03-18T14:39:43.653 INFO:teuthology.orchestra.run.smithi076.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 490 kB/s | 24 kB 00:00 2024-03-18T14:39:43.673 INFO:teuthology.orchestra.run.smithi077.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-03-18T14:39:43.696 INFO:teuthology.orchestra.run.smithi076.stdout:(64/174): python3-jsonpointer-2.0-4.el9.noarch. 433 kB/s | 18 kB 00:00 2024-03-18T14:39:43.732 INFO:teuthology.orchestra.run.smithi077.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-18T14:39:43.737 INFO:teuthology.orchestra.run.smithi076.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-18T14:39:43.779 INFO:teuthology.orchestra.run.smithi076.stdout:(66/174): python3-markupsafe-1.1.1-12.el9.x86_6 756 kB/s | 32 kB 00:00 2024-03-18T14:39:43.990 INFO:teuthology.orchestra.run.smithi077.stdout:(62/174): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-18T14:39:44.005 INFO:teuthology.orchestra.run.smithi076.stdout:(67/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-18T14:39:44.024 INFO:teuthology.orchestra.run.smithi077.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 739 kB/s | 24 kB 00:00 2024-03-18T14:39:44.055 INFO:teuthology.orchestra.run.smithi076.stdout:(68/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.3 MB/s | 217 kB 00:00 2024-03-18T14:39:44.057 INFO:teuthology.orchestra.run.smithi077.stdout:(64/174): python3-jsonpointer-2.0-4.el9.noarch. 542 kB/s | 18 kB 00:00 2024-03-18T14:39:44.099 INFO:teuthology.orchestra.run.smithi077.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-18T14:39:44.105 INFO:teuthology.orchestra.run.smithi076.stdout:(69/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-18T14:39:44.133 INFO:teuthology.orchestra.run.smithi077.stdout:(66/174): python3-markupsafe-1.1.1-12.el9.x86_6 948 kB/s | 32 kB 00:00 2024-03-18T14:39:44.147 INFO:teuthology.orchestra.run.smithi076.stdout:(70/174): python3-packaging-20.9-5.el9.noarch.r 1.6 MB/s | 69 kB 00:00 2024-03-18T14:39:44.181 INFO:teuthology.orchestra.run.smithi076.stdout:(71/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-18T14:39:44.248 INFO:teuthology.orchestra.run.smithi076.stdout:(72/174): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-18T14:39:44.267 INFO:teuthology.orchestra.run.smithi077.stdout:(67/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-18T14:39:44.281 INFO:teuthology.orchestra.run.smithi076.stdout:(73/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.9 MB/s | 132 kB 00:00 2024-03-18T14:39:44.300 INFO:teuthology.orchestra.run.smithi077.stdout:(68/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 6.4 MB/s | 217 kB 00:00 2024-03-18T14:39:44.323 INFO:teuthology.orchestra.run.smithi076.stdout:(74/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.9 MB/s | 211 kB 00:00 2024-03-18T14:39:44.342 INFO:teuthology.orchestra.run.smithi077.stdout:(69/174): python3-oauthlib-3.1.1-5.el9.noarch.r 4.1 MB/s | 174 kB 00:00 2024-03-18T14:39:44.376 INFO:teuthology.orchestra.run.smithi077.stdout:(70/174): python3-packaging-20.9-5.el9.noarch.r 2.0 MB/s | 69 kB 00:00 2024-03-18T14:39:44.382 INFO:teuthology.orchestra.run.smithi076.stdout:(75/174): python3-pytz-2021.1-5.el9.noarch.rpm 814 kB/s | 47 kB 00:00 2024-03-18T14:39:44.410 INFO:teuthology.orchestra.run.smithi077.stdout:(71/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-18T14:39:44.415 INFO:teuthology.orchestra.run.smithi076.stdout:(76/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-18T14:39:44.452 INFO:teuthology.orchestra.run.smithi077.stdout:(72/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-18T14:39:44.510 INFO:teuthology.orchestra.run.smithi077.stdout:(73/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-18T14:39:44.577 INFO:teuthology.orchestra.run.smithi077.stdout:(74/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-03-18T14:39:44.627 INFO:teuthology.orchestra.run.smithi077.stdout:(75/174): python3-pytz-2021.1-5.el9.noarch.rpm 950 kB/s | 47 kB 00:00 2024-03-18T14:39:44.661 INFO:teuthology.orchestra.run.smithi077.stdout:(76/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-18T14:39:44.791 INFO:teuthology.orchestra.run.smithi076.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-18T14:39:44.849 INFO:teuthology.orchestra.run.smithi076.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 665 kB/s | 39 kB 00:00 2024-03-18T14:39:44.900 INFO:teuthology.orchestra.run.smithi076.stdout:(79/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 816 kB/s | 41 kB 00:00 2024-03-18T14:39:44.958 INFO:teuthology.orchestra.run.smithi076.stdout:(80/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-18T14:39:44.992 INFO:teuthology.orchestra.run.smithi076.stdout:(81/174): uuid-1.6.2-55.el9.x86_64.rpm 1.6 MB/s | 56 kB 00:00 2024-03-18T14:39:45.026 INFO:teuthology.orchestra.run.smithi076.stdout:(82/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-18T14:39:45.059 INFO:teuthology.orchestra.run.smithi076.stdout:(83/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-18T14:39:45.093 INFO:teuthology.orchestra.run.smithi076.stdout:(84/174): fuse-2.9.9-15.el9.x86_64.rpm 2.3 MB/s | 79 kB 00:00 2024-03-18T14:39:45.103 INFO:teuthology.orchestra.run.smithi077.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-18T14:39:45.134 INFO:teuthology.orchestra.run.smithi076.stdout:(85/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-18T14:39:45.136 INFO:teuthology.orchestra.run.smithi077.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-18T14:39:45.170 INFO:teuthology.orchestra.run.smithi077.stdout:(79/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.2 MB/s | 41 kB 00:00 2024-03-18T14:39:45.210 INFO:teuthology.orchestra.run.smithi076.stdout:(86/174): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.1 MB 00:00 2024-03-18T14:39:45.229 INFO:teuthology.orchestra.run.smithi077.stdout:(80/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-18T14:39:45.260 INFO:teuthology.orchestra.run.smithi076.stdout:(87/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-18T14:39:45.271 INFO:teuthology.orchestra.run.smithi077.stdout:(81/174): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2024-03-18T14:39:45.302 INFO:teuthology.orchestra.run.smithi076.stdout:(88/174): python3-pycparser-2.20-6.el9.noarch.r 2.9 MB/s | 124 kB 00:00 2024-03-18T14:39:45.304 INFO:teuthology.orchestra.run.smithi077.stdout:(82/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-18T14:39:45.338 INFO:teuthology.orchestra.run.smithi077.stdout:(83/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-18T14:39:45.344 INFO:teuthology.orchestra.run.smithi076.stdout:(89/174): python3-urllib3-1.26.5-5.el9.noarch.r 4.4 MB/s | 187 kB 00:00 2024-03-18T14:39:45.363 INFO:teuthology.orchestra.run.smithi077.stdout:(84/174): fuse-2.9.9-15.el9.x86_64.rpm 3.1 MB/s | 79 kB 00:00 2024-03-18T14:39:45.405 INFO:teuthology.orchestra.run.smithi077.stdout:(85/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-18T14:39:45.464 INFO:teuthology.orchestra.run.smithi077.stdout:(86/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-18T14:39:45.506 INFO:teuthology.orchestra.run.smithi077.stdout:(87/174): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-18T14:39:45.548 INFO:teuthology.orchestra.run.smithi077.stdout:(88/174): python3-pycparser-2.20-6.el9.noarch.r 2.9 MB/s | 124 kB 00:00 2024-03-18T14:39:45.589 INFO:teuthology.orchestra.run.smithi077.stdout:(89/174): python3-urllib3-1.26.5-5.el9.noarch.r 4.4 MB/s | 187 kB 00:00 2024-03-18T14:39:45.782 INFO:teuthology.orchestra.run.smithi077.stdout:(90/174): jq-1.6-16.el9.x86_64.rpm 977 kB/s | 187 kB 00:00 2024-03-18T14:39:45.940 INFO:teuthology.orchestra.run.smithi077.stdout:(91/174): libgfortran-11.4.1-3.el9.x86_64.rpm 5.0 MB/s | 803 kB 00:00 2024-03-18T14:39:46.007 INFO:teuthology.orchestra.run.smithi077.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-03-18T14:39:46.065 INFO:teuthology.orchestra.run.smithi077.stdout:(93/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-18T14:39:46.165 INFO:teuthology.orchestra.run.smithi077.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 215 kB/s | 21 kB 00:00 2024-03-18T14:39:46.207 INFO:teuthology.orchestra.run.smithi077.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 612 kB/s | 25 kB 00:00 2024-03-18T14:39:46.219 INFO:teuthology.orchestra.run.smithi076.stdout:(90/174): jq-1.6-16.el9.x86_64.rpm 214 kB/s | 187 kB 00:00 2024-03-18T14:39:46.408 INFO:teuthology.orchestra.run.smithi077.stdout:(96/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 21 MB/s | 4.1 MB 00:00 2024-03-18T14:39:46.449 INFO:teuthology.orchestra.run.smithi077.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 5.8 MB/s | 245 kB 00:00 2024-03-18T14:39:46.491 INFO:teuthology.orchestra.run.smithi077.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 5.8 MB/s | 248 kB 00:00 2024-03-18T14:39:46.533 INFO:teuthology.orchestra.run.smithi077.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 5.1 MB/s | 216 kB 00:00 2024-03-18T14:39:46.575 INFO:teuthology.orchestra.run.smithi077.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 840 kB/s | 35 kB 00:00 2024-03-18T14:39:46.617 INFO:teuthology.orchestra.run.smithi077.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-03-18T14:39:46.667 INFO:teuthology.orchestra.run.smithi077.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-18T14:39:46.709 INFO:teuthology.orchestra.run.smithi077.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 537 kB/s | 22 kB 00:00 2024-03-18T14:39:46.760 INFO:teuthology.orchestra.run.smithi077.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 17 MB/s | 863 kB 00:00 2024-03-18T14:39:47.185 INFO:teuthology.orchestra.run.smithi077.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-03-18T14:39:47.285 INFO:teuthology.orchestra.run.smithi077.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 3.0 MB/s | 309 kB 00:00 2024-03-18T14:39:47.319 INFO:teuthology.orchestra.run.smithi174.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 777 kB/s | 7.4 MB 00:09 2024-03-18T14:39:47.352 INFO:teuthology.orchestra.run.smithi077.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 292 kB/s | 19 kB 00:00 2024-03-18T14:39:47.678 INFO:teuthology.orchestra.run.smithi077.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-18T14:39:47.778 INFO:teuthology.orchestra.run.smithi077.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-18T14:39:47.878 INFO:teuthology.orchestra.run.smithi077.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-03-18T14:39:47.911 INFO:teuthology.orchestra.run.smithi076.stdout:(91/174): libgfortran-11.4.1-3.el9.x86_64.rpm 475 kB/s | 803 kB 00:01 2024-03-18T14:39:47.945 INFO:teuthology.orchestra.run.smithi077.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-18T14:39:48.020 INFO:teuthology.orchestra.run.smithi077.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-03-18T14:39:48.113 INFO:teuthology.orchestra.run.smithi077.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2024-03-18T14:39:48.204 INFO:teuthology.orchestra.run.smithi077.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 5.9 MB/s | 548 kB 00:00 2024-03-18T14:39:48.271 INFO:teuthology.orchestra.run.smithi077.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 651 kB/s | 43 kB 00:00 2024-03-18T14:39:48.338 INFO:teuthology.orchestra.run.smithi077.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 484 kB/s | 32 kB 00:00 2024-03-18T14:39:48.405 INFO:teuthology.orchestra.run.smithi077.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 225 kB/s | 15 kB 00:00 2024-03-18T14:39:48.853 INFO:teuthology.orchestra.run.smithi076.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 207 kB/s | 195 kB 00:00 2024-03-18T14:39:49.628 INFO:teuthology.orchestra.run.smithi076.stdout:(93/174): python3-requests-2.25.1-8.el9.noarch. 161 kB/s | 125 kB 00:00 2024-03-18T14:39:49.720 INFO:teuthology.orchestra.run.smithi076.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 235 kB/s | 21 kB 00:00 2024-03-18T14:39:49.762 INFO:teuthology.orchestra.run.smithi076.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 610 kB/s | 25 kB 00:00 2024-03-18T14:39:50.012 INFO:teuthology.orchestra.run.smithi076.stdout:(96/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 17 MB/s | 4.1 MB 00:00 2024-03-18T14:39:50.062 INFO:teuthology.orchestra.run.smithi076.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-03-18T14:39:50.112 INFO:teuthology.orchestra.run.smithi076.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-03-18T14:39:50.163 INFO:teuthology.orchestra.run.smithi076.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 216 kB 00:00 2024-03-18T14:39:50.204 INFO:teuthology.orchestra.run.smithi076.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 841 kB/s | 35 kB 00:00 2024-03-18T14:39:50.255 INFO:teuthology.orchestra.run.smithi076.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 889 kB/s | 44 kB 00:00 2024-03-18T14:39:50.313 INFO:teuthology.orchestra.run.smithi076.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-18T14:39:50.372 INFO:teuthology.orchestra.run.smithi076.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 381 kB/s | 22 kB 00:00 2024-03-18T14:39:50.448 INFO:teuthology.orchestra.run.smithi076.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 11 MB/s | 863 kB 00:00 2024-03-18T14:39:50.555 INFO:teuthology.orchestra.run.smithi077.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 80 kB/s | 172 kB 00:02 2024-03-18T14:39:50.655 INFO:teuthology.orchestra.run.smithi077.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-03-18T14:39:50.747 INFO:teuthology.orchestra.run.smithi077.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-18T14:39:50.814 INFO:teuthology.orchestra.run.smithi076.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.5 MB/s | 548 kB 00:00 2024-03-18T14:39:50.839 INFO:teuthology.orchestra.run.smithi077.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 385 kB/s | 35 kB 00:00 2024-03-18T14:39:50.906 INFO:teuthology.orchestra.run.smithi076.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 3.3 MB/s | 309 kB 00:00 2024-03-18T14:39:50.914 INFO:teuthology.orchestra.run.smithi077.stdout:(122/174): python3-google-auth-2.28.0-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-03-18T14:39:50.981 INFO:teuthology.orchestra.run.smithi076.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 259 kB/s | 19 kB 00:00 2024-03-18T14:39:51.056 INFO:teuthology.orchestra.run.smithi077.stdout:(123/174): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-03-18T14:39:51.149 INFO:teuthology.orchestra.run.smithi077.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-03-18T14:39:51.249 INFO:teuthology.orchestra.run.smithi077.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-03-18T14:39:51.341 INFO:teuthology.orchestra.run.smithi077.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-03-18T14:39:51.432 INFO:teuthology.orchestra.run.smithi076.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 4.4 MB 00:00 2024-03-18T14:39:51.433 INFO:teuthology.orchestra.run.smithi077.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 253 kB/s | 23 kB 00:00 2024-03-18T14:39:51.508 INFO:teuthology.orchestra.run.smithi077.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 259 kB/s | 19 kB 00:00 2024-03-18T14:39:51.523 INFO:teuthology.orchestra.run.smithi076.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 284 kB/s | 26 kB 00:00 2024-03-18T14:39:51.615 INFO:teuthology.orchestra.run.smithi076.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 530 kB/s | 49 kB 00:00 2024-03-18T14:39:51.616 INFO:teuthology.orchestra.run.smithi077.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 182 kB/s | 20 kB 00:00 2024-03-18T14:39:51.674 INFO:teuthology.orchestra.run.smithi076.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-18T14:39:51.750 INFO:teuthology.orchestra.run.smithi077.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 7.7 MB/s | 1.0 MB 00:00 2024-03-18T14:39:51.850 INFO:teuthology.orchestra.run.smithi077.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 463 kB/s | 46 kB 00:00 2024-03-18T14:39:51.950 INFO:teuthology.orchestra.run.smithi077.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 790 kB/s | 79 kB 00:00 2024-03-18T14:39:52.042 INFO:teuthology.orchestra.run.smithi077.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 630 kB/s | 58 kB 00:00 2024-03-18T14:39:52.116 INFO:teuthology.orchestra.run.smithi076.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 342 kB/s | 151 kB 00:00 2024-03-18T14:39:52.159 INFO:teuthology.orchestra.run.smithi077.stdout:(134/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-18T14:39:52.234 INFO:teuthology.orchestra.run.smithi077.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 219 kB/s | 16 kB 00:00 2024-03-18T14:39:52.318 INFO:teuthology.orchestra.run.smithi077.stdout:(136/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-03-18T14:39:52.393 INFO:teuthology.orchestra.run.smithi077.stdout:(137/174): python3-pyperclip-1.8.2-2.el9.noarch 376 kB/s | 28 kB 00:00 2024-03-18T14:39:52.460 INFO:teuthology.orchestra.run.smithi077.stdout:(138/174): python3-repoze-lru-0.7-16.el9.noarch 463 kB/s | 31 kB 00:00 2024-03-18T14:39:52.544 INFO:teuthology.orchestra.run.smithi077.stdout:(139/174): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-03-18T14:39:52.582 INFO:teuthology.orchestra.run.smithi076.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.8 MB/s | 838 kB 00:00 2024-03-18T14:39:52.619 INFO:teuthology.orchestra.run.smithi077.stdout:(140/174): python3-rsa-4.9-2.el9.noarch.rpm 788 kB/s | 59 kB 00:00 2024-03-18T14:39:52.694 INFO:teuthology.orchestra.run.smithi077.stdout:(141/174): python3-tempora-5.0.0-2.el9.noarch.r 477 kB/s | 36 kB 00:00 2024-03-18T14:39:52.716 INFO:teuthology.orchestra.run.smithi076.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-03-18T14:39:52.761 INFO:teuthology.orchestra.run.smithi077.stdout:(142/174): python3-typing-extensions-4.4.0-2.el 765 kB/s | 51 kB 00:00 2024-03-18T14:39:52.816 INFO:teuthology.orchestra.run.smithi076.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 435 kB/s | 43 kB 00:00 2024-03-18T14:39:52.887 INFO:teuthology.orchestra.run.smithi077.stdout:(143/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-18T14:39:52.891 INFO:teuthology.orchestra.run.smithi076.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 429 kB/s | 32 kB 00:00 2024-03-18T14:39:52.962 INFO:teuthology.orchestra.run.smithi077.stdout:(144/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-18T14:39:52.967 INFO:teuthology.orchestra.run.smithi076.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 200 kB/s | 15 kB 00:00 2024-03-18T14:39:53.054 INFO:teuthology.orchestra.run.smithi077.stdout:(145/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.6 MB/s | 427 kB 00:00 2024-03-18T14:39:53.067 INFO:teuthology.orchestra.run.smithi076.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-03-18T14:39:53.137 INFO:teuthology.orchestra.run.smithi077.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 239 kB/s | 20 kB 00:00 2024-03-18T14:39:53.159 INFO:teuthology.orchestra.run.smithi076.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-03-18T14:39:53.221 INFO:teuthology.orchestra.run.smithi077.stdout:(147/174): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-18T14:39:53.234 INFO:teuthology.orchestra.run.smithi076.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-03-18T14:39:53.284 INFO:teuthology.orchestra.run.smithi076.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 708 kB/s | 35 kB 00:00 2024-03-18T14:39:53.338 INFO:teuthology.orchestra.run.smithi077.stdout:(148/174): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-18T14:39:53.393 INFO:teuthology.orchestra.run.smithi076.stdout:(122/174): python3-google-auth-2.28.0-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-03-18T14:39:53.660 INFO:teuthology.orchestra.run.smithi076.stdout:(123/174): python3-grpcio-1.46.7-10.el9.x86_64. 7.6 MB/s | 2.0 MB 00:00 2024-03-18T14:39:53.718 INFO:teuthology.orchestra.run.smithi076.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-03-18T14:39:53.802 INFO:teuthology.orchestra.run.smithi076.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-03-18T14:39:53.861 INFO:teuthology.orchestra.run.smithi076.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 302 kB/s | 18 kB 00:00 2024-03-18T14:39:53.928 INFO:teuthology.orchestra.run.smithi076.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 347 kB/s | 23 kB 00:00 2024-03-18T14:39:53.978 INFO:teuthology.orchestra.run.smithi076.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 390 kB/s | 19 kB 00:00 2024-03-18T14:39:54.061 INFO:teuthology.orchestra.run.smithi076.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 236 kB/s | 20 kB 00:00 2024-03-18T14:39:54.212 INFO:teuthology.orchestra.run.smithi076.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-03-18T14:39:54.260 INFO:teuthology.orchestra.run.smithi174.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 36 kB/s | 248 kB 00:06 2024-03-18T14:39:54.270 INFO:teuthology.orchestra.run.smithi076.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 797 kB/s | 46 kB 00:00 2024-03-18T14:39:54.387 INFO:teuthology.orchestra.run.smithi076.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 676 kB/s | 79 kB 00:00 2024-03-18T14:39:54.402 INFO:teuthology.orchestra.run.smithi174.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 348 kB/s | 49 kB 00:00 2024-03-18T14:39:54.462 INFO:teuthology.orchestra.run.smithi076.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 769 kB/s | 58 kB 00:00 2024-03-18T14:39:54.502 INFO:teuthology.orchestra.run.smithi174.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 164 kB/s | 16 kB 00:00 2024-03-18T14:39:54.563 INFO:teuthology.orchestra.run.smithi076.stdout:(134/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-18T14:39:54.646 INFO:teuthology.orchestra.run.smithi076.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 197 kB/s | 16 kB 00:00 2024-03-18T14:39:54.747 INFO:teuthology.orchestra.run.smithi076.stdout:(136/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 898 kB/s | 90 kB 00:00 2024-03-18T14:39:54.805 INFO:teuthology.orchestra.run.smithi076.stdout:(137/174): python3-pyperclip-1.8.2-2.el9.noarch 483 kB/s | 28 kB 00:00 2024-03-18T14:39:54.881 INFO:teuthology.orchestra.run.smithi076.stdout:(138/174): python3-repoze-lru-0.7-16.el9.noarch 411 kB/s | 31 kB 00:00 2024-03-18T14:39:54.973 INFO:teuthology.orchestra.run.smithi076.stdout:(139/174): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-03-18T14:39:55.056 INFO:teuthology.orchestra.run.smithi076.stdout:(140/174): python3-rsa-4.9-2.el9.noarch.rpm 710 kB/s | 59 kB 00:00 2024-03-18T14:39:55.102 INFO:teuthology.orchestra.run.smithi174.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 440 kB/s | 264 kB 00:00 2024-03-18T14:39:55.202 INFO:teuthology.orchestra.run.smithi174.stdout:(36/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 394 kB/s | 17 MB 00:43 2024-03-18T14:39:55.256 INFO:teuthology.orchestra.run.smithi076.stdout:(141/174): python3-tempora-5.0.0-2.el9.noarch.r 179 kB/s | 36 kB 00:00 2024-03-18T14:39:55.349 INFO:teuthology.orchestra.run.smithi076.stdout:(142/174): python3-typing-extensions-4.4.0-2.el 556 kB/s | 51 kB 00:00 2024-03-18T14:39:55.353 INFO:teuthology.orchestra.run.smithi174.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 122 kB/s | 18 kB 00:00 2024-03-18T14:39:55.440 INFO:teuthology.orchestra.run.smithi076.stdout:(143/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-18T14:39:55.532 INFO:teuthology.orchestra.run.smithi076.stdout:(144/174): python3-websocket-client-1.2.3-2.el9 978 kB/s | 90 kB 00:00 2024-03-18T14:39:55.666 INFO:teuthology.orchestra.run.smithi076.stdout:(145/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-03-18T14:39:55.733 INFO:teuthology.orchestra.run.smithi076.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 300 kB/s | 20 kB 00:00 2024-03-18T14:39:55.761 INFO:teuthology.orchestra.run.smithi174.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 1.6 MB/s | 662 kB 00:00 2024-03-18T14:39:55.833 INFO:teuthology.orchestra.run.smithi076.stdout:(147/174): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-18T14:39:55.861 INFO:teuthology.orchestra.run.smithi174.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 9.0 MB/s | 920 kB 00:00 2024-03-18T14:39:55.895 INFO:teuthology.orchestra.run.smithi174.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-18T14:39:55.945 INFO:teuthology.orchestra.run.smithi174.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-18T14:39:55.987 INFO:teuthology.orchestra.run.smithi174.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 760 kB/s | 32 kB 00:00 2024-03-18T14:39:56.008 INFO:teuthology.orchestra.run.smithi076.stdout:(148/174): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-18T14:39:56.137 INFO:teuthology.orchestra.run.smithi174.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 20 MB/s | 3.0 MB 00:00 2024-03-18T14:39:56.188 INFO:teuthology.orchestra.run.smithi174.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 343 kB/s | 17 kB 00:00 2024-03-18T14:39:56.230 INFO:teuthology.orchestra.run.smithi174.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 242 kB/s | 10 kB 00:00 2024-03-18T14:39:56.271 INFO:teuthology.orchestra.run.smithi174.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-18T14:39:56.313 INFO:teuthology.orchestra.run.smithi174.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-18T14:39:56.372 INFO:teuthology.orchestra.run.smithi174.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-18T14:39:56.439 INFO:teuthology.orchestra.run.smithi174.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-18T14:39:56.471 INFO:teuthology.orchestra.run.smithi174.stdout:(50/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 547 kB/s | 748 kB 00:01 2024-03-18T14:39:56.496 INFO:teuthology.orchestra.run.smithi174.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 813 kB/s | 46 kB 00:00 2024-03-18T14:39:56.530 INFO:teuthology.orchestra.run.smithi174.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-18T14:39:56.555 INFO:teuthology.orchestra.run.smithi174.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-18T14:39:56.597 INFO:teuthology.orchestra.run.smithi174.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-18T14:39:56.614 INFO:teuthology.orchestra.run.smithi174.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 580 kB/s | 34 kB 00:00 2024-03-18T14:39:56.856 INFO:teuthology.orchestra.run.smithi174.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 18 MB/s | 4.7 MB 00:00 2024-03-18T14:39:56.931 INFO:teuthology.orchestra.run.smithi174.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 4.8 MB/s | 1.5 MB 00:00 2024-03-18T14:39:56.965 INFO:teuthology.orchestra.run.smithi174.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 7.4 MB/s | 811 kB 00:00 2024-03-18T14:39:56.990 INFO:teuthology.orchestra.run.smithi174.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-18T14:39:57.274 INFO:teuthology.orchestra.run.smithi174.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 18 MB/s | 5.7 MB 00:00 2024-03-18T14:39:57.308 INFO:teuthology.orchestra.run.smithi174.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 3.2 MB/s | 1.0 MB 00:00 2024-03-18T14:39:57.333 INFO:teuthology.orchestra.run.smithi174.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-18T14:39:57.652 INFO:teuthology.orchestra.run.smithi174.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-18T14:39:57.675 INFO:teuthology.orchestra.run.smithi174.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 71 kB/s | 24 kB 00:00 2024-03-18T14:39:57.725 INFO:teuthology.orchestra.run.smithi174.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 247 kB/s | 18 kB 00:00 2024-03-18T14:39:57.751 INFO:teuthology.orchestra.run.smithi174.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-18T14:39:57.793 INFO:teuthology.orchestra.run.smithi174.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 473 kB/s | 32 kB 00:00 2024-03-18T14:39:58.068 INFO:teuthology.orchestra.run.smithi174.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 4.8 MB 00:00 2024-03-18T14:39:58.093 INFO:teuthology.orchestra.run.smithi174.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 726 kB/s | 217 kB 00:00 2024-03-18T14:39:58.135 INFO:teuthology.orchestra.run.smithi174.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 174 kB 00:00 2024-03-18T14:39:58.160 INFO:teuthology.orchestra.run.smithi174.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-18T14:39:58.202 INFO:teuthology.orchestra.run.smithi174.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 612 kB/s | 41 kB 00:00 2024-03-18T14:39:58.228 INFO:teuthology.orchestra.run.smithi174.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 3.4 MB/s | 238 kB 00:00 2024-03-18T14:39:58.253 INFO:teuthology.orchestra.run.smithi174.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-18T14:39:58.278 INFO:teuthology.orchestra.run.smithi174.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-18T14:39:58.303 INFO:teuthology.orchestra.run.smithi174.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 946 kB/s | 47 kB 00:00 2024-03-18T14:39:58.328 INFO:teuthology.orchestra.run.smithi174.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 852 kB/s | 43 kB 00:00 2024-03-18T14:39:58.662 INFO:teuthology.orchestra.run.smithi174.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 116 kB/s | 39 kB 00:00 2024-03-18T14:39:58.979 INFO:teuthology.orchestra.run.smithi174.stdout:(79/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 130 kB/s | 41 kB 00:00 2024-03-18T14:39:59.129 INFO:teuthology.orchestra.run.smithi174.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-03-18T14:39:59.154 INFO:teuthology.orchestra.run.smithi174.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.7 MB/s | 300 kB 00:00 2024-03-18T14:39:59.180 INFO:teuthology.orchestra.run.smithi174.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-18T14:39:59.205 INFO:teuthology.orchestra.run.smithi174.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-18T14:39:59.230 INFO:teuthology.orchestra.run.smithi174.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-18T14:39:59.256 INFO:teuthology.orchestra.run.smithi174.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-18T14:39:59.289 INFO:teuthology.orchestra.run.smithi174.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-18T14:39:59.356 INFO:teuthology.orchestra.run.smithi174.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-18T14:39:59.382 INFO:teuthology.orchestra.run.smithi174.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-18T14:39:59.407 INFO:teuthology.orchestra.run.smithi174.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-18T14:39:59.440 INFO:teuthology.orchestra.run.smithi174.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 187 kB 00:00 2024-03-18T14:39:59.499 INFO:teuthology.orchestra.run.smithi174.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-03-18T14:39:59.549 INFO:teuthology.orchestra.run.smithi174.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 7.2 MB/s | 803 kB 00:00 2024-03-18T14:39:59.591 INFO:teuthology.orchestra.run.smithi174.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-03-18T14:39:59.633 INFO:teuthology.orchestra.run.smithi174.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-03-18T14:40:00.038 INFO:teuthology.orchestra.run.smithi077.stdout:(149/174): ceph-mgr-diskprediction-local-19.0.0 324 kB/s | 7.4 MB 00:23 2024-03-18T14:40:00.350 INFO:teuthology.orchestra.run.smithi174.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 28 kB/s | 21 kB 00:00 2024-03-18T14:40:00.542 INFO:teuthology.orchestra.run.smithi174.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 28 kB/s | 25 kB 00:00 2024-03-18T14:40:00.558 INFO:teuthology.orchestra.run.smithi076.stdout:(149/174): librados2-19.0.0-2162.g3d78f5ba.el9. 764 kB/s | 3.4 MB 00:04 2024-03-18T14:40:00.913 INFO:teuthology.orchestra.run.smithi077.stdout:(150/174): librados2-19.0.0-2162.g3d78f5ba.el9. 459 kB/s | 3.4 MB 00:07 2024-03-18T14:40:00.954 INFO:teuthology.orchestra.run.smithi077.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-18T14:40:01.563 INFO:teuthology.orchestra.run.smithi077.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 3.8 MB/s | 2.3 MB 00:00 2024-03-18T14:40:01.897 INFO:teuthology.orchestra.run.smithi077.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 5.5 MB/s | 1.8 MB 00:00 2024-03-18T14:40:01.933 INFO:teuthology.orchestra.run.smithi174.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 176 kB/s | 245 kB 00:01 2024-03-18T14:40:01.980 INFO:teuthology.orchestra.run.smithi077.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 510 kB/s | 42 kB 00:00 2024-03-18T14:40:02.064 INFO:teuthology.orchestra.run.smithi077.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-18T14:40:02.156 INFO:teuthology.orchestra.run.smithi077.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 778 kB/s | 71 kB 00:00 2024-03-18T14:40:02.223 INFO:teuthology.orchestra.run.smithi077.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 548 kB/s | 36 kB 00:00 2024-03-18T14:40:02.298 INFO:teuthology.orchestra.run.smithi077.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.5 MB/s | 192 kB 00:00 2024-03-18T14:40:02.381 INFO:teuthology.orchestra.run.smithi077.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.0 MB/s | 256 kB 00:00 2024-03-18T14:40:02.457 INFO:teuthology.orchestra.run.smithi077.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 307 kB/s | 23 kB 00:00 2024-03-18T14:40:02.524 INFO:teuthology.orchestra.run.smithi077.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 108 kB 00:00 2024-03-18T14:40:02.607 INFO:teuthology.orchestra.run.smithi077.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 155 kB 00:00 2024-03-18T14:40:02.691 INFO:teuthology.orchestra.run.smithi077.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 136 kB 00:00 2024-03-18T14:40:02.775 INFO:teuthology.orchestra.run.smithi077.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 767 kB/s | 64 kB 00:00 2024-03-18T14:40:02.866 INFO:teuthology.orchestra.run.smithi077.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 314 kB/s | 29 kB 00:00 2024-03-18T14:40:03.092 INFO:teuthology.orchestra.run.smithi174.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 214 kB/s | 248 kB 00:01 2024-03-18T14:40:03.517 INFO:teuthology.orchestra.run.smithi077.stdout:(166/174): openssl-3.0.7-27.el9.x86_64.rpm 1.9 MB/s | 1.2 MB 00:00 2024-03-18T14:40:04.034 INFO:teuthology.orchestra.run.smithi174.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 230 kB/s | 216 kB 00:00 2024-03-18T14:40:04.142 INFO:teuthology.orchestra.run.smithi077.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 3.5 MB/s | 2.2 MB 00:00 2024-03-18T14:40:04.533 INFO:teuthology.orchestra.run.smithi076.stdout:(150/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 810 kB/s | 3.1 MB 00:03 2024-03-18T14:40:04.592 INFO:teuthology.orchestra.run.smithi076.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 5.5 MB/s | 321 kB 00:00 2024-03-18T14:40:04.709 INFO:teuthology.orchestra.run.smithi077.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.0 MB/s | 2.3 MB 00:00 2024-03-18T14:40:04.725 INFO:teuthology.orchestra.run.smithi174.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 50 kB/s | 35 kB 00:00 2024-03-18T14:40:04.834 INFO:teuthology.orchestra.run.smithi077.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 3.6 MB/s | 465 kB 00:00 2024-03-18T14:40:04.910 INFO:teuthology.orchestra.run.smithi077.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 416 kB/s | 31 kB 00:00 2024-03-18T14:40:05.302 INFO:teuthology.orchestra.run.smithi077.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 2.7 MB/s | 1.1 MB 00:00 2024-03-18T14:40:05.527 INFO:teuthology.orchestra.run.smithi077.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 330 kB/s | 74 kB 00:00 2024-03-18T14:40:05.567 INFO:teuthology.orchestra.run.smithi174.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 53 kB/s | 44 kB 00:00 2024-03-18T14:40:06.192 INFO:teuthology.orchestra.run.smithi076.stdout:(152/174): ceph-mgr-diskprediction-local-19.0.0 263 kB/s | 7.4 MB 00:28 2024-03-18T14:40:06.385 INFO:teuthology.orchestra.run.smithi174.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 115 kB/s | 94 kB 00:00 2024-03-18T14:40:07.077 INFO:teuthology.orchestra.run.smithi174.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 32 kB/s | 22 kB 00:00 2024-03-18T14:40:07.452 INFO:teuthology.orchestra.run.smithi174.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 596 kB/s | 4.1 MB 00:07 2024-03-18T14:40:08.002 INFO:teuthology.orchestra.run.smithi174.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 998 kB/s | 548 kB 00:00 2024-03-18T14:40:08.152 INFO:teuthology.orchestra.run.smithi174.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 2.0 MB/s | 309 kB 00:00 2024-03-18T14:40:08.211 INFO:teuthology.orchestra.run.smithi174.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 333 kB/s | 19 kB 00:00 2024-03-18T14:40:08.777 INFO:teuthology.orchestra.run.smithi077.stdout:(173/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 368 kB/s | 3.1 MB 00:08 2024-03-18T14:40:09.017 INFO:teuthology.orchestra.run.smithi076.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 530 kB/s | 2.3 MB 00:04 2024-03-18T14:40:09.119 INFO:teuthology.orchestra.run.smithi174.stdout:(108/174): protobuf-compiler-3.14.0-13.el9.x86_ 423 kB/s | 863 kB 00:02 2024-03-18T14:40:09.270 INFO:teuthology.orchestra.run.smithi174.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 173 kB/s | 26 kB 00:00 2024-03-18T14:40:09.453 INFO:teuthology.orchestra.run.smithi174.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 265 kB/s | 49 kB 00:00 2024-03-18T14:40:09.545 INFO:teuthology.orchestra.run.smithi174.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 735 kB/s | 67 kB 00:00 2024-03-18T14:40:09.671 INFO:teuthology.orchestra.run.smithi174.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-03-18T14:40:09.767 INFO:teuthology.orchestra.run.smithi076.stdout:(154/174): NetworkManager-libnm-1.46.0-1.el9.x8 521 kB/s | 1.8 MB 00:03 2024-03-18T14:40:09.817 INFO:teuthology.orchestra.run.smithi076.stdout:(155/174): NetworkManager-team-1.46.0-1.el9.x86 53 kB/s | 42 kB 00:00 2024-03-18T14:40:10.138 INFO:teuthology.orchestra.run.smithi174.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.8 MB/s | 838 kB 00:00 2024-03-18T14:40:10.338 INFO:teuthology.orchestra.run.smithi174.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 2.1 MB/s | 4.4 MB 00:02 2024-03-18T14:40:10.471 INFO:teuthology.orchestra.run.smithi174.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 326 kB/s | 43 kB 00:00 2024-03-18T14:40:10.530 INFO:teuthology.orchestra.run.smithi174.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 1.4 MB/s | 548 kB 00:00 2024-03-18T14:40:10.555 INFO:teuthology.orchestra.run.smithi174.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 386 kB/s | 32 kB 00:00 2024-03-18T14:40:10.572 INFO:teuthology.orchestra.run.smithi174.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 359 kB/s | 15 kB 00:00 2024-03-18T14:40:10.664 INFO:teuthology.orchestra.run.smithi174.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-03-18T14:40:10.692 INFO:teuthology.orchestra.run.smithi076.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 82 kB/s | 71 kB 00:00 2024-03-18T14:40:10.831 INFO:teuthology.orchestra.run.smithi174.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 1.4 MB/s | 358 kB 00:00 2024-03-18T14:40:10.923 INFO:teuthology.orchestra.run.smithi174.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.1 MB/s | 303 kB 00:00 2024-03-18T14:40:10.948 INFO:teuthology.orchestra.run.smithi174.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 302 kB/s | 35 kB 00:00 2024-03-18T14:40:11.009 INFO:teuthology.orchestra.run.smithi076.stdout:(157/174): NetworkManager-tui-1.46.0-1.el9.x86_ 200 kB/s | 248 kB 00:01 2024-03-18T14:40:11.057 INFO:teuthology.orchestra.run.smithi174.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 1.5 MB/s | 203 kB 00:00 2024-03-18T14:40:11.157 INFO:teuthology.orchestra.run.smithi174.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-18T14:40:11.232 INFO:teuthology.orchestra.run.smithi174.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 143 kB/s | 11 kB 00:00 2024-03-18T14:40:11.316 INFO:teuthology.orchestra.run.smithi174.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 212 kB/s | 18 kB 00:00 2024-03-18T14:40:11.399 INFO:teuthology.orchestra.run.smithi174.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 278 kB/s | 23 kB 00:00 2024-03-18T14:40:11.458 INFO:teuthology.orchestra.run.smithi174.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 334 kB/s | 19 kB 00:00 2024-03-18T14:40:11.517 INFO:teuthology.orchestra.run.smithi174.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 337 kB/s | 20 kB 00:00 2024-03-18T14:40:11.518 INFO:teuthology.orchestra.run.smithi076.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 44 kB/s | 36 kB 00:00 2024-03-18T14:40:12.233 INFO:teuthology.orchestra.run.smithi174.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 1.4 MB/s | 1.0 MB 00:00 2024-03-18T14:40:12.235 INFO:teuthology.orchestra.run.smithi076.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 160 kB/s | 192 kB 00:01 2024-03-18T14:40:12.367 INFO:teuthology.orchestra.run.smithi174.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 347 kB/s | 46 kB 00:00 2024-03-18T14:40:12.426 INFO:teuthology.orchestra.run.smithi174.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-18T14:40:12.484 INFO:teuthology.orchestra.run.smithi174.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 993 kB/s | 58 kB 00:00 2024-03-18T14:40:12.751 INFO:teuthology.orchestra.run.smithi076.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 207 kB/s | 256 kB 00:01 2024-03-18T14:40:12.759 INFO:teuthology.orchestra.run.smithi174.stdout:(134/174): python3-pecan-1.4.2-3.el9.noarch.rpm 990 kB/s | 272 kB 00:00 2024-03-18T14:40:12.818 INFO:teuthology.orchestra.run.smithi174.stdout:(135/174): python3-grpcio-1.46.7-10.el9.x86_64. 1.1 MB/s | 2.0 MB 00:01 2024-03-18T14:40:12.843 INFO:teuthology.orchestra.run.smithi174.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-03-18T14:40:12.869 INFO:teuthology.orchestra.run.smithi174.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-03-18T14:40:12.885 INFO:teuthology.orchestra.run.smithi076.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 35 kB/s | 23 kB 00:00 2024-03-18T14:40:12.910 INFO:teuthology.orchestra.run.smithi174.stdout:(138/174): python3-repoze-lru-0.7-16.el9.noarch 742 kB/s | 31 kB 00:00 2024-03-18T14:40:12.969 INFO:teuthology.orchestra.run.smithi174.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 225 kB/s | 28 kB 00:00 2024-03-18T14:40:12.994 INFO:teuthology.orchestra.run.smithi174.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-03-18T14:40:13.036 INFO:teuthology.orchestra.run.smithi174.stdout:(141/174): python3-tempora-5.0.0-2.el9.noarch.r 864 kB/s | 36 kB 00:00 2024-03-18T14:40:13.061 INFO:teuthology.orchestra.run.smithi174.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 642 kB/s | 59 kB 00:00 2024-03-18T14:40:13.087 INFO:teuthology.orchestra.run.smithi174.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-18T14:40:13.146 INFO:teuthology.orchestra.run.smithi174.stdout:(144/174): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-03-18T14:40:13.229 INFO:teuthology.orchestra.run.smithi174.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-03-18T14:40:13.279 INFO:teuthology.orchestra.run.smithi174.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-03-18T14:40:13.304 INFO:teuthology.orchestra.run.smithi174.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-03-18T14:40:13.310 INFO:teuthology.orchestra.run.smithi077.stdout:(174/174): ceph-test-19.0.0-2162.g3d78f5ba.el9. 749 kB/s | 48 MB 01:05 2024-03-18T14:40:13.313 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-03-18T14:40:13.313 INFO:teuthology.orchestra.run.smithi077.stdout:Total 2.9 MB/s | 224 MB 01:16 2024-03-18T14:40:13.347 INFO:teuthology.orchestra.run.smithi174.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-18T14:40:13.502 INFO:teuthology.orchestra.run.smithi076.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 144 kB/s | 108 kB 00:00 2024-03-18T14:40:13.764 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-03-18T14:40:13.794 INFO:teuthology.orchestra.run.smithi076.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 170 kB/s | 155 kB 00:00 2024-03-18T14:40:13.913 INFO:teuthology.orchestra.run.smithi174.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 2.6 MB/s | 1.6 MB 00:00 2024-03-18T14:40:14.032 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-03-18T14:40:14.032 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-03-18T14:40:14.302 INFO:teuthology.orchestra.run.smithi076.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 170 kB/s | 136 kB 00:00 2024-03-18T14:40:14.494 INFO:teuthology.orchestra.run.smithi076.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 91 kB/s | 64 kB 00:00 2024-03-18T14:40:14.953 INFO:teuthology.orchestra.run.smithi076.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 44 kB/s | 29 kB 00:00 2024-03-18T14:40:15.811 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-03-18T14:40:15.811 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-03-18T14:40:17.403 INFO:teuthology.orchestra.run.smithi076.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 426 kB/s | 1.2 MB 00:02 2024-03-18T14:40:18.796 INFO:teuthology.orchestra.run.smithi174.stdout:(150/174): librados2-19.0.0-2162.g3d78f5ba.el9. 638 kB/s | 3.4 MB 00:05 2024-03-18T14:40:18.846 INFO:teuthology.orchestra.run.smithi174.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-18T14:40:19.030 INFO:teuthology.orchestra.run.smithi174.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 13 MB/s | 2.3 MB 00:00 2024-03-18T14:40:19.180 INFO:teuthology.orchestra.run.smithi174.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 12 MB/s | 1.8 MB 00:00 2024-03-18T14:40:19.239 INFO:teuthology.orchestra.run.smithi174.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 727 kB/s | 42 kB 00:00 2024-03-18T14:40:19.314 INFO:teuthology.orchestra.run.smithi174.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 3.2 MB/s | 248 kB 00:00 2024-03-18T14:40:19.373 INFO:teuthology.orchestra.run.smithi174.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-18T14:40:19.394 INFO:teuthology.orchestra.run.smithi076.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 497 kB/s | 2.2 MB 00:04 2024-03-18T14:40:19.423 INFO:teuthology.orchestra.run.smithi174.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 731 kB/s | 36 kB 00:00 2024-03-18T14:40:19.507 INFO:teuthology.orchestra.run.smithi174.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.3 MB/s | 192 kB 00:00 2024-03-18T14:40:19.599 INFO:teuthology.orchestra.run.smithi174.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 256 kB 00:00 2024-03-18T14:40:19.649 INFO:teuthology.orchestra.run.smithi174.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 462 kB/s | 23 kB 00:00 2024-03-18T14:40:19.716 INFO:teuthology.orchestra.run.smithi174.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 108 kB 00:00 2024-03-18T14:40:19.783 INFO:teuthology.orchestra.run.smithi174.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 155 kB 00:00 2024-03-18T14:40:19.850 INFO:teuthology.orchestra.run.smithi174.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 136 kB 00:00 2024-03-18T14:40:19.908 INFO:teuthology.orchestra.run.smithi174.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-03-18T14:40:19.959 INFO:teuthology.orchestra.run.smithi174.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 575 kB/s | 29 kB 00:00 2024-03-18T14:40:20.109 INFO:teuthology.orchestra.run.smithi174.stdout:(166/174): openssl-3.0.7-27.el9.x86_64.rpm 8.1 MB/s | 1.2 MB 00:00 2024-03-18T14:40:20.326 INFO:teuthology.orchestra.run.smithi174.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 10 MB/s | 2.2 MB 00:00 2024-03-18T14:40:20.534 INFO:teuthology.orchestra.run.smithi174.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 11 MB/s | 2.3 MB 00:00 2024-03-18T14:40:20.635 INFO:teuthology.orchestra.run.smithi174.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.5 MB/s | 465 kB 00:00 2024-03-18T14:40:20.685 INFO:teuthology.orchestra.run.smithi174.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 626 kB/s | 31 kB 00:00 2024-03-18T14:40:20.835 INFO:teuthology.orchestra.run.smithi174.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.2 MB/s | 1.1 MB 00:00 2024-03-18T14:40:20.920 INFO:teuthology.orchestra.run.smithi076.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 305 kB/s | 465 kB 00:01 2024-03-18T14:40:21.620 INFO:teuthology.orchestra.run.smithi076.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 555 kB/s | 2.3 MB 00:04 2024-03-18T14:40:21.670 INFO:teuthology.orchestra.run.smithi076.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 41 kB/s | 31 kB 00:00 2024-03-18T14:40:21.677 INFO:teuthology.orchestra.run.smithi174.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 88 kB/s | 74 kB 00:00 2024-03-18T14:40:21.845 INFO:teuthology.orchestra.run.smithi076.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 426 kB/s | 74 kB 00:00 2024-03-18T14:40:22.633 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-03-18T14:40:22.747 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T14:40:22.773 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T14:40:22.804 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T14:40:22.813 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T14:40:22.853 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T14:40:22.908 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T14:40:22.941 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T14:40:22.966 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T14:40:23.008 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T14:40:23.338 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T14:40:23.392 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T14:40:23.517 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T14:40:23.685 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T14:40:23.759 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T14:40:23.788 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T14:40:23.832 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T14:40:23.945 INFO:teuthology.orchestra.run.smithi076.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 473 kB/s | 1.1 MB 00:02 2024-03-18T14:40:24.020 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T14:40:24.043 INFO:teuthology.orchestra.run.smithi174.stdout:(173/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 318 kB/s | 3.1 MB 00:10 2024-03-18T14:40:24.069 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T14:40:24.097 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T14:40:24.201 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T14:40:24.223 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T14:40:24.289 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T14:40:24.329 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T14:40:24.352 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T14:40:24.368 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T14:40:24.543 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T14:40:24.574 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T14:40:24.606 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T14:40:24.703 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T14:40:24.783 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T14:40:24.844 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T14:40:24.916 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T14:40:24.961 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T14:40:25.158 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T14:40:25.266 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T14:40:25.361 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T14:40:25.399 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T14:40:25.426 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T14:40:25.461 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T14:40:25.535 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T14:40:25.590 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T14:40:25.628 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T14:40:25.692 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T14:40:25.727 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T14:40:25.771 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T14:40:25.795 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T14:40:25.822 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T14:40:25.891 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T14:40:25.928 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T14:40:25.971 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T14:40:26.000 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T14:40:26.069 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T14:40:26.114 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T14:40:26.187 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T14:40:26.329 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T14:40:26.383 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T14:40:26.903 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T14:40:26.978 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T14:40:27.058 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T14:40:27.107 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T14:40:27.114 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T14:40:27.147 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T14:40:27.195 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T14:40:27.233 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T14:40:27.263 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T14:40:27.501 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T14:40:27.697 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T14:40:27.766 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T14:40:27.810 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T14:40:27.952 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T14:40:27.995 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T14:40:28.051 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T14:40:28.123 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T14:40:28.162 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T14:40:28.198 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T14:40:28.246 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T14:40:28.281 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T14:40:28.324 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T14:40:28.367 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T14:40:28.502 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T14:40:28.547 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T14:40:28.577 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T14:40:28.619 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T14:40:28.714 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T14:40:28.752 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T14:40:28.792 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T14:40:28.818 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T14:40:28.842 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T14:40:28.915 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T14:40:29.036 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T14:40:29.155 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T14:40:29.201 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T14:40:29.249 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T14:40:29.304 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T14:40:29.327 INFO:teuthology.orchestra.run.smithi174.stdout:(174/174): ceph-test-19.0.0-2162.g3d78f5ba.el9. 659 kB/s | 48 MB 01:13 2024-03-18T14:40:29.331 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-18T14:40:29.331 INFO:teuthology.orchestra.run.smithi174.stdout:Total 2.4 MB/s | 224 MB 01:33 2024-03-18T14:40:29.488 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T14:40:29.538 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T14:40:29.667 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T14:40:29.703 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T14:40:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T14:40:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T14:40:29.775 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-18T14:40:29.802 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T14:40:29.837 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T14:40:29.876 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T14:40:29.907 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T14:40:29.937 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T14:40:30.015 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T14:40:30.045 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-18T14:40:30.046 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-18T14:40:30.048 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T14:40:30.081 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T14:40:30.123 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T14:40:30.157 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T14:40:30.220 INFO:teuthology.orchestra.run.smithi076.stdout:(174/174): ceph-test-19.0.0-2162.g3d78f5ba.el9. 648 kB/s | 48 MB 01:15 2024-03-18T14:40:30.224 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-18T14:40:30.224 INFO:teuthology.orchestra.run.smithi076.stdout:Total 2.4 MB/s | 224 MB 01:35 2024-03-18T14:40:30.246 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T14:40:30.335 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T14:40:30.380 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T14:40:30.412 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T14:40:30.458 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T14:40:30.556 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T14:40:30.613 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T14:40:30.666 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-18T14:40:30.978 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-18T14:40:30.979 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-18T14:40:31.335 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T14:40:31.376 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T14:40:31.418 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T14:40:31.467 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T14:40:31.554 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T14:40:31.594 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T14:40:31.752 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-18T14:40:31.752 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-18T14:40:31.772 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T14:40:31.800 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T14:40:31.877 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T14:40:32.502 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T14:40:32.624 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T14:40:32.998 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-18T14:40:32.999 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-18T14:40:33.842 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T14:40:33.934 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T14:40:33.964 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T14:40:33.996 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T14:40:34.023 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T14:40:34.338 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T14:40:34.370 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T14:40:34.443 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T14:40:34.475 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T14:40:34.511 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T14:40:34.974 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T14:40:35.007 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T14:40:35.069 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T14:40:35.091 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T14:40:37.266 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T14:40:37.296 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T14:40:37.829 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T14:40:37.891 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T14:40:37.963 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T14:40:37.964 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T14:40:37.964 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T14:40:37.964 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-18T14:40:37.992 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T14:40:38.626 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-18T14:40:38.738 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T14:40:38.764 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T14:40:38.795 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T14:40:38.804 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T14:40:38.844 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T14:40:38.898 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T14:40:38.931 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T14:40:38.957 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T14:40:38.999 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T14:40:39.324 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T14:40:39.375 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T14:40:39.497 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T14:40:39.526 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T14:40:39.600 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T14:40:39.630 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T14:40:39.667 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T14:40:39.842 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T14:40:39.903 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T14:40:39.922 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T14:40:40.027 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T14:40:40.039 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-18T14:40:40.057 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T14:40:40.121 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T14:40:40.150 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T14:40:40.152 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T14:40:40.174 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T14:40:40.180 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T14:40:40.190 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T14:40:40.218 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T14:40:40.228 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T14:40:40.268 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T14:40:40.331 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T14:40:40.342 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T14:40:40.363 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T14:40:40.372 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T14:40:40.398 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T14:40:40.412 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T14:40:40.452 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T14:40:40.519 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T14:40:40.614 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T14:40:40.674 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T14:40:40.743 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T14:40:40.778 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T14:40:40.800 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T14:40:40.890 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T14:40:41.011 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T14:40:41.011 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T14:40:41.041 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T14:40:41.122 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T14:40:41.142 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T14:40:41.175 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T14:40:41.217 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T14:40:41.221 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T14:40:41.247 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T14:40:41.298 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T14:40:41.334 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T14:40:41.406 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T14:40:41.408 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T14:40:41.459 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T14:40:41.470 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T14:40:41.487 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T14:40:41.508 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T14:40:41.573 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T14:40:41.582 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T14:40:41.607 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T14:40:41.611 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T14:40:41.651 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T14:40:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T14:40:41.683 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T14:40:41.711 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T14:40:41.717 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T14:40:41.740 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T14:40:41.755 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T14:40:41.781 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T14:40:41.816 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T14:40:41.859 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T14:40:41.888 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T14:40:41.948 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T14:40:41.953 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T14:40:41.981 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T14:40:41.994 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T14:40:42.004 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T14:40:42.067 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T14:40:42.188 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T14:40:42.201 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T14:40:42.255 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T14:40:42.315 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T14:40:42.375 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T14:40:42.482 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T14:40:42.526 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T14:40:42.713 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T14:40:42.773 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T14:40:42.814 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T14:40:42.850 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T14:40:42.899 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T14:40:42.920 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T14:40:42.941 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T14:40:42.970 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T14:40:42.975 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T14:40:42.978 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T14:40:43.010 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T14:40:43.010 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T14:40:43.050 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T14:40:43.082 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T14:40:43.088 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T14:40:43.118 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T14:40:43.128 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T14:40:43.176 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T14:40:43.239 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T14:40:43.276 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T14:40:43.319 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T14:40:43.352 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T14:40:43.352 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T14:40:43.380 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T14:40:43.447 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T14:40:43.485 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T14:40:43.513 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T14:40:43.531 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T14:40:43.556 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T14:40:43.561 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T14:40:43.605 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T14:40:43.616 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T14:40:43.662 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T14:40:43.735 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T14:40:43.747 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T14:40:43.791 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T14:40:43.843 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T14:40:43.866 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T14:40:43.907 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T14:40:43.914 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T14:40:43.939 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T14:40:43.975 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T14:40:44.022 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T14:40:44.058 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T14:40:44.101 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T14:40:44.143 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T14:40:44.276 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T14:40:44.324 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T14:40:44.354 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T14:40:44.395 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T14:40:44.421 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T14:40:44.490 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T14:40:44.509 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T14:40:44.528 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T14:40:44.568 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T14:40:44.592 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T14:40:44.594 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T14:40:44.619 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T14:40:44.639 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T14:40:44.646 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T14:40:44.680 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T14:40:44.690 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T14:40:44.720 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T14:40:44.757 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T14:40:44.788 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T14:40:44.828 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T14:40:44.933 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T14:40:44.977 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T14:40:45.026 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T14:40:45.066 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T14:40:45.081 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T14:40:45.253 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T14:40:45.264 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T14:40:45.314 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T14:40:45.331 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T14:40:45.375 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T14:40:45.443 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T14:40:45.478 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T14:40:45.510 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T14:40:45.511 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T14:40:45.541 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T14:40:45.553 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T14:40:45.569 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T14:40:45.604 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T14:40:45.608 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T14:40:45.643 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T14:40:45.674 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T14:40:45.682 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T14:40:45.704 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T14:40:45.712 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T14:40:45.749 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T14:40:45.779 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T14:40:45.796 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T14:40:45.814 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T14:40:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T14:40:45.848 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T14:40:45.875 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T14:40:45.889 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T14:40:45.917 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T14:40:45.924 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T14:40:46.012 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T14:40:46.049 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T14:40:46.094 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T14:40:46.097 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T14:40:46.128 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T14:40:46.138 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T14:40:46.169 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T14:40:46.177 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T14:40:46.225 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T14:40:46.262 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T14:40:46.302 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T14:40:46.322 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T14:40:46.343 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T14:40:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T14:40:46.370 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T14:40:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T14:40:46.523 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T14:40:46.644 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T14:40:46.752 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T14:40:46.793 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T14:40:46.841 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T14:40:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T14:40:47.080 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T14:40:47.092 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T14:40:47.131 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T14:40:47.142 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T14:40:47.176 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T14:40:47.224 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T14:40:47.258 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T14:40:47.296 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T14:40:47.313 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T14:40:47.328 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T14:40:47.352 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T14:40:47.358 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T14:40:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T14:40:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T14:40:47.460 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T14:40:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T14:40:47.522 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T14:40:47.530 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T14:40:47.558 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T14:40:47.597 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T14:40:47.632 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T14:40:47.635 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T14:40:47.665 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T14:40:47.707 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T14:40:47.741 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T14:40:47.830 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T14:40:47.909 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T14:40:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T14:40:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T14:40:48.043 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T14:40:48.139 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T14:40:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T14:40:48.277 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T14:40:48.388 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T14:40:48.911 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T14:40:48.954 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T14:40:48.988 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T14:40:49.036 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T14:40:49.115 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T14:40:49.148 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T14:40:49.334 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T14:40:49.371 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T14:40:49.448 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T14:40:49.599 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T14:40:49.674 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T14:40:49.705 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T14:40:49.736 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T14:40:49.749 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T14:40:49.749 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2024-03-18T14:40:49.749 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2024-03-18T14:40:49.749 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2024-03-18T14:40:49.749 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2024-03-18T14:40:49.749 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2024-03-18T14:40:49.750 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /root 2024-03-18T14:40:49.750 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2024-03-18T14:40:49.750 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-18T14:40:49.763 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T14:40:49.914 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T14:40:49.928 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T14:40:49.928 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:49.928 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T14:40:49.928 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-18T14:40:50.086 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T14:40:50.092 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T14:40:50.201 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T14:40:50.235 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T14:40:50.318 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T14:40:50.348 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T14:40:50.385 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T14:40:50.849 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T14:40:50.888 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T14:40:50.959 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T14:40:50.990 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T14:40:51.408 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T14:40:51.433 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T14:40:51.433 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:51.433 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T14:40:51.433 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T14:40:51.433 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T14:40:51.433 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-18T14:40:51.438 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T14:40:51.513 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T14:40:51.546 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T14:40:51.575 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T14:40:51.602 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T14:40:51.658 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T14:40:51.680 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T14:40:51.680 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:51.680 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T14:40:51.680 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T14:40:51.680 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T14:40:51.680 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-18T14:40:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T14:40:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T14:40:52.048 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T14:40:52.096 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T14:40:52.125 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T14:40:52.133 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T14:40:52.147 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T14:40:52.148 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:52.148 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T14:40:52.148 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T14:40:52.148 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T14:40:52.148 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-18T14:40:52.235 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T14:40:52.596 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T14:40:52.628 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T14:40:52.645 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T14:40:52.676 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T14:40:52.699 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T14:40:52.699 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T14:40:52.721 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T14:40:52.751 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T14:40:52.782 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T14:40:53.015 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T14:40:53.049 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T14:40:53.259 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T14:40:53.311 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T14:40:53.852 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T14:40:53.889 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T14:40:53.969 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T14:40:53.970 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T14:40:53.970 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T14:40:53.970 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:40:53.998 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T14:40:54.014 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T14:40:54.042 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T14:40:54.141 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T14:40:54.296 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T14:40:54.327 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T14:40:54.350 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T14:40:54.350 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:54.350 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T14:40:54.350 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T14:40:54.350 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T14:40:54.350 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-18T14:40:54.394 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T14:40:54.406 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T14:40:54.451 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T14:40:54.991 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T14:40:55.034 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T14:40:55.322 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T14:40:55.355 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T14:40:55.377 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T14:40:55.377 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:55.378 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T14:40:55.378 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T14:40:55.378 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T14:40:55.378 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-18T14:40:55.422 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T14:40:55.444 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T14:40:55.444 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:55.444 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T14:40:55.444 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-18T14:40:55.489 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T14:40:55.511 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T14:40:55.511 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:55.511 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T14:40:55.512 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T14:40:55.512 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T14:40:55.512 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-18T14:40:55.588 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T14:40:55.630 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T14:40:55.706 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T14:40:55.706 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T14:40:55.706 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T14:40:55.706 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:40:55.739 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T14:40:55.823 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T14:40:55.845 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T14:40:55.845 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:55.845 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T14:40:55.846 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T14:40:55.846 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T14:40:55.846 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-18T14:41:00.294 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T14:41:00.338 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T14:41:00.366 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T14:41:00.436 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T14:41:00.615 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T14:41:00.745 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T14:41:00.782 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T14:41:00.821 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T14:41:00.874 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T14:41:00.913 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T14:41:00.942 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T14:41:00.991 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T14:41:01.021 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T14:41:01.066 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T14:41:01.136 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T14:41:01.170 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T14:41:01.771 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T14:41:01.810 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T14:41:01.829 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T14:41:02.627 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T14:41:02.704 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T14:41:02.745 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T14:41:02.828 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T14:41:02.875 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T14:41:02.900 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T14:41:02.925 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T14:41:02.951 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T14:41:02.951 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T14:41:03.031 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T14:41:03.061 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T14:41:03.061 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T14:41:03.129 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T14:41:03.155 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T14:41:03.196 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T14:41:03.222 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T14:41:03.246 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T14:41:03.279 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T14:41:03.327 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T14:41:03.368 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T14:41:03.393 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T14:41:03.418 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T14:41:03.444 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T14:41:03.469 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T14:41:03.475 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T14:41:03.475 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T14:41:03.655 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T14:41:03.655 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T14:41:03.738 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T14:41:03.763 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T14:41:03.788 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T14:41:03.821 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T14:41:03.847 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T14:41:03.912 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T14:41:05.624 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T14:41:05.624 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /sys 2024-03-18T14:41:05.625 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /proc 2024-03-18T14:41:05.625 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /mnt 2024-03-18T14:41:05.625 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /var/tmp 2024-03-18T14:41:05.625 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /home 2024-03-18T14:41:05.625 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /root 2024-03-18T14:41:05.625 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /tmp 2024-03-18T14:41:05.625 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:41:05.803 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T14:41:05.817 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T14:41:05.818 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:05.818 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T14:41:05.818 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:41:07.314 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T14:41:07.339 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T14:41:07.339 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:07.339 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T14:41:07.339 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T14:41:07.339 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T14:41:07.339 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:41:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T14:41:07.587 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T14:41:07.587 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:07.587 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T14:41:07.587 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T14:41:07.587 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T14:41:07.587 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T14:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /sys 2024-03-18T14:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /proc 2024-03-18T14:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /mnt 2024-03-18T14:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /var/tmp 2024-03-18T14:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /home 2024-03-18T14:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /root 2024-03-18T14:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /tmp 2024-03-18T14:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:41:07.989 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T14:41:08.004 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T14:41:08.004 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:08.004 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T14:41:08.004 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:41:08.023 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T14:41:08.046 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T14:41:08.046 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:08.046 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T14:41:08.046 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T14:41:08.046 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T14:41:08.046 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:41:08.133 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T14:41:08.592 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T14:41:08.616 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T14:41:08.616 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T14:41:08.616 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T14:41:08.616 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T14:41:08.616 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T14:41:08.616 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T14:41:08.616 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T14:41:08.616 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T14:41:08.616 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T14:41:08.616 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T14:41:08.617 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T14:41:08.617 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T14:41:08.617 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T14:41:08.617 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T14:41:08.617 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T14:41:08.617 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T14:41:08.617 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T14:41:08.617 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T14:41:08.617 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T14:41:08.617 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T14:41:08.617 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T14:41:08.618 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T14:41:08.618 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T14:41:08.618 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T14:41:08.618 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T14:41:08.618 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T14:41:08.618 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T14:41:08.618 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T14:41:08.618 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T14:41:08.618 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T14:41:08.618 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T14:41:08.618 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T14:41:08.618 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T14:41:08.619 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T14:41:08.619 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T14:41:08.619 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T14:41:08.619 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T14:41:08.619 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T14:41:08.619 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T14:41:08.619 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T14:41:08.619 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T14:41:08.619 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T14:41:08.619 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T14:41:08.619 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T14:41:08.619 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T14:41:08.619 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T14:41:08.619 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T14:41:08.620 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T14:41:08.620 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T14:41:08.620 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T14:41:08.620 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T14:41:08.620 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T14:41:08.620 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T14:41:08.620 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T14:41:08.620 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T14:41:08.620 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T14:41:08.620 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T14:41:08.620 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T14:41:08.620 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T14:41:08.620 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T14:41:08.620 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T14:41:08.620 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T14:41:08.621 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T14:41:08.621 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T14:41:08.621 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T14:41:08.621 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T14:41:08.621 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T14:41:08.621 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T14:41:08.621 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T14:41:08.621 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T14:41:08.621 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T14:41:08.621 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T14:41:08.621 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T14:41:08.621 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T14:41:08.621 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T14:41:08.621 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T14:41:08.621 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T14:41:08.621 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T14:41:08.622 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T14:41:08.622 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T14:41:08.622 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T14:41:08.622 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T14:41:08.622 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T14:41:08.622 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T14:41:08.622 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T14:41:08.622 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T14:41:08.622 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T14:41:08.622 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T14:41:08.622 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T14:41:08.622 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T14:41:08.622 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T14:41:08.622 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T14:41:08.622 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T14:41:08.623 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T14:41:08.623 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T14:41:08.623 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T14:41:08.623 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T14:41:08.623 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T14:41:08.623 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T14:41:08.623 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T14:41:08.623 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T14:41:08.623 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T14:41:08.623 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T14:41:08.623 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T14:41:08.623 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T14:41:08.623 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T14:41:08.623 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T14:41:08.623 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T14:41:08.624 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T14:41:08.624 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T14:41:08.624 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T14:41:08.624 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T14:41:08.624 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T14:41:08.624 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T14:41:08.624 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T14:41:08.624 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T14:41:08.624 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T14:41:08.624 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T14:41:08.624 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T14:41:08.624 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T14:41:08.624 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T14:41:08.624 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T14:41:08.624 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T14:41:08.624 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T14:41:08.625 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T14:41:08.625 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T14:41:08.625 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T14:41:08.625 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T14:41:08.625 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T14:41:08.625 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T14:41:08.625 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T14:41:08.625 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T14:41:08.625 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T14:41:08.625 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T14:41:08.625 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T14:41:08.625 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T14:41:08.625 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T14:41:08.625 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T14:41:08.625 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T14:41:08.625 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T14:41:08.626 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T14:41:08.626 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T14:41:08.626 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T14:41:08.626 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T14:41:08.626 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T14:41:08.626 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T14:41:08.626 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T14:41:08.626 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T14:41:08.626 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T14:41:08.626 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T14:41:08.626 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T14:41:08.626 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T14:41:08.626 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T14:41:08.626 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T14:41:08.626 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T14:41:08.627 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T14:41:08.627 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T14:41:08.627 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T14:41:08.627 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T14:41:08.627 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T14:41:08.627 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T14:41:08.627 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T14:41:08.627 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T14:41:08.627 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T14:41:08.627 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T14:41:08.627 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T14:41:08.627 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T14:41:08.627 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T14:41:08.627 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T14:41:08.627 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T14:41:08.628 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T14:41:08.628 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T14:41:08.628 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T14:41:08.628 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T14:41:08.628 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T14:41:08.628 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T14:41:08.628 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T14:41:08.628 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T14:41:08.628 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T14:41:08.628 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T14:41:08.628 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T14:41:08.628 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T14:41:08.628 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T14:41:08.628 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T14:41:08.628 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T14:41:08.629 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T14:41:08.629 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T14:41:08.629 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T14:41:08.629 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T14:41:08.629 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T14:41:08.629 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T14:41:08.629 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T14:41:08.629 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T14:41:08.629 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T14:41:08.629 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T14:41:08.629 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T14:41:08.629 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T14:41:08.629 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T14:41:08.630 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T14:41:08.655 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T14:41:08.715 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T14:41:08.747 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T14:41:08.979 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T14:41:09.014 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T14:41:09.474 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T14:41:09.499 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T14:41:09.499 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:09.499 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T14:41:09.499 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T14:41:09.499 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T14:41:09.499 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:41:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T14:41:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T14:41:09.755 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:09.755 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T14:41:09.756 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T14:41:09.756 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T14:41:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:41:09.971 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T14:41:09.997 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T14:41:10.096 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T14:41:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T14:41:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T14:41:10.241 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:10.241 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T14:41:10.242 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T14:41:10.242 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T14:41:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:41:10.251 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T14:41:10.274 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T14:41:10.297 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T14:41:10.297 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:10.297 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T14:41:10.297 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T14:41:10.297 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T14:41:10.297 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:41:10.329 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T14:41:10.342 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T14:41:10.352 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T14:41:10.398 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T14:41:10.746 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T14:41:10.778 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T14:41:10.810 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T14:41:10.869 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T14:41:10.892 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T14:41:11.128 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T14:41:11.160 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T14:41:11.265 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T14:41:11.302 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T14:41:11.324 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T14:41:11.324 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:11.324 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T14:41:11.324 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T14:41:11.324 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T14:41:11.324 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:41:11.369 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T14:41:11.391 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T14:41:11.391 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:11.391 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T14:41:11.392 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:41:11.436 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T14:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T14:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T14:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T14:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T14:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:41:11.506 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T14:41:11.506 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-18T14:41:11.506 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2024-03-18T14:41:11.506 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T14:41:11.506 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T14:41:11.506 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T14:41:11.506 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T14:41:11.506 INFO:teuthology.orchestra.run.smithi077.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T14:41:11.506 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T14:41:11.507 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T14:41:11.507 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T14:41:11.507 INFO:teuthology.orchestra.run.smithi077.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T14:41:11.507 INFO:teuthology.orchestra.run.smithi077.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T14:41:11.507 INFO:teuthology.orchestra.run.smithi077.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T14:41:11.507 INFO:teuthology.orchestra.run.smithi077.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T14:41:11.507 INFO:teuthology.orchestra.run.smithi077.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T14:41:11.507 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.507 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.507 INFO:teuthology.orchestra.run.smithi077.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T14:41:11.507 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T14:41:11.507 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T14:41:11.507 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T14:41:11.507 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T14:41:11.508 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T14:41:11.508 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T14:41:11.508 INFO:teuthology.orchestra.run.smithi077.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T14:41:11.508 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T14:41:11.508 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-03-18T14:41:11.508 INFO:teuthology.orchestra.run.smithi077.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T14:41:11.508 INFO:teuthology.orchestra.run.smithi077.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T14:41:11.508 INFO:teuthology.orchestra.run.smithi077.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T14:41:11.508 INFO:teuthology.orchestra.run.smithi077.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T14:41:11.508 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.508 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.508 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.508 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.508 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:11.508 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.508 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.509 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.509 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:11.509 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:11.509 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:11.509 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:11.509 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:11.509 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.509 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.509 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:11.509 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.509 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.509 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.509 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:11.509 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:11.510 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.510 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.510 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:11.510 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T14:41:11.510 INFO:teuthology.orchestra.run.smithi077.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T14:41:11.510 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T14:41:11.510 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T14:41:11.510 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T14:41:11.510 INFO:teuthology.orchestra.run.smithi077.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T14:41:11.510 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T14:41:11.510 INFO:teuthology.orchestra.run.smithi077.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T14:41:11.510 INFO:teuthology.orchestra.run.smithi077.stdout: jq-1.6-16.el9.x86_64 2024-03-18T14:41:11.511 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T14:41:11.511 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T14:41:11.511 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T14:41:11.511 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.511 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.511 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.511 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T14:41:11.511 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T14:41:11.511 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T14:41:11.511 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T14:41:11.511 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T14:41:11.511 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.511 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.511 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T14:41:11.511 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.511 INFO:teuthology.orchestra.run.smithi077.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T14:41:11.512 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T14:41:11.512 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T14:41:11.512 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T14:41:11.512 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T14:41:11.512 INFO:teuthology.orchestra.run.smithi077.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T14:41:11.512 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T14:41:11.512 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T14:41:11.512 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T14:41:11.512 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T14:41:11.512 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T14:41:11.513 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T14:41:11.513 INFO:teuthology.orchestra.run.smithi077.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T14:41:11.513 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T14:41:11.513 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T14:41:11.513 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T14:41:11.513 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T14:41:11.513 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T14:41:11.513 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T14:41:11.513 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T14:41:11.513 INFO:teuthology.orchestra.run.smithi077.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T14:41:11.513 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T14:41:11.513 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T14:41:11.513 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T14:41:11.514 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.514 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.514 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.514 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T14:41:11.514 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T14:41:11.514 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T14:41:11.514 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T14:41:11.514 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T14:41:11.514 INFO:teuthology.orchestra.run.smithi077.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T14:41:11.514 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T14:41:11.515 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T14:41:11.515 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T14:41:11.515 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T14:41:11.515 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T14:41:11.515 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T14:41:11.515 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T14:41:11.515 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T14:41:11.515 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T14:41:11.515 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T14:41:11.515 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T14:41:11.515 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T14:41:11.515 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T14:41:11.515 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T14:41:11.515 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T14:41:11.515 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T14:41:11.516 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T14:41:11.516 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T14:41:11.516 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T14:41:11.516 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T14:41:11.516 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T14:41:11.516 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T14:41:11.516 INFO:teuthology.orchestra.run.smithi077.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T14:41:11.516 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T14:41:11.516 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T14:41:11.516 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T14:41:11.516 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T14:41:11.516 INFO:teuthology.orchestra.run.smithi077.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T14:41:11.516 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T14:41:11.516 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T14:41:11.516 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T14:41:11.516 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T14:41:11.517 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T14:41:11.517 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T14:41:11.517 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.517 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.517 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T14:41:11.517 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T14:41:11.517 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T14:41:11.517 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.517 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T14:41:11.517 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T14:41:11.517 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T14:41:11.517 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T14:41:11.517 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T14:41:11.517 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T14:41:11.517 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T14:41:11.518 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T14:41:11.518 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T14:41:11.518 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T14:41:11.518 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T14:41:11.518 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T14:41:11.518 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T14:41:11.518 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T14:41:11.518 INFO:teuthology.orchestra.run.smithi077.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T14:41:11.518 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.518 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.518 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:11.518 INFO:teuthology.orchestra.run.smithi077.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T14:41:11.518 INFO:teuthology.orchestra.run.smithi077.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T14:41:11.518 INFO:teuthology.orchestra.run.smithi077.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T14:41:11.518 INFO:teuthology.orchestra.run.smithi077.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T14:41:11.519 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T14:41:11.519 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T14:41:11.519 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T14:41:11.519 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-18T14:41:11.519 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-03-18T14:41:11.762 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T14:41:11.784 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T14:41:11.784 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:11.784 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T14:41:11.784 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T14:41:11.784 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T14:41:11.784 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:41:12.008 DEBUG:teuthology.parallel:result is None 2024-03-18T14:41:12.125 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T14:41:12.161 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T14:41:12.260 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T14:41:12.424 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T14:41:12.454 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T14:41:12.477 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T14:41:12.477 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:12.478 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T14:41:12.478 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T14:41:12.478 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T14:41:12.478 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:41:12.522 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T14:41:12.532 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T14:41:12.580 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T14:41:13.451 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T14:41:13.482 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T14:41:13.504 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T14:41:13.504 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:13.504 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T14:41:13.504 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T14:41:13.504 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T14:41:13.505 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:41:13.549 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T14:41:13.571 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T14:41:13.571 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:13.571 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T14:41:13.571 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:41:13.617 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T14:41:13.638 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T14:41:13.638 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:13.639 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T14:41:13.639 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T14:41:13.639 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T14:41:13.639 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:41:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T14:41:13.973 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T14:41:13.973 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:41:13.973 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T14:41:13.973 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T14:41:13.974 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T14:41:13.974 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:41:16.233 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T14:41:16.282 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T14:41:16.312 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T14:41:16.339 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T14:41:16.520 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T14:41:16.630 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T14:41:16.671 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T14:41:16.710 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T14:41:16.763 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T14:41:16.806 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T14:41:16.838 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T14:41:16.870 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T14:41:16.892 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T14:41:16.946 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T14:41:17.012 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T14:41:17.048 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T14:41:17.634 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T14:41:17.670 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T14:41:17.690 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T14:41:18.432 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T14:41:18.476 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T14:41:18.503 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T14:41:18.511 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T14:41:18.539 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T14:41:18.596 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T14:41:18.639 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T14:41:18.719 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T14:41:18.720 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T14:41:18.760 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T14:41:18.785 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T14:41:18.810 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T14:41:18.831 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T14:41:18.837 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T14:41:18.837 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T14:41:18.879 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T14:41:18.916 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T14:41:18.938 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T14:41:18.946 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T14:41:18.946 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T14:41:18.996 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T14:41:19.022 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T14:41:19.039 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T14:41:19.048 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T14:41:19.071 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T14:41:19.090 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T14:41:19.104 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T14:41:19.115 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T14:41:19.140 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T14:41:19.142 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T14:41:19.173 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T14:41:19.212 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T14:41:19.216 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T14:41:19.253 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T14:41:19.278 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T14:41:19.299 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T14:41:19.303 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T14:41:19.329 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T14:41:19.340 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T14:41:19.354 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T14:41:19.360 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T14:41:19.360 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T14:41:19.540 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T14:41:19.540 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T14:41:19.623 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T14:41:19.648 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T14:41:19.673 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T14:41:19.706 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T14:41:19.732 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T14:41:19.797 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T14:41:19.922 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T14:41:19.963 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T14:41:19.982 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T14:41:20.798 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T14:41:20.884 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T14:41:20.925 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T14:41:21.015 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T14:41:21.054 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T14:41:21.079 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T14:41:21.105 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T14:41:21.131 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T14:41:21.131 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T14:41:21.227 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T14:41:21.257 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T14:41:21.257 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T14:41:21.341 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T14:41:21.376 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T14:41:21.409 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T14:41:21.435 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T14:41:21.460 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T14:41:21.494 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T14:41:21.541 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T14:41:21.573 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T14:41:21.598 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T14:41:21.624 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T14:41:21.648 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T14:41:21.674 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T14:41:21.680 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T14:41:21.680 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T14:41:21.860 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T14:41:21.860 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T14:41:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T14:41:21.976 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T14:41:22.001 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T14:41:22.035 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T14:41:22.069 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T14:41:22.136 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T14:41:24.467 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T14:41:24.468 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T14:41:24.468 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T14:41:24.468 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T14:41:24.468 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T14:41:24.468 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T14:41:24.468 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T14:41:24.468 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T14:41:24.468 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T14:41:24.468 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T14:41:24.468 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T14:41:24.468 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T14:41:24.468 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T14:41:24.468 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T14:41:24.468 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T14:41:24.468 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T14:41:24.468 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T14:41:24.469 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T14:41:24.469 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T14:41:24.469 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T14:41:24.469 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T14:41:24.469 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T14:41:24.469 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T14:41:24.469 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T14:41:24.469 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T14:41:24.469 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T14:41:24.469 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T14:41:24.469 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T14:41:24.469 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T14:41:24.469 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T14:41:24.469 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T14:41:24.470 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T14:41:24.470 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T14:41:24.483 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T14:41:24.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T14:41:24.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T14:41:24.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T14:41:24.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T14:41:24.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T14:41:24.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T14:41:24.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T14:41:24.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T14:41:24.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T14:41:24.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T14:41:24.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T14:41:24.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T14:41:24.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T14:41:24.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T14:41:24.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T14:41:24.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T14:41:24.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T14:41:24.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T14:41:24.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T14:41:24.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T14:41:24.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T14:41:24.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T14:41:24.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T14:41:24.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T14:41:24.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T14:41:24.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T14:41:24.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T14:41:24.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T14:41:24.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T14:41:24.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T14:41:24.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T14:41:24.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T14:41:24.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T14:41:24.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T14:41:24.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T14:41:24.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T14:41:24.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T14:41:24.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T14:41:24.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T14:41:24.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T14:41:24.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T14:41:24.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T14:41:24.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T14:41:24.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T14:41:24.487 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T14:41:24.487 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T14:41:24.487 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T14:41:24.487 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T14:41:24.487 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T14:41:24.487 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T14:41:24.487 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T14:41:24.487 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T14:41:24.487 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T14:41:24.487 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T14:41:24.487 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T14:41:24.487 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T14:41:24.487 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T14:41:24.487 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T14:41:24.487 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T14:41:24.488 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T14:41:24.488 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T14:41:24.488 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T14:41:24.488 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T14:41:24.488 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T14:41:24.488 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T14:41:24.488 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T14:41:24.488 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T14:41:24.488 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T14:41:24.488 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T14:41:24.488 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T14:41:24.488 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T14:41:24.488 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T14:41:24.488 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T14:41:24.488 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T14:41:24.488 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T14:41:24.489 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T14:41:24.489 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T14:41:24.489 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T14:41:24.489 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T14:41:24.489 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T14:41:24.489 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T14:41:24.489 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T14:41:24.489 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T14:41:24.489 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T14:41:24.489 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T14:41:24.489 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T14:41:24.489 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T14:41:24.489 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T14:41:24.489 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T14:41:24.489 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T14:41:24.489 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T14:41:24.490 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T14:41:24.490 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T14:41:24.490 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T14:41:24.490 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T14:41:24.490 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T14:41:24.490 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T14:41:24.490 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T14:41:24.490 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T14:41:24.490 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T14:41:24.490 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T14:41:24.490 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T14:41:24.490 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T14:41:24.490 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T14:41:24.490 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T14:41:24.490 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T14:41:24.491 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T14:41:24.491 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T14:41:24.491 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T14:41:24.491 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T14:41:24.491 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T14:41:24.491 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T14:41:24.491 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T14:41:24.491 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T14:41:24.491 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T14:41:24.491 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T14:41:24.491 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T14:41:24.491 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T14:41:24.491 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T14:41:24.491 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T14:41:24.491 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T14:41:24.492 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T14:41:24.492 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T14:41:24.492 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T14:41:24.492 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T14:41:24.492 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T14:41:24.492 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T14:41:24.492 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T14:41:24.492 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T14:41:24.492 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T14:41:24.492 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T14:41:24.492 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T14:41:24.492 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T14:41:24.492 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T14:41:24.492 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T14:41:24.492 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T14:41:24.492 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T14:41:24.493 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T14:41:24.493 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T14:41:24.493 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T14:41:24.493 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T14:41:24.493 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T14:41:24.493 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T14:41:24.493 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T14:41:24.493 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T14:41:24.493 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T14:41:24.493 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T14:41:24.493 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T14:41:24.493 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T14:41:24.493 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T14:41:24.493 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T14:41:24.493 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T14:41:24.493 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T14:41:24.494 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T14:41:24.494 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T14:41:24.494 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T14:41:24.494 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T14:41:24.494 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T14:41:24.494 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T14:41:24.494 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T14:41:24.494 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T14:41:24.494 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T14:41:24.494 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T14:41:24.494 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T14:41:26.955 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T14:41:26.955 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T14:41:26.955 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T14:41:26.955 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T14:41:26.955 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T14:41:26.955 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T14:41:26.955 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T14:41:26.956 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T14:41:26.956 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T14:41:26.956 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T14:41:26.956 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T14:41:26.956 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T14:41:26.956 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T14:41:26.956 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T14:41:26.956 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T14:41:26.956 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T14:41:26.956 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T14:41:26.956 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T14:41:26.956 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T14:41:26.956 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T14:41:26.956 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T14:41:26.956 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T14:41:26.957 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T14:41:26.957 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T14:41:26.957 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T14:41:26.957 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T14:41:26.957 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T14:41:26.957 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T14:41:26.957 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T14:41:26.957 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T14:41:26.957 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T14:41:26.957 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T14:41:26.957 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T14:41:26.957 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T14:41:26.957 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T14:41:26.957 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T14:41:26.957 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T14:41:26.957 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T14:41:26.958 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T14:41:26.958 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T14:41:26.958 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T14:41:26.958 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T14:41:26.958 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T14:41:26.958 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T14:41:26.958 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T14:41:26.958 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T14:41:26.958 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T14:41:26.958 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T14:41:26.958 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T14:41:26.958 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T14:41:26.958 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T14:41:26.958 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T14:41:26.958 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T14:41:26.959 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T14:41:26.959 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T14:41:26.959 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T14:41:26.959 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T14:41:26.959 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T14:41:26.959 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T14:41:26.959 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T14:41:26.959 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T14:41:26.959 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T14:41:26.959 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T14:41:26.959 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T14:41:26.959 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T14:41:26.959 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T14:41:26.959 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T14:41:26.959 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T14:41:26.959 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T14:41:26.960 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T14:41:26.960 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T14:41:26.960 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T14:41:26.960 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T14:41:26.960 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T14:41:26.960 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T14:41:26.960 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T14:41:26.960 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T14:41:26.960 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T14:41:26.960 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T14:41:26.960 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T14:41:26.960 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T14:41:26.960 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T14:41:26.960 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T14:41:26.960 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T14:41:26.960 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T14:41:26.960 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T14:41:26.961 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T14:41:26.961 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T14:41:26.961 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T14:41:26.961 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T14:41:26.961 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T14:41:26.961 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T14:41:26.961 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T14:41:26.961 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T14:41:26.961 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T14:41:26.961 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T14:41:26.961 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T14:41:26.961 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T14:41:26.961 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T14:41:26.961 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T14:41:26.961 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T14:41:26.961 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T14:41:26.962 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T14:41:26.962 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T14:41:26.962 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T14:41:26.962 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T14:41:26.962 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T14:41:26.962 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T14:41:26.962 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T14:41:26.962 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T14:41:26.962 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T14:41:26.962 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T14:41:26.962 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T14:41:26.962 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T14:41:26.962 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T14:41:26.962 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T14:41:26.962 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T14:41:26.963 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T14:41:26.963 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T14:41:26.963 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T14:41:26.963 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T14:41:26.963 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T14:41:26.963 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T14:41:26.963 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T14:41:26.963 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T14:41:26.963 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T14:41:26.963 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T14:41:26.963 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T14:41:26.963 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T14:41:26.963 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T14:41:26.963 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T14:41:26.963 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T14:41:26.963 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T14:41:26.964 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T14:41:26.964 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T14:41:26.964 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T14:41:26.964 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T14:41:26.964 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T14:41:26.964 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T14:41:26.964 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T14:41:26.964 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T14:41:26.964 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T14:41:26.964 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T14:41:26.964 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T14:41:26.964 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T14:41:26.964 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T14:41:26.964 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T14:41:26.964 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T14:41:26.964 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T14:41:26.965 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T14:41:26.965 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T14:41:26.965 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T14:41:26.965 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T14:41:26.965 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T14:41:26.965 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T14:41:26.965 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T14:41:26.965 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T14:41:26.965 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T14:41:26.965 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T14:41:26.965 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T14:41:26.965 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T14:41:26.965 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T14:41:26.965 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T14:41:26.965 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T14:41:26.965 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T14:41:26.965 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T14:41:26.966 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T14:41:26.966 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T14:41:26.966 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T14:41:26.966 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T14:41:26.966 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T14:41:26.966 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T14:41:26.966 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T14:41:26.966 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T14:41:26.966 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T14:41:26.966 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T14:41:26.966 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T14:41:26.966 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T14:41:26.966 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T14:41:26.966 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T14:41:26.966 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T14:41:26.967 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T14:41:26.967 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T14:41:26.967 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T14:41:26.967 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T14:41:26.967 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T14:41:26.967 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T14:41:26.967 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T14:41:26.967 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T14:41:26.967 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T14:41:26.967 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T14:41:26.967 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T14:41:26.967 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T14:41:26.967 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T14:41:26.967 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T14:41:26.967 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T14:41:26.967 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T14:41:26.967 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T14:41:27.455 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T14:41:27.455 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:41:27.455 INFO:teuthology.orchestra.run.smithi174.stdout:Upgraded: 2024-03-18T14:41:27.455 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T14:41:27.456 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T14:41:27.456 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T14:41:27.456 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T14:41:27.456 INFO:teuthology.orchestra.run.smithi174.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T14:41:27.456 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T14:41:27.456 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T14:41:27.456 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T14:41:27.456 INFO:teuthology.orchestra.run.smithi174.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T14:41:27.456 INFO:teuthology.orchestra.run.smithi174.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T14:41:27.456 INFO:teuthology.orchestra.run.smithi174.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T14:41:27.456 INFO:teuthology.orchestra.run.smithi174.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T14:41:27.456 INFO:teuthology.orchestra.run.smithi174.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T14:41:27.456 INFO:teuthology.orchestra.run.smithi174.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.457 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.457 INFO:teuthology.orchestra.run.smithi174.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T14:41:27.457 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T14:41:27.457 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T14:41:27.457 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T14:41:27.457 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T14:41:27.457 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T14:41:27.457 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T14:41:27.457 INFO:teuthology.orchestra.run.smithi174.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T14:41:27.457 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T14:41:27.457 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-18T14:41:27.457 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T14:41:27.458 INFO:teuthology.orchestra.run.smithi174.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T14:41:27.458 INFO:teuthology.orchestra.run.smithi174.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T14:41:27.458 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T14:41:27.458 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.458 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.458 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.458 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.458 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:27.458 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.458 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.458 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.458 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:27.458 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:27.458 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:27.458 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:27.458 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:27.459 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.459 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.459 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:27.459 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.459 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.459 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.459 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:27.459 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:27.459 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.459 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.459 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:27.459 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T14:41:27.459 INFO:teuthology.orchestra.run.smithi174.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T14:41:27.459 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T14:41:27.459 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T14:41:27.460 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T14:41:27.460 INFO:teuthology.orchestra.run.smithi174.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T14:41:27.460 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T14:41:27.460 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T14:41:27.460 INFO:teuthology.orchestra.run.smithi174.stdout: jq-1.6-16.el9.x86_64 2024-03-18T14:41:27.460 INFO:teuthology.orchestra.run.smithi174.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T14:41:27.460 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T14:41:27.460 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T14:41:27.460 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.460 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.460 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.460 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T14:41:27.460 INFO:teuthology.orchestra.run.smithi174.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T14:41:27.460 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T14:41:27.461 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T14:41:27.461 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T14:41:27.461 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.461 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.461 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T14:41:27.461 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.461 INFO:teuthology.orchestra.run.smithi174.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T14:41:27.461 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T14:41:27.461 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T14:41:27.461 INFO:teuthology.orchestra.run.smithi174.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T14:41:27.461 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T14:41:27.461 INFO:teuthology.orchestra.run.smithi174.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T14:41:27.461 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T14:41:27.461 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T14:41:27.461 INFO:teuthology.orchestra.run.smithi174.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T14:41:27.461 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T14:41:27.462 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T14:41:27.462 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T14:41:27.462 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T14:41:27.462 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T14:41:27.462 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T14:41:27.462 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T14:41:27.462 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T14:41:27.462 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T14:41:27.462 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T14:41:27.462 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T14:41:27.462 INFO:teuthology.orchestra.run.smithi174.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T14:41:27.462 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T14:41:27.462 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T14:41:27.462 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T14:41:27.462 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T14:41:27.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T14:41:27.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T14:41:27.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T14:41:27.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T14:41:27.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T14:41:27.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T14:41:27.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T14:41:27.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T14:41:27.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T14:41:27.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T14:41:27.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T14:41:27.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T14:41:27.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T14:41:27.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T14:41:27.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T14:41:27.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T14:41:27.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T14:41:27.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T14:41:27.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T14:41:27.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T14:41:27.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T14:41:27.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T14:41:27.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T14:41:27.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T14:41:27.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T14:41:27.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T14:41:27.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T14:41:27.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T14:41:27.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T14:41:27.465 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T14:41:27.465 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T14:41:27.465 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T14:41:27.465 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T14:41:27.465 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T14:41:27.465 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T14:41:27.465 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T14:41:27.465 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T14:41:27.465 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T14:41:27.465 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T14:41:27.465 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.465 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.465 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T14:41:27.465 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T14:41:27.465 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T14:41:27.466 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.466 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T14:41:27.466 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T14:41:27.466 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T14:41:27.466 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T14:41:27.466 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T14:41:27.466 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T14:41:27.466 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T14:41:27.466 INFO:teuthology.orchestra.run.smithi174.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T14:41:27.466 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T14:41:27.466 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T14:41:27.466 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T14:41:27.466 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T14:41:27.466 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T14:41:27.466 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T14:41:27.466 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T14:41:27.467 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.467 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.467 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:27.467 INFO:teuthology.orchestra.run.smithi174.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T14:41:27.467 INFO:teuthology.orchestra.run.smithi174.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T14:41:27.467 INFO:teuthology.orchestra.run.smithi174.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T14:41:27.467 INFO:teuthology.orchestra.run.smithi174.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T14:41:27.467 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T14:41:27.467 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T14:41:27.467 INFO:teuthology.orchestra.run.smithi174.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T14:41:27.467 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:41:27.468 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-18T14:41:27.921 DEBUG:teuthology.parallel:result is None 2024-03-18T14:41:29.948 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T14:41:29.948 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:41:29.948 INFO:teuthology.orchestra.run.smithi076.stdout:Upgraded: 2024-03-18T14:41:29.948 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T14:41:29.948 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T14:41:29.948 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T14:41:29.949 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T14:41:29.949 INFO:teuthology.orchestra.run.smithi076.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T14:41:29.949 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T14:41:29.949 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T14:41:29.949 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T14:41:29.949 INFO:teuthology.orchestra.run.smithi076.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T14:41:29.949 INFO:teuthology.orchestra.run.smithi076.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T14:41:29.949 INFO:teuthology.orchestra.run.smithi076.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T14:41:29.949 INFO:teuthology.orchestra.run.smithi076.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T14:41:29.949 INFO:teuthology.orchestra.run.smithi076.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T14:41:29.949 INFO:teuthology.orchestra.run.smithi076.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.949 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.949 INFO:teuthology.orchestra.run.smithi076.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T14:41:29.949 INFO:teuthology.orchestra.run.smithi076.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T14:41:29.949 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T14:41:29.950 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T14:41:29.950 INFO:teuthology.orchestra.run.smithi076.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T14:41:29.950 INFO:teuthology.orchestra.run.smithi076.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T14:41:29.950 INFO:teuthology.orchestra.run.smithi076.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T14:41:29.950 INFO:teuthology.orchestra.run.smithi076.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T14:41:29.950 INFO:teuthology.orchestra.run.smithi076.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T14:41:29.950 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-18T14:41:29.950 INFO:teuthology.orchestra.run.smithi076.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T14:41:29.950 INFO:teuthology.orchestra.run.smithi076.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T14:41:29.950 INFO:teuthology.orchestra.run.smithi076.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T14:41:29.950 INFO:teuthology.orchestra.run.smithi076.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T14:41:29.950 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.950 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.950 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.951 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.951 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:29.951 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.951 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.951 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.951 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:29.951 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:29.951 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:29.951 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:29.951 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:29.951 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.951 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.951 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:29.951 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.951 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.951 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.952 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:29.952 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:29.952 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.952 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.952 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:41:29.952 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T14:41:29.952 INFO:teuthology.orchestra.run.smithi076.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T14:41:29.952 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T14:41:29.952 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T14:41:29.952 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T14:41:29.952 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T14:41:29.952 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T14:41:29.952 INFO:teuthology.orchestra.run.smithi076.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T14:41:29.952 INFO:teuthology.orchestra.run.smithi076.stdout: jq-1.6-16.el9.x86_64 2024-03-18T14:41:29.952 INFO:teuthology.orchestra.run.smithi076.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T14:41:29.952 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T14:41:29.953 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T14:41:29.953 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.953 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.953 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.953 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T14:41:29.953 INFO:teuthology.orchestra.run.smithi076.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T14:41:29.953 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T14:41:29.953 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T14:41:29.953 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T14:41:29.953 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.953 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.953 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T14:41:29.953 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.953 INFO:teuthology.orchestra.run.smithi076.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T14:41:29.953 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T14:41:29.954 INFO:teuthology.orchestra.run.smithi076.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T14:41:29.954 INFO:teuthology.orchestra.run.smithi076.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T14:41:29.954 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T14:41:29.954 INFO:teuthology.orchestra.run.smithi076.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T14:41:29.954 INFO:teuthology.orchestra.run.smithi076.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T14:41:29.954 INFO:teuthology.orchestra.run.smithi076.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T14:41:29.954 INFO:teuthology.orchestra.run.smithi076.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T14:41:29.954 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T14:41:29.954 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T14:41:29.954 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T14:41:29.954 INFO:teuthology.orchestra.run.smithi076.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T14:41:29.954 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T14:41:29.954 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T14:41:29.954 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T14:41:29.955 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T14:41:29.955 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T14:41:29.955 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T14:41:29.955 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T14:41:29.955 INFO:teuthology.orchestra.run.smithi076.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T14:41:29.955 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T14:41:29.955 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T14:41:29.955 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T14:41:29.955 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.955 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.955 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.955 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T14:41:29.955 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T14:41:29.955 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T14:41:29.955 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T14:41:29.955 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T14:41:29.956 INFO:teuthology.orchestra.run.smithi076.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T14:41:29.956 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T14:41:29.956 INFO:teuthology.orchestra.run.smithi076.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T14:41:29.956 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T14:41:29.956 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T14:41:29.956 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T14:41:29.956 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T14:41:29.956 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T14:41:29.956 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T14:41:29.956 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T14:41:29.956 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T14:41:29.956 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T14:41:29.956 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T14:41:29.956 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T14:41:29.956 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T14:41:29.956 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T14:41:29.957 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T14:41:29.957 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T14:41:29.957 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T14:41:29.957 INFO:teuthology.orchestra.run.smithi076.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T14:41:29.957 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T14:41:29.957 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T14:41:29.957 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T14:41:29.957 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T14:41:29.957 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T14:41:29.957 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T14:41:29.957 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T14:41:29.957 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T14:41:29.957 INFO:teuthology.orchestra.run.smithi076.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T14:41:29.957 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T14:41:29.957 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T14:41:29.957 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T14:41:29.957 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T14:41:29.958 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T14:41:29.958 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T14:41:29.958 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.958 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.958 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T14:41:29.958 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T14:41:29.958 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T14:41:29.958 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.958 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T14:41:29.958 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T14:41:29.959 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T14:41:29.959 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T14:41:29.959 INFO:teuthology.orchestra.run.smithi076.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T14:41:29.959 INFO:teuthology.orchestra.run.smithi076.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T14:41:29.959 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T14:41:29.959 INFO:teuthology.orchestra.run.smithi076.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T14:41:29.959 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T14:41:29.959 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T14:41:29.959 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T14:41:29.959 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T14:41:29.959 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T14:41:29.959 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T14:41:29.959 INFO:teuthology.orchestra.run.smithi076.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T14:41:29.959 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.960 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.960 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:41:29.960 INFO:teuthology.orchestra.run.smithi076.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T14:41:29.960 INFO:teuthology.orchestra.run.smithi076.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T14:41:29.960 INFO:teuthology.orchestra.run.smithi076.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T14:41:29.960 INFO:teuthology.orchestra.run.smithi076.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T14:41:29.960 INFO:teuthology.orchestra.run.smithi076.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T14:41:29.960 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T14:41:29.960 INFO:teuthology.orchestra.run.smithi076.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T14:41:29.960 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:41:29.960 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-18T14:41:30.452 DEBUG:teuthology.parallel:result is None 2024-03-18T14:41:30.453 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:41:30.663 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T14:41:30.684 INFO:teuthology.orchestra.run.smithi076.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T14:41:30.685 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T14:41:30.685 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T14:41:30.686 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:41:30.822 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T14:41:30.847 INFO:teuthology.orchestra.run.smithi077.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T14:41:30.848 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T14:41:30.848 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T14:41:30.849 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:41:31.062 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T14:41:31.087 INFO:teuthology.orchestra.run.smithi174.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T14:41:31.087 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T14:41:31.087 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T14:41:31.088 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-18T14:41:31.088 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T14:41:31.089 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T14:41:31.118 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-18T14:41:31.118 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T14:41:31.152 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-18T14:41:31.152 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T14:41:31.186 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-18T14:41:31.186 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T14:41:31.186 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T14:41:31.213 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T14:41:31.279 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-18T14:41:31.279 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T14:41:31.311 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T14:41:31.381 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-18T14:41:31.381 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T14:41:31.411 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T14:41:31.480 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-18T14:41:31.481 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T14:41:31.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T14:41:31.507 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T14:41:31.573 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-18T14:41:31.573 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T14:41:31.600 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T14:41:31.667 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-18T14:41:31.667 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T14:41:31.698 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T14:41:31.769 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-18T14:41:31.769 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T14:41:31.769 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T14:41:31.797 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T14:41:31.863 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-18T14:41:31.863 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T14:41:31.890 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T14:41:31.957 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-18T14:41:31.957 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T14:41:31.988 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T14:41:32.059 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-18T14:41:32.145 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'mons down, quorum', 'out of quorum', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-18T14:41:32.145 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:41:32.146 INFO:tasks.cephadm:Cluster fsid is 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:41:32.146 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-18T14:41:32.146 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.76', 'mon.b': '172.21.15.77', 'mon.c': '172.21.15.174'} 2024-03-18T14:41:32.146 INFO:tasks.cephadm:First mon is mon.a on smithi076 2024-03-18T14:41:32.146 INFO:tasks.cephadm:First mgr is x 2024-03-18T14:41:32.146 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-18T14:41:32.146 DEBUG:teuthology.orchestra.run.smithi076:> sudo hostname $(hostname -s) 2024-03-18T14:41:32.174 DEBUG:teuthology.orchestra.run.smithi077:> sudo hostname $(hostname -s) 2024-03-18T14:41:32.201 DEBUG:teuthology.orchestra.run.smithi174:> sudo hostname $(hostname -s) 2024-03-18T14:41:32.232 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-18T14:41:32.232 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:41:32.441 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77658/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2162-g3d78f5ba', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2162.g3d78f5ba'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-15 01:33:04.210510', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240314.183243', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-18T14:41:32.576 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-batrick-testing-20240314.183243, sha1 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:41:32.577 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm 2024-03-18T14:41:32.577 DEBUG:teuthology.orchestra.run.smithi076:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T14:41:34.673 INFO:teuthology.orchestra.run.smithi076.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 14:41 /home/ubuntu/cephtest/cephadm 2024-03-18T14:41:34.673 DEBUG:teuthology.orchestra.run.smithi077:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T14:41:36.573 INFO:teuthology.orchestra.run.smithi077.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 14:41 /home/ubuntu/cephtest/cephadm 2024-03-18T14:41:36.573 DEBUG:teuthology.orchestra.run.smithi174:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T14:41:39.012 INFO:teuthology.orchestra.run.smithi174.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 14:41 /home/ubuntu/cephtest/cephadm 2024-03-18T14:41:39.012 DEBUG:teuthology.orchestra.run.smithi076:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T14:41:39.030 DEBUG:teuthology.orchestra.run.smithi077:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T14:41:39.049 DEBUG:teuthology.orchestra.run.smithi174:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T14:41:39.083 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af on all hosts... 2024-03-18T14:41:39.083 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T14:41:39.085 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T14:41:39.092 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T14:41:39.933 INFO:teuthology.orchestra.run.smithi076.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T14:41:39.958 INFO:teuthology.orchestra.run.smithi174.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T14:41:39.974 INFO:teuthology.orchestra.run.smithi077.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T14:42:45.438 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T14:42:45.439 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T14:42:45.439 INFO:teuthology.orchestra.run.smithi076.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T14:42:45.439 INFO:teuthology.orchestra.run.smithi076.stdout: "repo_digests": [ 2024-03-18T14:42:45.439 INFO:teuthology.orchestra.run.smithi076.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T14:42:45.439 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-18T14:42:45.439 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T14:42:53.715 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-18T14:42:53.716 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T14:42:53.716 INFO:teuthology.orchestra.run.smithi174.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T14:42:53.716 INFO:teuthology.orchestra.run.smithi174.stdout: "repo_digests": [ 2024-03-18T14:42:53.716 INFO:teuthology.orchestra.run.smithi174.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T14:42:53.716 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-18T14:42:53.716 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-18T14:42:54.687 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-18T14:42:54.687 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T14:42:54.688 INFO:teuthology.orchestra.run.smithi077.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T14:42:54.688 INFO:teuthology.orchestra.run.smithi077.stdout: "repo_digests": [ 2024-03-18T14:42:54.688 INFO:teuthology.orchestra.run.smithi077.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T14:42:54.688 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-18T14:42:54.688 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-18T14:42:54.712 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /etc/ceph 2024-03-18T14:42:54.747 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph 2024-03-18T14:42:54.776 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph 2024-03-18T14:42:54.811 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 777 /etc/ceph 2024-03-18T14:42:54.842 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /etc/ceph 2024-03-18T14:42:54.868 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /etc/ceph 2024-03-18T14:42:54.899 INFO:tasks.cephadm:Writing seed config... 2024-03-18T14:42:54.900 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-18T14:42:54.900 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-18T14:42:54.900 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-18T14:42:54.900 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-18T14:42:54.900 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-18T14:42:54.900 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-18T14:42:54.900 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-18T14:42:54.901 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-18T14:42:54.901 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-18T14:42:54.901 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-18T14:42:54.901 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-18T14:42:54.901 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-18T14:42:54.901 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-18T14:42:54.901 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-18T14:42:54.901 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-18T14:42:54.901 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-18T14:42:54.901 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-18T14:42:54.901 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-18T14:42:54.901 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-18T14:42:54.901 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-18T14:42:54.902 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-18T14:42:54.902 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-18T14:42:54.902 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-18T14:42:54.902 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-18T14:42:54.902 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-18T14:42:54.902 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-18T14:42:54.902 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-18T14:42:54.902 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-18T14:42:54.902 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-18T14:42:54.902 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-18T14:42:54.902 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-18T14:42:54.902 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-18T14:42:54.902 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-18T14:42:54.902 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-18T14:42:54.902 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-18T14:42:54.902 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-18T14:42:54.903 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-18T14:42:54.903 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-18T14:42:54.903 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-18T14:42:54.903 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-18T14:42:54.903 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-18T14:42:54.903 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-18T14:42:54.903 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-18T14:42:54.903 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-18T14:42:54.904 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T14:42:54.904 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-18T14:42:54.921 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 9cc83b72-e535-11ee-95c9-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-18T14:42:54.921 DEBUG:teuthology.orchestra.run.smithi076:mon.a> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mon.a.service 2024-03-18T14:42:54.964 DEBUG:teuthology.orchestra.run.smithi076:mgr.x> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mgr.x.service 2024-03-18T14:42:55.006 INFO:tasks.cephadm:Bootstrapping... 2024-03-18T14:42:55.007 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -v bootstrap --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.76 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-18T14:42:55.202 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-18T14:42:55.202 INFO:teuthology.orchestra.run.smithi076.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', '-v', 'bootstrap', '--fsid', '9cc83b72-e535-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.76', '--skip-admin-label'] 2024-03-18T14:42:55.227 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T14:42:55.227 INFO:teuthology.orchestra.run.smithi076.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-18T14:42:55.227 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying podman|docker is present... 2024-03-18T14:42:55.251 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T14:42:55.251 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying lvm2 is present... 2024-03-18T14:42:55.251 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying time synchronization is in place... 2024-03-18T14:42:55.258 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T14:42:55.259 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T14:42:55.264 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T14:42:55.264 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-03-18T14:42:55.270 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout enabled 2024-03-18T14:42:55.276 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout active 2024-03-18T14:42:55.276 INFO:teuthology.orchestra.run.smithi076.stdout:Unit chronyd.service is enabled and running 2024-03-18T14:42:55.276 INFO:teuthology.orchestra.run.smithi076.stdout:Repeating the final host check... 2024-03-18T14:42:55.299 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T14:42:55.299 INFO:teuthology.orchestra.run.smithi076.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-18T14:42:55.299 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl is present 2024-03-18T14:42:55.299 INFO:teuthology.orchestra.run.smithi076.stdout:lvcreate is present 2024-03-18T14:42:55.306 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T14:42:55.306 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T14:42:55.311 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T14:42:55.311 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-03-18T14:42:55.317 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout enabled 2024-03-18T14:42:55.323 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout active 2024-03-18T14:42:55.324 INFO:teuthology.orchestra.run.smithi076.stdout:Unit chronyd.service is enabled and running 2024-03-18T14:42:55.324 INFO:teuthology.orchestra.run.smithi076.stdout:Host looks OK 2024-03-18T14:42:55.324 INFO:teuthology.orchestra.run.smithi076.stdout:Cluster fsid: 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:42:55.324 INFO:teuthology.orchestra.run.smithi076.stdout:Acquiring lock 140332014542752 on /run/cephadm/9cc83b72-e535-11ee-95c9-87774f69a715.lock 2024-03-18T14:42:55.324 INFO:teuthology.orchestra.run.smithi076.stdout:Lock 140332014542752 acquired on /run/cephadm/9cc83b72-e535-11ee-95c9-87774f69a715.lock 2024-03-18T14:42:55.324 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying IP 172.21.15.76 port 3300 ... 2024-03-18T14:42:55.325 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying IP 172.21.15.76 port 6789 ... 2024-03-18T14:42:55.325 INFO:teuthology.orchestra.run.smithi076.stdout:Base mon IP(s) is [172.21.15.76:3300, 172.21.15.76:6789], mon addrv is [v2:172.21.15.76:3300,v1:172.21.15.76:6789] 2024-03-18T14:42:55.327 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.76 metric 100 2024-03-18T14:42:55.327 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.76 metric 100 2024-03-18T14:42:55.330 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-18T14:42:55.330 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-18T14:42:55.330 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1769sec hoplimit 64 pref medium 2024-03-18T14:42:55.332 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-18T14:42:55.333 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-18T14:42:55.333 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T14:42:55.333 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-18T14:42:55.333 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:47e1/64 scope link 2024-03-18T14:42:55.333 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T14:42:55.333 INFO:teuthology.orchestra.run.smithi076.stdout:Mon IP `172.21.15.76` is in CIDR network `172.21.0.0/20` 2024-03-18T14:42:55.334 INFO:teuthology.orchestra.run.smithi076.stdout:Mon IP `172.21.15.76` is in CIDR network `172.21.0.0/20` 2024-03-18T14:42:55.334 INFO:teuthology.orchestra.run.smithi076.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-18T14:42:55.334 INFO:teuthology.orchestra.run.smithi076.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-18T14:42:55.335 INFO:teuthology.orchestra.run.smithi076.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T14:42:58.980 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stdout be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-18T14:42:58.980 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T14:42:58.980 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Getting image source signatures 2024-03-18T14:42:58.980 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Copying blob sha256:66487374cb611c95435850c17b53ebb64c22ed9555f72cd43d678b60262a12b1 2024-03-18T14:42:58.980 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-18T14:42:58.980 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Copying config sha256:be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-18T14:42:58.980 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-18T14:43:04.639 INFO:teuthology.orchestra.run.smithi076.stdout:ceph: stdout ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-18T14:43:04.640 INFO:teuthology.orchestra.run.smithi076.stdout:Ceph version: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-18T14:43:04.640 INFO:teuthology.orchestra.run.smithi076.stdout:Extracting ceph user uid/gid from container image... 2024-03-18T14:43:05.578 INFO:teuthology.orchestra.run.smithi076.stdout:stat: stdout 167 167 2024-03-18T14:43:05.578 INFO:teuthology.orchestra.run.smithi076.stdout:Creating initial keys... 2024-03-18T14:43:06.816 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph-authtool: stdout AQD5UvhliL/KNxAAY3RokCIb7VexGpMHLCY1PA== 2024-03-18T14:43:07.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph-authtool: stdout AQD7UvhljdXrBxAA90cOVvzlulleAB7pTHjpOA== 2024-03-18T14:43:08.635 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph-authtool: stdout AQD8Uvhlobs3BRAATbxMh+s9Vewq86/EwrSiDw== 2024-03-18T14:43:08.637 INFO:teuthology.orchestra.run.smithi076.stdout:Creating initial monmap... 2024-03-18T14:43:09.674 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T14:43:09.674 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-18T14:43:09.674 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:43:09.674 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T14:43:09.674 INFO:teuthology.orchestra.run.smithi076.stdout:monmaptool for a [v2:172.21.15.76:3300,v1:172.21.15.76:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T14:43:09.674 INFO:teuthology.orchestra.run.smithi076.stdout:setting min_mon_release = quincy 2024-03-18T14:43:09.674 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: set fsid to 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:43:09.674 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T14:43:09.675 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:43:09.675 INFO:teuthology.orchestra.run.smithi076.stdout:Creating mon... 2024-03-18T14:43:11.509 INFO:teuthology.orchestra.run.smithi076.stdout:create mon.a on 2024-03-18T14:43:11.705 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-18T14:43:11.839 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-18T14:43:12.029 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-9cc83b72-e535-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-9cc83b72-e535-11ee-95c9-87774f69a715.target. 2024-03-18T14:43:12.029 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-9cc83b72-e535-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-9cc83b72-e535-11ee-95c9-87774f69a715.target. 2024-03-18T14:43:12.276 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mon.a 2024-03-18T14:43:12.276 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-18T14:43:12.450 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9cc83b72-e535-11ee-95c9-87774f69a715.target.wants/ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-9cc83b72-e535-11ee-95c9-87774f69a715@.service. 2024-03-18T14:43:12.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:12 smithi076 systemd[1]: Started Ceph mon.a for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:43:12.919 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T14:43:12.919 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout disabled 2024-03-18T14:43:12.927 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T14:43:12.927 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-03-18T14:43:12.927 INFO:teuthology.orchestra.run.smithi076.stdout:firewalld.service is not enabled 2024-03-18T14:43:12.927 INFO:teuthology.orchestra.run.smithi076.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T14:43:12.928 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mon to start... 2024-03-18T14:43:12.928 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mon... 2024-03-18T14:43:14.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:13 smithi076 ceph-mon[31823]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T14:43:14.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:13 smithi076 ceph-mon[31823]: monmap epoch 1 2024-03-18T14:43:14.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:13 smithi076 ceph-mon[31823]: fsid 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:43:14.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:13 smithi076 ceph-mon[31823]: last_changed 2024-03-18T14:43:08.949643+0000 2024-03-18T14:43:14.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:13 smithi076 ceph-mon[31823]: created 2024-03-18T14:43:08.949643+0000 2024-03-18T14:43:14.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:13 smithi076 ceph-mon[31823]: min_mon_release 19 (squid) 2024-03-18T14:43:14.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:13 smithi076 ceph-mon[31823]: election_strategy: 1 2024-03-18T14:43:14.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:13 smithi076 ceph-mon[31823]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-03-18T14:43:14.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:13 smithi076 ceph-mon[31823]: fsmap 2024-03-18T14:43:14.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:13 smithi076 ceph-mon[31823]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T14:43:14.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:13 smithi076 ceph-mon[31823]: mgrmap e1: no daemons active 2024-03-18T14:43:14.010 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:13 smithi076 ceph-mon[31823]: from='client.? 172.21.15.76:0/1611756402' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-18T14:43:14.011 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout cluster: 2024-03-18T14:43:14.017 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout id: 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:43:14.018 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-18T14:43:14.018 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-18T14:43:14.018 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout services: 2024-03-18T14:43:14.018 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.55484s) 2024-03-18T14:43:14.018 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-18T14:43:14.018 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-18T14:43:14.018 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-18T14:43:14.018 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout data: 2024-03-18T14:43:14.018 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-18T14:43:14.018 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-18T14:43:14.018 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-18T14:43:14.019 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout pgs: 2024-03-18T14:43:14.019 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-18T14:43:14.019 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.376+0000 7f895d95e700 1 Processor -- start 2024-03-18T14:43:14.019 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.376+0000 7f895d95e700 1 -- start start 2024-03-18T14:43:14.019 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.376+0000 7f895d95e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581080a0 0x7f89581084c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.019 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.376+0000 7f895d95e700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8958108a90 con 0x7f89581080a0 2024-03-18T14:43:14.019 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.376+0000 7f8956ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581080a0 0x7f89581084c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.019 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.376+0000 7f8956ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581080a0 0x7f89581084c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45048/0 (socket says 172.21.15.76:45048) 2024-03-18T14:43:14.019 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.376+0000 7f8956ffd700 1 -- 172.21.15.76:0/763060388 learned_addr learned my addr 172.21.15.76:0/763060388 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:14.019 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.376+0000 7f8956ffd700 1 -- 172.21.15.76:0/763060388 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8958106d00 con 0x7f89581080a0 2024-03-18T14:43:14.019 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.377+0000 7f8956ffd700 1 --2- 172.21.15.76:0/763060388 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581080a0 0x7f89581084c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f89500098a0 tx=0x7f8950009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a63fbb78ab5d920d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:14.019 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.377+0000 7f8955ffb700 1 -- 172.21.15.76:0/763060388 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8950004160 con 0x7f89581080a0 2024-03-18T14:43:14.019 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.377+0000 7f8955ffb700 1 -- 172.21.15.76:0/763060388 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8950004300 con 0x7f89581080a0 2024-03-18T14:43:14.019 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.377+0000 7f8955ffb700 1 -- 172.21.15.76:0/763060388 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89500036a0 con 0x7f89581080a0 2024-03-18T14:43:14.020 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.377+0000 7f895d95e700 1 -- 172.21.15.76:0/763060388 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581080a0 msgr2=0x7f89581084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:14.020 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.377+0000 7f895d95e700 1 --2- 172.21.15.76:0/763060388 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581080a0 0x7f89581084c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f89500098a0 tx=0x7f8950009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.020 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.377+0000 7f895d95e700 1 -- 172.21.15.76:0/763060388 shutdown_connections 2024-03-18T14:43:14.020 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.377+0000 7f895d95e700 1 --2- 172.21.15.76:0/763060388 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581080a0 0x7f89581084c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.020 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.377+0000 7f895d95e700 1 -- 172.21.15.76:0/763060388 >> 172.21.15.76:0/763060388 conn(0x7f89581017b0 msgr2=0x7f8958103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:14.020 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.377+0000 7f895d95e700 1 -- 172.21.15.76:0/763060388 shutdown_connections 2024-03-18T14:43:14.020 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.377+0000 7f895d95e700 1 -- 172.21.15.76:0/763060388 wait complete. 2024-03-18T14:43:14.020 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.378+0000 7f895d95e700 1 Processor -- start 2024-03-18T14:43:14.020 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.378+0000 7f895d95e700 1 -- start start 2024-03-18T14:43:14.020 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.379+0000 7f895d95e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581080a0 0x7f895819b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.020 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.379+0000 7f895d95e700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8958107230 con 0x7f89581080a0 2024-03-18T14:43:14.020 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.379+0000 7f8956ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581080a0 0x7f895819b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.020 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.379+0000 7f8956ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581080a0 0x7f895819b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45060/0 (socket says 172.21.15.76:45060) 2024-03-18T14:43:14.021 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.379+0000 7f8956ffd700 1 -- 172.21.15.76:0/1611756402 learned_addr learned my addr 172.21.15.76:0/1611756402 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:14.021 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.379+0000 7f8956ffd700 1 -- 172.21.15.76:0/1611756402 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895819b7d0 con 0x7f89581080a0 2024-03-18T14:43:14.021 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.379+0000 7f8956ffd700 1 --2- 172.21.15.76:0/1611756402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581080a0 0x7f895819b290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8950000c00 tx=0x7f8950004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:14.021 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.379+0000 7f895c95c700 1 -- 172.21.15.76:0/1611756402 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8950003db0 con 0x7f89581080a0 2024-03-18T14:43:14.021 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.379+0000 7f895c95c700 1 -- 172.21.15.76:0/1611756402 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8950011630 con 0x7f89581080a0 2024-03-18T14:43:14.021 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.379+0000 7f895d95e700 1 -- 172.21.15.76:0/1611756402 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f895819ba60 con 0x7f89581080a0 2024-03-18T14:43:14.021 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.379+0000 7f895c95c700 1 -- 172.21.15.76:0/1611756402 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8950011950 con 0x7f89581080a0 2024-03-18T14:43:14.021 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.379+0000 7f895d95e700 1 -- 172.21.15.76:0/1611756402 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f895819e750 con 0x7f89581080a0 2024-03-18T14:43:14.021 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.379+0000 7f895c95c700 1 -- 172.21.15.76:0/1611756402 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8950011af0 con 0x7f89581080a0 2024-03-18T14:43:14.021 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.380+0000 7f895c95c700 1 -- 172.21.15.76:0/1611756402 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f895001c790 con 0x7f89581080a0 2024-03-18T14:43:14.021 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.380+0000 7f895d95e700 1 -- 172.21.15.76:0/1611756402 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8958063900 con 0x7f89581080a0 2024-03-18T14:43:14.021 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.383+0000 7f895c95c700 1 -- 172.21.15.76:0/1611756402 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8950032020 con 0x7f89581080a0 2024-03-18T14:43:14.021 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.452+0000 7f895d95e700 1 -- 172.21.15.76:0/1611756402 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f895802dcf0 con 0x7f89581080a0 2024-03-18T14:43:14.021 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.452+0000 7f895c95c700 1 -- 172.21.15.76:0/1611756402 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7f8950004260 con 0x7f89581080a0 2024-03-18T14:43:14.022 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.452+0000 7f895d95e700 1 -- 172.21.15.76:0/1611756402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581080a0 msgr2=0x7f895819b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:14.022 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.452+0000 7f895d95e700 1 --2- 172.21.15.76:0/1611756402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581080a0 0x7f895819b290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8950000c00 tx=0x7f8950004750 comp rx=0 tx=0).stop 2024-03-18T14:43:14.022 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.452+0000 7f895d95e700 1 -- 172.21.15.76:0/1611756402 shutdown_connections 2024-03-18T14:43:14.022 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.452+0000 7f895d95e700 1 --2- 172.21.15.76:0/1611756402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581080a0 0x7f895819b290 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.022 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.452+0000 7f895d95e700 1 -- 172.21.15.76:0/1611756402 >> 172.21.15.76:0/1611756402 conn(0x7f89581017b0 msgr2=0x7f8958191ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:14.022 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.453+0000 7f895d95e700 1 -- 172.21.15.76:0/1611756402 shutdown_connections 2024-03-18T14:43:14.022 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:13.453+0000 7f895d95e700 1 -- 172.21.15.76:0/1611756402 wait complete. 2024-03-18T14:43:14.022 INFO:teuthology.orchestra.run.smithi076.stdout:mon is available 2024-03-18T14:43:14.022 INFO:teuthology.orchestra.run.smithi076.stdout:Assimilating anything we can from ceph.conf... 2024-03-18T14:43:15.039 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-18T14:43:15.039 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T14:43:15.039 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout fsid = 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:43:15.039 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.76:3300,v1:172.21.15.76:6789] 2024-03-18T14:43:15.039 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T14:43:15.039 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T14:43:15.039 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T14:43:15.039 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T14:43:15.040 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-18T14:43:15.040 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T14:43:15.040 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T14:43:15.040 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-18T14:43:15.040 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T14:43:15.040 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T14:43:15.040 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T14:43:15.040 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T14:43:15.040 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T14:43:15.040 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T14:43:15.040 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.417+0000 7f47ada0d700 1 Processor -- start 2024-03-18T14:43:15.040 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.417+0000 7f47ada0d700 1 -- start start 2024-03-18T14:43:15.040 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.417+0000 7f47ada0d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47a8108600 0x7f47a810a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.040 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.417+0000 7f47ada0d700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f47a810af30 con 0x7f47a8108600 2024-03-18T14:43:15.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.418+0000 7f47a6ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47a8108600 0x7f47a810a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.418+0000 7f47a6ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47a8108600 0x7f47a810a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45074/0 (socket says 172.21.15.76:45074) 2024-03-18T14:43:15.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.418+0000 7f47a6ffd700 1 -- 172.21.15.76:0/3566549204 learned_addr learned my addr 172.21.15.76:0/3566549204 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:15.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.418+0000 7f47a6ffd700 1 -- 172.21.15.76:0/3566549204 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a810b0b0 con 0x7f47a8108600 2024-03-18T14:43:15.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.418+0000 7f47a6ffd700 1 --2- 172.21.15.76:0/3566549204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47a8108600 0x7f47a810a9f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f47980098a0 tx=0x7f4798009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f1ce1c22614ee8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:15.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.418+0000 7f47a5ffb700 1 -- 172.21.15.76:0/3566549204 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4798004160 con 0x7f47a8108600 2024-03-18T14:43:15.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.418+0000 7f47a5ffb700 1 -- 172.21.15.76:0/3566549204 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4798004300 con 0x7f47a8108600 2024-03-18T14:43:15.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.418+0000 7f47a5ffb700 1 -- 172.21.15.76:0/3566549204 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47980036a0 con 0x7f47a8108600 2024-03-18T14:43:15.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.419+0000 7f47ada0d700 1 -- 172.21.15.76:0/3566549204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47a8108600 msgr2=0x7f47a810a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.419+0000 7f47ada0d700 1 --2- 172.21.15.76:0/3566549204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47a8108600 0x7f47a810a9f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f47980098a0 tx=0x7f4798009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.419+0000 7f47ada0d700 1 -- 172.21.15.76:0/3566549204 shutdown_connections 2024-03-18T14:43:15.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.419+0000 7f47ada0d700 1 --2- 172.21.15.76:0/3566549204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47a8108600 0x7f47a810a9f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.419+0000 7f47ada0d700 1 -- 172.21.15.76:0/3566549204 >> 172.21.15.76:0/3566549204 conn(0x7f47a8101ff0 msgr2=0x7f47a8104430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:15.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.419+0000 7f47ada0d700 1 -- 172.21.15.76:0/3566549204 shutdown_connections 2024-03-18T14:43:15.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.419+0000 7f47ada0d700 1 -- 172.21.15.76:0/3566549204 wait complete. 2024-03-18T14:43:15.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.419+0000 7f47ada0d700 1 Processor -- start 2024-03-18T14:43:15.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.420+0000 7f47ada0d700 1 -- start start 2024-03-18T14:43:15.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.420+0000 7f47ada0d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47a8108600 0x7f47a819b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.420+0000 7f47ada0d700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f47a810b4c0 con 0x7f47a8108600 2024-03-18T14:43:15.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.420+0000 7f47a6ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47a8108600 0x7f47a819b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.420+0000 7f47a6ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47a8108600 0x7f47a819b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45082/0 (socket says 172.21.15.76:45082) 2024-03-18T14:43:15.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.420+0000 7f47a6ffd700 1 -- 172.21.15.76:0/2364810602 learned_addr learned my addr 172.21.15.76:0/2364810602 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:15.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.420+0000 7f47a6ffd700 1 -- 172.21.15.76:0/2364810602 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a819b8d0 con 0x7f47a8108600 2024-03-18T14:43:15.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.420+0000 7f47a6ffd700 1 --2- 172.21.15.76:0/2364810602 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47a8108600 0x7f47a819b390 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4798000c00 tx=0x7f4798004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:15.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.420+0000 7f47aca0b700 1 -- 172.21.15.76:0/2364810602 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4798003db0 con 0x7f47a8108600 2024-03-18T14:43:15.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.420+0000 7f47aca0b700 1 -- 172.21.15.76:0/2364810602 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4798011630 con 0x7f47a8108600 2024-03-18T14:43:15.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.420+0000 7f47aca0b700 1 -- 172.21.15.76:0/2364810602 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4798011950 con 0x7f47a8108600 2024-03-18T14:43:15.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.420+0000 7f47ada0d700 1 -- 172.21.15.76:0/2364810602 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47a819bb60 con 0x7f47a8108600 2024-03-18T14:43:15.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.420+0000 7f47ada0d700 1 -- 172.21.15.76:0/2364810602 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47a819e850 con 0x7f47a8108600 2024-03-18T14:43:15.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.421+0000 7f47aca0b700 1 -- 172.21.15.76:0/2364810602 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f479801a460 con 0x7f47a8108600 2024-03-18T14:43:15.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.421+0000 7f47aca0b700 1 -- 172.21.15.76:0/2364810602 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f479801b780 con 0x7f47a8108600 2024-03-18T14:43:15.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.421+0000 7f47ada0d700 1 -- 172.21.15.76:0/2364810602 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47a8063900 con 0x7f47a8108600 2024-03-18T14:43:15.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.424+0000 7f47aca0b700 1 -- 172.21.15.76:0/2364810602 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f479801b980 con 0x7f47a8108600 2024-03-18T14:43:15.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.492+0000 7f47ada0d700 1 -- 172.21.15.76:0/2364810602 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f47a802dcc0 con 0x7f47a8108600 2024-03-18T14:43:15.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.495+0000 7f47aca0b700 1 -- 172.21.15.76:0/2364810602 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f479801bb60 con 0x7f47a8108600 2024-03-18T14:43:15.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.496+0000 7f47aca0b700 1 -- 172.21.15.76:0/2364810602 <== mon.0 v2:172.21.15.76:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f479801bd40 con 0x7f47a8108600 2024-03-18T14:43:15.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.496+0000 7f47ada0d700 1 -- 172.21.15.76:0/2364810602 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47a8108600 msgr2=0x7f47a819b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.496+0000 7f47ada0d700 1 --2- 172.21.15.76:0/2364810602 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47a8108600 0x7f47a819b390 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4798000c00 tx=0x7f4798004750 comp rx=0 tx=0).stop 2024-03-18T14:43:15.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.496+0000 7f47ada0d700 1 -- 172.21.15.76:0/2364810602 shutdown_connections 2024-03-18T14:43:15.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.496+0000 7f47ada0d700 1 --2- 172.21.15.76:0/2364810602 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47a8108600 0x7f47a819b390 secure :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4798000c00 tx=0x7f4798004750 comp rx=0 tx=0).stop 2024-03-18T14:43:15.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.496+0000 7f47ada0d700 1 -- 172.21.15.76:0/2364810602 >> 172.21.15.76:0/2364810602 conn(0x7f47a8101ff0 msgr2=0x7f47a8104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:15.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.496+0000 7f47ada0d700 1 -- 172.21.15.76:0/2364810602 shutdown_connections 2024-03-18T14:43:15.044 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:14.496+0000 7f47ada0d700 1 -- 172.21.15.76:0/2364810602 wait complete. 2024-03-18T14:43:15.044 INFO:teuthology.orchestra.run.smithi076.stdout:Generating new minimal ceph.conf... 2024-03-18T14:43:15.157 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:14 smithi076 ceph-mon[31823]: from='client.? 172.21.15.76:0/2364810602' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T14:43:15.158 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:14 smithi076 ceph-mon[31823]: from='client.? 172.21.15.76:0/2364810602' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-18T14:43:16.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:15 smithi076 ceph-mon[31823]: from='client.? 172.21.15.76:0/1323056391' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:43:16.452 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.660+0000 7f492a9e6700 1 Processor -- start 2024-03-18T14:43:16.452 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.660+0000 7f492a9e6700 1 -- start start 2024-03-18T14:43:16.452 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.660+0000 7f492a9e6700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4924108180 0x7f49241085a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.452 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.660+0000 7f492a9e6700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f49241066c0 con 0x7f4924108180 2024-03-18T14:43:16.452 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.660+0000 7f4923fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4924108180 0x7f49241085a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.452 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.660+0000 7f4923fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4924108180 0x7f49241085a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45090/0 (socket says 172.21.15.76:45090) 2024-03-18T14:43:16.452 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.660+0000 7f4923fff700 1 -- 172.21.15.76:0/2759863865 learned_addr learned my addr 172.21.15.76:0/2759863865 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:16.452 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.661+0000 7f4923fff700 1 -- 172.21.15.76:0/2759863865 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4924106840 con 0x7f4924108180 2024-03-18T14:43:16.452 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.661+0000 7f4923fff700 1 --2- 172.21.15.76:0/2759863865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4924108180 0x7f49241085a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f49140098a0 tx=0x7f4914009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab514d56202a9db server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:16.453 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.661+0000 7f49237fe700 1 -- 172.21.15.76:0/2759863865 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4914004160 con 0x7f4924108180 2024-03-18T14:43:16.453 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.661+0000 7f49237fe700 1 -- 172.21.15.76:0/2759863865 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4914004300 con 0x7f4924108180 2024-03-18T14:43:16.453 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.661+0000 7f49237fe700 1 -- 172.21.15.76:0/2759863865 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4914011b50 con 0x7f4924108180 2024-03-18T14:43:16.453 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.661+0000 7f492a9e6700 1 -- 172.21.15.76:0/2759863865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4924108180 msgr2=0x7f49241085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.453 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.661+0000 7f492a9e6700 1 --2- 172.21.15.76:0/2759863865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4924108180 0x7f49241085a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f49140098a0 tx=0x7f4914009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.453 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.661+0000 7f492a9e6700 1 -- 172.21.15.76:0/2759863865 shutdown_connections 2024-03-18T14:43:16.453 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.661+0000 7f492a9e6700 1 --2- 172.21.15.76:0/2759863865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4924108180 0x7f49241085a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.453 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.661+0000 7f492a9e6700 1 -- 172.21.15.76:0/2759863865 >> 172.21.15.76:0/2759863865 conn(0x7f4924102000 msgr2=0x7f4924104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:16.453 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.662+0000 7f492a9e6700 1 -- 172.21.15.76:0/2759863865 shutdown_connections 2024-03-18T14:43:16.453 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.662+0000 7f492a9e6700 1 -- 172.21.15.76:0/2759863865 wait complete. 2024-03-18T14:43:16.453 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.662+0000 7f492a9e6700 1 Processor -- start 2024-03-18T14:43:16.453 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.662+0000 7f492a9e6700 1 -- start start 2024-03-18T14:43:16.453 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.662+0000 7f492a9e6700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4924108180 0x7f492419bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.453 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.662+0000 7f492a9e6700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f492410b050 con 0x7f4924108180 2024-03-18T14:43:16.454 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.662+0000 7f4923fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4924108180 0x7f492419bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.454 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.662+0000 7f4923fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4924108180 0x7f492419bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45092/0 (socket says 172.21.15.76:45092) 2024-03-18T14:43:16.454 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.662+0000 7f4923fff700 1 -- 172.21.15.76:0/1323056391 learned_addr learned my addr 172.21.15.76:0/1323056391 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:16.454 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.663+0000 7f4923fff700 1 -- 172.21.15.76:0/1323056391 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f492419c240 con 0x7f4924108180 2024-03-18T14:43:16.454 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.663+0000 7f4923fff700 1 --2- 172.21.15.76:0/1323056391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4924108180 0x7f492419bd00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4914009fb0 tx=0x7f4914011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:16.454 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.663+0000 7f4921ffb700 1 -- 172.21.15.76:0/1323056391 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49140038b0 con 0x7f4924108180 2024-03-18T14:43:16.454 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.663+0000 7f4921ffb700 1 -- 172.21.15.76:0/1323056391 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4914011e50 con 0x7f4924108180 2024-03-18T14:43:16.454 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.663+0000 7f492a9e6700 1 -- 172.21.15.76:0/1323056391 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f492419c4d0 con 0x7f4924108180 2024-03-18T14:43:16.454 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.663+0000 7f4921ffb700 1 -- 172.21.15.76:0/1323056391 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4914019550 con 0x7f4924108180 2024-03-18T14:43:16.454 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.663+0000 7f492a9e6700 1 -- 172.21.15.76:0/1323056391 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f492419f1b0 con 0x7f4924108180 2024-03-18T14:43:16.454 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.663+0000 7f4921ffb700 1 -- 172.21.15.76:0/1323056391 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4914019be0 con 0x7f4924108180 2024-03-18T14:43:16.454 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.663+0000 7f4921ffb700 1 -- 172.21.15.76:0/1323056391 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4914024730 con 0x7f4924108180 2024-03-18T14:43:16.454 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.664+0000 7f492a9e6700 1 -- 172.21.15.76:0/1323056391 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4924063900 con 0x7f4924108180 2024-03-18T14:43:16.455 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.666+0000 7f4921ffb700 1 -- 172.21.15.76:0/1323056391 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4914003a50 con 0x7f4924108180 2024-03-18T14:43:16.455 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.734+0000 7f492a9e6700 1 -- 172.21.15.76:0/1323056391 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f492402dcc0 con 0x7f4924108180 2024-03-18T14:43:16.455 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.735+0000 7f4921ffb700 1 -- 172.21.15.76:0/1323056391 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f49140249d0 con 0x7f4924108180 2024-03-18T14:43:16.455 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.736+0000 7f492a9e6700 1 -- 172.21.15.76:0/1323056391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4924108180 msgr2=0x7f492419bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.455 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.736+0000 7f492a9e6700 1 --2- 172.21.15.76:0/1323056391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4924108180 0x7f492419bd00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4914009fb0 tx=0x7f4914011bf0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.455 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.736+0000 7f492a9e6700 1 -- 172.21.15.76:0/1323056391 shutdown_connections 2024-03-18T14:43:16.455 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.736+0000 7f492a9e6700 1 --2- 172.21.15.76:0/1323056391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4924108180 0x7f492419bd00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.455 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.736+0000 7f492a9e6700 1 -- 172.21.15.76:0/1323056391 >> 172.21.15.76:0/1323056391 conn(0x7f4924102000 msgr2=0x7f4924102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:16.455 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.736+0000 7f492a9e6700 1 -- 172.21.15.76:0/1323056391 shutdown_connections 2024-03-18T14:43:16.455 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:15.736+0000 7f492a9e6700 1 -- 172.21.15.76:0/1323056391 wait complete. 2024-03-18T14:43:16.455 INFO:teuthology.orchestra.run.smithi076.stdout:Restarting the monitor... 2024-03-18T14:43:16.707 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:16 smithi076 systemd[1]: Stopping Ceph mon.a for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T14:43:16.989 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:16 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a[31819]: 2024-03-18T14:43:16.704+0000 7f08077dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T14:43:16.989 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:16 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a[31819]: 2024-03-18T14:43:16.705+0000 7f08077dc700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-18T14:43:16.990 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:16 smithi076 podman[32071]: 2024-03-18 14:43:16.736400638 +0000 UTC m=+0.129116551 container died e23477297fe43e76f9e919f5b60d4fb6f1f4d726ad534f028676c647ec172319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-18T14:43:17.390 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:17 smithi076 podman[32071]: 2024-03-18 14:43:17.088340911 +0000 UTC m=+0.481056828 container cleanup e23477297fe43e76f9e919f5b60d4fb6f1f4d726ad534f028676c647ec172319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T14:43:17.390 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:17 smithi076 bash[32071]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a 2024-03-18T14:43:17.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:17 smithi076 podman[32085]: 2024-03-18 14:43:17.389168027 +0000 UTC m=+0.653423289 container remove e23477297fe43e76f9e919f5b60d4fb6f1f4d726ad534f028676c647ec172319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-18T14:43:17.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:17 smithi076 systemd[1]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-18T14:43:17.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:17 smithi076 systemd[1]: Stopped Ceph mon.a for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:43:17.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:17 smithi076 systemd[1]: Starting Ceph mon.a for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T14:43:18.043 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:17 smithi076 podman[32185]: 2024-03-18 14:43:17.669440138 +0000 UTC m=+0.017793413 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:43:18.043 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:17 smithi076 podman[32185]: 2024-03-18 14:43:17.795038983 +0000 UTC m=+0.143392282 container create 3df3aa06ebbe1ec071dca70665ddb9ab25137b99c794a5e752c9ce2e1f69f75e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-18T14:43:18.043 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 podman[32185]: 2024-03-18 14:43:18.042451759 +0000 UTC m=+0.390805016 container init 3df3aa06ebbe1ec071dca70665ddb9ab25137b99c794a5e752c9ce2e1f69f75e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-18T14:43:18.131 INFO:teuthology.orchestra.run.smithi076.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-18T14:43:18.314 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 podman[32185]: 2024-03-18 14:43:18.045479015 +0000 UTC m=+0.393832281 container start 3df3aa06ebbe1ec071dca70665ddb9ab25137b99c794a5e752c9ce2e1f69f75e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T14:43:18.314 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: set uid:gid to 167:167 (ceph:ceph) 2024-03-18T14:43:18.314 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-18T14:43:18.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: pidfile_write: ignore empty --pid-file 2024-03-18T14:43:18.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: load: jerasure load: lrc 2024-03-18T14:43:18.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: RocksDB version: 7.9.2 2024-03-18T14:43:18.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Git sha 0 2024-03-18T14:43:18.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-18T14:43:18.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: DB SUMMARY 2024-03-18T14:43:18.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: DB Session ID: T3OA4AMDPFYTCHFPNOGY 2024-03-18T14:43:18.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: CURRENT file: CURRENT 2024-03-18T14:43:18.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: IDENTITY file: IDENTITY 2024-03-18T14:43:18.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-18T14:43:18.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-18T14:43:18.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104194 ; 2024-03-18T14:43:18.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.error_if_exists: 0 2024-03-18T14:43:18.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.create_if_missing: 0 2024-03-18T14:43:18.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.paranoid_checks: 1 2024-03-18T14:43:18.316 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-18T14:43:18.316 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-18T14:43:18.316 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-18T14:43:18.316 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.env: 0x563b0c3e6c60 2024-03-18T14:43:18.316 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.fs: PosixFileSystem 2024-03-18T14:43:18.316 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.info_log: 0x563b0e425ac0 2024-03-18T14:43:18.316 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_file_opening_threads: 16 2024-03-18T14:43:18.316 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.statistics: (nil) 2024-03-18T14:43:18.316 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.use_fsync: 0 2024-03-18T14:43:18.316 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_log_file_size: 0 2024-03-18T14:43:18.316 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-18T14:43:18.316 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-18T14:43:18.316 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.keep_log_file_num: 1000 2024-03-18T14:43:18.316 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.recycle_log_file_num: 0 2024-03-18T14:43:18.316 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.allow_fallocate: 1 2024-03-18T14:43:18.318 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.allow_mmap_reads: 0 2024-03-18T14:43:18.318 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.allow_mmap_writes: 0 2024-03-18T14:43:18.318 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.use_direct_reads: 0 2024-03-18T14:43:18.318 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-18T14:43:18.318 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.create_missing_column_families: 0 2024-03-18T14:43:18.318 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.db_log_dir: 2024-03-18T14:43:18.318 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.wal_dir: 2024-03-18T14:43:18.318 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-18T14:43:18.318 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-18T14:43:18.318 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-18T14:43:18.318 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-18T14:43:18.318 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-18T14:43:18.318 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-18T14:43:18.318 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.advise_random_on_open: 1 2024-03-18T14:43:18.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.db_write_buffer_size: 0 2024-03-18T14:43:18.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.write_buffer_manager: 0x563b0e3f81e0 2024-03-18T14:43:18.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-18T14:43:18.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-18T14:43:18.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-18T14:43:18.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.rate_limiter: (nil) 2024-03-18T14:43:18.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-18T14:43:18.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.wal_recovery_mode: 2 2024-03-18T14:43:18.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.enable_thread_tracking: 0 2024-03-18T14:43:18.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.enable_pipelined_write: 0 2024-03-18T14:43:18.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.unordered_write: 0 2024-03-18T14:43:18.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-18T14:43:18.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-18T14:43:18.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-18T14:43:18.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-18T14:43:18.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.row_cache: None 2024-03-18T14:43:18.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.wal_filter: None 2024-03-18T14:43:18.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-18T14:43:18.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.allow_ingest_behind: 0 2024-03-18T14:43:18.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.two_write_queues: 0 2024-03-18T14:43:18.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.manual_wal_flush: 0 2024-03-18T14:43:18.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.wal_compression: 0 2024-03-18T14:43:18.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.atomic_flush: 0 2024-03-18T14:43:18.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-18T14:43:18.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-18T14:43:18.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-18T14:43:18.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.log_readahead_size: 0 2024-03-18T14:43:18.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-18T14:43:18.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.best_efforts_recovery: 0 2024-03-18T14:43:18.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-18T14:43:18.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-18T14:43:18.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.allow_data_in_errors: 0 2024-03-18T14:43:18.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.db_host_id: __hostname__ 2024-03-18T14:43:18.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-18T14:43:18.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_background_jobs: 2 2024-03-18T14:43:18.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_background_compactions: -1 2024-03-18T14:43:18.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_subcompactions: 1 2024-03-18T14:43:18.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-18T14:43:18.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-18T14:43:18.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-18T14:43:18.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_total_wal_size: 0 2024-03-18T14:43:18.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-18T14:43:18.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-18T14:43:18.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-18T14:43:18.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-18T14:43:18.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_open_files: -1 2024-03-18T14:43:18.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.bytes_per_sync: 0 2024-03-18T14:43:18.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-18T14:43:18.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-18T14:43:18.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compaction_readahead_size: 0 2024-03-18T14:43:18.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_background_flushes: -1 2024-03-18T14:43:18.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Compression algorithms supported: 2024-03-18T14:43:18.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-18T14:43:18.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: kZSTD supported: 0 2024-03-18T14:43:18.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: kXpressCompression supported: 0 2024-03-18T14:43:18.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: kLZ4HCCompression supported: 1 2024-03-18T14:43:18.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: kZlibCompression supported: 1 2024-03-18T14:43:18.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: kSnappyCompression supported: 1 2024-03-18T14:43:18.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: kLZ4Compression supported: 1 2024-03-18T14:43:18.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: kBZip2Compression supported: 0 2024-03-18T14:43:18.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-18T14:43:18.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-18T14:43:18.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-18T14:43:18.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-18T14:43:18.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-18T14:43:18.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.merge_operator: 2024-03-18T14:43:18.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compaction_filter: None 2024-03-18T14:43:18.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compaction_filter_factory: None 2024-03-18T14:43:18.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.sst_partitioner_factory: None 2024-03-18T14:43:18.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-18T14:43:18.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-18T14:43:18.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563b0e425bc0) 2024-03-18T14:43:18.324 INFO:journalctl@ceph.mon.a.smithi076.stdout: cache_index_and_filter_blocks: 1 2024-03-18T14:43:18.324 INFO:journalctl@ceph.mon.a.smithi076.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-18T14:43:18.324 INFO:journalctl@ceph.mon.a.smithi076.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-18T14:43:18.324 INFO:journalctl@ceph.mon.a.smithi076.stdout: pin_top_level_index_and_filter: 1 2024-03-18T14:43:18.324 INFO:journalctl@ceph.mon.a.smithi076.stdout: index_type: 0 2024-03-18T14:43:18.324 INFO:journalctl@ceph.mon.a.smithi076.stdout: data_block_index_type: 0 2024-03-18T14:43:18.324 INFO:journalctl@ceph.mon.a.smithi076.stdout: index_shortening: 1 2024-03-18T14:43:18.324 INFO:journalctl@ceph.mon.a.smithi076.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-18T14:43:18.324 INFO:journalctl@ceph.mon.a.smithi076.stdout: checksum: 4 2024-03-18T14:43:18.324 INFO:journalctl@ceph.mon.a.smithi076.stdout: no_block_cache: 0 2024-03-18T14:43:18.324 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_cache: 0x563b0e42b090 2024-03-18T14:43:18.325 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_cache_name: BinnedLRUCache 2024-03-18T14:43:18.325 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_cache_options: 2024-03-18T14:43:18.325 INFO:journalctl@ceph.mon.a.smithi076.stdout: capacity : 536870912 2024-03-18T14:43:18.325 INFO:journalctl@ceph.mon.a.smithi076.stdout: num_shard_bits : 4 2024-03-18T14:43:18.325 INFO:journalctl@ceph.mon.a.smithi076.stdout: strict_capacity_limit : 0 2024-03-18T14:43:18.325 INFO:journalctl@ceph.mon.a.smithi076.stdout: high_pri_pool_ratio: 0.000 2024-03-18T14:43:18.325 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_cache_compressed: (nil) 2024-03-18T14:43:18.325 INFO:journalctl@ceph.mon.a.smithi076.stdout: persistent_cache: (nil) 2024-03-18T14:43:18.325 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_size: 4096 2024-03-18T14:43:18.325 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_size_deviation: 10 2024-03-18T14:43:18.325 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_restart_interval: 16 2024-03-18T14:43:18.325 INFO:journalctl@ceph.mon.a.smithi076.stdout: index_block_restart_interval: 1 2024-03-18T14:43:18.325 INFO:journalctl@ceph.mon.a.smithi076.stdout: metadata_block_size: 4096 2024-03-18T14:43:18.325 INFO:journalctl@ceph.mon.a.smithi076.stdout: partition_filters: 0 2024-03-18T14:43:18.325 INFO:journalctl@ceph.mon.a.smithi076.stdout: use_delta_encoding: 1 2024-03-18T14:43:18.326 INFO:journalctl@ceph.mon.a.smithi076.stdout: filter_policy: bloomfilter 2024-03-18T14:43:18.326 INFO:journalctl@ceph.mon.a.smithi076.stdout: whole_key_filtering: 1 2024-03-18T14:43:18.326 INFO:journalctl@ceph.mon.a.smithi076.stdout: verify_compression: 0 2024-03-18T14:43:18.326 INFO:journalctl@ceph.mon.a.smithi076.stdout: read_amp_bytes_per_bit: 0 2024-03-18T14:43:18.326 INFO:journalctl@ceph.mon.a.smithi076.stdout: format_version: 5 2024-03-18T14:43:18.326 INFO:journalctl@ceph.mon.a.smithi076.stdout: enable_index_compression: 1 2024-03-18T14:43:18.326 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_align: 0 2024-03-18T14:43:18.326 INFO:journalctl@ceph.mon.a.smithi076.stdout: max_auto_readahead_size: 262144 2024-03-18T14:43:18.326 INFO:journalctl@ceph.mon.a.smithi076.stdout: prepopulate_block_cache: 0 2024-03-18T14:43:18.326 INFO:journalctl@ceph.mon.a.smithi076.stdout: initial_auto_readahead_size: 8192 2024-03-18T14:43:18.326 INFO:journalctl@ceph.mon.a.smithi076.stdout: num_file_reads_for_auto_readahead: 2 2024-03-18T14:43:18.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.write_buffer_size: 33554432 2024-03-18T14:43:18.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_write_buffer_number: 2 2024-03-18T14:43:18.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compression: NoCompression 2024-03-18T14:43:18.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.bottommost_compression: Disabled 2024-03-18T14:43:18.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T14:43:18.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T14:43:18.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.num_levels: 7 2024-03-18T14:43:18.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T14:43:18.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T14:43:18.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T14:43:18.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T14:43:18.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T14:43:18.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T14:43:18.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T14:43:18.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T14:43:18.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T14:43:18.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T14:43:18.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-18T14:43:18.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-18T14:43:18.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T14:43:18.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T14:43:18.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T14:43:18.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T14:43:18.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T14:43:18.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-18T14:43:18.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T14:43:18.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T14:43:18.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-18T14:43:18.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T14:43:18.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T14:43:18.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T14:43:18.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T14:43:18.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T14:43:18.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T14:43:18.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T14:43:18.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T14:43:18.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T14:43:18.329 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T14:43:18.329 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T14:43:18.329 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T14:43:18.329 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T14:43:18.329 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T14:43:18.329 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T14:43:18.329 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T14:43:18.329 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T14:43:18.329 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-18T14:43:18.329 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.arena_block_size: 1048576 2024-03-18T14:43:18.329 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T14:43:18.329 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T14:43:18.329 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T14:43:18.329 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T14:43:18.330 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T14:43:18.330 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T14:43:18.330 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T14:43:18.330 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T14:43:18.330 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T14:43:18.330 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T14:43:18.330 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T14:43:18.330 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T14:43:18.330 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T14:43:18.330 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.table_properties_collectors: 2024-03-18T14:43:18.330 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.inplace_update_support: 0 2024-03-18T14:43:18.330 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T14:43:18.330 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T14:43:18.330 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T14:43:18.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T14:43:18.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.bloom_locality: 0 2024-03-18T14:43:18.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.max_successive_merges: 0 2024-03-18T14:43:18.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T14:43:18.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T14:43:18.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T14:43:18.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T14:43:18.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.ttl: 2592000 2024-03-18T14:43:18.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T14:43:18.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-18T14:43:18.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-18T14:43:18.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.enable_blob_files: false 2024-03-18T14:43:18.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.min_blob_size: 0 2024-03-18T14:43:18.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T14:43:18.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T14:43:18.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T14:43:18.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T14:43:18.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-18T14:43:18.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-18T14:43:18.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.blob_file_starting_level: 0 2024-03-18T14:43:18.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-18T14:43:18.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-18T14:43:18.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-18T14:43:18.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 62ead443-f226-454f-bfdf-cb37576a6b85 2024-03-18T14:43:18.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710772998073785, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-18T14:43:18.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-18T14:43:18.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710772998074790, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99378, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 322, "table_properties": {"data_size": 97363, "index_size": 297, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13945, "raw_average_key_size": 49, "raw_value_size": 89734, "raw_average_value_size": 319, "num_data_blocks": 12, "num_entries": 281, "num_filter_entries": 281, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710772998, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "62ead443-f226-454f-bfdf-cb37576a6b85", "db_session_id": "T3OA4AMDPFYTCHFPNOGY", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-18T14:43:18.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710772998074879, "job": 1, "event": "recovery_finished"} 2024-03-18T14:43:18.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-18T14:43:18.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T14:43:18.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563b0e51e000 2024-03-18T14:43:18.333 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: DB pointer 0x563b0e508000 2024-03-18T14:43:18.333 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-18T14:43:18.333 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-18T14:43:18.333 INFO:journalctl@ceph.mon.a.smithi076.stdout: ** DB Stats ** 2024-03-18T14:43:18.333 INFO:journalctl@ceph.mon.a.smithi076.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T14:43:18.333 INFO:journalctl@ceph.mon.a.smithi076.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T14:43:18.333 INFO:journalctl@ceph.mon.a.smithi076.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T14:43:18.333 INFO:journalctl@ceph.mon.a.smithi076.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T14:43:18.333 INFO:journalctl@ceph.mon.a.smithi076.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T14:43:18.333 INFO:journalctl@ceph.mon.a.smithi076.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T14:43:18.333 INFO:journalctl@ceph.mon.a.smithi076.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T14:43:18.333 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-03-18T14:43:18.333 INFO:journalctl@ceph.mon.a.smithi076.stdout: ** Compaction Stats [default] ** 2024-03-18T14:43:18.334 INFO:journalctl@ceph.mon.a.smithi076.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T14:43:18.334 INFO:journalctl@ceph.mon.a.smithi076.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-18T14:43:18.334 INFO:journalctl@ceph.mon.a.smithi076.stdout: L0 2/0 98.88 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T14:43:18.334 INFO:journalctl@ceph.mon.a.smithi076.stdout: Sum 2/0 98.88 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T14:43:18.334 INFO:journalctl@ceph.mon.a.smithi076.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T14:43:18.334 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-03-18T14:43:18.334 INFO:journalctl@ceph.mon.a.smithi076.stdout: ** Compaction Stats [default] ** 2024-03-18T14:43:18.334 INFO:journalctl@ceph.mon.a.smithi076.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T14:43:18.334 INFO:journalctl@ceph.mon.a.smithi076.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T14:43:18.334 INFO:journalctl@ceph.mon.a.smithi076.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 147.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T14:43:18.334 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-03-18T14:43:18.334 INFO:journalctl@ceph.mon.a.smithi076.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-18T14:43:18.334 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-03-18T14:43:18.334 INFO:journalctl@ceph.mon.a.smithi076.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T14:43:18.335 INFO:journalctl@ceph.mon.a.smithi076.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T14:43:18.335 INFO:journalctl@ceph.mon.a.smithi076.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T14:43:18.335 INFO:journalctl@ceph.mon.a.smithi076.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T14:43:18.335 INFO:journalctl@ceph.mon.a.smithi076.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T14:43:18.335 INFO:journalctl@ceph.mon.a.smithi076.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T14:43:18.335 INFO:journalctl@ceph.mon.a.smithi076.stdout: Cumulative compaction: 0.00 GB write, 29.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T14:43:18.335 INFO:journalctl@ceph.mon.a.smithi076.stdout: Interval compaction: 0.00 GB write, 29.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T14:43:18.335 INFO:journalctl@ceph.mon.a.smithi076.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T14:43:18.335 INFO:journalctl@ceph.mon.a.smithi076.stdout: Block cache BinnedLRUCache@0x563b0e42b090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-18T14:43:18.335 INFO:journalctl@ceph.mon.a.smithi076.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-18T14:43:18.335 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-03-18T14:43:18.335 INFO:journalctl@ceph.mon.a.smithi076.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T14:43:18.335 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: starting mon.a rank 0 at public addrs [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] at bind addrs [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:43:18.335 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: mon.a@-1(???) e1 preinit fsid 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:43:18.335 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: mon.a@-1(???).mds e1 new map 2024-03-18T14:43:18.335 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: mon.a@-1(???).mds e1 print_map 2024-03-18T14:43:18.335 INFO:journalctl@ceph.mon.a.smithi076.stdout: e1 2024-03-18T14:43:18.336 INFO:journalctl@ceph.mon.a.smithi076.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T14:43:18.336 INFO:journalctl@ceph.mon.a.smithi076.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T14:43:18.336 INFO:journalctl@ceph.mon.a.smithi076.stdout: legacy client fscid: -1 2024-03-18T14:43:18.336 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-03-18T14:43:18.336 INFO:journalctl@ceph.mon.a.smithi076.stdout: No filesystems configured 2024-03-18T14:43:18.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T14:43:18.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T14:43:18.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T14:43:18.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T14:43:18.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-18T14:43:18.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 bash[32185]: 3df3aa06ebbe1ec071dca70665ddb9ab25137b99c794a5e752c9ce2e1f69f75e 2024-03-18T14:43:18.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 systemd[1]: Started Ceph mon.a for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:43:18.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T14:43:18.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: monmap epoch 1 2024-03-18T14:43:18.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: fsid 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:43:18.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: last_changed 2024-03-18T14:43:08.949643+0000 2024-03-18T14:43:18.337 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: created 2024-03-18T14:43:08.949643+0000 2024-03-18T14:43:18.337 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: min_mon_release 19 (squid) 2024-03-18T14:43:18.337 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: election_strategy: 1 2024-03-18T14:43:18.337 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-03-18T14:43:18.337 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: fsmap 2024-03-18T14:43:18.337 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T14:43:18.337 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:18 smithi076 ceph-mon[32205]: mgrmap e1: no daemons active 2024-03-18T14:43:19.181 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.512+0000 7fecc7993700 1 Processor -- start 2024-03-18T14:43:19.181 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.512+0000 7fecc7993700 1 -- start start 2024-03-18T14:43:19.181 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.512+0000 7fecc7993700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecc0107ba0 0x7fecc0107fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:19.181 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.512+0000 7fecc7993700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fecc0108590 con 0x7fecc0107ba0 2024-03-18T14:43:19.181 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.512+0000 7fecc572f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecc0107ba0 0x7fecc0107fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:19.181 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.512+0000 7fecc572f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecc0107ba0 0x7fecc0107fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55320/0 (socket says 172.21.15.76:55320) 2024-03-18T14:43:19.181 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.512+0000 7fecc572f700 1 -- 172.21.15.76:0/529687200 learned_addr learned my addr 172.21.15.76:0/529687200 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:19.181 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.513+0000 7fecc572f700 1 -- 172.21.15.76:0/529687200 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc0108da0 con 0x7fecc0107ba0 2024-03-18T14:43:19.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.513+0000 7fecc572f700 1 --2- 172.21.15.76:0/529687200 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecc0107ba0 0x7fecc0107fc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0098a0 tx=0x7fecbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ae2d3e8581c01c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:19.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.513+0000 7fecb7fff700 1 -- 172.21.15.76:0/529687200 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fecbc004160 con 0x7fecc0107ba0 2024-03-18T14:43:19.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.513+0000 7fecb7fff700 1 -- 172.21.15.76:0/529687200 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fecbc004300 con 0x7fecc0107ba0 2024-03-18T14:43:19.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.513+0000 7fecb7fff700 1 -- 172.21.15.76:0/529687200 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fecbc011b50 con 0x7fecc0107ba0 2024-03-18T14:43:19.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.514+0000 7fecc7993700 1 -- 172.21.15.76:0/529687200 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecc0107ba0 msgr2=0x7fecc0107fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:19.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.514+0000 7fecc7993700 1 --2- 172.21.15.76:0/529687200 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecc0107ba0 0x7fecc0107fc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0098a0 tx=0x7fecbc009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:19.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.514+0000 7fecc7993700 1 -- 172.21.15.76:0/529687200 shutdown_connections 2024-03-18T14:43:19.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.514+0000 7fecc7993700 1 --2- 172.21.15.76:0/529687200 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecc0107ba0 0x7fecc0107fc0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:19.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.514+0000 7fecc7993700 1 -- 172.21.15.76:0/529687200 >> 172.21.15.76:0/529687200 conn(0x7fecc0103180 msgr2=0x7fecc01055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:19.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.514+0000 7fecc7993700 1 -- 172.21.15.76:0/529687200 shutdown_connections 2024-03-18T14:43:19.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.514+0000 7fecc7993700 1 -- 172.21.15.76:0/529687200 wait complete. 2024-03-18T14:43:19.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.514+0000 7fecc7993700 1 Processor -- start 2024-03-18T14:43:19.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.514+0000 7fecc7993700 1 -- start start 2024-03-18T14:43:19.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.515+0000 7fecc7993700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecc0107ba0 0x7fecc019bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:19.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.515+0000 7fecc7993700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fecc010c150 con 0x7fecc0107ba0 2024-03-18T14:43:19.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.515+0000 7fecc572f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecc0107ba0 0x7fecc019bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:19.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.515+0000 7fecc572f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecc0107ba0 0x7fecc019bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55332/0 (socket says 172.21.15.76:55332) 2024-03-18T14:43:19.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.515+0000 7fecc572f700 1 -- 172.21.15.76:0/2425255836 learned_addr learned my addr 172.21.15.76:0/2425255836 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:19.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.515+0000 7fecc572f700 1 -- 172.21.15.76:0/2425255836 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc019c0b0 con 0x7fecc0107ba0 2024-03-18T14:43:19.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.515+0000 7fecc572f700 1 --2- 172.21.15.76:0/2425255836 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecc0107ba0 0x7fecc019bb70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fecbc011670 tx=0x7fecbc011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:19.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.515+0000 7fecb67fc700 1 -- 172.21.15.76:0/2425255836 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fecbc0039c0 con 0x7fecc0107ba0 2024-03-18T14:43:19.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.515+0000 7fecb67fc700 1 -- 172.21.15.76:0/2425255836 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fecbc011e50 con 0x7fecc0107ba0 2024-03-18T14:43:19.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.515+0000 7fecc7993700 1 -- 172.21.15.76:0/2425255836 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecc019c340 con 0x7fecc0107ba0 2024-03-18T14:43:19.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.515+0000 7fecb67fc700 1 -- 172.21.15.76:0/2425255836 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fecbc01a5c0 con 0x7fecc0107ba0 2024-03-18T14:43:19.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.515+0000 7fecc7993700 1 -- 172.21.15.76:0/2425255836 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecc019f030 con 0x7fecc0107ba0 2024-03-18T14:43:19.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.515+0000 7fecb67fc700 1 -- 172.21.15.76:0/2425255836 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fecbc01ac00 con 0x7fecc0107ba0 2024-03-18T14:43:19.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.516+0000 7fecb67fc700 1 -- 172.21.15.76:0/2425255836 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fecbc019700 con 0x7fecc0107ba0 2024-03-18T14:43:19.184 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.516+0000 7fecc7993700 1 -- 172.21.15.76:0/2425255836 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecc0063900 con 0x7fecc0107ba0 2024-03-18T14:43:19.184 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.519+0000 7fecb67fc700 1 -- 172.21.15.76:0/2425255836 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fecbc003b60 con 0x7fecc0107ba0 2024-03-18T14:43:19.184 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.586+0000 7fecc7993700 1 -- 172.21.15.76:0/2425255836 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fecc019f200 con 0x7fecc0107ba0 2024-03-18T14:43:19.184 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.588+0000 7fecb67fc700 1 -- 172.21.15.76:0/2425255836 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fecbc019900 con 0x7fecc0107ba0 2024-03-18T14:43:19.184 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.589+0000 7fecc7993700 1 -- 172.21.15.76:0/2425255836 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecc0107ba0 msgr2=0x7fecc019bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:19.184 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.589+0000 7fecc7993700 1 --2- 172.21.15.76:0/2425255836 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecc0107ba0 0x7fecc019bb70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fecbc011670 tx=0x7fecbc011b60 comp rx=0 tx=0).stop 2024-03-18T14:43:19.184 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.589+0000 7fecc7993700 1 -- 172.21.15.76:0/2425255836 shutdown_connections 2024-03-18T14:43:19.184 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.589+0000 7fecc7993700 1 --2- 172.21.15.76:0/2425255836 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecc0107ba0 0x7fecc019bb70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:19.184 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.589+0000 7fecc7993700 1 -- 172.21.15.76:0/2425255836 >> 172.21.15.76:0/2425255836 conn(0x7fecc0103180 msgr2=0x7fecc0103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:19.184 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.589+0000 7fecc7993700 1 -- 172.21.15.76:0/2425255836 shutdown_connections 2024-03-18T14:43:19.184 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:18.589+0000 7fecc7993700 1 -- 172.21.15.76:0/2425255836 wait complete. 2024-03-18T14:43:19.184 INFO:teuthology.orchestra.run.smithi076.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-18T14:43:19.198 INFO:teuthology.orchestra.run.smithi076.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-18T14:43:19.198 INFO:teuthology.orchestra.run.smithi076.stdout:Creating mgr... 2024-03-18T14:43:19.199 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-18T14:43:19.199 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-18T14:43:19.443 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mgr.x 2024-03-18T14:43:19.443 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-18T14:43:19.584 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9cc83b72-e535-11ee-95c9-87774f69a715.target.wants/ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-9cc83b72-e535-11ee-95c9-87774f69a715@.service. 2024-03-18T14:43:19.815 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:19 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2425255836' entity='client.admin' 2024-03-18T14:43:19.815 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:19 smithi076 systemd[1]: Starting Ceph mgr.x for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T14:43:20.086 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:19 smithi076 podman[32420]: 2024-03-18 14:43:19.717168737 +0000 UTC m=+0.017675171 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:43:20.086 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:19 smithi076 podman[32420]: 2024-03-18 14:43:19.816847601 +0000 UTC m=+0.117354060 container create bb92b6931123a9d1e01f482e87010c49b25f6de40966af28c840f29f100783c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-18T14:43:20.087 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:20 smithi076 podman[32420]: 2024-03-18 14:43:20.018943194 +0000 UTC m=+0.319449607 container init bb92b6931123a9d1e01f482e87010c49b25f6de40966af28c840f29f100783c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-18T14:43:20.087 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:20 smithi076 podman[32420]: 2024-03-18 14:43:20.021265045 +0000 UTC m=+0.321771465 container start bb92b6931123a9d1e01f482e87010c49b25f6de40966af28c840f29f100783c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T14:43:20.087 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:20 smithi076 bash[32420]: bb92b6931123a9d1e01f482e87010c49b25f6de40966af28c840f29f100783c6 2024-03-18T14:43:20.101 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T14:43:20.101 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout disabled 2024-03-18T14:43:20.109 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T14:43:20.109 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-03-18T14:43:20.109 INFO:teuthology.orchestra.run.smithi076.stdout:firewalld.service is not enabled 2024-03-18T14:43:20.109 INFO:teuthology.orchestra.run.smithi076.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T14:43:20.118 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T14:43:20.118 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout disabled 2024-03-18T14:43:20.126 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T14:43:20.126 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-03-18T14:43:20.126 INFO:teuthology.orchestra.run.smithi076.stdout:firewalld.service is not enabled 2024-03-18T14:43:20.126 INFO:teuthology.orchestra.run.smithi076.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-18T14:43:20.127 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mgr to start... 2024-03-18T14:43:20.127 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mgr... 2024-03-18T14:43:20.337 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:20 smithi076 systemd[1]: Started Ceph mgr.x for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:43:20.610 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:20 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:20.584+0000 7f7858403200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T14:43:20.909 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:20 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2983125618' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T14:43:20.909 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:20 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:20.682+0000 7f7858403200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T14:43:21.181 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-18T14:43:21.181 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-03-18T14:43:21.181 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsid": "9cc83b72-e535-11ee-95c9-87774f69a715", 2024-03-18T14:43:21.181 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T14:43:21.181 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T14:43:21.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T14:43:21.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T14:43:21.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:21.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T14:43:21.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T14:43:21.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 0 2024-03-18T14:43:21.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-18T14:43:21.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T14:43:21.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T14:43:21.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-18T14:43:21.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-18T14:43:21.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T14:43:21.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T14:43:21.182 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T14:43:21.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T14:43:21.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:21.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T14:43:21.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T14:43:21.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T14:43:21.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T14:43:21.183 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T14:43:21.191 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T14:43:21.191 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T14:43:21.191 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T14:43:21.191 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:21.191 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T14:43:21.191 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T14:43:21.191 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T14:43:21.191 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T14:43:21.191 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T14:43:21.191 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T14:43:21.192 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T14:43:21.192 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T14:43:21.192 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T14:43:21.192 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:21.192 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T14:43:21.192 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T14:43:21.192 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T14:43:21.192 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T14:43:21.192 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:21.192 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T14:43:21.192 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T14:43:21.192 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T14:43:21.192 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T14:43:21.192 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T14:43:21.193 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T14:43:21.193 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T14:43:21.193 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-18T14:43:21.193 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T14:43:21.193 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:21.193 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T14:43:21.193 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T14:43:21.193 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T14:43:12.899758+0000", 2024-03-18T14:43:21.193 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T14:43:21.193 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:21.193 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T14:43:21.193 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-03-18T14:43:21.193 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.530+0000 7f8c1d746700 1 Processor -- start 2024-03-18T14:43:21.193 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.530+0000 7f8c1d746700 1 -- start start 2024-03-18T14:43:21.194 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.530+0000 7f8c1d746700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c18108180 0x7f8c181085a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.194 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.530+0000 7f8c1d746700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8c181066c0 con 0x7f8c18108180 2024-03-18T14:43:21.194 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.530+0000 7f8c16ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c18108180 0x7f8c181085a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.194 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.531+0000 7f8c16ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c18108180 0x7f8c181085a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55366/0 (socket says 172.21.15.76:55366) 2024-03-18T14:43:21.194 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.531+0000 7f8c16ffd700 1 -- 172.21.15.76:0/4161433 learned_addr learned my addr 172.21.15.76:0/4161433 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:21.194 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.531+0000 7f8c16ffd700 1 -- 172.21.15.76:0/4161433 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c18106840 con 0x7f8c18108180 2024-03-18T14:43:21.194 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.531+0000 7f8c16ffd700 1 --2- 172.21.15.76:0/4161433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c18108180 0x7f8c181085a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0098a0 tx=0x7f8c0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce0409dd9e4496a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:21.194 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.531+0000 7f8c15ffb700 1 -- 172.21.15.76:0/4161433 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c0c004160 con 0x7f8c18108180 2024-03-18T14:43:21.194 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.531+0000 7f8c15ffb700 1 -- 172.21.15.76:0/4161433 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8c0c004300 con 0x7f8c18108180 2024-03-18T14:43:21.194 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.531+0000 7f8c15ffb700 1 -- 172.21.15.76:0/4161433 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c0c011b50 con 0x7f8c18108180 2024-03-18T14:43:21.194 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.532+0000 7f8c1d746700 1 -- 172.21.15.76:0/4161433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c18108180 msgr2=0x7f8c181085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.194 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.532+0000 7f8c1d746700 1 --2- 172.21.15.76:0/4161433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c18108180 0x7f8c181085a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0098a0 tx=0x7f8c0c009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.194 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.532+0000 7f8c1d746700 1 -- 172.21.15.76:0/4161433 shutdown_connections 2024-03-18T14:43:21.194 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.532+0000 7f8c1d746700 1 --2- 172.21.15.76:0/4161433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c18108180 0x7f8c181085a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.195 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.532+0000 7f8c1d746700 1 -- 172.21.15.76:0/4161433 >> 172.21.15.76:0/4161433 conn(0x7f8c18102000 msgr2=0x7f8c18104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:21.195 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.532+0000 7f8c1d746700 1 -- 172.21.15.76:0/4161433 shutdown_connections 2024-03-18T14:43:21.195 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.532+0000 7f8c1d746700 1 -- 172.21.15.76:0/4161433 wait complete. 2024-03-18T14:43:21.195 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.533+0000 7f8c1d746700 1 Processor -- start 2024-03-18T14:43:21.195 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.533+0000 7f8c1d746700 1 -- start start 2024-03-18T14:43:21.195 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.533+0000 7f8c1d746700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c18108180 0x7f8c1819dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.195 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.533+0000 7f8c1d746700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1810b050 con 0x7f8c18108180 2024-03-18T14:43:21.195 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.533+0000 7f8c16ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c18108180 0x7f8c1819dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.195 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.533+0000 7f8c16ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c18108180 0x7f8c1819dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55380/0 (socket says 172.21.15.76:55380) 2024-03-18T14:43:21.195 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.533+0000 7f8c16ffd700 1 -- 172.21.15.76:0/2983125618 learned_addr learned my addr 172.21.15.76:0/2983125618 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:21.195 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.533+0000 7f8c16ffd700 1 -- 172.21.15.76:0/2983125618 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c1819e420 con 0x7f8c18108180 2024-03-18T14:43:21.195 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.534+0000 7f8c16ffd700 1 --2- 172.21.15.76:0/2983125618 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c18108180 0x7f8c1819dee0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c009fb0 tx=0x7f8c0c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:21.195 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.534+0000 7f8bfffff700 1 -- 172.21.15.76:0/2983125618 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c0c0038b0 con 0x7f8c18108180 2024-03-18T14:43:21.195 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.534+0000 7f8bfffff700 1 -- 172.21.15.76:0/2983125618 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8c0c011e50 con 0x7f8c18108180 2024-03-18T14:43:21.195 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.534+0000 7f8c1d746700 1 -- 172.21.15.76:0/2983125618 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c1819e6b0 con 0x7f8c18108180 2024-03-18T14:43:21.196 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.534+0000 7f8c1d746700 1 -- 172.21.15.76:0/2983125618 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c181a13a0 con 0x7f8c18108180 2024-03-18T14:43:21.196 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.534+0000 7f8bfffff700 1 -- 172.21.15.76:0/2983125618 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c0c019550 con 0x7f8c18108180 2024-03-18T14:43:21.196 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.534+0000 7f8bfffff700 1 -- 172.21.15.76:0/2983125618 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8c0c019b90 con 0x7f8c18108180 2024-03-18T14:43:21.196 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.534+0000 7f8bfffff700 1 -- 172.21.15.76:0/2983125618 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8c0c024730 con 0x7f8c18108180 2024-03-18T14:43:21.196 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.535+0000 7f8c1d746700 1 -- 172.21.15.76:0/2983125618 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c18063900 con 0x7f8c18108180 2024-03-18T14:43:21.196 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.537+0000 7f8bfffff700 1 -- 172.21.15.76:0/2983125618 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8c0c02dda0 con 0x7f8c18108180 2024-03-18T14:43:21.196 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.608+0000 7f8c1d746700 1 -- 172.21.15.76:0/2983125618 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f8c1802dd30 con 0x7f8c18108180 2024-03-18T14:43:21.196 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.609+0000 7f8bfffff700 1 -- 172.21.15.76:0/2983125618 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f8c0c032030 con 0x7f8c18108180 2024-03-18T14:43:21.196 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.610+0000 7f8c1d746700 1 -- 172.21.15.76:0/2983125618 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c18108180 msgr2=0x7f8c1819dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.196 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.610+0000 7f8c1d746700 1 --2- 172.21.15.76:0/2983125618 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c18108180 0x7f8c1819dee0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c009fb0 tx=0x7f8c0c011bf0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.196 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.610+0000 7f8c1d746700 1 -- 172.21.15.76:0/2983125618 shutdown_connections 2024-03-18T14:43:21.196 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.610+0000 7f8c1d746700 1 --2- 172.21.15.76:0/2983125618 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c18108180 0x7f8c1819dee0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.196 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.610+0000 7f8c1d746700 1 -- 172.21.15.76:0/2983125618 >> 172.21.15.76:0/2983125618 conn(0x7f8c18102000 msgr2=0x7f8c18102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:21.196 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.610+0000 7f8c1d746700 1 -- 172.21.15.76:0/2983125618 shutdown_connections 2024-03-18T14:43:21.196 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:20.610+0000 7f8c1d746700 1 -- 172.21.15.76:0/2983125618 wait complete. 2024-03-18T14:43:21.197 INFO:teuthology.orchestra.run.smithi076.stdout:mgr not available, waiting (1/15)... 2024-03-18T14:43:21.556 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:21 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:21.326+0000 7f7858403200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T14:43:21.556 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:21 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:21.416+0000 7f7858403200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T14:43:21.556 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:21 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:21.554+0000 7f7858403200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T14:43:21.912 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:21 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:21.648+0000 7f7858403200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T14:43:21.912 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:21 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:21.728+0000 7f7858403200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T14:43:22.277 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:21 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:21.993+0000 7f7858403200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T14:43:22.278 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:22 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:22.100+0000 7f7858403200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T14:43:22.278 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:22 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:22.183+0000 7f7858403200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T14:43:22.278 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:22 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:22.275+0000 7f7858403200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T14:43:22.662 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:22 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:22.448+0000 7f7858403200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T14:43:23.629 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:23 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:23.410+0000 7f7858403200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T14:43:23.629 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:23 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:23.542+0000 7f7858403200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T14:43:23.884 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:23 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/900119372' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T14:43:24.162 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:23 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:23.882+0000 7f7858403200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T14:43:24.162 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:23 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:23.972+0000 7f7858403200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T14:43:24.258 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-18T14:43:24.276 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-03-18T14:43:24.276 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsid": "9cc83b72-e535-11ee-95c9-87774f69a715", 2024-03-18T14:43:24.277 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T14:43:24.277 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T14:43:24.277 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T14:43:24.277 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T14:43:24.277 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:24.277 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T14:43:24.278 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T14:43:24.278 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 0 2024-03-18T14:43:24.278 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-18T14:43:24.278 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T14:43:24.278 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T14:43:24.279 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-18T14:43:24.279 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-18T14:43:24.279 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T14:43:24.279 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T14:43:24.279 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T14:43:24.279 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T14:43:24.280 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:24.280 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T14:43:24.280 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T14:43:24.280 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T14:43:24.280 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T14:43:24.280 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T14:43:24.281 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T14:43:24.281 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T14:43:24.281 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T14:43:24.281 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:24.281 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T14:43:24.282 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T14:43:24.282 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T14:43:24.282 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T14:43:24.282 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T14:43:24.282 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T14:43:24.282 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T14:43:24.283 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T14:43:24.283 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T14:43:24.283 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:24.283 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T14:43:24.283 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T14:43:24.283 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T14:43:24.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T14:43:24.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:24.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T14:43:24.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T14:43:24.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T14:43:24.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T14:43:24.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T14:43:24.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T14:43:24.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T14:43:24.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-18T14:43:24.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T14:43:24.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:24.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T14:43:24.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T14:43:24.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T14:43:12.899758+0000", 2024-03-18T14:43:24.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T14:43:24.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:24.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T14:43:24.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-03-18T14:43:24.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.633+0000 7f9e401d2700 1 Processor -- start 2024-03-18T14:43:24.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.634+0000 7f9e401d2700 1 -- start start 2024-03-18T14:43:24.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.634+0000 7f9e401d2700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e38106a60 0x7f9e38106e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:24.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.634+0000 7f9e401d2700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9e38107450 con 0x7f9e38106a60 2024-03-18T14:43:24.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.634+0000 7f9e3df6e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e38106a60 0x7f9e38106e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:24.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.634+0000 7f9e3df6e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e38106a60 0x7f9e38106e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55388/0 (socket says 172.21.15.76:55388) 2024-03-18T14:43:24.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.634+0000 7f9e3df6e700 1 -- 172.21.15.76:0/2177864952 learned_addr learned my addr 172.21.15.76:0/2177864952 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:24.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.634+0000 7f9e3df6e700 1 -- 172.21.15.76:0/2177864952 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e38107c60 con 0x7f9e38106a60 2024-03-18T14:43:24.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.634+0000 7f9e3df6e700 1 --2- 172.21.15.76:0/2177864952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e38106a60 0x7f9e38106e80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9e280098a0 tx=0x7f9e28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d089d5d0d05176f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:24.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.635+0000 7f9e3cf6c700 1 -- 172.21.15.76:0/2177864952 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e28004160 con 0x7f9e38106a60 2024-03-18T14:43:24.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.635+0000 7f9e3cf6c700 1 -- 172.21.15.76:0/2177864952 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9e28004300 con 0x7f9e38106a60 2024-03-18T14:43:24.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.635+0000 7f9e3cf6c700 1 -- 172.21.15.76:0/2177864952 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e28011b50 con 0x7f9e38106a60 2024-03-18T14:43:24.287 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.635+0000 7f9e401d2700 1 -- 172.21.15.76:0/2177864952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e38106a60 msgr2=0x7f9e38106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:24.287 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.635+0000 7f9e401d2700 1 --2- 172.21.15.76:0/2177864952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e38106a60 0x7f9e38106e80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9e280098a0 tx=0x7f9e28009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.287 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.636+0000 7f9e401d2700 1 -- 172.21.15.76:0/2177864952 shutdown_connections 2024-03-18T14:43:24.287 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.636+0000 7f9e401d2700 1 --2- 172.21.15.76:0/2177864952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e38106a60 0x7f9e38106e80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.287 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.636+0000 7f9e401d2700 1 -- 172.21.15.76:0/2177864952 >> 172.21.15.76:0/2177864952 conn(0x7f9e38102040 msgr2=0x7f9e38104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:24.287 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.636+0000 7f9e401d2700 1 -- 172.21.15.76:0/2177864952 shutdown_connections 2024-03-18T14:43:24.287 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.636+0000 7f9e401d2700 1 -- 172.21.15.76:0/2177864952 wait complete. 2024-03-18T14:43:24.287 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.636+0000 7f9e401d2700 1 Processor -- start 2024-03-18T14:43:24.287 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.636+0000 7f9e401d2700 1 -- start start 2024-03-18T14:43:24.287 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.636+0000 7f9e401d2700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e38106a60 0x7f9e3819bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:24.287 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.636+0000 7f9e401d2700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3810b010 con 0x7f9e38106a60 2024-03-18T14:43:24.287 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.637+0000 7f9e3df6e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e38106a60 0x7f9e3819bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:24.287 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.637+0000 7f9e3df6e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e38106a60 0x7f9e3819bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55404/0 (socket says 172.21.15.76:55404) 2024-03-18T14:43:24.287 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.637+0000 7f9e3df6e700 1 -- 172.21.15.76:0/900119372 learned_addr learned my addr 172.21.15.76:0/900119372 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:24.288 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.637+0000 7f9e3df6e700 1 -- 172.21.15.76:0/900119372 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e3819c0f0 con 0x7f9e38106a60 2024-03-18T14:43:24.288 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.637+0000 7f9e3df6e700 1 --2- 172.21.15.76:0/900119372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e38106a60 0x7f9e3819bbb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e28011670 tx=0x7f9e28011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:24.288 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.637+0000 7f9e2effd700 1 -- 172.21.15.76:0/900119372 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e280039c0 con 0x7f9e38106a60 2024-03-18T14:43:24.288 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.637+0000 7f9e2effd700 1 -- 172.21.15.76:0/900119372 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9e28011e50 con 0x7f9e38106a60 2024-03-18T14:43:24.288 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.637+0000 7f9e401d2700 1 -- 172.21.15.76:0/900119372 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e3819c380 con 0x7f9e38106a60 2024-03-18T14:43:24.288 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.637+0000 7f9e2effd700 1 -- 172.21.15.76:0/900119372 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e2801a5c0 con 0x7f9e38106a60 2024-03-18T14:43:24.288 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.637+0000 7f9e401d2700 1 -- 172.21.15.76:0/900119372 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e3819f020 con 0x7f9e38106a60 2024-03-18T14:43:24.288 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.637+0000 7f9e2effd700 1 -- 172.21.15.76:0/900119372 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9e2801ac50 con 0x7f9e38106a60 2024-03-18T14:43:24.288 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.638+0000 7f9e2effd700 1 -- 172.21.15.76:0/900119372 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9e28019700 con 0x7f9e38106a60 2024-03-18T14:43:24.288 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.638+0000 7f9e401d2700 1 -- 172.21.15.76:0/900119372 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e38063900 con 0x7f9e38106a60 2024-03-18T14:43:24.288 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.640+0000 7f9e2effd700 1 -- 172.21.15.76:0/900119372 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9e28003b60 con 0x7f9e38106a60 2024-03-18T14:43:24.288 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.713+0000 7f9e401d2700 1 -- 172.21.15.76:0/900119372 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f9e380008d0 con 0x7f9e38106a60 2024-03-18T14:43:24.288 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.713+0000 7f9e2effd700 1 -- 172.21.15.76:0/900119372 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f9e280199a0 con 0x7f9e38106a60 2024-03-18T14:43:24.288 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.714+0000 7f9e401d2700 1 -- 172.21.15.76:0/900119372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e38106a60 msgr2=0x7f9e3819bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:24.288 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.714+0000 7f9e401d2700 1 --2- 172.21.15.76:0/900119372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e38106a60 0x7f9e3819bbb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e28011670 tx=0x7f9e28011b60 comp rx=0 tx=0).stop 2024-03-18T14:43:24.289 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.714+0000 7f9e401d2700 1 -- 172.21.15.76:0/900119372 shutdown_connections 2024-03-18T14:43:24.289 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.714+0000 7f9e401d2700 1 --2- 172.21.15.76:0/900119372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e38106a60 0x7f9e3819bbb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.289 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.714+0000 7f9e401d2700 1 -- 172.21.15.76:0/900119372 >> 172.21.15.76:0/900119372 conn(0x7f9e38102040 msgr2=0x7f9e38102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:24.289 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.714+0000 7f9e401d2700 1 -- 172.21.15.76:0/900119372 shutdown_connections 2024-03-18T14:43:24.289 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:23.714+0000 7f9e401d2700 1 -- 172.21.15.76:0/900119372 wait complete. 2024-03-18T14:43:24.289 INFO:teuthology.orchestra.run.smithi076.stdout:mgr not available, waiting (2/15)... 2024-03-18T14:43:24.633 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:24 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:24.167+0000 7f7858403200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:43:24.633 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:24 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:24.352+0000 7f7858403200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T14:43:24.887 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:24 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:24.631+0000 7f7858403200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:43:24.887 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:24 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:24.885+0000 7f7858403200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T14:43:25.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:25 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:25.118+0000 7f7858403200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T14:43:26.387 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:25 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:25.919+0000 7f7858403200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T14:43:26.651 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:26 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:26.512+0000 7f7858403200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T14:43:26.651 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:26 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:26.610+0000 7f7858403200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T14:43:26.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:26 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/849460157' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T14:43:26.912 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:26 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:26.839+0000 7f7858403200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T14:43:27.441 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-18T14:43:27.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-03-18T14:43:27.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsid": "9cc83b72-e535-11ee-95c9-87774f69a715", 2024-03-18T14:43:27.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T14:43:27.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T14:43:27.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T14:43:27.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T14:43:27.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:27.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T14:43:27.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T14:43:27.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 0 2024-03-18T14:43:27.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-18T14:43:27.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T14:43:27.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T14:43:27.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-18T14:43:27.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-18T14:43:27.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T14:43:27.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T14:43:27.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T14:43:27.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T14:43:27.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:27.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T14:43:27.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T14:43:27.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T14:43:27.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T14:43:27.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T14:43:27.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T14:43:27.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T14:43:27.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T14:43:27.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:27.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T14:43:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T14:43:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T14:43:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T14:43:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T14:43:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T14:43:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T14:43:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T14:43:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T14:43:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T14:43:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T14:43:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T14:43:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T14:43:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T14:43:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T14:43:27.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T14:43:27.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T14:43:27.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T14:43:27.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T14:43:27.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T14:43:27.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-18T14:43:27.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T14:43:27.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:27.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T14:43:27.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T14:43:27.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T14:43:12.899758+0000", 2024-03-18T14:43:27.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T14:43:27.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:27.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T14:43:27.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-03-18T14:43:27.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.681+0000 7f624b4d1700 1 Processor -- start 2024-03-18T14:43:27.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.681+0000 7f624b4d1700 1 -- start start 2024-03-18T14:43:27.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.681+0000 7f624b4d1700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6244107b60 0x7f6244107f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.681+0000 7f624b4d1700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6244108550 con 0x7f6244107b60 2024-03-18T14:43:27.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.682+0000 7f624926d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6244107b60 0x7f6244107f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.682+0000 7f624926d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6244107b60 0x7f6244107f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55410/0 (socket says 172.21.15.76:55410) 2024-03-18T14:43:27.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.682+0000 7f624926d700 1 -- 172.21.15.76:0/4046405799 learned_addr learned my addr 172.21.15.76:0/4046405799 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:27.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.682+0000 7f624926d700 1 -- 172.21.15.76:0/4046405799 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6244108d60 con 0x7f6244107b60 2024-03-18T14:43:27.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.682+0000 7f624926d700 1 --2- 172.21.15.76:0/4046405799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6244107b60 0x7f6244107f80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f62400098a0 tx=0x7f6240009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c3f4165e8fdb1b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:27.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.682+0000 7f623bfff700 1 -- 172.21.15.76:0/4046405799 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6240004160 con 0x7f6244107b60 2024-03-18T14:43:27.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.682+0000 7f623bfff700 1 -- 172.21.15.76:0/4046405799 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6240004300 con 0x7f6244107b60 2024-03-18T14:43:27.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.682+0000 7f623bfff700 1 -- 172.21.15.76:0/4046405799 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6240011b50 con 0x7f6244107b60 2024-03-18T14:43:27.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.683+0000 7f624b4d1700 1 -- 172.21.15.76:0/4046405799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6244107b60 msgr2=0x7f6244107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.683+0000 7f624b4d1700 1 --2- 172.21.15.76:0/4046405799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6244107b60 0x7f6244107f80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f62400098a0 tx=0x7f6240009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.683+0000 7f624b4d1700 1 -- 172.21.15.76:0/4046405799 shutdown_connections 2024-03-18T14:43:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.683+0000 7f624b4d1700 1 --2- 172.21.15.76:0/4046405799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6244107b60 0x7f6244107f80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.683+0000 7f624b4d1700 1 -- 172.21.15.76:0/4046405799 >> 172.21.15.76:0/4046405799 conn(0x7f62441030e0 msgr2=0x7f6244105540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.683+0000 7f624b4d1700 1 -- 172.21.15.76:0/4046405799 shutdown_connections 2024-03-18T14:43:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.683+0000 7f624b4d1700 1 -- 172.21.15.76:0/4046405799 wait complete. 2024-03-18T14:43:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.684+0000 7f624b4d1700 1 Processor -- start 2024-03-18T14:43:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.684+0000 7f624b4d1700 1 -- start start 2024-03-18T14:43:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.684+0000 7f624b4d1700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6244107b60 0x7f624419bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.684+0000 7f624b4d1700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f624410c110 con 0x7f6244107b60 2024-03-18T14:43:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.684+0000 7f624926d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6244107b60 0x7f624419bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.684+0000 7f624926d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6244107b60 0x7f624419bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55412/0 (socket says 172.21.15.76:55412) 2024-03-18T14:43:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.684+0000 7f624926d700 1 -- 172.21.15.76:0/849460157 learned_addr learned my addr 172.21.15.76:0/849460157 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.684+0000 7f624926d700 1 -- 172.21.15.76:0/849460157 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f624419c0c0 con 0x7f6244107b60 2024-03-18T14:43:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.684+0000 7f624926d700 1 --2- 172.21.15.76:0/849460157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6244107b60 0x7f624419bb80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6240009fb0 tx=0x7f6240011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.684+0000 7f623a7fc700 1 -- 172.21.15.76:0/849460157 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62400038b0 con 0x7f6244107b60 2024-03-18T14:43:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.684+0000 7f623a7fc700 1 -- 172.21.15.76:0/849460157 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6240011e50 con 0x7f6244107b60 2024-03-18T14:43:27.448 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.684+0000 7f623a7fc700 1 -- 172.21.15.76:0/849460157 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6240019550 con 0x7f6244107b60 2024-03-18T14:43:27.448 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.684+0000 7f624b4d1700 1 -- 172.21.15.76:0/849460157 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f624419c350 con 0x7f6244107b60 2024-03-18T14:43:27.448 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.685+0000 7f624b4d1700 1 -- 172.21.15.76:0/849460157 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f624419f040 con 0x7f6244107b60 2024-03-18T14:43:27.448 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.685+0000 7f623a7fc700 1 -- 172.21.15.76:0/849460157 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6240019be0 con 0x7f6244107b60 2024-03-18T14:43:27.448 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.685+0000 7f623a7fc700 1 -- 172.21.15.76:0/849460157 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6240024730 con 0x7f6244107b60 2024-03-18T14:43:27.448 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.685+0000 7f624b4d1700 1 -- 172.21.15.76:0/849460157 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6244063900 con 0x7f6244107b60 2024-03-18T14:43:27.448 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.688+0000 7f623a7fc700 1 -- 172.21.15.76:0/849460157 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6240003a50 con 0x7f6244107b60 2024-03-18T14:43:27.448 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.759+0000 7f624b4d1700 1 -- 172.21.15.76:0/849460157 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f624419f210 con 0x7f6244107b60 2024-03-18T14:43:27.448 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.760+0000 7f623a7fc700 1 -- 172.21.15.76:0/849460157 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f62400249d0 con 0x7f6244107b60 2024-03-18T14:43:27.448 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.761+0000 7f624b4d1700 1 -- 172.21.15.76:0/849460157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6244107b60 msgr2=0x7f624419bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.448 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.761+0000 7f624b4d1700 1 --2- 172.21.15.76:0/849460157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6244107b60 0x7f624419bb80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6240009fb0 tx=0x7f6240011bf0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.448 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.761+0000 7f624b4d1700 1 -- 172.21.15.76:0/849460157 shutdown_connections 2024-03-18T14:43:27.448 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.761+0000 7f624b4d1700 1 --2- 172.21.15.76:0/849460157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6244107b60 0x7f624419bb80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.448 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.761+0000 7f624b4d1700 1 -- 172.21.15.76:0/849460157 >> 172.21.15.76:0/849460157 conn(0x7f62441030e0 msgr2=0x7f6244103cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:27.448 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.761+0000 7f624b4d1700 1 -- 172.21.15.76:0/849460157 shutdown_connections 2024-03-18T14:43:27.449 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:26.761+0000 7f624b4d1700 1 -- 172.21.15.76:0/849460157 wait complete. 2024-03-18T14:43:27.449 INFO:teuthology.orchestra.run.smithi076.stdout:mgr not available, waiting (3/15)... 2024-03-18T14:43:28.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:27 smithi076 ceph-mon[32205]: Activating manager daemon x 2024-03-18T14:43:28.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:27 smithi076 ceph-mon[32205]: mgrmap e2: x(active, starting, since 0.0033489s) 2024-03-18T14:43:28.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:27 smithi076 ceph-mon[32205]: from='mgr.14100 172.21.15.76:0/1181363' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T14:43:28.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:27 smithi076 ceph-mon[32205]: from='mgr.14100 172.21.15.76:0/1181363' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T14:43:28.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:27 smithi076 ceph-mon[32205]: from='mgr.14100 172.21.15.76:0/1181363' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T14:43:28.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:27 smithi076 ceph-mon[32205]: from='mgr.14100 172.21.15.76:0/1181363' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:43:28.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:27 smithi076 ceph-mon[32205]: from='mgr.14100 172.21.15.76:0/1181363' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T14:43:28.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:27 smithi076 ceph-mon[32205]: Manager daemon x is now available 2024-03-18T14:43:28.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:27 smithi076 ceph-mon[32205]: from='mgr.14100 172.21.15.76:0/1181363' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T14:43:28.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:27 smithi076 ceph-mon[32205]: from='mgr.14100 172.21.15.76:0/1181363' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T14:43:28.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:27 smithi076 ceph-mon[32205]: from='mgr.14100 172.21.15.76:0/1181363' entity='mgr.x' 2024-03-18T14:43:28.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:27 smithi076 ceph-mon[32205]: from='mgr.14100 172.21.15.76:0/1181363' entity='mgr.x' 2024-03-18T14:43:28.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:27 smithi076 ceph-mon[32205]: from='mgr.14100 172.21.15.76:0/1181363' entity='mgr.x' 2024-03-18T14:43:29.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:28 smithi076 ceph-mon[32205]: mgrmap e3: x(active, since 1.0071s) 2024-03-18T14:43:30.681 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-18T14:43:30.681 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-03-18T14:43:30.681 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsid": "9cc83b72-e535-11ee-95c9-87774f69a715", 2024-03-18T14:43:30.681 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T14:43:30.682 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T14:43:30.682 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T14:43:30.682 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T14:43:30.682 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:30.682 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T14:43:30.682 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T14:43:30.682 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 0 2024-03-18T14:43:30.682 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-18T14:43:30.682 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T14:43:30.682 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T14:43:30.682 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-18T14:43:30.682 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-18T14:43:30.682 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T14:43:30.682 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T14:43:30.682 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T14:43:30.683 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T14:43:30.683 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:30.683 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T14:43:30.683 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T14:43:30.683 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T14:43:30.683 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T14:43:30.683 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T14:43:30.690 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T14:43:30.690 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T14:43:30.691 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T14:43:30.691 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:30.691 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T14:43:30.691 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T14:43:30.691 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T14:43:30.691 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T14:43:30.691 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T14:43:30.691 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T14:43:30.691 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T14:43:30.691 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T14:43:30.691 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T14:43:30.691 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:30.691 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T14:43:30.691 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T14:43:30.691 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T14:43:30.691 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T14:43:30.692 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:30.692 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T14:43:30.692 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T14:43:30.692 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T14:43:30.692 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T14:43:30.692 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T14:43:30.692 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T14:43:30.692 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T14:43:30.692 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-18T14:43:30.692 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T14:43:30.692 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:30.692 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T14:43:30.692 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T14:43:30.692 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T14:43:12.899758+0000", 2024-03-18T14:43:30.692 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T14:43:30.692 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-18T14:43:30.693 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T14:43:30.693 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-03-18T14:43:30.693 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.874+0000 7f1b7766c700 1 Processor -- start 2024-03-18T14:43:30.693 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.874+0000 7f1b7766c700 1 -- start start 2024-03-18T14:43:30.693 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.874+0000 7f1b7766c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b70107ba0 0x7f1b70107fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:30.693 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.874+0000 7f1b7766c700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1b70108590 con 0x7f1b70107ba0 2024-03-18T14:43:30.693 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.874+0000 7f1b75408700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b70107ba0 0x7f1b70107fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:30.693 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.875+0000 7f1b75408700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b70107ba0 0x7f1b70107fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54320/0 (socket says 172.21.15.76:54320) 2024-03-18T14:43:30.693 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.875+0000 7f1b75408700 1 -- 172.21.15.76:0/3581022717 learned_addr learned my addr 172.21.15.76:0/3581022717 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:30.693 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.875+0000 7f1b75408700 1 -- 172.21.15.76:0/3581022717 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b70108da0 con 0x7f1b70107ba0 2024-03-18T14:43:30.693 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.875+0000 7f1b75408700 1 --2- 172.21.15.76:0/3581022717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b70107ba0 0x7f1b70107fc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1b680098a0 tx=0x7f1b68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b274c14554a932d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:30.693 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.875+0000 7f1b6ffff700 1 -- 172.21.15.76:0/3581022717 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b68004160 con 0x7f1b70107ba0 2024-03-18T14:43:30.693 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.875+0000 7f1b6ffff700 1 -- 172.21.15.76:0/3581022717 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1b68004300 con 0x7f1b70107ba0 2024-03-18T14:43:30.693 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.875+0000 7f1b6ffff700 1 -- 172.21.15.76:0/3581022717 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b68011b50 con 0x7f1b70107ba0 2024-03-18T14:43:30.693 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.876+0000 7f1b7766c700 1 -- 172.21.15.76:0/3581022717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b70107ba0 msgr2=0x7f1b70107fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:30.693 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.876+0000 7f1b7766c700 1 --2- 172.21.15.76:0/3581022717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b70107ba0 0x7f1b70107fc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1b680098a0 tx=0x7f1b68009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:30.694 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.876+0000 7f1b7766c700 1 -- 172.21.15.76:0/3581022717 shutdown_connections 2024-03-18T14:43:30.694 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.876+0000 7f1b7766c700 1 --2- 172.21.15.76:0/3581022717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b70107ba0 0x7f1b70107fc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:30.694 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.876+0000 7f1b7766c700 1 -- 172.21.15.76:0/3581022717 >> 172.21.15.76:0/3581022717 conn(0x7f1b70103180 msgr2=0x7f1b701055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:30.694 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.876+0000 7f1b7766c700 1 -- 172.21.15.76:0/3581022717 shutdown_connections 2024-03-18T14:43:30.694 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.876+0000 7f1b7766c700 1 -- 172.21.15.76:0/3581022717 wait complete. 2024-03-18T14:43:30.694 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.876+0000 7f1b7766c700 1 Processor -- start 2024-03-18T14:43:30.694 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.876+0000 7f1b7766c700 1 -- start start 2024-03-18T14:43:30.694 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.877+0000 7f1b7766c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b70107ba0 0x7f1b7019dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:30.694 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.877+0000 7f1b7766c700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7010c150 con 0x7f1b70107ba0 2024-03-18T14:43:30.694 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.877+0000 7f1b75408700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b70107ba0 0x7f1b7019dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:30.694 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.877+0000 7f1b75408700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b70107ba0 0x7f1b7019dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54332/0 (socket says 172.21.15.76:54332) 2024-03-18T14:43:30.694 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.877+0000 7f1b75408700 1 -- 172.21.15.76:0/1239701599 learned_addr learned my addr 172.21.15.76:0/1239701599 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:30.694 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.877+0000 7f1b75408700 1 -- 172.21.15.76:0/1239701599 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b7019e270 con 0x7f1b70107ba0 2024-03-18T14:43:30.694 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.877+0000 7f1b75408700 1 --2- 172.21.15.76:0/1239701599 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b70107ba0 0x7f1b7019dd30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1b68011670 tx=0x7f1b68011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:30.694 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.877+0000 7f1b6e7fc700 1 -- 172.21.15.76:0/1239701599 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b680039c0 con 0x7f1b70107ba0 2024-03-18T14:43:30.695 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.877+0000 7f1b6e7fc700 1 -- 172.21.15.76:0/1239701599 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1b68011e50 con 0x7f1b70107ba0 2024-03-18T14:43:30.695 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.877+0000 7f1b7766c700 1 -- 172.21.15.76:0/1239701599 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b7019e500 con 0x7f1b70107ba0 2024-03-18T14:43:30.695 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.877+0000 7f1b6e7fc700 1 -- 172.21.15.76:0/1239701599 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b6801a5c0 con 0x7f1b70107ba0 2024-03-18T14:43:30.695 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.877+0000 7f1b7766c700 1 -- 172.21.15.76:0/1239701599 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b701a11f0 con 0x7f1b70107ba0 2024-03-18T14:43:30.695 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.878+0000 7f1b6e7fc700 1 -- 172.21.15.76:0/1239701599 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f1b6801a870 con 0x7f1b70107ba0 2024-03-18T14:43:30.695 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.878+0000 7f1b7766c700 1 -- 172.21.15.76:0/1239701599 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b70063900 con 0x7f1b70107ba0 2024-03-18T14:43:30.695 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.878+0000 7f1b6e7fc700 1 --2- 172.21.15.76:0/1239701599 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7f1b58038ed0 0x7f1b5803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:30.695 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.878+0000 7f1b6e7fc700 1 -- 172.21.15.76:0/1239701599 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1b6804d100 con 0x7f1b70107ba0 2024-03-18T14:43:30.695 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.879+0000 7f1b74c07700 1 --2- 172.21.15.76:0/1239701599 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7f1b58038ed0 0x7f1b5803b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:30.695 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.879+0000 7f1b74c07700 1 --2- 172.21.15.76:0/1239701599 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7f1b58038ed0 0x7f1b5803b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1b64006fd0 tx=0x7f1b64006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:30.695 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:29.884+0000 7f1b6e7fc700 1 -- 172.21.15.76:0/1239701599 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1b68019c70 con 0x7f1b70107ba0 2024-03-18T14:43:30.695 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:30.154+0000 7f1b7766c700 1 -- 172.21.15.76:0/1239701599 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f1b701a13c0 con 0x7f1b70107ba0 2024-03-18T14:43:30.695 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:30.155+0000 7f1b6e7fc700 1 -- 172.21.15.76:0/1239701599 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f1b701a13c0 con 0x7f1b70107ba0 2024-03-18T14:43:30.695 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:30.157+0000 7f1b7766c700 1 -- 172.21.15.76:0/1239701599 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7f1b58038ed0 msgr2=0x7f1b5803b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:30.695 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:30.157+0000 7f1b7766c700 1 --2- 172.21.15.76:0/1239701599 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7f1b58038ed0 0x7f1b5803b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1b64006fd0 tx=0x7f1b64006e40 comp rx=0 tx=0).stop 2024-03-18T14:43:30.696 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:30.157+0000 7f1b7766c700 1 -- 172.21.15.76:0/1239701599 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b70107ba0 msgr2=0x7f1b7019dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:30.696 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:30.157+0000 7f1b7766c700 1 --2- 172.21.15.76:0/1239701599 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b70107ba0 0x7f1b7019dd30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1b68011670 tx=0x7f1b68011b60 comp rx=0 tx=0).stop 2024-03-18T14:43:30.696 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:30.157+0000 7f1b7766c700 1 -- 172.21.15.76:0/1239701599 shutdown_connections 2024-03-18T14:43:30.696 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:30.157+0000 7f1b7766c700 1 --2- 172.21.15.76:0/1239701599 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7f1b58038ed0 0x7f1b5803b390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:30.696 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:30.157+0000 7f1b7766c700 1 --2- 172.21.15.76:0/1239701599 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b70107ba0 0x7f1b7019dd30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:30.696 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:30.157+0000 7f1b7766c700 1 -- 172.21.15.76:0/1239701599 >> 172.21.15.76:0/1239701599 conn(0x7f1b70103180 msgr2=0x7f1b70103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:30.696 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:30.157+0000 7f1b7766c700 1 -- 172.21.15.76:0/1239701599 shutdown_connections 2024-03-18T14:43:30.696 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:30.157+0000 7f1b7766c700 1 -- 172.21.15.76:0/1239701599 wait complete. 2024-03-18T14:43:30.696 INFO:teuthology.orchestra.run.smithi076.stdout:mgr is available 2024-03-18T14:43:31.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:30 smithi076 ceph-mon[32205]: mgrmap e4: x(active, since 3s) 2024-03-18T14:43:31.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:30 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1239701599' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T14:43:31.908 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-18T14:43:31.908 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T14:43:31.908 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout fsid = 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:43:31.908 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T14:43:31.909 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T14:43:31.909 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T14:43:31.909 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T14:43:31.909 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-18T14:43:31.909 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T14:43:31.909 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T14:43:31.909 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-18T14:43:31.909 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T14:43:31.909 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T14:43:31.909 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T14:43:31.909 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T14:43:31.909 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T14:43:31.909 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T14:43:31.909 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.092+0000 7fc90bd73700 1 Processor -- start 2024-03-18T14:43:31.909 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.092+0000 7fc90bd73700 1 -- start start 2024-03-18T14:43:31.909 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.092+0000 7fc90bd73700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc904106ab0 0x7fc904106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:31.910 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.092+0000 7fc90bd73700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc9041074a0 con 0x7fc904106ab0 2024-03-18T14:43:31.910 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.092+0000 7fc909b0f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc904106ab0 0x7fc904106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:31.910 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.092+0000 7fc909b0f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc904106ab0 0x7fc904106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54336/0 (socket says 172.21.15.76:54336) 2024-03-18T14:43:31.910 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.092+0000 7fc909b0f700 1 -- 172.21.15.76:0/4066410626 learned_addr learned my addr 172.21.15.76:0/4066410626 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:31.910 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.092+0000 7fc909b0f700 1 -- 172.21.15.76:0/4066410626 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc904107cb0 con 0x7fc904106ab0 2024-03-18T14:43:31.910 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.093+0000 7fc909b0f700 1 --2- 172.21.15.76:0/4066410626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc904106ab0 0x7fc904106ed0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc9000098a0 tx=0x7fc900009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a59ca5e70853446 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:31.910 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.093+0000 7fc908b0d700 1 -- 172.21.15.76:0/4066410626 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc900004160 con 0x7fc904106ab0 2024-03-18T14:43:31.910 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.093+0000 7fc908b0d700 1 -- 172.21.15.76:0/4066410626 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc900004300 con 0x7fc904106ab0 2024-03-18T14:43:31.910 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.093+0000 7fc908b0d700 1 -- 172.21.15.76:0/4066410626 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc900011b50 con 0x7fc904106ab0 2024-03-18T14:43:31.910 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.093+0000 7fc90bd73700 1 -- 172.21.15.76:0/4066410626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc904106ab0 msgr2=0x7fc904106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:31.910 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.093+0000 7fc90bd73700 1 --2- 172.21.15.76:0/4066410626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc904106ab0 0x7fc904106ed0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc9000098a0 tx=0x7fc900009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:31.910 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.093+0000 7fc90bd73700 1 -- 172.21.15.76:0/4066410626 shutdown_connections 2024-03-18T14:43:31.911 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.093+0000 7fc90bd73700 1 --2- 172.21.15.76:0/4066410626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc904106ab0 0x7fc904106ed0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:31.911 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.093+0000 7fc90bd73700 1 -- 172.21.15.76:0/4066410626 >> 172.21.15.76:0/4066410626 conn(0x7fc904102050 msgr2=0x7fc904104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:31.911 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.093+0000 7fc90bd73700 1 -- 172.21.15.76:0/4066410626 shutdown_connections 2024-03-18T14:43:31.911 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.093+0000 7fc90bd73700 1 -- 172.21.15.76:0/4066410626 wait complete. 2024-03-18T14:43:31.911 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.094+0000 7fc90bd73700 1 Processor -- start 2024-03-18T14:43:31.911 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.094+0000 7fc90bd73700 1 -- start start 2024-03-18T14:43:31.911 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.094+0000 7fc90bd73700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc904106ab0 0x7fc904191c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:31.911 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.094+0000 7fc90bd73700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc90410b060 con 0x7fc904106ab0 2024-03-18T14:43:31.911 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.094+0000 7fc909b0f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc904106ab0 0x7fc904191c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:31.911 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.094+0000 7fc909b0f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc904106ab0 0x7fc904191c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54340/0 (socket says 172.21.15.76:54340) 2024-03-18T14:43:31.911 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.094+0000 7fc909b0f700 1 -- 172.21.15.76:0/1936937765 learned_addr learned my addr 172.21.15.76:0/1936937765 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:31.911 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.094+0000 7fc909b0f700 1 -- 172.21.15.76:0/1936937765 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9041921a0 con 0x7fc904106ab0 2024-03-18T14:43:31.911 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.095+0000 7fc909b0f700 1 --2- 172.21.15.76:0/1936937765 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc904106ab0 0x7fc904191c60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc900009fb0 tx=0x7fc900011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:31.911 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.095+0000 7fc8faffd700 1 -- 172.21.15.76:0/1936937765 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9000038b0 con 0x7fc904106ab0 2024-03-18T14:43:31.912 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.095+0000 7fc90bd73700 1 -- 172.21.15.76:0/1936937765 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc904190390 con 0x7fc904106ab0 2024-03-18T14:43:31.912 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.095+0000 7fc8faffd700 1 -- 172.21.15.76:0/1936937765 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc900011e50 con 0x7fc904106ab0 2024-03-18T14:43:31.912 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.095+0000 7fc8faffd700 1 -- 172.21.15.76:0/1936937765 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc900019550 con 0x7fc904106ab0 2024-03-18T14:43:31.912 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.095+0000 7fc90bd73700 1 -- 172.21.15.76:0/1936937765 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9041908d0 con 0x7fc904106ab0 2024-03-18T14:43:31.912 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.096+0000 7fc8faffd700 1 -- 172.21.15.76:0/1936937765 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7fc900019850 con 0x7fc904106ab0 2024-03-18T14:43:31.912 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.096+0000 7fc90bd73700 1 -- 172.21.15.76:0/1936937765 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc904063900 con 0x7fc904106ab0 2024-03-18T14:43:31.912 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.096+0000 7fc8faffd700 1 --2- 172.21.15.76:0/1936937765 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fc8f0038e80 0x7fc8f003b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:31.912 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.096+0000 7fc8faffd700 1 -- 172.21.15.76:0/1936937765 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc90004d150 con 0x7fc904106ab0 2024-03-18T14:43:31.912 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.096+0000 7fc90930e700 1 --2- 172.21.15.76:0/1936937765 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fc8f0038e80 0x7fc8f003b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:31.912 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.097+0000 7fc90930e700 1 --2- 172.21.15.76:0/1936937765 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fc8f0038e80 0x7fc8f003b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4006fd0 tx=0x7fc8f4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:31.912 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.101+0000 7fc8faffd700 1 -- 172.21.15.76:0/1936937765 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc900003ac0 con 0x7fc904106ab0 2024-03-18T14:43:31.912 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.292+0000 7fc90bd73700 1 -- 172.21.15.76:0/1936937765 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fc9041959e0 con 0x7fc904106ab0 2024-03-18T14:43:31.912 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.293+0000 7fc8faffd700 1 -- 172.21.15.76:0/1936937765 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7fc90002d440 con 0x7fc904106ab0 2024-03-18T14:43:31.912 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.295+0000 7fc90bd73700 1 -- 172.21.15.76:0/1936937765 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fc8f0038e80 msgr2=0x7fc8f003b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:31.913 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.295+0000 7fc90bd73700 1 --2- 172.21.15.76:0/1936937765 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fc8f0038e80 0x7fc8f003b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4006fd0 tx=0x7fc8f4006e40 comp rx=0 tx=0).stop 2024-03-18T14:43:31.913 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.295+0000 7fc90bd73700 1 -- 172.21.15.76:0/1936937765 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc904106ab0 msgr2=0x7fc904191c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:31.913 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.295+0000 7fc90bd73700 1 --2- 172.21.15.76:0/1936937765 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc904106ab0 0x7fc904191c60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc900009fb0 tx=0x7fc900011bf0 comp rx=0 tx=0).stop 2024-03-18T14:43:31.913 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.296+0000 7fc90bd73700 1 -- 172.21.15.76:0/1936937765 shutdown_connections 2024-03-18T14:43:31.913 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.296+0000 7fc90bd73700 1 --2- 172.21.15.76:0/1936937765 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fc8f0038e80 0x7fc8f003b340 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:31.913 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.296+0000 7fc90bd73700 1 --2- 172.21.15.76:0/1936937765 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc904106ab0 0x7fc904191c60 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:31.913 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.296+0000 7fc90bd73700 1 -- 172.21.15.76:0/1936937765 >> 172.21.15.76:0/1936937765 conn(0x7fc904102050 msgr2=0x7fc904102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:31.913 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.296+0000 7fc90bd73700 1 -- 172.21.15.76:0/1936937765 shutdown_connections 2024-03-18T14:43:31.913 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:31.296+0000 7fc90bd73700 1 -- 172.21.15.76:0/1936937765 wait complete. 2024-03-18T14:43:31.913 INFO:teuthology.orchestra.run.smithi076.stdout:Enabling cephadm module... 2024-03-18T14:43:32.138 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:31 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1936937765' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T14:43:32.868 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:32 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1817750573' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-18T14:43:33.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:32 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: ignoring --setuser ceph since I am not root 2024-03-18T14:43:33.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:32 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: ignoring --setgroup ceph since I am not root 2024-03-18T14:43:33.387 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.354+0000 7ff2b432b700 1 Processor -- start 2024-03-18T14:43:33.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.354+0000 7ff2b432b700 1 -- start start 2024-03-18T14:43:33.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.354+0000 7ff2b432b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2ac106ab0 0x7ff2ac106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:33.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.354+0000 7ff2b432b700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ac1074a0 con 0x7ff2ac106ab0 2024-03-18T14:43:33.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.354+0000 7ff2b20c7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2ac106ab0 0x7ff2ac106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:33.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.354+0000 7ff2b20c7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2ac106ab0 0x7ff2ac106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54342/0 (socket says 172.21.15.76:54342) 2024-03-18T14:43:33.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.354+0000 7ff2b20c7700 1 -- 172.21.15.76:0/3816989333 learned_addr learned my addr 172.21.15.76:0/3816989333 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:33.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.355+0000 7ff2b20c7700 1 -- 172.21.15.76:0/3816989333 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2ac107cb0 con 0x7ff2ac106ab0 2024-03-18T14:43:33.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.355+0000 7ff2b20c7700 1 --2- 172.21.15.76:0/3816989333 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2ac106ab0 0x7ff2ac106ed0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80098a0 tx=0x7ff2a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b182f95a557826ef server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:33.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.355+0000 7ff2b10c5700 1 -- 172.21.15.76:0/3816989333 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2a8004160 con 0x7ff2ac106ab0 2024-03-18T14:43:33.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.355+0000 7ff2b10c5700 1 -- 172.21.15.76:0/3816989333 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff2a8004300 con 0x7ff2ac106ab0 2024-03-18T14:43:33.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.355+0000 7ff2b10c5700 1 -- 172.21.15.76:0/3816989333 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2a8011b50 con 0x7ff2ac106ab0 2024-03-18T14:43:33.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.355+0000 7ff2b432b700 1 -- 172.21.15.76:0/3816989333 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2ac106ab0 msgr2=0x7ff2ac106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:33.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.355+0000 7ff2b432b700 1 --2- 172.21.15.76:0/3816989333 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2ac106ab0 0x7ff2ac106ed0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80098a0 tx=0x7ff2a8009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.356+0000 7ff2b432b700 1 -- 172.21.15.76:0/3816989333 shutdown_connections 2024-03-18T14:43:33.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.356+0000 7ff2b432b700 1 --2- 172.21.15.76:0/3816989333 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2ac106ab0 0x7ff2ac106ed0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.356+0000 7ff2b432b700 1 -- 172.21.15.76:0/3816989333 >> 172.21.15.76:0/3816989333 conn(0x7ff2ac102050 msgr2=0x7ff2ac104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:33.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.356+0000 7ff2b432b700 1 -- 172.21.15.76:0/3816989333 shutdown_connections 2024-03-18T14:43:33.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.356+0000 7ff2b432b700 1 -- 172.21.15.76:0/3816989333 wait complete. 2024-03-18T14:43:33.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.356+0000 7ff2b432b700 1 Processor -- start 2024-03-18T14:43:33.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.356+0000 7ff2b432b700 1 -- start start 2024-03-18T14:43:33.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.356+0000 7ff2b432b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2ac106ab0 0x7ff2ac19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:33.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.356+0000 7ff2b432b700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ac10b060 con 0x7ff2ac106ab0 2024-03-18T14:43:33.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.357+0000 7ff2b20c7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2ac106ab0 0x7ff2ac19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:33.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.357+0000 7ff2b20c7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2ac106ab0 0x7ff2ac19aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54356/0 (socket says 172.21.15.76:54356) 2024-03-18T14:43:33.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.357+0000 7ff2b20c7700 1 -- 172.21.15.76:0/1817750573 learned_addr learned my addr 172.21.15.76:0/1817750573 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:33.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.357+0000 7ff2b20c7700 1 -- 172.21.15.76:0/1817750573 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2ac19b010 con 0x7ff2ac106ab0 2024-03-18T14:43:33.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.357+0000 7ff2b20c7700 1 --2- 172.21.15.76:0/1817750573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2ac106ab0 0x7ff2ac19aad0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8011850 tx=0x7ff2a8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:33.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.357+0000 7ff2a37fe700 1 -- 172.21.15.76:0/1817750573 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2a8003af0 con 0x7ff2ac106ab0 2024-03-18T14:43:33.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.357+0000 7ff2a37fe700 1 -- 172.21.15.76:0/1817750573 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff2a8020070 con 0x7ff2ac106ab0 2024-03-18T14:43:33.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.357+0000 7ff2b432b700 1 -- 172.21.15.76:0/1817750573 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2ac19b2a0 con 0x7ff2ac106ab0 2024-03-18T14:43:33.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.357+0000 7ff2a37fe700 1 -- 172.21.15.76:0/1817750573 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2a80195a0 con 0x7ff2ac106ab0 2024-03-18T14:43:33.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.357+0000 7ff2b432b700 1 -- 172.21.15.76:0/1817750573 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2ac19df90 con 0x7ff2ac106ab0 2024-03-18T14:43:33.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.358+0000 7ff2a37fe700 1 -- 172.21.15.76:0/1817750573 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7ff2a8019850 con 0x7ff2ac106ab0 2024-03-18T14:43:33.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.358+0000 7ff2a37fe700 1 --2- 172.21.15.76:0/1817750573 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7ff298038f20 0x7ff29803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:33.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.358+0000 7ff2b432b700 1 -- 172.21.15.76:0/1817750573 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2ac063900 con 0x7ff2ac106ab0 2024-03-18T14:43:33.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.358+0000 7ff2b18c6700 1 --2- 172.21.15.76:0/1817750573 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7ff298038f20 0x7ff29803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:33.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.358+0000 7ff2a37fe700 1 -- 172.21.15.76:0/1817750573 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff2a8011e00 con 0x7ff2ac106ab0 2024-03-18T14:43:33.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.359+0000 7ff2b18c6700 1 --2- 172.21.15.76:0/1817750573 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7ff298038f20 0x7ff29803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff29c006fd0 tx=0x7ff29c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:33.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.363+0000 7ff2a37fe700 1 -- 172.21.15.76:0/1817750573 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff2a8003c90 con 0x7ff2ac106ab0 2024-03-18T14:43:33.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.615+0000 7ff2b432b700 1 -- 172.21.15.76:0/1817750573 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7ff2ac19e160 con 0x7ff2ac106ab0 2024-03-18T14:43:33.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.863+0000 7ff2a37fe700 1 -- 172.21.15.76:0/1817750573 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7ff2ac19e160 con 0x7ff2ac106ab0 2024-03-18T14:43:33.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.864+0000 7ff2a37fe700 1 -- 172.21.15.76:0/1817750573 <== mon.0 v2:172.21.15.76:3300/0 8 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7ff2a800b430 con 0x7ff2ac106ab0 2024-03-18T14:43:33.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.866+0000 7ff2b432b700 1 -- 172.21.15.76:0/1817750573 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7ff298038f20 msgr2=0x7ff29803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:33.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.866+0000 7ff2b432b700 1 --2- 172.21.15.76:0/1817750573 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7ff298038f20 0x7ff29803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff29c006fd0 tx=0x7ff29c006e40 comp rx=0 tx=0).stop 2024-03-18T14:43:33.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.866+0000 7ff2b432b700 1 -- 172.21.15.76:0/1817750573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2ac106ab0 msgr2=0x7ff2ac19aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:33.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.866+0000 7ff2b432b700 1 --2- 172.21.15.76:0/1817750573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2ac106ab0 0x7ff2ac19aad0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8011850 tx=0x7ff2a8011c10 comp rx=0 tx=0).stop 2024-03-18T14:43:33.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.866+0000 7ff2b432b700 1 -- 172.21.15.76:0/1817750573 shutdown_connections 2024-03-18T14:43:33.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.866+0000 7ff2b432b700 1 --2- 172.21.15.76:0/1817750573 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7ff298038f20 0x7ff29803b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.866+0000 7ff2b432b700 1 --2- 172.21.15.76:0/1817750573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2ac106ab0 0x7ff2ac19aad0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.866+0000 7ff2b432b700 1 -- 172.21.15.76:0/1817750573 >> 172.21.15.76:0/1817750573 conn(0x7ff2ac102050 msgr2=0x7ff2ac102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:33.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.867+0000 7ff2b432b700 1 -- 172.21.15.76:0/1817750573 shutdown_connections 2024-03-18T14:43:33.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:32.867+0000 7ff2b432b700 1 -- 172.21.15.76:0/1817750573 wait complete. 2024-03-18T14:43:33.692 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:33 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:33.439+0000 7f9b4765c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T14:43:33.693 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:33 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:33.534+0000 7f9b4765c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T14:43:34.088 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:33 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1817750573' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-18T14:43:34.088 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:33 smithi076 ceph-mon[32205]: mgrmap e5: x(active, since 6s) 2024-03-18T14:43:34.412 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:34 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:34.205+0000 7f9b4765c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T14:43:34.412 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:34 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:34.299+0000 7f9b4765c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T14:43:34.745 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:34 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:34.442+0000 7f9b4765c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T14:43:34.745 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:34 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:34.539+0000 7f9b4765c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T14:43:34.745 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:34 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:34.623+0000 7f9b4765c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T14:43:34.754 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-03-18T14:43:34.754 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-18T14:43:34.754 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T14:43:34.754 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-18T14:43:34.754 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T14:43:34.754 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-03-18T14:43:34.754 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.881+0000 7f124eb27700 1 Processor -- start 2024-03-18T14:43:34.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.881+0000 7f124eb27700 1 -- start start 2024-03-18T14:43:34.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.881+0000 7f124eb27700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f124807de80 0x7f124807c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.881+0000 7f124eb27700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f124807c8b0 con 0x7f124807de80 2024-03-18T14:43:34.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.881+0000 7f124c8c3700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f124807de80 0x7f124807c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.881+0000 7f124c8c3700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f124807de80 0x7f124807c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54376/0 (socket says 172.21.15.76:54376) 2024-03-18T14:43:34.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.881+0000 7f124c8c3700 1 -- 172.21.15.76:0/4161780479 learned_addr learned my addr 172.21.15.76:0/4161780479 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:34.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.882+0000 7f124c8c3700 1 -- 172.21.15.76:0/4161780479 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f124807ca30 con 0x7f124807de80 2024-03-18T14:43:34.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.882+0000 7f124c8c3700 1 --2- 172.21.15.76:0/4161780479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f124807de80 0x7f124807c2e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f123c0098a0 tx=0x7f123c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c89eeeb752a36b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:34.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.882+0000 7f12477fe700 1 -- 172.21.15.76:0/4161780479 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f123c004160 con 0x7f124807de80 2024-03-18T14:43:34.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.882+0000 7f12477fe700 1 -- 172.21.15.76:0/4161780479 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f123c004300 con 0x7f124807de80 2024-03-18T14:43:34.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.882+0000 7f12477fe700 1 -- 172.21.15.76:0/4161780479 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f123c011b50 con 0x7f124807de80 2024-03-18T14:43:34.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.883+0000 7f124eb27700 1 -- 172.21.15.76:0/4161780479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f124807de80 msgr2=0x7f124807c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.883+0000 7f124eb27700 1 --2- 172.21.15.76:0/4161780479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f124807de80 0x7f124807c2e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f123c0098a0 tx=0x7f123c009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.883+0000 7f124eb27700 1 -- 172.21.15.76:0/4161780479 shutdown_connections 2024-03-18T14:43:34.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.883+0000 7f124eb27700 1 --2- 172.21.15.76:0/4161780479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f124807de80 0x7f124807c2e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.883+0000 7f124eb27700 1 -- 172.21.15.76:0/4161780479 >> 172.21.15.76:0/4161780479 conn(0x7f1248103100 msgr2=0x7f1248105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:34.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.883+0000 7f124eb27700 1 -- 172.21.15.76:0/4161780479 shutdown_connections 2024-03-18T14:43:34.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.883+0000 7f124eb27700 1 -- 172.21.15.76:0/4161780479 wait complete. 2024-03-18T14:43:34.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.884+0000 7f124eb27700 1 Processor -- start 2024-03-18T14:43:34.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.884+0000 7f124eb27700 1 -- start start 2024-03-18T14:43:34.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.884+0000 7f124eb27700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f124807de80 0x7f124819dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.884+0000 7f124eb27700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f124810e170 con 0x7f124807de80 2024-03-18T14:43:34.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.884+0000 7f124c8c3700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f124807de80 0x7f124819dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.884+0000 7f124c8c3700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f124807de80 0x7f124819dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54380/0 (socket says 172.21.15.76:54380) 2024-03-18T14:43:34.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.884+0000 7f124c8c3700 1 -- 172.21.15.76:0/1396099656 learned_addr learned my addr 172.21.15.76:0/1396099656 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:34.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.884+0000 7f124c8c3700 1 -- 172.21.15.76:0/1396099656 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f124819e270 con 0x7f124807de80 2024-03-18T14:43:34.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.884+0000 7f124c8c3700 1 --2- 172.21.15.76:0/1396099656 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f124807de80 0x7f124819dd30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f123c011670 tx=0x7f123c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:34.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.885+0000 7f1245ffb700 1 -- 172.21.15.76:0/1396099656 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f123c0039c0 con 0x7f124807de80 2024-03-18T14:43:34.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.885+0000 7f1245ffb700 1 -- 172.21.15.76:0/1396099656 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f123c011e50 con 0x7f124807de80 2024-03-18T14:43:34.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.885+0000 7f1245ffb700 1 -- 172.21.15.76:0/1396099656 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f123c01a5c0 con 0x7f124807de80 2024-03-18T14:43:34.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.885+0000 7f124eb27700 1 -- 172.21.15.76:0/1396099656 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f124819e500 con 0x7f124807de80 2024-03-18T14:43:34.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.885+0000 7f124eb27700 1 -- 172.21.15.76:0/1396099656 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12481a11f0 con 0x7f124807de80 2024-03-18T14:43:34.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.886+0000 7f124eb27700 1 -- 172.21.15.76:0/1396099656 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1248063900 con 0x7f124807de80 2024-03-18T14:43:34.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.886+0000 7f1245ffb700 1 -- 172.21.15.76:0/1396099656 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f123c024430 con 0x7f124807de80 2024-03-18T14:43:34.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.886+0000 7f1245ffb700 1 --2- 172.21.15.76:0/1396099656 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7f1238038ed0 0x7f123803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.886+0000 7f1245ffb700 1 -- 172.21.15.76:0/1396099656 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f123c04d1a0 con 0x7f124807de80 2024-03-18T14:43:34.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.886+0000 7f1247fff700 1 -- 172.21.15.76:0/1396099656 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7f1238038ed0 msgr2=0x7f123803b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/216126402 2024-03-18T14:43:34.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.886+0000 7f1247fff700 1 --2- 172.21.15.76:0/1396099656 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7f1238038ed0 0x7f123803b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T14:43:34.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:33.891+0000 7f1245ffb700 1 -- 172.21.15.76:0/1396099656 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f123c017ca0 con 0x7f124807de80 2024-03-18T14:43:34.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:34.087+0000 7f1247fff700 1 -- 172.21.15.76:0/1396099656 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7f1238038ed0 msgr2=0x7f123803b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/216126402 2024-03-18T14:43:34.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:34.087+0000 7f1247fff700 1 --2- 172.21.15.76:0/1396099656 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7f1238038ed0 0x7f123803b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T14:43:34.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:34.149+0000 7f124eb27700 1 -- 172.21.15.76:0/1396099656 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f12481a13c0 con 0x7f124807de80 2024-03-18T14:43:34.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:34.149+0000 7f1245ffb700 1 -- 172.21.15.76:0/1396099656 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f12481a13c0 con 0x7f124807de80 2024-03-18T14:43:34.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:34.151+0000 7f124eb27700 1 -- 172.21.15.76:0/1396099656 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7f1238038ed0 msgr2=0x7f123803b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T14:43:34.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:34.151+0000 7f124eb27700 1 --2- 172.21.15.76:0/1396099656 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7f1238038ed0 0x7f123803b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:34.151+0000 7f124eb27700 1 -- 172.21.15.76:0/1396099656 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f124807de80 msgr2=0x7f124819dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:34.151+0000 7f124eb27700 1 --2- 172.21.15.76:0/1396099656 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f124807de80 0x7f124819dd30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f123c011670 tx=0x7f123c011b60 comp rx=0 tx=0).stop 2024-03-18T14:43:34.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:34.151+0000 7f124eb27700 1 -- 172.21.15.76:0/1396099656 shutdown_connections 2024-03-18T14:43:34.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:34.151+0000 7f124eb27700 1 --2- 172.21.15.76:0/1396099656 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7f1238038ed0 0x7f123803b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:34.151+0000 7f124eb27700 1 --2- 172.21.15.76:0/1396099656 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f124807de80 0x7f124819dd30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:34.151+0000 7f124eb27700 1 -- 172.21.15.76:0/1396099656 >> 172.21.15.76:0/1396099656 conn(0x7f1248103100 msgr2=0x7f1248103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:34.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:34.151+0000 7f124eb27700 1 -- 172.21.15.76:0/1396099656 shutdown_connections 2024-03-18T14:43:34.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:34.151+0000 7f124eb27700 1 -- 172.21.15.76:0/1396099656 wait complete. 2024-03-18T14:43:34.758 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for the mgr to restart... 2024-03-18T14:43:34.759 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mgr epoch 5... 2024-03-18T14:43:34.997 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:34 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1396099656' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T14:43:34.997 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:34 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:34.894+0000 7f9b4765c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T14:43:35.373 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:35 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:35.007+0000 7f9b4765c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T14:43:35.374 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:35 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:35.094+0000 7f9b4765c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T14:43:35.374 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:35 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:35.194+0000 7f9b4765c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T14:43:35.662 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:35 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:35.372+0000 7f9b4765c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T14:43:36.623 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:36 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:36.332+0000 7f9b4765c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T14:43:36.624 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:36 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:36.463+0000 7f9b4765c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T14:43:36.888 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:36 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:36.796+0000 7f9b4765c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T14:43:36.888 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:36 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:36.886+0000 7f9b4765c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T14:43:37.162 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:37 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:37.082+0000 7f9b4765c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:43:37.554 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:37 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:37.267+0000 7f9b4765c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T14:43:37.808 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:37 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:37.552+0000 7f9b4765c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:43:37.808 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:37 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:37.807+0000 7f9b4765c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T14:43:38.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:38 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:38.042+0000 7f9b4765c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T14:43:39.162 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:38 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:38.837+0000 7f9b4765c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T14:43:39.662 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:39 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:39.404+0000 7f9b4765c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T14:43:39.662 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:39 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:39.501+0000 7f9b4765c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T14:43:40.162 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:39 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:39.717+0000 7f9b4765c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T14:43:40.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:39 smithi076 ceph-mon[32205]: Active manager daemon x restarted 2024-03-18T14:43:40.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:39 smithi076 ceph-mon[32205]: Activating manager daemon x 2024-03-18T14:43:40.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:39 smithi076 ceph-mon[32205]: osdmap e2: 0 total, 0 up, 0 in 2024-03-18T14:43:40.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:39 smithi076 ceph-mon[32205]: mgrmap e6: x(active, starting, since 0.00350485s) 2024-03-18T14:43:40.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:39 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:43:40.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:39 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T14:43:40.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:39 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T14:43:40.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:39 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T14:43:40.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:39 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T14:43:40.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:39 smithi076 ceph-mon[32205]: Manager daemon x is now available 2024-03-18T14:43:40.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:39 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' 2024-03-18T14:43:40.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:39 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' 2024-03-18T14:43:40.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:39 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:43:40.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:39 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:43:41.101 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:40 smithi076 ceph-mon[32205]: Found migration_current of "None". Setting to last migration. 2024-03-18T14:43:41.101 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:40 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:43:41.101 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:40 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T14:43:41.101 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:40 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T14:43:41.101 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:40 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' 2024-03-18T14:43:41.101 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:40 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' 2024-03-18T14:43:41.101 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:40 smithi076 ceph-mon[32205]: mgrmap e7: x(active, since 1.00535s) 2024-03-18T14:43:41.386 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-03-18T14:43:41.387 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-18T14:43:41.387 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T14:43:41.387 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-03-18T14:43:41.387 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.214+0000 7fa156b0b700 1 Processor -- start 2024-03-18T14:43:41.387 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.214+0000 7fa156b0b700 1 -- start start 2024-03-18T14:43:41.387 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.214+0000 7fa156b0b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa150108630 0x7fa15010aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:41.387 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.214+0000 7fa156b0b700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa15007be50 con 0x7fa150108630 2024-03-18T14:43:41.387 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.214+0000 7fa1548a7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa150108630 0x7fa15010aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:41.387 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.214+0000 7fa1548a7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa150108630 0x7fa15010aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54384/0 (socket says 172.21.15.76:54384) 2024-03-18T14:43:41.387 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.214+0000 7fa1548a7700 1 -- 172.21.15.76:0/1578668402 learned_addr learned my addr 172.21.15.76:0/1578668402 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:41.387 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.214+0000 7fa1548a7700 1 -- 172.21.15.76:0/1578668402 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa15010af60 con 0x7fa150108630 2024-03-18T14:43:41.387 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.215+0000 7fa1548a7700 1 --2- 172.21.15.76:0/1578668402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa150108630 0x7fa15010aa20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa1440098a0 tx=0x7fa144009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75aa8d9449e9294 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:41.387 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.215+0000 7fa14f7fe700 1 -- 172.21.15.76:0/1578668402 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa144004160 con 0x7fa150108630 2024-03-18T14:43:41.387 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.215+0000 7fa14f7fe700 1 -- 172.21.15.76:0/1578668402 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa144004300 con 0x7fa150108630 2024-03-18T14:43:41.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.215+0000 7fa14f7fe700 1 -- 172.21.15.76:0/1578668402 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa144011b50 con 0x7fa150108630 2024-03-18T14:43:41.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.215+0000 7fa156b0b700 1 -- 172.21.15.76:0/1578668402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa150108630 msgr2=0x7fa15010aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:41.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.216+0000 7fa156b0b700 1 --2- 172.21.15.76:0/1578668402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa150108630 0x7fa15010aa20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa1440098a0 tx=0x7fa144009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:41.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.216+0000 7fa156b0b700 1 -- 172.21.15.76:0/1578668402 shutdown_connections 2024-03-18T14:43:41.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.216+0000 7fa156b0b700 1 --2- 172.21.15.76:0/1578668402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa150108630 0x7fa15010aa20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:41.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.216+0000 7fa156b0b700 1 -- 172.21.15.76:0/1578668402 >> 172.21.15.76:0/1578668402 conn(0x7fa150102020 msgr2=0x7fa150104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:41.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.216+0000 7fa156b0b700 1 -- 172.21.15.76:0/1578668402 shutdown_connections 2024-03-18T14:43:41.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.216+0000 7fa156b0b700 1 -- 172.21.15.76:0/1578668402 wait complete. 2024-03-18T14:43:41.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.216+0000 7fa156b0b700 1 Processor -- start 2024-03-18T14:43:41.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.216+0000 7fa156b0b700 1 -- start start 2024-03-18T14:43:41.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.216+0000 7fa156b0b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa150108630 0x7fa150107eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:41.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.216+0000 7fa156b0b700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa15010bf10 con 0x7fa150108630 2024-03-18T14:43:41.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.217+0000 7fa1548a7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa150108630 0x7fa150107eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:41.388 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.217+0000 7fa1548a7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa150108630 0x7fa150107eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54400/0 (socket says 172.21.15.76:54400) 2024-03-18T14:43:41.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.217+0000 7fa1548a7700 1 -- 172.21.15.76:0/2144683991 learned_addr learned my addr 172.21.15.76:0/2144683991 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:41.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.217+0000 7fa1548a7700 1 -- 172.21.15.76:0/2144683991 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1501083f0 con 0x7fa150108630 2024-03-18T14:43:41.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.217+0000 7fa1548a7700 1 --2- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa150108630 0x7fa150107eb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa144009fb0 tx=0x7fa144011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:41.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.217+0000 7fa14dffb700 1 -- 172.21.15.76:0/2144683991 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa1440038b0 con 0x7fa150108630 2024-03-18T14:43:41.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.217+0000 7fa14dffb700 1 -- 172.21.15.76:0/2144683991 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa144011e50 con 0x7fa150108630 2024-03-18T14:43:41.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.217+0000 7fa156b0b700 1 -- 172.21.15.76:0/2144683991 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1501065e0 con 0x7fa150108630 2024-03-18T14:43:41.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.217+0000 7fa14dffb700 1 -- 172.21.15.76:0/2144683991 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa144019420 con 0x7fa150108630 2024-03-18T14:43:41.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.217+0000 7fa156b0b700 1 -- 172.21.15.76:0/2144683991 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa150106b20 con 0x7fa150108630 2024-03-18T14:43:41.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.218+0000 7fa14dffb700 1 -- 172.21.15.76:0/2144683991 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fa1440196d0 con 0x7fa150108630 2024-03-18T14:43:41.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.219+0000 7fa14dffb700 1 --2- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fa140038f20 0x7fa14003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:41.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.219+0000 7fa14dffb700 1 -- 172.21.15.76:0/2144683991 --> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa14003baf0 con 0x7fa140038f20 2024-03-18T14:43:41.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.219+0000 7fa14dffb700 1 -- 172.21.15.76:0/2144683991 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa14404cfb0 con 0x7fa150108630 2024-03-18T14:43:41.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.219+0000 7fa14ffff700 1 -- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fa140038f20 msgr2=0x7fa14003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/216126402 2024-03-18T14:43:41.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.219+0000 7fa14ffff700 1 --2- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fa140038f20 0x7fa14003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T14:43:41.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.420+0000 7fa14ffff700 1 -- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fa140038f20 msgr2=0x7fa14003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/216126402 2024-03-18T14:43:41.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.420+0000 7fa14ffff700 1 --2- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fa140038f20 0x7fa14003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T14:43:41.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.821+0000 7fa14ffff700 1 -- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fa140038f20 msgr2=0x7fa14003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/216126402 2024-03-18T14:43:41.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:35.821+0000 7fa14ffff700 1 --2- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fa140038f20 0x7fa14003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T14:43:41.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:36.622+0000 7fa14ffff700 1 -- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fa140038f20 msgr2=0x7fa14003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/216126402 2024-03-18T14:43:41.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:36.622+0000 7fa14ffff700 1 --2- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fa140038f20 0x7fa14003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T14:43:41.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:38.223+0000 7fa14ffff700 1 -- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fa140038f20 msgr2=0x7fa14003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/216126402 2024-03-18T14:43:41.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:38.223+0000 7fa14ffff700 1 --2- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fa140038f20 0x7fa14003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T14:43:41.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:39.732+0000 7fa14dffb700 1 -- 172.21.15.76:0/2144683991 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mgrmap(e 6) ==== 45873+0+0 (secure 0 0 0) 0x7fa144003a50 con 0x7fa150108630 2024-03-18T14:43:41.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:39.732+0000 7fa14dffb700 1 -- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fa140038f20 msgr2=0x7fa14003b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T14:43:41.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:39.732+0000 7fa14dffb700 1 --2- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fa140038f20 0x7fa14003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:41.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.733+0000 7fa14dffb700 1 -- 172.21.15.76:0/2144683991 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7fa14404ec30 con 0x7fa150108630 2024-03-18T14:43:41.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.733+0000 7fa14dffb700 1 --2- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fa14003c810 0x7fa14003ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:41.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.733+0000 7fa14dffb700 1 -- 172.21.15.76:0/2144683991 --> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa144012ea0 con 0x7fa14003c810 2024-03-18T14:43:41.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.733+0000 7fa14ffff700 1 --2- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fa14003c810 0x7fa14003ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:41.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.734+0000 7fa14ffff700 1 --2- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fa14003c810 0x7fa14003ec00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa13c003a10 tx=0x7fa13c0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:41.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.735+0000 7fa14dffb700 1 -- 172.21.15.76:0/2144683991 <== mgr.14120 v2:172.21.15.76:6800/3054039863 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fa144012ea0 con 0x7fa14003c810 2024-03-18T14:43:41.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.746+0000 7fa156b0b700 1 -- 172.21.15.76:0/2144683991 --> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fa150063900 con 0x7fa14003c810 2024-03-18T14:43:41.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.746+0000 7fa14dffb700 1 -- 172.21.15.76:0/2144683991 <== mgr.14120 v2:172.21.15.76:6800/3054039863 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fa150063900 con 0x7fa14003c810 2024-03-18T14:43:41.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.746+0000 7fa156b0b700 1 -- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fa14003c810 msgr2=0x7fa14003ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:41.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.746+0000 7fa156b0b700 1 --2- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fa14003c810 0x7fa14003ec00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa13c003a10 tx=0x7fa13c0092b0 comp rx=0 tx=0).stop 2024-03-18T14:43:41.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.746+0000 7fa156b0b700 1 -- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa150108630 msgr2=0x7fa150107eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:41.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.746+0000 7fa156b0b700 1 --2- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa150108630 0x7fa150107eb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa144009fb0 tx=0x7fa144011bf0 comp rx=0 tx=0).stop 2024-03-18T14:43:41.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.746+0000 7fa156b0b700 1 -- 172.21.15.76:0/2144683991 shutdown_connections 2024-03-18T14:43:41.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.746+0000 7fa156b0b700 1 --2- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fa14003c810 0x7fa14003ec00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:41.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.746+0000 7fa156b0b700 1 --2- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa150108630 0x7fa150107eb0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:41.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.746+0000 7fa156b0b700 1 --2- 172.21.15.76:0/2144683991 >> [v2:172.21.15.76:6800/216126402,v1:172.21.15.76:6801/216126402] conn(0x7fa140038f20 0x7fa14003b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:41.392 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.746+0000 7fa156b0b700 1 -- 172.21.15.76:0/2144683991 >> 172.21.15.76:0/2144683991 conn(0x7fa150102020 msgr2=0x7fa150102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:41.392 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.746+0000 7fa156b0b700 1 -- 172.21.15.76:0/2144683991 shutdown_connections 2024-03-18T14:43:41.392 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:40.746+0000 7fa156b0b700 1 -- 172.21.15.76:0/2144683991 wait complete. 2024-03-18T14:43:41.392 INFO:teuthology.orchestra.run.smithi076.stdout:mgr epoch 5 is available 2024-03-18T14:43:41.392 INFO:teuthology.orchestra.run.smithi076.stdout:Setting orchestrator backend to cephadm... 2024-03-18T14:43:42.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:41 smithi076 ceph-mon[32205]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T14:43:42.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:41 smithi076 ceph-mon[32205]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T14:43:42.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:41 smithi076 ceph-mon[32205]: [18/Mar/2024:14:43:40] ENGINE Bus STARTING 2024-03-18T14:43:42.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:41 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:43:42.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.802+0000 7f2c9af8c700 1 Processor -- start 2024-03-18T14:43:42.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.802+0000 7f2c9af8c700 1 -- start start 2024-03-18T14:43:42.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.802+0000 7f2c9af8c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c94106ab0 0x7f2c94106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.802+0000 7f2c9af8c700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2c941074a0 con 0x7f2c94106ab0 2024-03-18T14:43:42.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.802+0000 7f2c98d28700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c94106ab0 0x7f2c94106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.802+0000 7f2c98d28700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c94106ab0 0x7f2c94106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59334/0 (socket says 172.21.15.76:59334) 2024-03-18T14:43:42.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.802+0000 7f2c98d28700 1 -- 172.21.15.76:0/2058236197 learned_addr learned my addr 172.21.15.76:0/2058236197 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:42.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.802+0000 7f2c98d28700 1 -- 172.21.15.76:0/2058236197 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c94107cb0 con 0x7f2c94106ab0 2024-03-18T14:43:42.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.802+0000 7f2c98d28700 1 --2- 172.21.15.76:0/2058236197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c94106ab0 0x7f2c94106ed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009e80 tx=0x7f2c8c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff8640298fd43f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:42.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.803+0000 7f2c937fe700 1 -- 172.21.15.76:0/2058236197 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c8c017070 con 0x7f2c94106ab0 2024-03-18T14:43:42.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.803+0000 7f2c937fe700 1 -- 172.21.15.76:0/2058236197 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2c8c00b8c0 con 0x7f2c94106ab0 2024-03-18T14:43:42.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.803+0000 7f2c937fe700 1 -- 172.21.15.76:0/2058236197 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c8c0045c0 con 0x7f2c94106ab0 2024-03-18T14:43:42.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.803+0000 7f2c9af8c700 1 -- 172.21.15.76:0/2058236197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c94106ab0 msgr2=0x7f2c94106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.803+0000 7f2c9af8c700 1 --2- 172.21.15.76:0/2058236197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c94106ab0 0x7f2c94106ed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009e80 tx=0x7f2c8c00b330 comp rx=0 tx=0).stop 2024-03-18T14:43:42.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.803+0000 7f2c9af8c700 1 -- 172.21.15.76:0/2058236197 shutdown_connections 2024-03-18T14:43:42.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.803+0000 7f2c9af8c700 1 --2- 172.21.15.76:0/2058236197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c94106ab0 0x7f2c94106ed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.803+0000 7f2c9af8c700 1 -- 172.21.15.76:0/2058236197 >> 172.21.15.76:0/2058236197 conn(0x7f2c94102090 msgr2=0x7f2c941044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:42.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.803+0000 7f2c9af8c700 1 -- 172.21.15.76:0/2058236197 shutdown_connections 2024-03-18T14:43:42.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.803+0000 7f2c9af8c700 1 -- 172.21.15.76:0/2058236197 wait complete. 2024-03-18T14:43:42.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.804+0000 7f2c9af8c700 1 Processor -- start 2024-03-18T14:43:42.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.804+0000 7f2c9af8c700 1 -- start start 2024-03-18T14:43:42.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.804+0000 7f2c9af8c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c94106ab0 0x7f2c9419bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.804+0000 7f2c9af8c700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9410b060 con 0x7f2c94106ab0 2024-03-18T14:43:42.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.804+0000 7f2c98d28700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c94106ab0 0x7f2c9419bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.804+0000 7f2c98d28700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c94106ab0 0x7f2c9419bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59338/0 (socket says 172.21.15.76:59338) 2024-03-18T14:43:42.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.804+0000 7f2c98d28700 1 -- 172.21.15.76:0/2188538356 learned_addr learned my addr 172.21.15.76:0/2188538356 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:42.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.804+0000 7f2c98d28700 1 -- 172.21.15.76:0/2188538356 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c9419c090 con 0x7f2c94106ab0 2024-03-18T14:43:42.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.804+0000 7f2c98d28700 1 --2- 172.21.15.76:0/2188538356 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c94106ab0 0x7f2c9419bb50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c0072a0 tx=0x7f2c8c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:42.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.805+0000 7f2c91ffb700 1 -- 172.21.15.76:0/2188538356 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c8c020020 con 0x7f2c94106ab0 2024-03-18T14:43:42.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.805+0000 7f2c91ffb700 1 -- 172.21.15.76:0/2188538356 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2c8c003b50 con 0x7f2c94106ab0 2024-03-18T14:43:42.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.805+0000 7f2c9af8c700 1 -- 172.21.15.76:0/2188538356 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c9419c320 con 0x7f2c94106ab0 2024-03-18T14:43:42.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.805+0000 7f2c91ffb700 1 -- 172.21.15.76:0/2188538356 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c8c017040 con 0x7f2c94106ab0 2024-03-18T14:43:42.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.805+0000 7f2c9af8c700 1 -- 172.21.15.76:0/2188538356 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c9419f010 con 0x7f2c94106ab0 2024-03-18T14:43:42.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.806+0000 7f2c91ffb700 1 -- 172.21.15.76:0/2188538356 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f2c8c023020 con 0x7f2c94106ab0 2024-03-18T14:43:42.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.806+0000 7f2c91ffb700 1 --2- 172.21.15.76:0/2188538356 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f2c80038ed0 0x7f2c8003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.806+0000 7f2c9af8c700 1 -- 172.21.15.76:0/2188538356 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c94063900 con 0x7f2c94106ab0 2024-03-18T14:43:42.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.806+0000 7f2c91ffb700 1 -- 172.21.15.76:0/2188538356 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2c8c00f050 con 0x7f2c94106ab0 2024-03-18T14:43:42.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.806+0000 7f2c93fff700 1 --2- 172.21.15.76:0/2188538356 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f2c80038ed0 0x7f2c8003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.806+0000 7f2c93fff700 1 --2- 172.21.15.76:0/2188538356 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f2c80038ed0 0x7f2c8003b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2c84006fd0 tx=0x7f2c84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:42.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:41.811+0000 7f2c91ffb700 1 -- 172.21.15.76:0/2188538356 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2c8c02a020 con 0x7f2c94106ab0 2024-03-18T14:43:42.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.021+0000 7f2c9af8c700 1 -- 172.21.15.76:0/2188538356 --> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f2c9402df50 con 0x7f2c80038ed0 2024-03-18T14:43:42.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.031+0000 7f2c91ffb700 1 -- 172.21.15.76:0/2188538356 <== mgr.14120 v2:172.21.15.76:6800/3054039863 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2c9402df50 con 0x7f2c80038ed0 2024-03-18T14:43:42.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.033+0000 7f2c9af8c700 1 -- 172.21.15.76:0/2188538356 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f2c80038ed0 msgr2=0x7f2c8003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.033+0000 7f2c9af8c700 1 --2- 172.21.15.76:0/2188538356 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f2c80038ed0 0x7f2c8003b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2c84006fd0 tx=0x7f2c84006e40 comp rx=0 tx=0).stop 2024-03-18T14:43:42.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.033+0000 7f2c9af8c700 1 -- 172.21.15.76:0/2188538356 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c94106ab0 msgr2=0x7f2c9419bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.033+0000 7f2c9af8c700 1 --2- 172.21.15.76:0/2188538356 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c94106ab0 0x7f2c9419bb50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c0072a0 tx=0x7f2c8c00bd50 comp rx=0 tx=0).stop 2024-03-18T14:43:42.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.033+0000 7f2c9af8c700 1 -- 172.21.15.76:0/2188538356 shutdown_connections 2024-03-18T14:43:42.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.033+0000 7f2c9af8c700 1 --2- 172.21.15.76:0/2188538356 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f2c80038ed0 0x7f2c8003b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.033+0000 7f2c9af8c700 1 --2- 172.21.15.76:0/2188538356 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c94106ab0 0x7f2c9419bb50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.033+0000 7f2c9af8c700 1 -- 172.21.15.76:0/2188538356 >> 172.21.15.76:0/2188538356 conn(0x7f2c94102090 msgr2=0x7f2c94102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:42.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.034+0000 7f2c9af8c700 1 -- 172.21.15.76:0/2188538356 shutdown_connections 2024-03-18T14:43:42.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.034+0000 7f2c9af8c700 1 -- 172.21.15.76:0/2188538356 wait complete. 2024-03-18T14:43:42.876 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:42 smithi076 ceph-mon[32205]: [18/Mar/2024:14:43:40] ENGINE Serving on http://172.21.15.76:8765 2024-03-18T14:43:42.876 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:42 smithi076 ceph-mon[32205]: [18/Mar/2024:14:43:41] ENGINE Serving on https://172.21.15.76:7150 2024-03-18T14:43:42.876 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:42 smithi076 ceph-mon[32205]: [18/Mar/2024:14:43:41] ENGINE Bus STARTED 2024-03-18T14:43:42.876 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:42 smithi076 ceph-mon[32205]: mgrmap e8: x(active, since 2s) 2024-03-18T14:43:42.876 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:42 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' 2024-03-18T14:43:42.877 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:42 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:43:43.712 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-18T14:43:43.712 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.980+0000 7fe1411f5700 1 Processor -- start 2024-03-18T14:43:43.712 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.980+0000 7fe1411f5700 1 -- start start 2024-03-18T14:43:43.712 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.980+0000 7fe1411f5700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe13c07de40 0x7fe13c07c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:43.712 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.980+0000 7fe1411f5700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c07c870 con 0x7fe13c07de40 2024-03-18T14:43:43.712 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.981+0000 7fe13ad9d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe13c07de40 0x7fe13c07c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:43.712 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.981+0000 7fe13ad9d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe13c07de40 0x7fe13c07c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59344/0 (socket says 172.21.15.76:59344) 2024-03-18T14:43:43.712 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.981+0000 7fe13ad9d700 1 -- 172.21.15.76:0/3763862890 learned_addr learned my addr 172.21.15.76:0/3763862890 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:43.713 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.981+0000 7fe13ad9d700 1 -- 172.21.15.76:0/3763862890 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe13c07c9f0 con 0x7fe13c07de40 2024-03-18T14:43:43.713 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.981+0000 7fe13ad9d700 1 --2- 172.21.15.76:0/3763862890 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe13c07de40 0x7fe13c07c2a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe1300098a0 tx=0x7fe130009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f3955371d6423a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:43.713 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.981+0000 7fe139d9b700 1 -- 172.21.15.76:0/3763862890 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe130004160 con 0x7fe13c07de40 2024-03-18T14:43:43.713 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.981+0000 7fe139d9b700 1 -- 172.21.15.76:0/3763862890 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe130004300 con 0x7fe13c07de40 2024-03-18T14:43:43.713 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.981+0000 7fe139d9b700 1 -- 172.21.15.76:0/3763862890 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe130011b50 con 0x7fe13c07de40 2024-03-18T14:43:43.713 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.982+0000 7fe1411f5700 1 -- 172.21.15.76:0/3763862890 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe13c07de40 msgr2=0x7fe13c07c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:43.713 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.982+0000 7fe1411f5700 1 --2- 172.21.15.76:0/3763862890 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe13c07de40 0x7fe13c07c2a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe1300098a0 tx=0x7fe130009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:43.713 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.982+0000 7fe1411f5700 1 -- 172.21.15.76:0/3763862890 shutdown_connections 2024-03-18T14:43:43.713 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.982+0000 7fe1411f5700 1 --2- 172.21.15.76:0/3763862890 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe13c07de40 0x7fe13c07c2a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:43.713 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.982+0000 7fe1411f5700 1 -- 172.21.15.76:0/3763862890 >> 172.21.15.76:0/3763862890 conn(0x7fe13c1030e0 msgr2=0x7fe13c105540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:43.713 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.982+0000 7fe1411f5700 1 -- 172.21.15.76:0/3763862890 shutdown_connections 2024-03-18T14:43:43.713 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.982+0000 7fe1411f5700 1 -- 172.21.15.76:0/3763862890 wait complete. 2024-03-18T14:43:43.713 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.982+0000 7fe1411f5700 1 Processor -- start 2024-03-18T14:43:43.714 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.983+0000 7fe1411f5700 1 -- start start 2024-03-18T14:43:43.714 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.983+0000 7fe1411f5700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe13c07de40 0x7fe13c19dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:43.714 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.983+0000 7fe1411f5700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c10e190 con 0x7fe13c07de40 2024-03-18T14:43:43.714 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.983+0000 7fe13ad9d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe13c07de40 0x7fe13c19dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:43.714 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.983+0000 7fe13ad9d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe13c07de40 0x7fe13c19dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59360/0 (socket says 172.21.15.76:59360) 2024-03-18T14:43:43.714 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.983+0000 7fe13ad9d700 1 -- 172.21.15.76:0/2973664336 learned_addr learned my addr 172.21.15.76:0/2973664336 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:43.714 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.983+0000 7fe13ad9d700 1 -- 172.21.15.76:0/2973664336 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe13c19e2d0 con 0x7fe13c07de40 2024-03-18T14:43:43.714 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.983+0000 7fe13ad9d700 1 --2- 172.21.15.76:0/2973664336 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe13c07de40 0x7fe13c19dd90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe130011b60 tx=0x7fe130011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:43.714 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.984+0000 7fe123fff700 1 -- 172.21.15.76:0/2973664336 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe130003970 con 0x7fe13c07de40 2024-03-18T14:43:43.714 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.984+0000 7fe1411f5700 1 -- 172.21.15.76:0/2973664336 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe13c19e560 con 0x7fe13c07de40 2024-03-18T14:43:43.714 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.984+0000 7fe123fff700 1 -- 172.21.15.76:0/2973664336 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe130003b10 con 0x7fe13c07de40 2024-03-18T14:43:43.714 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.984+0000 7fe123fff700 1 -- 172.21.15.76:0/2973664336 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe130024610 con 0x7fe13c07de40 2024-03-18T14:43:43.714 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.984+0000 7fe1411f5700 1 -- 172.21.15.76:0/2973664336 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe13c1a1210 con 0x7fe13c07de40 2024-03-18T14:43:43.714 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.985+0000 7fe123fff700 1 -- 172.21.15.76:0/2973664336 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fe130024910 con 0x7fe13c07de40 2024-03-18T14:43:43.714 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.985+0000 7fe1411f5700 1 -- 172.21.15.76:0/2973664336 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe13c063900 con 0x7fe13c07de40 2024-03-18T14:43:43.715 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.985+0000 7fe123fff700 1 --2- 172.21.15.76:0/2973664336 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fe124038f20 0x7fe12403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:43.715 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.985+0000 7fe123fff700 1 -- 172.21.15.76:0/2973664336 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe130011e00 con 0x7fe13c07de40 2024-03-18T14:43:43.715 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.985+0000 7fe13a59c700 1 --2- 172.21.15.76:0/2973664336 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fe124038f20 0x7fe12403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:43.715 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.985+0000 7fe13a59c700 1 --2- 172.21.15.76:0/2973664336 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fe124038f20 0x7fe12403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe12c006fd0 tx=0x7fe12c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:43.715 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:42.990+0000 7fe123fff700 1 -- 172.21.15.76:0/2973664336 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe13000ba00 con 0x7fe13c07de40 2024-03-18T14:43:43.715 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:43.191+0000 7fe1411f5700 1 -- 172.21.15.76:0/2973664336 --> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fe13c1043f0 con 0x7fe124038f20 2024-03-18T14:43:43.715 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:43.192+0000 7fe123fff700 1 -- 172.21.15.76:0/2973664336 <== mgr.14120 v2:172.21.15.76:6800/3054039863 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fe13c1043f0 con 0x7fe124038f20 2024-03-18T14:43:43.715 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:43.194+0000 7fe1411f5700 1 -- 172.21.15.76:0/2973664336 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fe124038f20 msgr2=0x7fe12403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:43.715 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:43.194+0000 7fe1411f5700 1 --2- 172.21.15.76:0/2973664336 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fe124038f20 0x7fe12403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe12c006fd0 tx=0x7fe12c006e40 comp rx=0 tx=0).stop 2024-03-18T14:43:43.715 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:43.194+0000 7fe1411f5700 1 -- 172.21.15.76:0/2973664336 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe13c07de40 msgr2=0x7fe13c19dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:43.715 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:43.194+0000 7fe1411f5700 1 --2- 172.21.15.76:0/2973664336 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe13c07de40 0x7fe13c19dd90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe130011b60 tx=0x7fe130011c40 comp rx=0 tx=0).stop 2024-03-18T14:43:43.715 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:43.194+0000 7fe1411f5700 1 -- 172.21.15.76:0/2973664336 shutdown_connections 2024-03-18T14:43:43.715 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:43.194+0000 7fe1411f5700 1 --2- 172.21.15.76:0/2973664336 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fe124038f20 0x7fe12403b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:43.715 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:43.194+0000 7fe1411f5700 1 --2- 172.21.15.76:0/2973664336 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe13c07de40 0x7fe13c19dd90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:43.715 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:43.194+0000 7fe1411f5700 1 -- 172.21.15.76:0/2973664336 >> 172.21.15.76:0/2973664336 conn(0x7fe13c1030e0 msgr2=0x7fe13c103ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:43.716 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:43.194+0000 7fe1411f5700 1 -- 172.21.15.76:0/2973664336 shutdown_connections 2024-03-18T14:43:43.716 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:43.194+0000 7fe1411f5700 1 -- 172.21.15.76:0/2973664336 wait complete. 2024-03-18T14:43:43.716 INFO:teuthology.orchestra.run.smithi076.stdout:Generating ssh key... 2024-03-18T14:43:44.116 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:43 smithi076 ceph-mon[32205]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:43:44.796 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: Generating public/private rsa key pair. 2024-03-18T14:43:44.796 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: Your identification has been saved in /tmp/tmp05cqudcj/key. 2024-03-18T14:43:44.797 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: Your public key has been saved in /tmp/tmp05cqudcj/key.pub. 2024-03-18T14:43:44.797 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: The key fingerprint is: 2024-03-18T14:43:44.797 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: SHA256:Qs70vMiWYN1DQJfwyNvCjKuQK38b+oYNOa8oltgPbME ceph-9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:43:44.797 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: The key's randomart image is: 2024-03-18T14:43:44.797 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: +---[RSA 3072]----+ 2024-03-18T14:43:44.797 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: | .+... | 2024-03-18T14:43:44.797 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: | . =. | 2024-03-18T14:43:44.797 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: | = o | 2024-03-18T14:43:44.797 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: | . X B | 2024-03-18T14:43:44.797 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: | E.+ X S | 2024-03-18T14:43:44.797 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: | o+o + = o | 2024-03-18T14:43:44.797 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: |+.=*o = . | 2024-03-18T14:43:44.797 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: |+Bo+=o | 2024-03-18T14:43:44.797 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: |*.=*+. | 2024-03-18T14:43:44.797 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: +----[SHA256]-----+ 2024-03-18T14:43:45.086 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-mon[32205]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:43:45.086 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' 2024-03-18T14:43:45.086 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:44 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' 2024-03-18T14:43:45.094 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.137+0000 7f86ea700700 1 Processor -- start 2024-03-18T14:43:45.094 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.137+0000 7f86ea700700 1 -- start start 2024-03-18T14:43:45.094 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.137+0000 7f86ea700700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86e4107ba0 0x7f86e4107fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.094 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.137+0000 7f86ea700700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f86e4108590 con 0x7f86e4107ba0 2024-03-18T14:43:45.094 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.137+0000 7f86e3fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86e4107ba0 0x7f86e4107fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.094 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.137+0000 7f86e3fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86e4107ba0 0x7f86e4107fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59374/0 (socket says 172.21.15.76:59374) 2024-03-18T14:43:45.094 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.137+0000 7f86e3fff700 1 -- 172.21.15.76:0/4219183232 learned_addr learned my addr 172.21.15.76:0/4219183232 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:45.094 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.137+0000 7f86e3fff700 1 -- 172.21.15.76:0/4219183232 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e4108da0 con 0x7f86e4107ba0 2024-03-18T14:43:45.094 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.138+0000 7f86e3fff700 1 --2- 172.21.15.76:0/4219183232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86e4107ba0 0x7f86e4107fc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f86d40098a0 tx=0x7f86d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39f58e7568e20d37 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:45.094 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.138+0000 7f86e2ffd700 1 -- 172.21.15.76:0/4219183232 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86d4004160 con 0x7f86e4107ba0 2024-03-18T14:43:45.094 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.138+0000 7f86e2ffd700 1 -- 172.21.15.76:0/4219183232 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f86d4004300 con 0x7f86e4107ba0 2024-03-18T14:43:45.095 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.138+0000 7f86e2ffd700 1 -- 172.21.15.76:0/4219183232 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86d4011b50 con 0x7f86e4107ba0 2024-03-18T14:43:45.095 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.138+0000 7f86ea700700 1 -- 172.21.15.76:0/4219183232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86e4107ba0 msgr2=0x7f86e4107fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.095 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.138+0000 7f86ea700700 1 --2- 172.21.15.76:0/4219183232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86e4107ba0 0x7f86e4107fc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f86d40098a0 tx=0x7f86d4009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.095 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.138+0000 7f86ea700700 1 -- 172.21.15.76:0/4219183232 shutdown_connections 2024-03-18T14:43:45.095 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.138+0000 7f86ea700700 1 --2- 172.21.15.76:0/4219183232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86e4107ba0 0x7f86e4107fc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.095 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.138+0000 7f86ea700700 1 -- 172.21.15.76:0/4219183232 >> 172.21.15.76:0/4219183232 conn(0x7f86e4103180 msgr2=0x7f86e41055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:45.095 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.138+0000 7f86ea700700 1 -- 172.21.15.76:0/4219183232 shutdown_connections 2024-03-18T14:43:45.095 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.138+0000 7f86ea700700 1 -- 172.21.15.76:0/4219183232 wait complete. 2024-03-18T14:43:45.095 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.139+0000 7f86ea700700 1 Processor -- start 2024-03-18T14:43:45.095 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.139+0000 7f86ea700700 1 -- start start 2024-03-18T14:43:45.095 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.139+0000 7f86ea700700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86e4107ba0 0x7f86e407db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.095 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.139+0000 7f86ea700700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f86e410c150 con 0x7f86e4107ba0 2024-03-18T14:43:45.095 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.139+0000 7f86e3fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86e4107ba0 0x7f86e407db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.095 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.139+0000 7f86e3fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86e4107ba0 0x7f86e407db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59386/0 (socket says 172.21.15.76:59386) 2024-03-18T14:43:45.095 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.139+0000 7f86e3fff700 1 -- 172.21.15.76:0/747276550 learned_addr learned my addr 172.21.15.76:0/747276550 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:45.096 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.140+0000 7f86e3fff700 1 -- 172.21.15.76:0/747276550 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e407e070 con 0x7f86e4107ba0 2024-03-18T14:43:45.096 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.140+0000 7f86e3fff700 1 --2- 172.21.15.76:0/747276550 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86e4107ba0 0x7f86e407db30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f86d4000c00 tx=0x7f86d4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:45.096 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.140+0000 7f86e17fa700 1 -- 172.21.15.76:0/747276550 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86d4003b20 con 0x7f86e4107ba0 2024-03-18T14:43:45.096 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.140+0000 7f86ea700700 1 -- 172.21.15.76:0/747276550 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86e407c260 con 0x7f86e4107ba0 2024-03-18T14:43:45.096 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.140+0000 7f86e17fa700 1 -- 172.21.15.76:0/747276550 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f86d4003cc0 con 0x7f86e4107ba0 2024-03-18T14:43:45.096 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.140+0000 7f86e17fa700 1 -- 172.21.15.76:0/747276550 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86d4024910 con 0x7f86e4107ba0 2024-03-18T14:43:45.096 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.140+0000 7f86ea700700 1 -- 172.21.15.76:0/747276550 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86e407c7a0 con 0x7f86e4107ba0 2024-03-18T14:43:45.096 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.141+0000 7f86e17fa700 1 -- 172.21.15.76:0/747276550 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f86d4022020 con 0x7f86e4107ba0 2024-03-18T14:43:45.096 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.141+0000 7f86ea700700 1 -- 172.21.15.76:0/747276550 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86e4063900 con 0x7f86e4107ba0 2024-03-18T14:43:45.096 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.141+0000 7f86e17fa700 1 --2- 172.21.15.76:0/747276550 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f86cc038f20 0x7f86cc03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.096 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.141+0000 7f86e17fa700 1 -- 172.21.15.76:0/747276550 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f86d4011e00 con 0x7f86e4107ba0 2024-03-18T14:43:45.096 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.141+0000 7f86e37fe700 1 --2- 172.21.15.76:0/747276550 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f86cc038f20 0x7f86cc03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.096 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.142+0000 7f86e37fe700 1 --2- 172.21.15.76:0/747276550 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f86cc038f20 0x7f86cc03b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f86d8006fd0 tx=0x7f86d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:45.096 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.146+0000 7f86e17fa700 1 -- 172.21.15.76:0/747276550 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f86d400b5a0 con 0x7f86e4107ba0 2024-03-18T14:43:45.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.343+0000 7f86ea700700 1 -- 172.21.15.76:0/747276550 --> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f86e41043f0 con 0x7f86cc038f20 2024-03-18T14:43:45.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.507+0000 7f86e17fa700 1 -- 172.21.15.76:0/747276550 <== mgr.14120 v2:172.21.15.76:6800/3054039863 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f86e41043f0 con 0x7f86cc038f20 2024-03-18T14:43:45.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.509+0000 7f86ea700700 1 -- 172.21.15.76:0/747276550 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f86cc038f20 msgr2=0x7f86cc03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.509+0000 7f86ea700700 1 --2- 172.21.15.76:0/747276550 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f86cc038f20 0x7f86cc03b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f86d8006fd0 tx=0x7f86d8006e40 comp rx=0 tx=0).stop 2024-03-18T14:43:45.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.509+0000 7f86ea700700 1 -- 172.21.15.76:0/747276550 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86e4107ba0 msgr2=0x7f86e407db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.509+0000 7f86ea700700 1 --2- 172.21.15.76:0/747276550 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86e4107ba0 0x7f86e407db30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f86d4000c00 tx=0x7f86d4011c10 comp rx=0 tx=0).stop 2024-03-18T14:43:45.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.509+0000 7f86ea700700 1 -- 172.21.15.76:0/747276550 shutdown_connections 2024-03-18T14:43:45.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.509+0000 7f86ea700700 1 --2- 172.21.15.76:0/747276550 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f86cc038f20 0x7f86cc03b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.509+0000 7f86ea700700 1 --2- 172.21.15.76:0/747276550 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86e4107ba0 0x7f86e407db30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.509+0000 7f86ea700700 1 -- 172.21.15.76:0/747276550 >> 172.21.15.76:0/747276550 conn(0x7f86e4103180 msgr2=0x7f86e4103ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:45.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.509+0000 7f86ea700700 1 -- 172.21.15.76:0/747276550 shutdown_connections 2024-03-18T14:43:45.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:44.509+0000 7f86ea700700 1 -- 172.21.15.76:0/747276550 wait complete. 2024-03-18T14:43:46.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:45 smithi076 ceph-mon[32205]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:43:46.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:45 smithi076 ceph-mon[32205]: Generating ssh key... 2024-03-18T14:43:46.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRnyV/3Ny3N7Baa21i7Z5gwNkbknoGqURhlqBmX1Q61L/ivn6U9pSk8IDaaqbdLbCw0FQzJAGhWAJC+COQVzWX4BzJbhtDBwHZjcHa06pfXtzBfWinXCXYy3iF6hQ9FXWIWtvraBUBb0akuPiyk2k6sSHTnxwPGhw5661FFbllcefPimf16C02MYx3EYQe6zP2vXJ2h7x1zHe8JT/kmokWBEHBIR8Jmhyytqp/kMAAVE5kg9rIbNSwR7kP30TyZR7TVDxK7o6SnBIwpArHx5K3QcSfetQHBbzlEEaqEjxZI6vZCqzmvzJ+/hnjZGeuPDptQiTZIYC8JubkTNpYE9jt5razrffXsp3EV0lW8kAbLLUcRSyTIKj8Gw7zjPGbpGVfhWtkylcIKYIhiaLBgr4FfCySd05Yux/pHx7Zhbss//m1lPtiRHtauXU1yyrV7wzmGdl7IX42EL0aoLQEaTQSZcCdg1g1IvkHvu1J3gHZ+2Q/fvKf2c2LZpzAZ62kdRU= ceph-9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:43:46.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.528+0000 7f5fb5e35700 1 Processor -- start 2024-03-18T14:43:46.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.529+0000 7f5fb5e35700 1 -- start start 2024-03-18T14:43:46.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.529+0000 7f5fb5e35700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fb007de80 0x7f5fb007c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.529+0000 7f5fb5e35700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb007c8b0 con 0x7f5fb007de80 2024-03-18T14:43:46.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.529+0000 7f5faf7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fb007de80 0x7f5fb007c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.529+0000 7f5faf7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fb007de80 0x7f5fb007c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59400/0 (socket says 172.21.15.76:59400) 2024-03-18T14:43:46.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.529+0000 7f5faf7fe700 1 -- 172.21.15.76:0/4106140238 learned_addr learned my addr 172.21.15.76:0/4106140238 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:46.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.529+0000 7f5faf7fe700 1 -- 172.21.15.76:0/4106140238 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb007ca30 con 0x7f5fb007de80 2024-03-18T14:43:46.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.529+0000 7f5faf7fe700 1 --2- 172.21.15.76:0/4106140238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fb007de80 0x7f5fb007c2e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5fa00098a0 tx=0x7f5fa0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c12317deca2760 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:46.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.530+0000 7f5fae7fc700 1 -- 172.21.15.76:0/4106140238 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fa0004160 con 0x7f5fb007de80 2024-03-18T14:43:46.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.530+0000 7f5fae7fc700 1 -- 172.21.15.76:0/4106140238 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5fa0004300 con 0x7f5fb007de80 2024-03-18T14:43:46.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.530+0000 7f5fae7fc700 1 -- 172.21.15.76:0/4106140238 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fa0011b50 con 0x7f5fb007de80 2024-03-18T14:43:46.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.530+0000 7f5fb5e35700 1 -- 172.21.15.76:0/4106140238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fb007de80 msgr2=0x7f5fb007c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.530+0000 7f5fb5e35700 1 --2- 172.21.15.76:0/4106140238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fb007de80 0x7f5fb007c2e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5fa00098a0 tx=0x7f5fa0009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.531+0000 7f5fb5e35700 1 -- 172.21.15.76:0/4106140238 shutdown_connections 2024-03-18T14:43:46.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.531+0000 7f5fb5e35700 1 --2- 172.21.15.76:0/4106140238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fb007de80 0x7f5fb007c2e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.531+0000 7f5fb5e35700 1 -- 172.21.15.76:0/4106140238 >> 172.21.15.76:0/4106140238 conn(0x7f5fb0103180 msgr2=0x7f5fb01055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:46.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.531+0000 7f5fb5e35700 1 -- 172.21.15.76:0/4106140238 shutdown_connections 2024-03-18T14:43:46.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.531+0000 7f5fb5e35700 1 -- 172.21.15.76:0/4106140238 wait complete. 2024-03-18T14:43:46.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.531+0000 7f5fb5e35700 1 Processor -- start 2024-03-18T14:43:46.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.531+0000 7f5fb5e35700 1 -- start start 2024-03-18T14:43:46.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.532+0000 7f5fb5e35700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fb007de80 0x7f5fb019dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.532+0000 7f5fb5e35700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb010e170 con 0x7f5fb007de80 2024-03-18T14:43:46.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.532+0000 7f5faf7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fb007de80 0x7f5fb019dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.532+0000 7f5faf7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fb007de80 0x7f5fb019dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59402/0 (socket says 172.21.15.76:59402) 2024-03-18T14:43:46.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.532+0000 7f5faf7fe700 1 -- 172.21.15.76:0/2577671610 learned_addr learned my addr 172.21.15.76:0/2577671610 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:46.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.532+0000 7f5faf7fe700 1 -- 172.21.15.76:0/2577671610 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb019e270 con 0x7f5fb007de80 2024-03-18T14:43:46.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.532+0000 7f5faf7fe700 1 --2- 172.21.15.76:0/2577671610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fb007de80 0x7f5fb019dd30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0000c00 tx=0x7f5fa0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:46.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.532+0000 7f5facff9700 1 -- 172.21.15.76:0/2577671610 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fa0003b20 con 0x7f5fb007de80 2024-03-18T14:43:46.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.532+0000 7f5facff9700 1 -- 172.21.15.76:0/2577671610 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5fa0003cc0 con 0x7f5fb007de80 2024-03-18T14:43:46.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.532+0000 7f5fb5e35700 1 -- 172.21.15.76:0/2577671610 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb019e500 con 0x7f5fb007de80 2024-03-18T14:43:46.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.532+0000 7f5facff9700 1 -- 172.21.15.76:0/2577671610 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fa0024910 con 0x7f5fb007de80 2024-03-18T14:43:46.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.532+0000 7f5fb5e35700 1 -- 172.21.15.76:0/2577671610 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb01a11f0 con 0x7f5fb007de80 2024-03-18T14:43:46.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.533+0000 7f5facff9700 1 -- 172.21.15.76:0/2577671610 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f5fa0022020 con 0x7f5fb007de80 2024-03-18T14:43:46.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.533+0000 7f5fb5e35700 1 -- 172.21.15.76:0/2577671610 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fb0063900 con 0x7f5fb007de80 2024-03-18T14:43:46.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.533+0000 7f5facff9700 1 --2- 172.21.15.76:0/2577671610 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f5f98038f20 0x7f5f9803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.303 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.533+0000 7f5facff9700 1 -- 172.21.15.76:0/2577671610 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5fa0011e00 con 0x7f5fb007de80 2024-03-18T14:43:46.303 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.534+0000 7f5faeffd700 1 --2- 172.21.15.76:0/2577671610 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f5f98038f20 0x7f5f9803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.303 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.534+0000 7f5faeffd700 1 --2- 172.21.15.76:0/2577671610 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f5f98038f20 0x7f5f9803b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4006fd0 tx=0x7f5fa4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:46.303 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.538+0000 7f5facff9700 1 -- 172.21.15.76:0/2577671610 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5fa00172b0 con 0x7f5fb007de80 2024-03-18T14:43:46.303 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.744+0000 7f5fb5e35700 1 -- 172.21.15.76:0/2577671610 --> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f5fb01043a0 con 0x7f5f98038f20 2024-03-18T14:43:46.303 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.745+0000 7f5facff9700 1 -- 172.21.15.76:0/2577671610 <== mgr.14120 v2:172.21.15.76:6800/3054039863 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f5fb01043a0 con 0x7f5f98038f20 2024-03-18T14:43:46.303 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.748+0000 7f5fb5e35700 1 -- 172.21.15.76:0/2577671610 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f5f98038f20 msgr2=0x7f5f9803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.303 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.748+0000 7f5fb5e35700 1 --2- 172.21.15.76:0/2577671610 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f5f98038f20 0x7f5f9803b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4006fd0 tx=0x7f5fa4006e40 comp rx=0 tx=0).stop 2024-03-18T14:43:46.303 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.748+0000 7f5fb5e35700 1 -- 172.21.15.76:0/2577671610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fb007de80 msgr2=0x7f5fb019dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.303 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.748+0000 7f5fb5e35700 1 --2- 172.21.15.76:0/2577671610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fb007de80 0x7f5fb019dd30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0000c00 tx=0x7f5fa0011c10 comp rx=0 tx=0).stop 2024-03-18T14:43:46.303 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.748+0000 7f5fb5e35700 1 -- 172.21.15.76:0/2577671610 shutdown_connections 2024-03-18T14:43:46.303 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.748+0000 7f5fb5e35700 1 --2- 172.21.15.76:0/2577671610 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f5f98038f20 0x7f5f9803b3e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.303 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.748+0000 7f5fb5e35700 1 --2- 172.21.15.76:0/2577671610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fb007de80 0x7f5fb019dd30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.303 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.748+0000 7f5fb5e35700 1 -- 172.21.15.76:0/2577671610 >> 172.21.15.76:0/2577671610 conn(0x7f5fb0103180 msgr2=0x7f5fb0103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:46.303 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.748+0000 7f5fb5e35700 1 -- 172.21.15.76:0/2577671610 shutdown_connections 2024-03-18T14:43:46.304 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:45.748+0000 7f5fb5e35700 1 -- 172.21.15.76:0/2577671610 wait complete. 2024-03-18T14:43:46.304 INFO:teuthology.orchestra.run.smithi076.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-18T14:43:46.304 INFO:teuthology.orchestra.run.smithi076.stdout:Adding key to root@localhost authorized_keys... 2024-03-18T14:43:46.304 INFO:teuthology.orchestra.run.smithi076.stdout:Adding host smithi076... 2024-03-18T14:43:47.053 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:46 smithi076 ceph-mon[32205]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:43:48.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:47 smithi076 ceph-mon[32205]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi076", "addr": "172.21.15.76", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:43:48.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:47 smithi076 ceph-mon[32205]: Deploying cephadm binary to smithi076 2024-03-18T14:43:51.607 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout Added host 'smithi076' with addr '172.21.15.76' 2024-03-18T14:43:51.607 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.712+0000 7f0e0193b700 1 Processor -- start 2024-03-18T14:43:51.607 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.712+0000 7f0e0193b700 1 -- start start 2024-03-18T14:43:51.607 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.712+0000 7f0e0193b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0dfc1086a0 0x7f0dfc10aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:51.607 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.712+0000 7f0e0193b700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc07bdf0 con 0x7f0dfc1086a0 2024-03-18T14:43:51.607 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.712+0000 7f0dfaffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0dfc1086a0 0x7f0dfc10aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:51.607 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.712+0000 7f0dfaffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0dfc1086a0 0x7f0dfc10aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59414/0 (socket says 172.21.15.76:59414) 2024-03-18T14:43:51.607 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.712+0000 7f0dfaffd700 1 -- 172.21.15.76:0/190279041 learned_addr learned my addr 172.21.15.76:0/190279041 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:51.607 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.712+0000 7f0dfaffd700 1 -- 172.21.15.76:0/190279041 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc10afd0 con 0x7f0dfc1086a0 2024-03-18T14:43:51.607 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.712+0000 7f0dfaffd700 1 --2- 172.21.15.76:0/190279041 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0dfc1086a0 0x7f0dfc10aa90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0dec0098a0 tx=0x7f0dec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68d43abfe76aef46 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:51.608 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.713+0000 7f0df9ffb700 1 -- 172.21.15.76:0/190279041 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dec004160 con 0x7f0dfc1086a0 2024-03-18T14:43:51.608 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.713+0000 7f0df9ffb700 1 -- 172.21.15.76:0/190279041 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0dec004300 con 0x7f0dfc1086a0 2024-03-18T14:43:51.608 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.713+0000 7f0df9ffb700 1 -- 172.21.15.76:0/190279041 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dec011b50 con 0x7f0dfc1086a0 2024-03-18T14:43:51.608 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.713+0000 7f0e0193b700 1 -- 172.21.15.76:0/190279041 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0dfc1086a0 msgr2=0x7f0dfc10aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:51.608 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.713+0000 7f0e0193b700 1 --2- 172.21.15.76:0/190279041 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0dfc1086a0 0x7f0dfc10aa90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0dec0098a0 tx=0x7f0dec009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.608 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.713+0000 7f0e0193b700 1 -- 172.21.15.76:0/190279041 shutdown_connections 2024-03-18T14:43:51.608 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.713+0000 7f0e0193b700 1 --2- 172.21.15.76:0/190279041 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0dfc1086a0 0x7f0dfc10aa90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.608 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.713+0000 7f0e0193b700 1 -- 172.21.15.76:0/190279041 >> 172.21.15.76:0/190279041 conn(0x7f0dfc102090 msgr2=0x7f0dfc1044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:51.608 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.713+0000 7f0e0193b700 1 -- 172.21.15.76:0/190279041 shutdown_connections 2024-03-18T14:43:51.608 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.713+0000 7f0e0193b700 1 -- 172.21.15.76:0/190279041 wait complete. 2024-03-18T14:43:51.608 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.714+0000 7f0e0193b700 1 Processor -- start 2024-03-18T14:43:51.608 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.714+0000 7f0e0193b700 1 -- start start 2024-03-18T14:43:51.608 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.714+0000 7f0e0193b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0dfc1086a0 0x7f0dfc107f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:51.608 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.714+0000 7f0e0193b700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc10bf90 con 0x7f0dfc1086a0 2024-03-18T14:43:51.609 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.714+0000 7f0dfaffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0dfc1086a0 0x7f0dfc107f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:51.609 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.714+0000 7f0dfaffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0dfc1086a0 0x7f0dfc107f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59428/0 (socket says 172.21.15.76:59428) 2024-03-18T14:43:51.609 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.714+0000 7f0dfaffd700 1 -- 172.21.15.76:0/1225260323 learned_addr learned my addr 172.21.15.76:0/1225260323 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:51.609 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.715+0000 7f0dfaffd700 1 -- 172.21.15.76:0/1225260323 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc108460 con 0x7f0dfc1086a0 2024-03-18T14:43:51.609 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.715+0000 7f0dfaffd700 1 --2- 172.21.15.76:0/1225260323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0dfc1086a0 0x7f0dfc107f20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0dec011670 tx=0x7f0dec011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:51.609 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.715+0000 7f0e00939700 1 -- 172.21.15.76:0/1225260323 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dec0039c0 con 0x7f0dfc1086a0 2024-03-18T14:43:51.609 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.715+0000 7f0e0193b700 1 -- 172.21.15.76:0/1225260323 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dfc106650 con 0x7f0dfc1086a0 2024-03-18T14:43:51.609 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.715+0000 7f0e00939700 1 -- 172.21.15.76:0/1225260323 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0dec011e50 con 0x7f0dfc1086a0 2024-03-18T14:43:51.609 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.715+0000 7f0e00939700 1 -- 172.21.15.76:0/1225260323 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dec01a5c0 con 0x7f0dfc1086a0 2024-03-18T14:43:51.609 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.715+0000 7f0e0193b700 1 -- 172.21.15.76:0/1225260323 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dfc106b90 con 0x7f0dfc1086a0 2024-03-18T14:43:51.609 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.716+0000 7f0e00939700 1 -- 172.21.15.76:0/1225260323 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f0dec01a8c0 con 0x7f0dfc1086a0 2024-03-18T14:43:51.609 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.716+0000 7f0e00939700 1 --2- 172.21.15.76:0/1225260323 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f0de4038ed0 0x7f0de403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:51.609 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.716+0000 7f0e00939700 1 -- 172.21.15.76:0/1225260323 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0dec04d430 con 0x7f0dfc1086a0 2024-03-18T14:43:51.609 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.716+0000 7f0e0193b700 1 -- 172.21.15.76:0/1225260323 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dfc063900 con 0x7f0dfc1086a0 2024-03-18T14:43:51.610 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.716+0000 7f0dfa7fc700 1 --2- 172.21.15.76:0/1225260323 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f0de4038ed0 0x7f0de403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:51.610 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.716+0000 7f0dfa7fc700 1 --2- 172.21.15.76:0/1225260323 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f0de4038ed0 0x7f0de403b390 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0df0006fd0 tx=0x7f0df0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:51.610 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.721+0000 7f0e00939700 1 -- 172.21.15.76:0/1225260323 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0dec019c70 con 0x7f0dfc1086a0 2024-03-18T14:43:51.610 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:46.920+0000 7f0e0193b700 1 -- 172.21.15.76:0/1225260323 --> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi076", "addr": "172.21.15.76", "target": ["mon-mgr", ""]}) -- 0x7f0dfc1074c0 con 0x7f0de4038ed0 2024-03-18T14:43:51.610 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:51.125+0000 7f0e00939700 1 -- 172.21.15.76:0/1225260323 <== mgr.14120 v2:172.21.15.76:6800/3054039863 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f0dfc1074c0 con 0x7f0de4038ed0 2024-03-18T14:43:51.610 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:51.128+0000 7f0e0193b700 1 -- 172.21.15.76:0/1225260323 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f0de4038ed0 msgr2=0x7f0de403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:51.610 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:51.128+0000 7f0e0193b700 1 --2- 172.21.15.76:0/1225260323 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f0de4038ed0 0x7f0de403b390 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0df0006fd0 tx=0x7f0df0006e40 comp rx=0 tx=0).stop 2024-03-18T14:43:51.610 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:51.128+0000 7f0e0193b700 1 -- 172.21.15.76:0/1225260323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0dfc1086a0 msgr2=0x7f0dfc107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:51.610 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:51.128+0000 7f0e0193b700 1 --2- 172.21.15.76:0/1225260323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0dfc1086a0 0x7f0dfc107f20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0dec011670 tx=0x7f0dec011b60 comp rx=0 tx=0).stop 2024-03-18T14:43:51.610 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:51.128+0000 7f0e0193b700 1 -- 172.21.15.76:0/1225260323 shutdown_connections 2024-03-18T14:43:51.610 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:51.128+0000 7f0e0193b700 1 --2- 172.21.15.76:0/1225260323 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f0de4038ed0 0x7f0de403b390 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.611 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:51.128+0000 7f0e0193b700 1 --2- 172.21.15.76:0/1225260323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0dfc1086a0 0x7f0dfc107f20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.611 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:51.128+0000 7f0e0193b700 1 -- 172.21.15.76:0/1225260323 >> 172.21.15.76:0/1225260323 conn(0x7f0dfc102090 msgr2=0x7f0dfc102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:51.611 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:51.128+0000 7f0e0193b700 1 -- 172.21.15.76:0/1225260323 shutdown_connections 2024-03-18T14:43:51.611 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:51.128+0000 7f0e0193b700 1 -- 172.21.15.76:0/1225260323 wait complete. 2024-03-18T14:43:51.611 INFO:teuthology.orchestra.run.smithi076.stdout:Deploying unmanaged mon service... 2024-03-18T14:43:52.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:52 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' 2024-03-18T14:43:52.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:52 smithi076 ceph-mon[32205]: Added host smithi076 2024-03-18T14:43:52.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:52 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:43:52.824 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-18T14:43:52.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.045+0000 7fc35822b700 1 Processor -- start 2024-03-18T14:43:52.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.045+0000 7fc35822b700 1 -- start start 2024-03-18T14:43:52.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.045+0000 7fc35822b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc350108690 0x7fc35010aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.045+0000 7fc35822b700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc35007bdd0 con 0x7fc350108690 2024-03-18T14:43:52.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.045+0000 7fc355fc7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc350108690 0x7fc35010aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.045+0000 7fc355fc7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc350108690 0x7fc35010aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53492/0 (socket says 172.21.15.76:53492) 2024-03-18T14:43:52.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.045+0000 7fc355fc7700 1 -- 172.21.15.76:0/3869510324 learned_addr learned my addr 172.21.15.76:0/3869510324 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:52.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.045+0000 7fc355fc7700 1 -- 172.21.15.76:0/3869510324 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35010afc0 con 0x7fc350108690 2024-03-18T14:43:52.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.046+0000 7fc355fc7700 1 --2- 172.21.15.76:0/3869510324 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc350108690 0x7fc35010aa80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc34c0098a0 tx=0x7fc34c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77bdc924bacbd6c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:52.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.046+0000 7fc354fc5700 1 -- 172.21.15.76:0/3869510324 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc34c004160 con 0x7fc350108690 2024-03-18T14:43:52.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.046+0000 7fc354fc5700 1 -- 172.21.15.76:0/3869510324 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc34c004300 con 0x7fc350108690 2024-03-18T14:43:52.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.046+0000 7fc354fc5700 1 -- 172.21.15.76:0/3869510324 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc34c011b50 con 0x7fc350108690 2024-03-18T14:43:52.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.046+0000 7fc35822b700 1 -- 172.21.15.76:0/3869510324 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc350108690 msgr2=0x7fc35010aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.046+0000 7fc35822b700 1 --2- 172.21.15.76:0/3869510324 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc350108690 0x7fc35010aa80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc34c0098a0 tx=0x7fc34c009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.046+0000 7fc35822b700 1 -- 172.21.15.76:0/3869510324 shutdown_connections 2024-03-18T14:43:52.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.046+0000 7fc35822b700 1 --2- 172.21.15.76:0/3869510324 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc350108690 0x7fc35010aa80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.046+0000 7fc35822b700 1 -- 172.21.15.76:0/3869510324 >> 172.21.15.76:0/3869510324 conn(0x7fc350102020 msgr2=0x7fc350104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:52.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.047+0000 7fc35822b700 1 -- 172.21.15.76:0/3869510324 shutdown_connections 2024-03-18T14:43:52.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.047+0000 7fc35822b700 1 -- 172.21.15.76:0/3869510324 wait complete. 2024-03-18T14:43:52.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.047+0000 7fc35822b700 1 Processor -- start 2024-03-18T14:43:52.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.047+0000 7fc35822b700 1 -- start start 2024-03-18T14:43:52.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.047+0000 7fc35822b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc350108690 0x7fc35019bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.047+0000 7fc35822b700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc35010bf80 con 0x7fc350108690 2024-03-18T14:43:52.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.047+0000 7fc355fc7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc350108690 0x7fc35019bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.047+0000 7fc355fc7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc350108690 0x7fc35019bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53496/0 (socket says 172.21.15.76:53496) 2024-03-18T14:43:52.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.047+0000 7fc355fc7700 1 -- 172.21.15.76:0/1795750415 learned_addr learned my addr 172.21.15.76:0/1795750415 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:52.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.048+0000 7fc355fc7700 1 -- 172.21.15.76:0/1795750415 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35019c0c0 con 0x7fc350108690 2024-03-18T14:43:52.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.048+0000 7fc355fc7700 1 --2- 172.21.15.76:0/1795750415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc350108690 0x7fc35019bb80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc34c009fb0 tx=0x7fc34c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:52.827 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.048+0000 7fc346ffd700 1 -- 172.21.15.76:0/1795750415 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc34c0038b0 con 0x7fc350108690 2024-03-18T14:43:52.827 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.048+0000 7fc346ffd700 1 -- 172.21.15.76:0/1795750415 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc34c011e50 con 0x7fc350108690 2024-03-18T14:43:52.827 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.048+0000 7fc35822b700 1 -- 172.21.15.76:0/1795750415 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc35019c350 con 0x7fc350108690 2024-03-18T14:43:52.827 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.048+0000 7fc346ffd700 1 -- 172.21.15.76:0/1795750415 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc34c019420 con 0x7fc350108690 2024-03-18T14:43:52.827 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.048+0000 7fc35822b700 1 -- 172.21.15.76:0/1795750415 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc35019f040 con 0x7fc350108690 2024-03-18T14:43:52.827 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.049+0000 7fc346ffd700 1 -- 172.21.15.76:0/1795750415 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fc34c01a430 con 0x7fc350108690 2024-03-18T14:43:52.827 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.049+0000 7fc35822b700 1 -- 172.21.15.76:0/1795750415 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc350063900 con 0x7fc350108690 2024-03-18T14:43:52.827 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.049+0000 7fc346ffd700 1 --2- 172.21.15.76:0/1795750415 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fc33c038ed0 0x7fc33c03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.827 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.049+0000 7fc346ffd700 1 -- 172.21.15.76:0/1795750415 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc34c04d140 con 0x7fc350108690 2024-03-18T14:43:52.827 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.049+0000 7fc3557c6700 1 --2- 172.21.15.76:0/1795750415 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fc33c038ed0 0x7fc33c03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.827 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.050+0000 7fc3557c6700 1 --2- 172.21.15.76:0/1795750415 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fc33c038ed0 0x7fc33c03b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc340006fd0 tx=0x7fc340006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:52.827 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.054+0000 7fc346ffd700 1 -- 172.21.15.76:0/1795750415 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc34c003b00 con 0x7fc350108690 2024-03-18T14:43:52.827 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.266+0000 7fc35822b700 1 -- 172.21.15.76:0/1795750415 --> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fc3501032e0 con 0x7fc33c038ed0 2024-03-18T14:43:52.827 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.271+0000 7fc346ffd700 1 -- 172.21.15.76:0/1795750415 <== mgr.14120 v2:172.21.15.76:6800/3054039863 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc3501032e0 con 0x7fc33c038ed0 2024-03-18T14:43:52.828 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.273+0000 7fc35822b700 1 -- 172.21.15.76:0/1795750415 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fc33c038ed0 msgr2=0x7fc33c03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.828 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.273+0000 7fc35822b700 1 --2- 172.21.15.76:0/1795750415 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fc33c038ed0 0x7fc33c03b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc340006fd0 tx=0x7fc340006e40 comp rx=0 tx=0).stop 2024-03-18T14:43:52.828 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.273+0000 7fc35822b700 1 -- 172.21.15.76:0/1795750415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc350108690 msgr2=0x7fc35019bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.828 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.273+0000 7fc35822b700 1 --2- 172.21.15.76:0/1795750415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc350108690 0x7fc35019bb80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc34c009fb0 tx=0x7fc34c011bf0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.828 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.273+0000 7fc35822b700 1 -- 172.21.15.76:0/1795750415 shutdown_connections 2024-03-18T14:43:52.828 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.273+0000 7fc35822b700 1 --2- 172.21.15.76:0/1795750415 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fc33c038ed0 0x7fc33c03b390 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.828 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.273+0000 7fc35822b700 1 --2- 172.21.15.76:0/1795750415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc350108690 0x7fc35019bb80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.828 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.273+0000 7fc35822b700 1 -- 172.21.15.76:0/1795750415 >> 172.21.15.76:0/1795750415 conn(0x7fc350102020 msgr2=0x7fc350102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:52.828 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.273+0000 7fc35822b700 1 -- 172.21.15.76:0/1795750415 shutdown_connections 2024-03-18T14:43:52.828 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:52.273+0000 7fc35822b700 1 -- 172.21.15.76:0/1795750415 wait complete. 2024-03-18T14:43:52.828 INFO:teuthology.orchestra.run.smithi076.stdout:Deploying unmanaged mgr service... 2024-03-18T14:43:53.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:53 smithi076 ceph-mon[32205]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:43:53.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:53 smithi076 ceph-mon[32205]: Saving service mon spec with placement count:5 2024-03-18T14:43:53.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:53 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' 2024-03-18T14:43:54.166 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-18T14:43:54.166 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.232+0000 7f1ca9650700 1 Processor -- start 2024-03-18T14:43:54.166 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.232+0000 7f1ca9650700 1 -- start start 2024-03-18T14:43:54.166 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.233+0000 7f1ca9650700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca4106a60 0x7f1ca4106e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:54.166 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.233+0000 7f1ca9650700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca4107450 con 0x7f1ca4106a60 2024-03-18T14:43:54.166 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.233+0000 7f1ca2ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca4106a60 0x7f1ca4106e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:54.166 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.233+0000 7f1ca2ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca4106a60 0x7f1ca4106e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53508/0 (socket says 172.21.15.76:53508) 2024-03-18T14:43:54.166 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.233+0000 7f1ca2ffd700 1 -- 172.21.15.76:0/3557195563 learned_addr learned my addr 172.21.15.76:0/3557195563 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:54.166 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.233+0000 7f1ca2ffd700 1 -- 172.21.15.76:0/3557195563 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca4107c60 con 0x7f1ca4106a60 2024-03-18T14:43:54.167 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.233+0000 7f1ca2ffd700 1 --2- 172.21.15.76:0/3557195563 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca4106a60 0x7f1ca4106e80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c940098a0 tx=0x7f1c94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b5fe5fb1c90de3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:54.167 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.234+0000 7f1ca1ffb700 1 -- 172.21.15.76:0/3557195563 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c94004160 con 0x7f1ca4106a60 2024-03-18T14:43:54.167 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.234+0000 7f1ca1ffb700 1 -- 172.21.15.76:0/3557195563 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1c94004300 con 0x7f1ca4106a60 2024-03-18T14:43:54.167 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.234+0000 7f1ca1ffb700 1 -- 172.21.15.76:0/3557195563 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c94011b50 con 0x7f1ca4106a60 2024-03-18T14:43:54.167 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.234+0000 7f1ca9650700 1 -- 172.21.15.76:0/3557195563 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca4106a60 msgr2=0x7f1ca4106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:54.167 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.234+0000 7f1ca9650700 1 --2- 172.21.15.76:0/3557195563 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca4106a60 0x7f1ca4106e80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c940098a0 tx=0x7f1c94009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:54.167 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.234+0000 7f1ca9650700 1 -- 172.21.15.76:0/3557195563 shutdown_connections 2024-03-18T14:43:54.167 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.234+0000 7f1ca9650700 1 --2- 172.21.15.76:0/3557195563 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca4106a60 0x7f1ca4106e80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:54.167 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.234+0000 7f1ca9650700 1 -- 172.21.15.76:0/3557195563 >> 172.21.15.76:0/3557195563 conn(0x7f1ca4102040 msgr2=0x7f1ca4104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:54.167 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.234+0000 7f1ca9650700 1 -- 172.21.15.76:0/3557195563 shutdown_connections 2024-03-18T14:43:54.167 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.234+0000 7f1ca9650700 1 -- 172.21.15.76:0/3557195563 wait complete. 2024-03-18T14:43:54.167 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.235+0000 7f1ca9650700 1 Processor -- start 2024-03-18T14:43:54.167 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.235+0000 7f1ca9650700 1 -- start start 2024-03-18T14:43:54.167 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.235+0000 7f1ca9650700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca4106a60 0x7f1ca41999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:54.167 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.235+0000 7f1ca9650700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca407af90 con 0x7f1ca4106a60 2024-03-18T14:43:54.168 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.235+0000 7f1ca2ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca4106a60 0x7f1ca41999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:54.168 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.235+0000 7f1ca2ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca4106a60 0x7f1ca41999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53522/0 (socket says 172.21.15.76:53522) 2024-03-18T14:43:54.168 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.235+0000 7f1ca2ffd700 1 -- 172.21.15.76:0/2802391631 learned_addr learned my addr 172.21.15.76:0/2802391631 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:54.168 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.236+0000 7f1ca2ffd700 1 -- 172.21.15.76:0/2802391631 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca4199f30 con 0x7f1ca4106a60 2024-03-18T14:43:54.168 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.236+0000 7f1ca2ffd700 1 --2- 172.21.15.76:0/2802391631 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca4106a60 0x7f1ca41999f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1c94011850 tx=0x7f1c94011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:54.168 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.236+0000 7f1c8bfff700 1 -- 172.21.15.76:0/2802391631 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c94003af0 con 0x7f1ca4106a60 2024-03-18T14:43:54.168 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.236+0000 7f1c8bfff700 1 -- 172.21.15.76:0/2802391631 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1c94020070 con 0x7f1ca4106a60 2024-03-18T14:43:54.168 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.236+0000 7f1ca9650700 1 -- 172.21.15.76:0/2802391631 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ca419a1c0 con 0x7f1ca4106a60 2024-03-18T14:43:54.168 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.236+0000 7f1c8bfff700 1 -- 172.21.15.76:0/2802391631 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c940195a0 con 0x7f1ca4106a60 2024-03-18T14:43:54.168 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.236+0000 7f1ca9650700 1 -- 172.21.15.76:0/2802391631 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ca419ce60 con 0x7f1ca4106a60 2024-03-18T14:43:54.168 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.237+0000 7f1c8bfff700 1 -- 172.21.15.76:0/2802391631 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f1c9401a430 con 0x7f1ca4106a60 2024-03-18T14:43:54.168 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.237+0000 7f1ca9650700 1 -- 172.21.15.76:0/2802391631 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ca4063900 con 0x7f1ca4106a60 2024-03-18T14:43:54.168 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.237+0000 7f1c8bfff700 1 --2- 172.21.15.76:0/2802391631 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f1c8c03b1d0 0x7f1c8c03d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:54.168 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.237+0000 7f1c8bfff700 1 -- 172.21.15.76:0/2802391631 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1c94011e00 con 0x7f1ca4106a60 2024-03-18T14:43:54.168 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.237+0000 7f1ca27fc700 1 --2- 172.21.15.76:0/2802391631 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f1c8c03b1d0 0x7f1c8c03d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:54.169 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.238+0000 7f1ca27fc700 1 --2- 172.21.15.76:0/2802391631 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f1c8c03b1d0 0x7f1c8c03d690 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1c98006fd0 tx=0x7f1c98006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:54.169 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.242+0000 7f1c8bfff700 1 -- 172.21.15.76:0/2802391631 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1c94024e00 con 0x7f1ca4106a60 2024-03-18T14:43:54.169 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.457+0000 7f1ca9650700 1 -- 172.21.15.76:0/2802391631 --> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f1ca4062710 con 0x7f1c8c03b1d0 2024-03-18T14:43:54.169 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.465+0000 7f1c8bfff700 1 -- 172.21.15.76:0/2802391631 <== mgr.14120 v2:172.21.15.76:6800/3054039863 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1ca4062710 con 0x7f1c8c03b1d0 2024-03-18T14:43:54.169 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.467+0000 7f1ca9650700 1 -- 172.21.15.76:0/2802391631 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f1c8c03b1d0 msgr2=0x7f1c8c03d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:54.169 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.467+0000 7f1ca9650700 1 --2- 172.21.15.76:0/2802391631 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f1c8c03b1d0 0x7f1c8c03d690 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1c98006fd0 tx=0x7f1c98006e40 comp rx=0 tx=0).stop 2024-03-18T14:43:54.169 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.467+0000 7f1ca9650700 1 -- 172.21.15.76:0/2802391631 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca4106a60 msgr2=0x7f1ca41999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:54.169 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.467+0000 7f1ca9650700 1 --2- 172.21.15.76:0/2802391631 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca4106a60 0x7f1ca41999f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1c94011850 tx=0x7f1c94011c10 comp rx=0 tx=0).stop 2024-03-18T14:43:54.169 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.467+0000 7f1ca9650700 1 -- 172.21.15.76:0/2802391631 shutdown_connections 2024-03-18T14:43:54.169 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.467+0000 7f1ca9650700 1 --2- 172.21.15.76:0/2802391631 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f1c8c03b1d0 0x7f1c8c03d690 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:54.169 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.467+0000 7f1ca9650700 1 --2- 172.21.15.76:0/2802391631 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca4106a60 0x7f1ca41999f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:54.169 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.467+0000 7f1ca9650700 1 -- 172.21.15.76:0/2802391631 >> 172.21.15.76:0/2802391631 conn(0x7f1ca4102040 msgr2=0x7f1ca4190000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:54.169 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.467+0000 7f1ca9650700 1 -- 172.21.15.76:0/2802391631 shutdown_connections 2024-03-18T14:43:54.169 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:53.467+0000 7f1ca9650700 1 -- 172.21.15.76:0/2802391631 wait complete. 2024-03-18T14:43:54.837 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:54 smithi076 ceph-mon[32205]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:43:54.837 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:54 smithi076 ceph-mon[32205]: Saving service mgr spec with placement count:2 2024-03-18T14:43:54.837 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:54 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' 2024-03-18T14:43:55.926 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:55 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' 2024-03-18T14:43:55.926 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:55 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' 2024-03-18T14:43:55.926 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:55 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/842756646' entity='client.admin' 2024-03-18T14:43:55.933 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.133+0000 7fb77d75b700 1 Processor -- start 2024-03-18T14:43:55.933 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.133+0000 7fb77d75b700 1 -- start start 2024-03-18T14:43:55.933 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.133+0000 7fb77d75b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb778108690 0x7fb77810aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:55.933 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.133+0000 7fb77d75b700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb77807bef0 con 0x7fb778108690 2024-03-18T14:43:55.933 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.134+0000 7fb776ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb778108690 0x7fb77810aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:55.933 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.134+0000 7fb776ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb778108690 0x7fb77810aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53534/0 (socket says 172.21.15.76:53534) 2024-03-18T14:43:55.934 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.134+0000 7fb776ffd700 1 -- 172.21.15.76:0/2572028952 learned_addr learned my addr 172.21.15.76:0/2572028952 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:55.934 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.134+0000 7fb776ffd700 1 -- 172.21.15.76:0/2572028952 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77810afc0 con 0x7fb778108690 2024-03-18T14:43:55.934 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.134+0000 7fb776ffd700 1 --2- 172.21.15.76:0/2572028952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb778108690 0x7fb77810aa80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb7700098a0 tx=0x7fb770009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced454b870be4b7a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:55.934 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.134+0000 7fb775ffb700 1 -- 172.21.15.76:0/2572028952 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb770004160 con 0x7fb778108690 2024-03-18T14:43:55.934 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.134+0000 7fb775ffb700 1 -- 172.21.15.76:0/2572028952 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb770004300 con 0x7fb778108690 2024-03-18T14:43:55.934 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.134+0000 7fb775ffb700 1 -- 172.21.15.76:0/2572028952 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb770011b20 con 0x7fb778108690 2024-03-18T14:43:55.934 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.135+0000 7fb77d75b700 1 -- 172.21.15.76:0/2572028952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb778108690 msgr2=0x7fb77810aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:55.934 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.135+0000 7fb77d75b700 1 --2- 172.21.15.76:0/2572028952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb778108690 0x7fb77810aa80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb7700098a0 tx=0x7fb770009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.934 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.135+0000 7fb77d75b700 1 -- 172.21.15.76:0/2572028952 shutdown_connections 2024-03-18T14:43:55.934 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.135+0000 7fb77d75b700 1 --2- 172.21.15.76:0/2572028952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb778108690 0x7fb77810aa80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.934 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.135+0000 7fb77d75b700 1 -- 172.21.15.76:0/2572028952 >> 172.21.15.76:0/2572028952 conn(0x7fb778102000 msgr2=0x7fb778104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:55.934 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.135+0000 7fb77d75b700 1 -- 172.21.15.76:0/2572028952 shutdown_connections 2024-03-18T14:43:55.934 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.135+0000 7fb77d75b700 1 -- 172.21.15.76:0/2572028952 wait complete. 2024-03-18T14:43:55.935 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.136+0000 7fb77d75b700 1 Processor -- start 2024-03-18T14:43:55.935 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.136+0000 7fb77d75b700 1 -- start start 2024-03-18T14:43:55.935 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.136+0000 7fb77d75b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb778108690 0x7fb77819bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:55.935 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.136+0000 7fb77d75b700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb77810c0e0 con 0x7fb778108690 2024-03-18T14:43:55.935 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.136+0000 7fb776ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb778108690 0x7fb77819bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:55.935 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.136+0000 7fb776ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb778108690 0x7fb77819bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53542/0 (socket says 172.21.15.76:53542) 2024-03-18T14:43:55.935 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.136+0000 7fb776ffd700 1 -- 172.21.15.76:0/842756646 learned_addr learned my addr 172.21.15.76:0/842756646 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:55.935 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.136+0000 7fb776ffd700 1 -- 172.21.15.76:0/842756646 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77819c170 con 0x7fb778108690 2024-03-18T14:43:55.935 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.136+0000 7fb776ffd700 1 --2- 172.21.15.76:0/842756646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb778108690 0x7fb77819bc30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb770011b60 tx=0x7fb770011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:55.935 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.137+0000 7fb75ffff700 1 -- 172.21.15.76:0/842756646 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb770003b60 con 0x7fb778108690 2024-03-18T14:43:55.935 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.137+0000 7fb75ffff700 1 -- 172.21.15.76:0/842756646 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb77001a910 con 0x7fb778108690 2024-03-18T14:43:55.935 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.137+0000 7fb75ffff700 1 -- 172.21.15.76:0/842756646 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb770024930 con 0x7fb778108690 2024-03-18T14:43:55.935 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.137+0000 7fb77d75b700 1 -- 172.21.15.76:0/842756646 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb77819c400 con 0x7fb778108690 2024-03-18T14:43:55.935 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.137+0000 7fb77d75b700 1 -- 172.21.15.76:0/842756646 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb77819c860 con 0x7fb778108690 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.138+0000 7fb77d75b700 1 -- 172.21.15.76:0/842756646 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb778063900 con 0x7fb778108690 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.138+0000 7fb75ffff700 1 -- 172.21.15.76:0/842756646 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fb770022020 con 0x7fb778108690 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.138+0000 7fb75ffff700 1 --2- 172.21.15.76:0/842756646 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fb760038f70 0x7fb76003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.138+0000 7fb75ffff700 1 -- 172.21.15.76:0/842756646 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb77004cff0 con 0x7fb778108690 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.138+0000 7fb7767fc700 1 --2- 172.21.15.76:0/842756646 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fb760038f70 0x7fb76003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.139+0000 7fb7767fc700 1 --2- 172.21.15.76:0/842756646 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fb760038f70 0x7fb76003b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb76c006fd0 tx=0x7fb76c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.143+0000 7fb75ffff700 1 -- 172.21.15.76:0/842756646 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb77001abe0 con 0x7fb778108690 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.347+0000 7fb77d75b700 1 -- 172.21.15.76:0/842756646 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fb77819f5a0 con 0x7fb778108690 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.351+0000 7fb75ffff700 1 -- 172.21.15.76:0/842756646 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fb77000bc70 con 0x7fb778108690 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.353+0000 7fb77d75b700 1 -- 172.21.15.76:0/842756646 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fb760038f70 msgr2=0x7fb76003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.353+0000 7fb77d75b700 1 --2- 172.21.15.76:0/842756646 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fb760038f70 0x7fb76003b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb76c006fd0 tx=0x7fb76c006e40 comp rx=0 tx=0).stop 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.353+0000 7fb77d75b700 1 -- 172.21.15.76:0/842756646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb778108690 msgr2=0x7fb77819bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.353+0000 7fb77d75b700 1 --2- 172.21.15.76:0/842756646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb778108690 0x7fb77819bc30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb770011b60 tx=0x7fb770011c40 comp rx=0 tx=0).stop 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.353+0000 7fb77d75b700 1 -- 172.21.15.76:0/842756646 shutdown_connections 2024-03-18T14:43:55.937 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.353+0000 7fb77d75b700 1 --2- 172.21.15.76:0/842756646 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7fb760038f70 0x7fb76003b430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.937 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.353+0000 7fb77d75b700 1 --2- 172.21.15.76:0/842756646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb778108690 0x7fb77819bc30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.937 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.353+0000 7fb77d75b700 1 -- 172.21.15.76:0/842756646 >> 172.21.15.76:0/842756646 conn(0x7fb778102000 msgr2=0x7fb778102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:55.937 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.353+0000 7fb77d75b700 1 -- 172.21.15.76:0/842756646 shutdown_connections 2024-03-18T14:43:55.937 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:55.353+0000 7fb77d75b700 1 -- 172.21.15.76:0/842756646 wait complete. 2024-03-18T14:43:57.650 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.387+0000 7f3877eec700 1 Processor -- start 2024-03-18T14:43:57.650 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.387+0000 7f3877eec700 1 -- start start 2024-03-18T14:43:57.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.387+0000 7f3877eec700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3870106a60 0x7f3870106e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.387+0000 7f3877eec700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3870107450 con 0x7f3870106a60 2024-03-18T14:43:57.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.388+0000 7f3875c88700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3870106a60 0x7f3870106e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:57.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.388+0000 7f3875c88700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3870106a60 0x7f3870106e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53554/0 (socket says 172.21.15.76:53554) 2024-03-18T14:43:57.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.388+0000 7f3875c88700 1 -- 172.21.15.76:0/3038129448 learned_addr learned my addr 172.21.15.76:0/3038129448 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:57.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.388+0000 7f3875c88700 1 -- 172.21.15.76:0/3038129448 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3870107c60 con 0x7f3870106a60 2024-03-18T14:43:57.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.388+0000 7f3875c88700 1 --2- 172.21.15.76:0/3038129448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3870106a60 0x7f3870106e80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f386c0098a0 tx=0x7f386c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b052cd1d57d55a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:57.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.388+0000 7f3874c86700 1 -- 172.21.15.76:0/3038129448 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f386c004160 con 0x7f3870106a60 2024-03-18T14:43:57.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.388+0000 7f3874c86700 1 -- 172.21.15.76:0/3038129448 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f386c004300 con 0x7f3870106a60 2024-03-18T14:43:57.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.388+0000 7f3874c86700 1 -- 172.21.15.76:0/3038129448 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f386c011b20 con 0x7f3870106a60 2024-03-18T14:43:57.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.389+0000 7f3877eec700 1 -- 172.21.15.76:0/3038129448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3870106a60 msgr2=0x7f3870106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.389+0000 7f3877eec700 1 --2- 172.21.15.76:0/3038129448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3870106a60 0x7f3870106e80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f386c0098a0 tx=0x7f386c009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.389+0000 7f3877eec700 1 -- 172.21.15.76:0/3038129448 shutdown_connections 2024-03-18T14:43:57.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.389+0000 7f3877eec700 1 --2- 172.21.15.76:0/3038129448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3870106a60 0x7f3870106e80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.389+0000 7f3877eec700 1 -- 172.21.15.76:0/3038129448 >> 172.21.15.76:0/3038129448 conn(0x7f3870102040 msgr2=0x7f3870104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:57.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.389+0000 7f3877eec700 1 -- 172.21.15.76:0/3038129448 shutdown_connections 2024-03-18T14:43:57.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.389+0000 7f3877eec700 1 -- 172.21.15.76:0/3038129448 wait complete. 2024-03-18T14:43:57.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.389+0000 7f3877eec700 1 Processor -- start 2024-03-18T14:43:57.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.389+0000 7f3877eec700 1 -- start start 2024-03-18T14:43:57.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.390+0000 7f3877eec700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3870106a60 0x7f387019a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.390+0000 7f3877eec700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f387010b010 con 0x7f3870106a60 2024-03-18T14:43:57.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.390+0000 7f3875c88700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3870106a60 0x7f387019a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:57.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.390+0000 7f3875c88700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3870106a60 0x7f387019a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53558/0 (socket says 172.21.15.76:53558) 2024-03-18T14:43:57.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.390+0000 7f3875c88700 1 -- 172.21.15.76:0/1820602141 learned_addr learned my addr 172.21.15.76:0/1820602141 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:57.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.390+0000 7f3875c88700 1 -- 172.21.15.76:0/1820602141 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f387019af00 con 0x7f3870106a60 2024-03-18T14:43:57.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.390+0000 7f3875c88700 1 --2- 172.21.15.76:0/1820602141 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3870106a60 0x7f387019a9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f386c0117e0 tx=0x7f386c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:57.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.390+0000 7f3866ffd700 1 -- 172.21.15.76:0/1820602141 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f386c003bf0 con 0x7f3870106a60 2024-03-18T14:43:57.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.390+0000 7f3866ffd700 1 -- 172.21.15.76:0/1820602141 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f386c01a910 con 0x7f3870106a60 2024-03-18T14:43:57.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.390+0000 7f3877eec700 1 -- 172.21.15.76:0/1820602141 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f387019b190 con 0x7f3870106a60 2024-03-18T14:43:57.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.390+0000 7f3877eec700 1 -- 172.21.15.76:0/1820602141 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f387019b5f0 con 0x7f3870106a60 2024-03-18T14:43:57.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.390+0000 7f3866ffd700 1 -- 172.21.15.76:0/1820602141 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f386c024970 con 0x7f3870106a60 2024-03-18T14:43:57.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.391+0000 7f3866ffd700 1 -- 172.21.15.76:0/1820602141 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f386c022020 con 0x7f3870106a60 2024-03-18T14:43:57.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.391+0000 7f3877eec700 1 -- 172.21.15.76:0/1820602141 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3870063900 con 0x7f3870106a60 2024-03-18T14:43:57.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.391+0000 7f3866ffd700 1 --2- 172.21.15.76:0/1820602141 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f385c038f20 0x7f385c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.391+0000 7f3866ffd700 1 -- 172.21.15.76:0/1820602141 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f386c04d620 con 0x7f3870106a60 2024-03-18T14:43:57.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.391+0000 7f3875487700 1 --2- 172.21.15.76:0/1820602141 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f385c038f20 0x7f385c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:57.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.392+0000 7f3875487700 1 --2- 172.21.15.76:0/1820602141 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f385c038f20 0x7f385c03b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3860006fd0 tx=0x7f3860006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:57.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.396+0000 7f3866ffd700 1 -- 172.21.15.76:0/1820602141 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f386c0172b0 con 0x7f3870106a60 2024-03-18T14:43:57.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.609+0000 7f3877eec700 1 -- 172.21.15.76:0/1820602141 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f387019dfc0 con 0x7f3870106a60 2024-03-18T14:43:57.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.612+0000 7f3866ffd700 1 -- 172.21.15.76:0/1820602141 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f386c00b5a0 con 0x7f3870106a60 2024-03-18T14:43:57.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.614+0000 7f3877eec700 1 -- 172.21.15.76:0/1820602141 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f385c038f20 msgr2=0x7f385c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.614+0000 7f3877eec700 1 --2- 172.21.15.76:0/1820602141 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f385c038f20 0x7f385c03b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3860006fd0 tx=0x7f3860006e40 comp rx=0 tx=0).stop 2024-03-18T14:43:57.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.614+0000 7f3877eec700 1 -- 172.21.15.76:0/1820602141 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3870106a60 msgr2=0x7f387019a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.614+0000 7f3877eec700 1 --2- 172.21.15.76:0/1820602141 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3870106a60 0x7f387019a9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f386c0117e0 tx=0x7f386c011ba0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.614+0000 7f3877eec700 1 -- 172.21.15.76:0/1820602141 shutdown_connections 2024-03-18T14:43:57.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.614+0000 7f3877eec700 1 --2- 172.21.15.76:0/1820602141 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f385c038f20 0x7f385c03b3e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.614+0000 7f3877eec700 1 --2- 172.21.15.76:0/1820602141 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3870106a60 0x7f387019a9c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.654 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.614+0000 7f3877eec700 1 -- 172.21.15.76:0/1820602141 >> 172.21.15.76:0/1820602141 conn(0x7f3870102040 msgr2=0x7f3870102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:57.654 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.615+0000 7f3877eec700 1 -- 172.21.15.76:0/1820602141 shutdown_connections 2024-03-18T14:43:57.654 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:56.615+0000 7f3877eec700 1 -- 172.21.15.76:0/1820602141 wait complete. 2024-03-18T14:43:57.654 INFO:teuthology.orchestra.run.smithi076.stdout:Enabling the dashboard module... 2024-03-18T14:43:57.895 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:57 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1820602141' entity='client.admin' 2024-03-18T14:43:57.895 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:57 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' 2024-03-18T14:43:59.653 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:59 smithi076 ceph-mon[32205]: from='mgr.14120 172.21.15.76:0/438893193' entity='mgr.x' 2024-03-18T14:43:59.653 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:43:59 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2845023633' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-18T14:43:59.654 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:59 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: ignoring --setuser ceph since I am not root 2024-03-18T14:43:59.654 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:59 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: ignoring --setgroup ceph since I am not root 2024-03-18T14:43:59.944 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.097+0000 7f33a2e2a700 1 Processor -- start 2024-03-18T14:43:59.944 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.097+0000 7f33a2e2a700 1 -- start start 2024-03-18T14:43:59.944 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.097+0000 7f33a2e2a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f339c108630 0x7f339c10aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:59.945 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.097+0000 7f33a2e2a700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f339c07be50 con 0x7f339c108630 2024-03-18T14:43:59.945 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.097+0000 7f33a0bc6700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f339c108630 0x7f339c10aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:59.945 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.097+0000 7f33a0bc6700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f339c108630 0x7f339c10aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53564/0 (socket says 172.21.15.76:53564) 2024-03-18T14:43:59.945 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.097+0000 7f33a0bc6700 1 -- 172.21.15.76:0/2384019442 learned_addr learned my addr 172.21.15.76:0/2384019442 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:59.945 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.098+0000 7f33a0bc6700 1 -- 172.21.15.76:0/2384019442 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f339c10af60 con 0x7f339c108630 2024-03-18T14:43:59.945 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.098+0000 7f33a0bc6700 1 --2- 172.21.15.76:0/2384019442 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f339c108630 0x7f339c10aa20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f33900098a0 tx=0x7f3390009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d97aa2e89f9186 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:59.945 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.098+0000 7f339b7fe700 1 -- 172.21.15.76:0/2384019442 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3390004160 con 0x7f339c108630 2024-03-18T14:43:59.945 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.098+0000 7f339b7fe700 1 -- 172.21.15.76:0/2384019442 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3390004300 con 0x7f339c108630 2024-03-18T14:43:59.945 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.098+0000 7f339b7fe700 1 -- 172.21.15.76:0/2384019442 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3390011b20 con 0x7f339c108630 2024-03-18T14:43:59.945 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.099+0000 7f33a2e2a700 1 -- 172.21.15.76:0/2384019442 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f339c108630 msgr2=0x7f339c10aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:59.945 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.099+0000 7f33a2e2a700 1 --2- 172.21.15.76:0/2384019442 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f339c108630 0x7f339c10aa20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f33900098a0 tx=0x7f3390009bb0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.945 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.099+0000 7f33a2e2a700 1 -- 172.21.15.76:0/2384019442 shutdown_connections 2024-03-18T14:43:59.945 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.099+0000 7f33a2e2a700 1 --2- 172.21.15.76:0/2384019442 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f339c108630 0x7f339c10aa20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.945 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.099+0000 7f33a2e2a700 1 -- 172.21.15.76:0/2384019442 >> 172.21.15.76:0/2384019442 conn(0x7f339c102020 msgr2=0x7f339c104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:59.946 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.099+0000 7f33a2e2a700 1 -- 172.21.15.76:0/2384019442 shutdown_connections 2024-03-18T14:43:59.946 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.099+0000 7f33a2e2a700 1 -- 172.21.15.76:0/2384019442 wait complete. 2024-03-18T14:43:59.946 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.099+0000 7f33a2e2a700 1 Processor -- start 2024-03-18T14:43:59.946 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.099+0000 7f33a2e2a700 1 -- start start 2024-03-18T14:43:59.946 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.100+0000 7f33a2e2a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f339c108630 0x7f339c107eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:59.946 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.100+0000 7f33a2e2a700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f339c10bfc0 con 0x7f339c108630 2024-03-18T14:43:59.946 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.100+0000 7f33a0bc6700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f339c108630 0x7f339c107eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:59.946 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.100+0000 7f33a0bc6700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f339c108630 0x7f339c107eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53570/0 (socket says 172.21.15.76:53570) 2024-03-18T14:43:59.946 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.100+0000 7f33a0bc6700 1 -- 172.21.15.76:0/2845023633 learned_addr learned my addr 172.21.15.76:0/2845023633 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:43:59.946 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.100+0000 7f33a0bc6700 1 -- 172.21.15.76:0/2845023633 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f339c1083f0 con 0x7f339c108630 2024-03-18T14:43:59.946 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.100+0000 7f33a0bc6700 1 --2- 172.21.15.76:0/2845023633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f339c108630 0x7f339c107eb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3390011b60 tx=0x7f3390011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:59.946 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.100+0000 7f3399ffb700 1 -- 172.21.15.76:0/2845023633 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3390003b60 con 0x7f339c108630 2024-03-18T14:43:59.946 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.100+0000 7f3399ffb700 1 -- 172.21.15.76:0/2845023633 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f339001a910 con 0x7f339c108630 2024-03-18T14:43:59.947 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.100+0000 7f3399ffb700 1 -- 172.21.15.76:0/2845023633 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3390024930 con 0x7f339c108630 2024-03-18T14:43:59.947 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.100+0000 7f33a2e2a700 1 -- 172.21.15.76:0/2845023633 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f339c1065e0 con 0x7f339c108630 2024-03-18T14:43:59.947 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.101+0000 7f33a2e2a700 1 -- 172.21.15.76:0/2845023633 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f339c106aa0 con 0x7f339c108630 2024-03-18T14:43:59.947 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.101+0000 7f3399ffb700 1 -- 172.21.15.76:0/2845023633 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f3390022020 con 0x7f339c108630 2024-03-18T14:43:59.947 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.102+0000 7f3399ffb700 1 --2- 172.21.15.76:0/2845023633 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f338c038f20 0x7f338c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:59.947 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.102+0000 7f33a2e2a700 1 -- 172.21.15.76:0/2845023633 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f339c063900 con 0x7f339c108630 2024-03-18T14:43:59.947 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.102+0000 7f3399ffb700 1 -- 172.21.15.76:0/2845023633 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f339004d010 con 0x7f339c108630 2024-03-18T14:43:59.947 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.102+0000 7f339bfff700 1 --2- 172.21.15.76:0/2845023633 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f338c038f20 0x7f338c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:59.947 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.102+0000 7f339bfff700 1 --2- 172.21.15.76:0/2845023633 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f338c038f20 0x7f338c03b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3388006fd0 tx=0x7f3388006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:59.947 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.107+0000 7f3399ffb700 1 -- 172.21.15.76:0/2845023633 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3390019c70 con 0x7f339c108630 2024-03-18T14:43:59.947 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:58.365+0000 7f33a2e2a700 1 -- 172.21.15.76:0/2845023633 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f339c106d80 con 0x7f339c108630 2024-03-18T14:43:59.947 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:59.340+0000 7f3399ffb700 1 -- 172.21.15.76:0/2845023633 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f339002d910 con 0x7f339c108630 2024-03-18T14:43:59.947 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:59.342+0000 7f3399ffb700 1 -- 172.21.15.76:0/2845023633 <== mon.0 v2:172.21.15.76:3300/0 8 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f339002d3f0 con 0x7f339c108630 2024-03-18T14:43:59.947 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:59.344+0000 7f33a2e2a700 1 -- 172.21.15.76:0/2845023633 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f338c038f20 msgr2=0x7f338c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:59.947 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:59.344+0000 7f33a2e2a700 1 --2- 172.21.15.76:0/2845023633 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f338c038f20 0x7f338c03b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3388006fd0 tx=0x7f3388006e40 comp rx=0 tx=0).stop 2024-03-18T14:43:59.948 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:59.344+0000 7f33a2e2a700 1 -- 172.21.15.76:0/2845023633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f339c108630 msgr2=0x7f339c107eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:59.948 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:59.344+0000 7f33a2e2a700 1 --2- 172.21.15.76:0/2845023633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f339c108630 0x7f339c107eb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3390011b60 tx=0x7f3390011c40 comp rx=0 tx=0).stop 2024-03-18T14:43:59.948 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:59.344+0000 7f33a2e2a700 1 -- 172.21.15.76:0/2845023633 shutdown_connections 2024-03-18T14:43:59.948 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:59.344+0000 7f33a2e2a700 1 --2- 172.21.15.76:0/2845023633 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f338c038f20 0x7f338c03b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.948 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:59.344+0000 7f33a2e2a700 1 --2- 172.21.15.76:0/2845023633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f339c108630 0x7f339c107eb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.948 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:59.344+0000 7f33a2e2a700 1 -- 172.21.15.76:0/2845023633 >> 172.21.15.76:0/2845023633 conn(0x7f339c102020 msgr2=0x7f339c102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:59.948 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:59.344+0000 7f33a2e2a700 1 -- 172.21.15.76:0/2845023633 shutdown_connections 2024-03-18T14:43:59.948 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:43:59.344+0000 7f33a2e2a700 1 -- 172.21.15.76:0/2845023633 wait complete. 2024-03-18T14:44:00.193 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:43:59 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:43:59.915+0000 7fa6bd21d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T14:44:00.193 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:00 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:00.016+0000 7fa6bd21d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T14:44:00.574 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:00 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2845023633' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-18T14:44:00.574 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:00 smithi076 ceph-mon[32205]: mgrmap e9: x(active, since 19s) 2024-03-18T14:44:00.912 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:00 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:00.685+0000 7fa6bd21d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T14:44:00.912 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:00 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:00.785+0000 7fa6bd21d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T14:44:01.295 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:00 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:00.929+0000 7fa6bd21d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T14:44:01.295 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:01 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:01.028+0000 7fa6bd21d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T14:44:01.295 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:01 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:01.112+0000 7fa6bd21d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T14:44:01.303 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-03-18T14:44:01.304 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-18T14:44:01.304 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T14:44:01.304 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-18T14:44:01.304 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T14:44:01.304 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-03-18T14:44:01.304 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.389+0000 7f8f2fca7700 1 Processor -- start 2024-03-18T14:44:01.304 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.389+0000 7f8f2fca7700 1 -- start start 2024-03-18T14:44:01.304 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.389+0000 7f8f2fca7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f28108620 0x7f8f2810aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:01.304 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.389+0000 7f8f2fca7700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2807bf10 con 0x7f8f28108620 2024-03-18T14:44:01.304 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.390+0000 7f8f2da43700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f28108620 0x7f8f2810aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:01.305 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.390+0000 7f8f2da43700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f28108620 0x7f8f2810aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48010/0 (socket says 172.21.15.76:48010) 2024-03-18T14:44:01.305 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.390+0000 7f8f2da43700 1 -- 172.21.15.76:0/3525467245 learned_addr learned my addr 172.21.15.76:0/3525467245 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:01.305 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.390+0000 7f8f2da43700 1 -- 172.21.15.76:0/3525467245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f2810af50 con 0x7f8f28108620 2024-03-18T14:44:01.305 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.390+0000 7f8f2da43700 1 --2- 172.21.15.76:0/3525467245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f28108620 0x7f8f2810aa10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8f180098a0 tx=0x7f8f18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcae2d4b6b8a2d2d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:01.305 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.390+0000 7f8f2ca41700 1 -- 172.21.15.76:0/3525467245 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f18004160 con 0x7f8f28108620 2024-03-18T14:44:01.305 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.390+0000 7f8f2ca41700 1 -- 172.21.15.76:0/3525467245 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f18004300 con 0x7f8f28108620 2024-03-18T14:44:01.305 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.390+0000 7f8f2ca41700 1 -- 172.21.15.76:0/3525467245 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f18011b20 con 0x7f8f28108620 2024-03-18T14:44:01.305 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.391+0000 7f8f2fca7700 1 -- 172.21.15.76:0/3525467245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f28108620 msgr2=0x7f8f2810aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:01.305 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.391+0000 7f8f2fca7700 1 --2- 172.21.15.76:0/3525467245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f28108620 0x7f8f2810aa10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8f180098a0 tx=0x7f8f18009bb0 comp rx=0 tx=0).stop 2024-03-18T14:44:01.305 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.391+0000 7f8f2fca7700 1 -- 172.21.15.76:0/3525467245 shutdown_connections 2024-03-18T14:44:01.305 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.391+0000 7f8f2fca7700 1 --2- 172.21.15.76:0/3525467245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f28108620 0x7f8f2810aa10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:01.305 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.391+0000 7f8f2fca7700 1 -- 172.21.15.76:0/3525467245 >> 172.21.15.76:0/3525467245 conn(0x7f8f28101ff0 msgr2=0x7f8f28104430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:01.305 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.391+0000 7f8f2fca7700 1 -- 172.21.15.76:0/3525467245 shutdown_connections 2024-03-18T14:44:01.306 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.391+0000 7f8f2fca7700 1 -- 172.21.15.76:0/3525467245 wait complete. 2024-03-18T14:44:01.306 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.392+0000 7f8f2fca7700 1 Processor -- start 2024-03-18T14:44:01.306 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.392+0000 7f8f2fca7700 1 -- start start 2024-03-18T14:44:01.306 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.392+0000 7f8f2fca7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f28108620 0x7f8f2819bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:01.306 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.392+0000 7f8f2fca7700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2810c070 con 0x7f8f28108620 2024-03-18T14:44:01.306 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.392+0000 7f8f2da43700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f28108620 0x7f8f2819bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:01.306 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.392+0000 7f8f2da43700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f28108620 0x7f8f2819bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48024/0 (socket says 172.21.15.76:48024) 2024-03-18T14:44:01.306 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.392+0000 7f8f2da43700 1 -- 172.21.15.76:0/2096739662 learned_addr learned my addr 172.21.15.76:0/2096739662 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:01.306 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.392+0000 7f8f2da43700 1 -- 172.21.15.76:0/2096739662 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f2819c130 con 0x7f8f28108620 2024-03-18T14:44:01.306 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.393+0000 7f8f2da43700 1 --2- 172.21.15.76:0/2096739662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f28108620 0x7f8f2819bbf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8f180116a0 tx=0x7f8f18011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:01.306 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.393+0000 7f8f1effd700 1 -- 172.21.15.76:0/2096739662 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f18003a40 con 0x7f8f28108620 2024-03-18T14:44:01.306 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.393+0000 7f8f1effd700 1 -- 172.21.15.76:0/2096739662 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f18003be0 con 0x7f8f28108620 2024-03-18T14:44:01.307 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.393+0000 7f8f1effd700 1 -- 172.21.15.76:0/2096739662 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f1801a750 con 0x7f8f28108620 2024-03-18T14:44:01.307 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.393+0000 7f8f2fca7700 1 -- 172.21.15.76:0/2096739662 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f2819c3c0 con 0x7f8f28108620 2024-03-18T14:44:01.307 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.393+0000 7f8f2fca7700 1 -- 172.21.15.76:0/2096739662 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f2819c820 con 0x7f8f28108620 2024-03-18T14:44:01.307 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.394+0000 7f8f1effd700 1 -- 172.21.15.76:0/2096739662 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f8f1801aa00 con 0x7f8f28108620 2024-03-18T14:44:01.307 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.394+0000 7f8f2fca7700 1 -- 172.21.15.76:0/2096739662 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f28063900 con 0x7f8f28108620 2024-03-18T14:44:01.307 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.394+0000 7f8f1effd700 1 --2- 172.21.15.76:0/2096739662 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f8f14038f80 0x7f8f1403b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:01.307 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.394+0000 7f8f1effd700 1 -- 172.21.15.76:0/2096739662 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8f1804d6e0 con 0x7f8f28108620 2024-03-18T14:44:01.307 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.394+0000 7f8f2d242700 1 -- 172.21.15.76:0/2096739662 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f8f14038f80 msgr2=0x7f8f1403b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3054039863 2024-03-18T14:44:01.307 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.394+0000 7f8f2d242700 1 --2- 172.21.15.76:0/2096739662 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f8f14038f80 0x7f8f1403b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T14:44:01.307 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.403+0000 7f8f1effd700 1 -- 172.21.15.76:0/2096739662 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8f18011e20 con 0x7f8f28108620 2024-03-18T14:44:01.307 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.595+0000 7f8f2d242700 1 -- 172.21.15.76:0/2096739662 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f8f14038f80 msgr2=0x7f8f1403b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3054039863 2024-03-18T14:44:01.307 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.595+0000 7f8f2d242700 1 --2- 172.21.15.76:0/2096739662 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f8f14038f80 0x7f8f1403b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T14:44:01.307 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.665+0000 7f8f2fca7700 1 -- 172.21.15.76:0/2096739662 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f8f2819f0b0 con 0x7f8f28108620 2024-03-18T14:44:01.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.665+0000 7f8f1effd700 1 -- 172.21.15.76:0/2096739662 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f8f2819f0b0 con 0x7f8f28108620 2024-03-18T14:44:01.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.667+0000 7f8f2fca7700 1 -- 172.21.15.76:0/2096739662 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f8f14038f80 msgr2=0x7f8f1403b440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T14:44:01.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.667+0000 7f8f2fca7700 1 --2- 172.21.15.76:0/2096739662 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f8f14038f80 0x7f8f1403b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:01.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.667+0000 7f8f2fca7700 1 -- 172.21.15.76:0/2096739662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f28108620 msgr2=0x7f8f2819bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:01.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.667+0000 7f8f2fca7700 1 --2- 172.21.15.76:0/2096739662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f28108620 0x7f8f2819bbf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8f180116a0 tx=0x7f8f18011ba0 comp rx=0 tx=0).stop 2024-03-18T14:44:01.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.667+0000 7f8f2fca7700 1 -- 172.21.15.76:0/2096739662 shutdown_connections 2024-03-18T14:44:01.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.667+0000 7f8f2fca7700 1 --2- 172.21.15.76:0/2096739662 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f8f14038f80 0x7f8f1403b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:01.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.667+0000 7f8f2fca7700 1 --2- 172.21.15.76:0/2096739662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f28108620 0x7f8f2819bbf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:01.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.667+0000 7f8f2fca7700 1 -- 172.21.15.76:0/2096739662 >> 172.21.15.76:0/2096739662 conn(0x7f8f28101ff0 msgr2=0x7f8f28102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:01.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.667+0000 7f8f2fca7700 1 -- 172.21.15.76:0/2096739662 shutdown_connections 2024-03-18T14:44:01.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:00.667+0000 7f8f2fca7700 1 -- 172.21.15.76:0/2096739662 wait complete. 2024-03-18T14:44:01.308 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for the mgr to restart... 2024-03-18T14:44:01.308 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mgr epoch 9... 2024-03-18T14:44:01.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:01 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2096739662' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T14:44:01.550 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:01 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:01.387+0000 7fa6bd21d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T14:44:01.550 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:01 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:01.499+0000 7fa6bd21d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T14:44:01.867 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:01 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:01.588+0000 7fa6bd21d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T14:44:01.867 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:01 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:01.687+0000 7fa6bd21d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T14:44:01.867 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:01 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:01.865+0000 7fa6bd21d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T14:44:03.155 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:02 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:02.823+0000 7fa6bd21d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T14:44:03.156 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:02 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:02.953+0000 7fa6bd21d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T14:44:03.412 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:03 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:03.285+0000 7fa6bd21d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T14:44:03.412 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:03 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:03.377+0000 7fa6bd21d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T14:44:03.911 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:03 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:03.576+0000 7fa6bd21d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:44:03.912 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:03 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:03.768+0000 7fa6bd21d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T14:44:04.315 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:04 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:04.055+0000 7fa6bd21d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:44:04.315 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:04 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:04.313+0000 7fa6bd21d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T14:44:04.662 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:04 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:04.550+0000 7fa6bd21d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T14:44:05.661 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:05 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:05.354+0000 7fa6bd21d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T14:44:06.240 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:05 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:05.921+0000 7fa6bd21d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T14:44:06.240 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:06 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:06.020+0000 7fa6bd21d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T14:44:06.240 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:06 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:06.238+0000 7fa6bd21d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T14:44:06.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:06 smithi076 ceph-mon[32205]: Active manager daemon x restarted 2024-03-18T14:44:06.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:06 smithi076 ceph-mon[32205]: Activating manager daemon x 2024-03-18T14:44:06.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:06 smithi076 ceph-mon[32205]: osdmap e3: 0 total, 0 up, 0 in 2024-03-18T14:44:06.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:06 smithi076 ceph-mon[32205]: mgrmap e10: x(active, starting, since 0.0034932s) 2024-03-18T14:44:06.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:44:06.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T14:44:06.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T14:44:06.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T14:44:06.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T14:44:06.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:06 smithi076 ceph-mon[32205]: Manager daemon x is now available 2024-03-18T14:44:06.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:44:06.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:44:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:07 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T14:44:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:07 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T14:44:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:07 smithi076 ceph-mon[32205]: mgrmap e11: x(active, since 1.0067s) 2024-03-18T14:44:08.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-03-18T14:44:08.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-18T14:44:08.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T14:44:08.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-03-18T14:44:08.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.747+0000 7f10f9a7c700 1 Processor -- start 2024-03-18T14:44:08.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.747+0000 7f10f9a7c700 1 -- start start 2024-03-18T14:44:08.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.747+0000 7f10f9a7c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f41064a0 0x7f10f41068c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.747+0000 7f10f9a7c700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f10f407bed0 con 0x7f10f41064a0 2024-03-18T14:44:08.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.747+0000 7f10f37fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f41064a0 0x7f10f41068c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.747+0000 7f10f37fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f41064a0 0x7f10f41068c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48040/0 (socket says 172.21.15.76:48040) 2024-03-18T14:44:08.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.747+0000 7f10f37fe700 1 -- 172.21.15.76:0/3051063369 learned_addr learned my addr 172.21.15.76:0/3051063369 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:08.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.748+0000 7f10f37fe700 1 -- 172.21.15.76:0/3051063369 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f4106e00 con 0x7f10f41064a0 2024-03-18T14:44:08.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.748+0000 7f10f37fe700 1 --2- 172.21.15.76:0/3051063369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f41064a0 0x7f10f41068c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f10e80098a0 tx=0x7f10e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22afff3096763c66 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:08.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.748+0000 7f10f27fc700 1 -- 172.21.15.76:0/3051063369 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10e8004160 con 0x7f10f41064a0 2024-03-18T14:44:08.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.748+0000 7f10f27fc700 1 -- 172.21.15.76:0/3051063369 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10e8004300 con 0x7f10f41064a0 2024-03-18T14:44:08.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.748+0000 7f10f27fc700 1 -- 172.21.15.76:0/3051063369 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10e8011b20 con 0x7f10f41064a0 2024-03-18T14:44:08.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.749+0000 7f10f9a7c700 1 -- 172.21.15.76:0/3051063369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f41064a0 msgr2=0x7f10f41068c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.749+0000 7f10f9a7c700 1 --2- 172.21.15.76:0/3051063369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f41064a0 0x7f10f41068c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f10e80098a0 tx=0x7f10e8009bb0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.749+0000 7f10f9a7c700 1 -- 172.21.15.76:0/3051063369 shutdown_connections 2024-03-18T14:44:08.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.749+0000 7f10f9a7c700 1 --2- 172.21.15.76:0/3051063369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f41064a0 0x7f10f41068c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.749+0000 7f10f9a7c700 1 -- 172.21.15.76:0/3051063369 >> 172.21.15.76:0/3051063369 conn(0x7f10f4102000 msgr2=0x7f10f4104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:08.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.749+0000 7f10f9a7c700 1 -- 172.21.15.76:0/3051063369 shutdown_connections 2024-03-18T14:44:08.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.749+0000 7f10f9a7c700 1 -- 172.21.15.76:0/3051063369 wait complete. 2024-03-18T14:44:08.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.749+0000 7f10f9a7c700 1 Processor -- start 2024-03-18T14:44:08.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.750+0000 7f10f9a7c700 1 -- start start 2024-03-18T14:44:08.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.750+0000 7f10f9a7c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f41064a0 0x7f10f419de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.150 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.750+0000 7f10f9a7c700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f10f4107ed0 con 0x7f10f41064a0 2024-03-18T14:44:08.150 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.750+0000 7f10f37fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f41064a0 0x7f10f419de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.150 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.750+0000 7f10f37fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f41064a0 0x7f10f419de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48056/0 (socket says 172.21.15.76:48056) 2024-03-18T14:44:08.150 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.750+0000 7f10f37fe700 1 -- 172.21.15.76:0/1367149773 learned_addr learned my addr 172.21.15.76:0/1367149773 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:08.150 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.750+0000 7f10f37fe700 1 -- 172.21.15.76:0/1367149773 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f419e350 con 0x7f10f41064a0 2024-03-18T14:44:08.150 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.750+0000 7f10f37fe700 1 --2- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f41064a0 0x7f10f419de10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f10e80116a0 tx=0x7f10e8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:08.150 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.750+0000 7f10f0ff9700 1 -- 172.21.15.76:0/1367149773 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10e8003a40 con 0x7f10f41064a0 2024-03-18T14:44:08.150 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.750+0000 7f10f0ff9700 1 -- 172.21.15.76:0/1367149773 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10e8003be0 con 0x7f10f41064a0 2024-03-18T14:44:08.150 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.750+0000 7f10f0ff9700 1 -- 172.21.15.76:0/1367149773 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10e801a750 con 0x7f10f41064a0 2024-03-18T14:44:08.150 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.750+0000 7f10f9a7c700 1 -- 172.21.15.76:0/1367149773 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f419e5e0 con 0x7f10f41064a0 2024-03-18T14:44:08.150 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.750+0000 7f10f9a7c700 1 -- 172.21.15.76:0/1367149773 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f419ea40 con 0x7f10f41064a0 2024-03-18T14:44:08.150 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.751+0000 7f10f0ff9700 1 -- 172.21.15.76:0/1367149773 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f10e801aa00 con 0x7f10f41064a0 2024-03-18T14:44:08.150 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.751+0000 7f10f0ff9700 1 --2- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f10dc038f80 0x7f10dc03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.150 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.751+0000 7f10f9a7c700 1 -- 172.21.15.76:0/1367149773 --> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f10f4063900 con 0x7f10dc038f80 2024-03-18T14:44:08.151 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.751+0000 7f10f0ff9700 1 -- 172.21.15.76:0/1367149773 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f10e804e370 con 0x7f10f41064a0 2024-03-18T14:44:08.151 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.751+0000 7f10f2ffd700 1 -- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f10dc038f80 msgr2=0x7f10dc03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3054039863 2024-03-18T14:44:08.151 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.751+0000 7f10f2ffd700 1 --2- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f10dc038f80 0x7f10dc03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T14:44:08.151 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.952+0000 7f10f2ffd700 1 -- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f10dc038f80 msgr2=0x7f10dc03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3054039863 2024-03-18T14:44:08.151 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:01.952+0000 7f10f2ffd700 1 --2- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f10dc038f80 0x7f10dc03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T14:44:08.151 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:02.353+0000 7f10f2ffd700 1 -- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f10dc038f80 msgr2=0x7f10dc03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3054039863 2024-03-18T14:44:08.151 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:02.353+0000 7f10f2ffd700 1 --2- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f10dc038f80 0x7f10dc03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T14:44:08.151 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:03.154+0000 7f10f2ffd700 1 -- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f10dc038f80 msgr2=0x7f10dc03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3054039863 2024-03-18T14:44:08.151 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:03.154+0000 7f10f2ffd700 1 --2- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f10dc038f80 0x7f10dc03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T14:44:08.151 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:04.755+0000 7f10f2ffd700 1 -- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f10dc038f80 msgr2=0x7f10dc03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3054039863 2024-03-18T14:44:08.151 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:04.755+0000 7f10f2ffd700 1 --2- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f10dc038f80 0x7f10dc03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T14:44:08.151 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:06.253+0000 7f10f0ff9700 1 -- 172.21.15.76:0/1367149773 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mgrmap(e 10) ==== 45886+0+0 (secure 0 0 0) 0x7f10e802d910 con 0x7f10f41064a0 2024-03-18T14:44:08.151 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:06.253+0000 7f10f0ff9700 1 -- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f10dc038f80 msgr2=0x7f10dc03b440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T14:44:08.151 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:06.253+0000 7f10f0ff9700 1 --2- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f10dc038f80 0x7f10dc03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.152 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.255+0000 7f10f0ff9700 1 -- 172.21.15.76:0/1367149773 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7f10e804f5b0 con 0x7f10f41064a0 2024-03-18T14:44:08.152 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.255+0000 7f10f0ff9700 1 --2- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f10dc03c840 0x7f10dc03ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.152 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.255+0000 7f10f0ff9700 1 -- 172.21.15.76:0/1367149773 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f10f4063900 con 0x7f10dc03c840 2024-03-18T14:44:08.152 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.255+0000 7f10f2ffd700 1 --2- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f10dc03c840 0x7f10dc03ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.152 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.256+0000 7f10f2ffd700 1 --2- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f10dc03c840 0x7f10dc03ec30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f10e4003a10 tx=0x7f10e4009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:08.152 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.257+0000 7f10f0ff9700 1 -- 172.21.15.76:0/1367149773 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f10f4063900 con 0x7f10dc03c840 2024-03-18T14:44:08.152 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.264+0000 7f10f9a7c700 1 -- 172.21.15.76:0/1367149773 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f10f4062710 con 0x7f10dc03c840 2024-03-18T14:44:08.152 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.265+0000 7f10f0ff9700 1 -- 172.21.15.76:0/1367149773 <== mgr.14152 v2:172.21.15.76:6800/3010549595 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f10f4062710 con 0x7f10dc03c840 2024-03-18T14:44:08.152 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.265+0000 7f10f9a7c700 1 -- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f10dc03c840 msgr2=0x7f10dc03ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.152 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.265+0000 7f10f9a7c700 1 --2- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f10dc03c840 0x7f10dc03ec30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f10e4003a10 tx=0x7f10e4009270 comp rx=0 tx=0).stop 2024-03-18T14:44:08.152 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.265+0000 7f10f9a7c700 1 -- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f41064a0 msgr2=0x7f10f419de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.152 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.265+0000 7f10f9a7c700 1 --2- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f41064a0 0x7f10f419de10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f10e80116a0 tx=0x7f10e8011ba0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.152 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.265+0000 7f10f9a7c700 1 -- 172.21.15.76:0/1367149773 shutdown_connections 2024-03-18T14:44:08.152 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.265+0000 7f10f9a7c700 1 --2- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f10dc03c840 0x7f10dc03ec30 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.153 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.265+0000 7f10f9a7c700 1 --2- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f41064a0 0x7f10f419de10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.153 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.265+0000 7f10f9a7c700 1 --2- 172.21.15.76:0/1367149773 >> [v2:172.21.15.76:6800/3054039863,v1:172.21.15.76:6801/3054039863] conn(0x7f10dc038f80 0x7f10dc03b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.153 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.265+0000 7f10f9a7c700 1 -- 172.21.15.76:0/1367149773 >> 172.21.15.76:0/1367149773 conn(0x7f10f4102000 msgr2=0x7f10f4194a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:08.153 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.265+0000 7f10f9a7c700 1 -- 172.21.15.76:0/1367149773 shutdown_connections 2024-03-18T14:44:08.153 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:07.265+0000 7f10f9a7c700 1 -- 172.21.15.76:0/1367149773 wait complete. 2024-03-18T14:44:08.153 INFO:teuthology.orchestra.run.smithi076.stdout:mgr epoch 9 is available 2024-03-18T14:44:08.153 INFO:teuthology.orchestra.run.smithi076.stdout:Generating a dashboard self-signed certificate... 2024-03-18T14:44:08.584 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:08 smithi076 ceph-mon[32205]: [18/Mar/2024:14:44:07] ENGINE Bus STARTING 2024-03-18T14:44:08.584 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:08 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:09.504 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-18T14:44:09.504 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.606+0000 7fd415527700 1 Processor -- start 2024-03-18T14:44:09.504 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.606+0000 7fd415527700 1 -- start start 2024-03-18T14:44:09.504 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.606+0000 7fd415527700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd410105e70 0x7fd410106290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:09.504 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.606+0000 7fd415527700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd410106860 con 0x7fd410105e70 2024-03-18T14:44:09.505 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.606+0000 7fd40effd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd410105e70 0x7fd410106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:09.505 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.606+0000 7fd40effd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd410105e70 0x7fd410106290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47870/0 (socket says 172.21.15.76:47870) 2024-03-18T14:44:09.505 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.606+0000 7fd40effd700 1 -- 172.21.15.76:0/834825481 learned_addr learned my addr 172.21.15.76:0/834825481 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:09.505 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.607+0000 7fd40effd700 1 -- 172.21.15.76:0/834825481 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd410108e00 con 0x7fd410105e70 2024-03-18T14:44:09.505 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.607+0000 7fd40effd700 1 --2- 172.21.15.76:0/834825481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd410105e70 0x7fd410106290 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd408009e80 tx=0x7fd40800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b43e7566103e930 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:09.505 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.607+0000 7fd40dffb700 1 -- 172.21.15.76:0/834825481 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd408017070 con 0x7fd410105e70 2024-03-18T14:44:09.505 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.607+0000 7fd40dffb700 1 -- 172.21.15.76:0/834825481 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd40800b8c0 con 0x7fd410105e70 2024-03-18T14:44:09.505 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.607+0000 7fd40dffb700 1 -- 172.21.15.76:0/834825481 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd4080045c0 con 0x7fd410105e70 2024-03-18T14:44:09.505 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.608+0000 7fd415527700 1 -- 172.21.15.76:0/834825481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd410105e70 msgr2=0x7fd410106290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:09.505 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.608+0000 7fd415527700 1 --2- 172.21.15.76:0/834825481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd410105e70 0x7fd410106290 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd408009e80 tx=0x7fd40800b330 comp rx=0 tx=0).stop 2024-03-18T14:44:09.505 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.608+0000 7fd415527700 1 -- 172.21.15.76:0/834825481 shutdown_connections 2024-03-18T14:44:09.505 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.608+0000 7fd415527700 1 --2- 172.21.15.76:0/834825481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd410105e70 0x7fd410106290 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.505 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.608+0000 7fd415527700 1 -- 172.21.15.76:0/834825481 >> 172.21.15.76:0/834825481 conn(0x7fd410101840 msgr2=0x7fd410103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:09.505 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.608+0000 7fd415527700 1 -- 172.21.15.76:0/834825481 shutdown_connections 2024-03-18T14:44:09.505 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.608+0000 7fd415527700 1 -- 172.21.15.76:0/834825481 wait complete. 2024-03-18T14:44:09.506 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.609+0000 7fd415527700 1 Processor -- start 2024-03-18T14:44:09.506 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.609+0000 7fd415527700 1 -- start start 2024-03-18T14:44:09.506 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.609+0000 7fd415527700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd410105e70 0x7fd41019bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:09.506 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.609+0000 7fd415527700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd41010c1b0 con 0x7fd410105e70 2024-03-18T14:44:09.506 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.609+0000 7fd40effd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd410105e70 0x7fd41019bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:09.506 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.609+0000 7fd40effd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd410105e70 0x7fd41019bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47876/0 (socket says 172.21.15.76:47876) 2024-03-18T14:44:09.506 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.609+0000 7fd40effd700 1 -- 172.21.15.76:0/2177541248 learned_addr learned my addr 172.21.15.76:0/2177541248 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:09.506 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.609+0000 7fd40effd700 1 -- 172.21.15.76:0/2177541248 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd41019c0a0 con 0x7fd410105e70 2024-03-18T14:44:09.506 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.609+0000 7fd40effd700 1 --2- 172.21.15.76:0/2177541248 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd410105e70 0x7fd41019bb60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd408009530 tx=0x7fd408003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:09.506 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.610+0000 7fd3f7fff700 1 -- 172.21.15.76:0/2177541248 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd408020020 con 0x7fd410105e70 2024-03-18T14:44:09.506 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.610+0000 7fd3f7fff700 1 -- 172.21.15.76:0/2177541248 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd40800bba0 con 0x7fd410105e70 2024-03-18T14:44:09.506 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.610+0000 7fd415527700 1 -- 172.21.15.76:0/2177541248 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd41019c330 con 0x7fd410105e70 2024-03-18T14:44:09.506 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.610+0000 7fd3f7fff700 1 -- 172.21.15.76:0/2177541248 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd408017040 con 0x7fd410105e70 2024-03-18T14:44:09.506 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.610+0000 7fd415527700 1 -- 172.21.15.76:0/2177541248 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd41019f020 con 0x7fd410105e70 2024-03-18T14:44:09.507 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.610+0000 7fd3f7fff700 1 -- 172.21.15.76:0/2177541248 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7fd408023020 con 0x7fd410105e70 2024-03-18T14:44:09.507 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.611+0000 7fd415527700 1 -- 172.21.15.76:0/2177541248 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd410063900 con 0x7fd410105e70 2024-03-18T14:44:09.507 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.611+0000 7fd3f7fff700 1 --2- 172.21.15.76:0/2177541248 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd3f8038ec0 0x7fd3f803b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:09.507 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.611+0000 7fd3f7fff700 1 -- 172.21.15.76:0/2177541248 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd40800f050 con 0x7fd410105e70 2024-03-18T14:44:09.507 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.611+0000 7fd40e7fc700 1 --2- 172.21.15.76:0/2177541248 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd3f8038ec0 0x7fd3f803b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:09.507 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.611+0000 7fd40e7fc700 1 --2- 172.21.15.76:0/2177541248 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd3f8038ec0 0x7fd3f803b380 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd404006fd0 tx=0x7fd404006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:09.507 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.616+0000 7fd3f7fff700 1 -- 172.21.15.76:0/2177541248 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd40802a020 con 0x7fd410105e70 2024-03-18T14:44:09.507 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.826+0000 7fd3f7fff700 1 -- 172.21.15.76:0/2177541248 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7fd40801b370 con 0x7fd410105e70 2024-03-18T14:44:09.507 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.831+0000 7fd415527700 1 -- 172.21.15.76:0/2177541248 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fd410102ab0 con 0x7fd3f8038ec0 2024-03-18T14:44:09.507 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.928+0000 7fd3f7fff700 1 -- 172.21.15.76:0/2177541248 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fd410102ab0 con 0x7fd3f8038ec0 2024-03-18T14:44:09.507 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.930+0000 7fd415527700 1 -- 172.21.15.76:0/2177541248 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd3f8038ec0 msgr2=0x7fd3f803b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:09.507 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.930+0000 7fd415527700 1 --2- 172.21.15.76:0/2177541248 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd3f8038ec0 0x7fd3f803b380 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd404006fd0 tx=0x7fd404006e40 comp rx=0 tx=0).stop 2024-03-18T14:44:09.507 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.930+0000 7fd415527700 1 -- 172.21.15.76:0/2177541248 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd410105e70 msgr2=0x7fd41019bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:09.507 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.930+0000 7fd415527700 1 --2- 172.21.15.76:0/2177541248 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd410105e70 0x7fd41019bb60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd408009530 tx=0x7fd408003d00 comp rx=0 tx=0).stop 2024-03-18T14:44:09.507 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.930+0000 7fd415527700 1 -- 172.21.15.76:0/2177541248 shutdown_connections 2024-03-18T14:44:09.508 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.930+0000 7fd415527700 1 --2- 172.21.15.76:0/2177541248 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd3f8038ec0 0x7fd3f803b380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.508 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.930+0000 7fd415527700 1 --2- 172.21.15.76:0/2177541248 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd410105e70 0x7fd41019bb60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.508 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.930+0000 7fd415527700 1 -- 172.21.15.76:0/2177541248 >> 172.21.15.76:0/2177541248 conn(0x7fd410101840 msgr2=0x7fd4101023a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:09.508 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.931+0000 7fd415527700 1 -- 172.21.15.76:0/2177541248 shutdown_connections 2024-03-18T14:44:09.508 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:08.931+0000 7fd415527700 1 -- 172.21.15.76:0/2177541248 wait complete. 2024-03-18T14:44:09.508 INFO:teuthology.orchestra.run.smithi076.stdout:Creating initial admin user... 2024-03-18T14:44:09.623 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:09 smithi076 ceph-mon[32205]: [18/Mar/2024:14:44:07] ENGINE Serving on https://172.21.15.76:7150 2024-03-18T14:44:09.623 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:09 smithi076 ceph-mon[32205]: [18/Mar/2024:14:44:07] ENGINE Serving on http://172.21.15.76:8765 2024-03-18T14:44:09.623 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:09 smithi076 ceph-mon[32205]: [18/Mar/2024:14:44:07] ENGINE Bus STARTED 2024-03-18T14:44:09.623 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:09 smithi076 ceph-mon[32205]: mgrmap e12: x(active, since 2s) 2024-03-18T14:44:09.623 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:09 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:09.623 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:09 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:10.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:10 smithi076 ceph-mon[32205]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:44:10.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:10 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:11.012 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$406ld1nVXf83FQAnetfn9OrewztXyLfqbkYYD9c55QYi7pn5XU9ve", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710773050, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-18T14:44:11.012 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.942+0000 7f76fdfe4700 1 Processor -- start 2024-03-18T14:44:11.012 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.942+0000 7f76fdfe4700 1 -- start start 2024-03-18T14:44:11.012 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.942+0000 7f76fdfe4700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76f8107b70 0x7f76f8107f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.942+0000 7f76fdfe4700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f76f8108560 con 0x7f76f8107b70 2024-03-18T14:44:11.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.942+0000 7f76f77fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76f8107b70 0x7f76f8107f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.942+0000 7f76f77fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76f8107b70 0x7f76f8107f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47886/0 (socket says 172.21.15.76:47886) 2024-03-18T14:44:11.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.942+0000 7f76f77fe700 1 -- 172.21.15.76:0/851819178 learned_addr learned my addr 172.21.15.76:0/851819178 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:11.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.942+0000 7f76f77fe700 1 -- 172.21.15.76:0/851819178 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f8108d70 con 0x7f76f8107b70 2024-03-18T14:44:11.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.943+0000 7f76f77fe700 1 --2- 172.21.15.76:0/851819178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76f8107b70 0x7f76f8107f90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f76e80098a0 tx=0x7f76e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21640270d428eb91 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:11.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.943+0000 7f76f67fc700 1 -- 172.21.15.76:0/851819178 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76e8004160 con 0x7f76f8107b70 2024-03-18T14:44:11.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.943+0000 7f76f67fc700 1 -- 172.21.15.76:0/851819178 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76e8004300 con 0x7f76f8107b70 2024-03-18T14:44:11.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.943+0000 7f76f67fc700 1 -- 172.21.15.76:0/851819178 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76e8011b20 con 0x7f76f8107b70 2024-03-18T14:44:11.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.943+0000 7f76fdfe4700 1 -- 172.21.15.76:0/851819178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76f8107b70 msgr2=0x7f76f8107f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.943+0000 7f76fdfe4700 1 --2- 172.21.15.76:0/851819178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76f8107b70 0x7f76f8107f90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f76e80098a0 tx=0x7f76e8009bb0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.943+0000 7f76fdfe4700 1 -- 172.21.15.76:0/851819178 shutdown_connections 2024-03-18T14:44:11.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.943+0000 7f76fdfe4700 1 --2- 172.21.15.76:0/851819178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76f8107b70 0x7f76f8107f90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.943+0000 7f76fdfe4700 1 -- 172.21.15.76:0/851819178 >> 172.21.15.76:0/851819178 conn(0x7f76f8103150 msgr2=0x7f76f8105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:11.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.944+0000 7f76fdfe4700 1 -- 172.21.15.76:0/851819178 shutdown_connections 2024-03-18T14:44:11.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.944+0000 7f76fdfe4700 1 -- 172.21.15.76:0/851819178 wait complete. 2024-03-18T14:44:11.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.944+0000 7f76fdfe4700 1 Processor -- start 2024-03-18T14:44:11.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.944+0000 7f76fdfe4700 1 -- start start 2024-03-18T14:44:11.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.945+0000 7f76fdfe4700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76f8107b70 0x7f76f807db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.945+0000 7f76fdfe4700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f76f810c120 con 0x7f76f8107b70 2024-03-18T14:44:11.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.945+0000 7f76f77fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76f8107b70 0x7f76f807db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.945+0000 7f76f77fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76f8107b70 0x7f76f807db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47902/0 (socket says 172.21.15.76:47902) 2024-03-18T14:44:11.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.945+0000 7f76f77fe700 1 -- 172.21.15.76:0/3482636889 learned_addr learned my addr 172.21.15.76:0/3482636889 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:11.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.945+0000 7f76f77fe700 1 -- 172.21.15.76:0/3482636889 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f807e070 con 0x7f76f8107b70 2024-03-18T14:44:11.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.945+0000 7f76f77fe700 1 --2- 172.21.15.76:0/3482636889 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76f8107b70 0x7f76f807db30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f76e8000c00 tx=0x7f76e8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:11.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.945+0000 7f76f4ff9700 1 -- 172.21.15.76:0/3482636889 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76e8003b60 con 0x7f76f8107b70 2024-03-18T14:44:11.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.945+0000 7f76f4ff9700 1 -- 172.21.15.76:0/3482636889 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76e801a910 con 0x7f76f8107b70 2024-03-18T14:44:11.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.945+0000 7f76fdfe4700 1 -- 172.21.15.76:0/3482636889 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76f807c260 con 0x7f76f8107b70 2024-03-18T14:44:11.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.945+0000 7f76f4ff9700 1 -- 172.21.15.76:0/3482636889 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76e8024930 con 0x7f76f8107b70 2024-03-18T14:44:11.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.945+0000 7f76fdfe4700 1 -- 172.21.15.76:0/3482636889 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76f807c720 con 0x7f76f8107b70 2024-03-18T14:44:11.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.946+0000 7f76f4ff9700 1 -- 172.21.15.76:0/3482636889 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f76e8022020 con 0x7f76f8107b70 2024-03-18T14:44:11.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.946+0000 7f76f4ff9700 1 --2- 172.21.15.76:0/3482636889 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f76e0038fd0 0x7f76e003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.946+0000 7f76fdfe4700 1 -- 172.21.15.76:0/3482636889 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76f8063900 con 0x7f76f8107b70 2024-03-18T14:44:11.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.946+0000 7f76f4ff9700 1 -- 172.21.15.76:0/3482636889 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f76e804d0e0 con 0x7f76f8107b70 2024-03-18T14:44:11.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.947+0000 7f76f6ffd700 1 --2- 172.21.15.76:0/3482636889 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f76e0038fd0 0x7f76e003b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.947+0000 7f76f6ffd700 1 --2- 172.21.15.76:0/3482636889 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f76e0038fd0 0x7f76e003b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f76ec006fd0 tx=0x7f76ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:11.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:09.951+0000 7f76f4ff9700 1 -- 172.21.15.76:0/3482636889 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f76e802ddb0 con 0x7f76f8107b70 2024-03-18T14:44:11.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:10.195+0000 7f76fdfe4700 1 -- 172.21.15.76:0/3482636889 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f76f807cec0 con 0x7f76e0038fd0 2024-03-18T14:44:11.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:10.424+0000 7f76f4ff9700 1 -- 172.21.15.76:0/3482636889 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f76f807cec0 con 0x7f76e0038fd0 2024-03-18T14:44:11.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:10.426+0000 7f76fdfe4700 1 -- 172.21.15.76:0/3482636889 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f76e0038fd0 msgr2=0x7f76e003b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:10.426+0000 7f76fdfe4700 1 --2- 172.21.15.76:0/3482636889 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f76e0038fd0 0x7f76e003b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f76ec006fd0 tx=0x7f76ec006e40 comp rx=0 tx=0).stop 2024-03-18T14:44:11.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:10.426+0000 7f76fdfe4700 1 -- 172.21.15.76:0/3482636889 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76f8107b70 msgr2=0x7f76f807db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:10.426+0000 7f76fdfe4700 1 --2- 172.21.15.76:0/3482636889 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76f8107b70 0x7f76f807db30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f76e8000c00 tx=0x7f76e8011c10 comp rx=0 tx=0).stop 2024-03-18T14:44:11.016 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:10.426+0000 7f76fdfe4700 1 -- 172.21.15.76:0/3482636889 shutdown_connections 2024-03-18T14:44:11.016 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:10.426+0000 7f76fdfe4700 1 --2- 172.21.15.76:0/3482636889 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f76e0038fd0 0x7f76e003b490 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.016 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:10.426+0000 7f76fdfe4700 1 --2- 172.21.15.76:0/3482636889 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76f8107b70 0x7f76f807db30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.016 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:10.426+0000 7f76fdfe4700 1 -- 172.21.15.76:0/3482636889 >> 172.21.15.76:0/3482636889 conn(0x7f76f8103150 msgr2=0x7f76f8103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:11.016 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:10.426+0000 7f76fdfe4700 1 -- 172.21.15.76:0/3482636889 shutdown_connections 2024-03-18T14:44:11.016 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:10.426+0000 7f76fdfe4700 1 -- 172.21.15.76:0/3482636889 wait complete. 2024-03-18T14:44:11.016 INFO:teuthology.orchestra.run.smithi076.stdout:Fetching dashboard port number... 2024-03-18T14:44:11.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:11 smithi076 ceph-mon[32205]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:44:11.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:11 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:12.298 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 8443 2024-03-18T14:44:12.298 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.451+0000 7f8a0522f700 1 Processor -- start 2024-03-18T14:44:12.298 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.451+0000 7f8a0522f700 1 -- start start 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.452+0000 7f8a0522f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a00106470 0x7f8a00106890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.452+0000 7f8a0522f700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0007bdf0 con 0x7f8a00106470 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.452+0000 7f89fed9d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a00106470 0x7f8a00106890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.452+0000 7f89fed9d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a00106470 0x7f8a00106890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47906/0 (socket says 172.21.15.76:47906) 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.452+0000 7f89fed9d700 1 -- 172.21.15.76:0/325117152 learned_addr learned my addr 172.21.15.76:0/325117152 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.452+0000 7f89fed9d700 1 -- 172.21.15.76:0/325117152 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a00106dd0 con 0x7f8a00106470 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.452+0000 7f89fed9d700 1 --2- 172.21.15.76:0/325117152 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a00106470 0x7f8a00106890 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f89f00098a0 tx=0x7f89f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c3a70e510b743e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.453+0000 7f89fdd9b700 1 -- 172.21.15.76:0/325117152 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89f0004160 con 0x7f8a00106470 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.453+0000 7f89fdd9b700 1 -- 172.21.15.76:0/325117152 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f0004300 con 0x7f8a00106470 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.453+0000 7f89fdd9b700 1 -- 172.21.15.76:0/325117152 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89f0011b20 con 0x7f8a00106470 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.453+0000 7f8a0522f700 1 -- 172.21.15.76:0/325117152 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a00106470 msgr2=0x7f8a00106890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.453+0000 7f8a0522f700 1 --2- 172.21.15.76:0/325117152 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a00106470 0x7f8a00106890 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f89f00098a0 tx=0x7f89f0009bb0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.453+0000 7f8a0522f700 1 -- 172.21.15.76:0/325117152 shutdown_connections 2024-03-18T14:44:12.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.453+0000 7f8a0522f700 1 --2- 172.21.15.76:0/325117152 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a00106470 0x7f8a00106890 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.453+0000 7f8a0522f700 1 -- 172.21.15.76:0/325117152 >> 172.21.15.76:0/325117152 conn(0x7f8a00102090 msgr2=0x7f8a001044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:12.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.453+0000 7f8a0522f700 1 -- 172.21.15.76:0/325117152 shutdown_connections 2024-03-18T14:44:12.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.453+0000 7f8a0522f700 1 -- 172.21.15.76:0/325117152 wait complete. 2024-03-18T14:44:12.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.454+0000 7f8a0522f700 1 Processor -- start 2024-03-18T14:44:12.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.454+0000 7f8a0522f700 1 -- start start 2024-03-18T14:44:12.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.454+0000 7f8a0522f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a00106470 0x7f8a0019dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.454+0000 7f8a0522f700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8a00107ea0 con 0x7f8a00106470 2024-03-18T14:44:12.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.454+0000 7f89fed9d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a00106470 0x7f8a0019dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.454+0000 7f89fed9d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a00106470 0x7f8a0019dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47916/0 (socket says 172.21.15.76:47916) 2024-03-18T14:44:12.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.454+0000 7f89fed9d700 1 -- 172.21.15.76:0/1805642551 learned_addr learned my addr 172.21.15.76:0/1805642551 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:12.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.455+0000 7f89fed9d700 1 -- 172.21.15.76:0/1805642551 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0019e1b0 con 0x7f8a00106470 2024-03-18T14:44:12.300 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.455+0000 7f89fed9d700 1 --2- 172.21.15.76:0/1805642551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a00106470 0x7f8a0019dc70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f89f00117e0 tx=0x7f89f0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:12.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.455+0000 7f89e7fff700 1 -- 172.21.15.76:0/1805642551 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89f0003bf0 con 0x7f8a00106470 2024-03-18T14:44:12.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.455+0000 7f89e7fff700 1 -- 172.21.15.76:0/1805642551 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f001a910 con 0x7f8a00106470 2024-03-18T14:44:12.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.455+0000 7f8a0522f700 1 -- 172.21.15.76:0/1805642551 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a0019e440 con 0x7f8a00106470 2024-03-18T14:44:12.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.455+0000 7f89e7fff700 1 -- 172.21.15.76:0/1805642551 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89f0024970 con 0x7f8a00106470 2024-03-18T14:44:12.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.455+0000 7f8a0522f700 1 -- 172.21.15.76:0/1805642551 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a0019e8a0 con 0x7f8a00106470 2024-03-18T14:44:12.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.456+0000 7f89e7fff700 1 -- 172.21.15.76:0/1805642551 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f89f0022020 con 0x7f8a00106470 2024-03-18T14:44:12.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.456+0000 7f8a0522f700 1 -- 172.21.15.76:0/1805642551 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a00063900 con 0x7f8a00106470 2024-03-18T14:44:12.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.456+0000 7f89e7fff700 1 --2- 172.21.15.76:0/1805642551 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89e8038f80 0x7f89e803b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.456+0000 7f89e7fff700 1 -- 172.21.15.76:0/1805642551 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f89f004d2a0 con 0x7f8a00106470 2024-03-18T14:44:12.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.456+0000 7f89fe59c700 1 --2- 172.21.15.76:0/1805642551 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89e8038f80 0x7f89e803b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.457+0000 7f89fe59c700 1 --2- 172.21.15.76:0/1805642551 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89e8038f80 0x7f89e803b440 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f89f4006fd0 tx=0x7f89f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:12.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.461+0000 7f89e7fff700 1 -- 172.21.15.76:0/1805642551 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f89f0011e20 con 0x7f8a00106470 2024-03-18T14:44:12.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.665+0000 7f8a0522f700 1 -- 172.21.15.76:0/1805642551 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f8a0002dfb0 con 0x7f8a00106470 2024-03-18T14:44:12.301 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.666+0000 7f89e7fff700 1 -- 172.21.15.76:0/1805642551 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f8a0002dfb0 con 0x7f8a00106470 2024-03-18T14:44:12.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.668+0000 7f8a0522f700 1 -- 172.21.15.76:0/1805642551 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89e8038f80 msgr2=0x7f89e803b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.668+0000 7f8a0522f700 1 --2- 172.21.15.76:0/1805642551 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89e8038f80 0x7f89e803b440 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f89f4006fd0 tx=0x7f89f4006e40 comp rx=0 tx=0).stop 2024-03-18T14:44:12.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.668+0000 7f8a0522f700 1 -- 172.21.15.76:0/1805642551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a00106470 msgr2=0x7f8a0019dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.668+0000 7f8a0522f700 1 --2- 172.21.15.76:0/1805642551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a00106470 0x7f8a0019dc70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f89f00117e0 tx=0x7f89f0011c10 comp rx=0 tx=0).stop 2024-03-18T14:44:12.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.668+0000 7f8a0522f700 1 -- 172.21.15.76:0/1805642551 shutdown_connections 2024-03-18T14:44:12.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.668+0000 7f8a0522f700 1 --2- 172.21.15.76:0/1805642551 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89e8038f80 0x7f89e803b440 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.668+0000 7f8a0522f700 1 --2- 172.21.15.76:0/1805642551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a00106470 0x7f8a0019dc70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.668+0000 7f8a0522f700 1 -- 172.21.15.76:0/1805642551 >> 172.21.15.76:0/1805642551 conn(0x7f8a00102090 msgr2=0x7f8a001948e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:12.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.668+0000 7f8a0522f700 1 -- 172.21.15.76:0/1805642551 shutdown_connections 2024-03-18T14:44:12.302 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:11.668+0000 7f8a0522f700 1 -- 172.21.15.76:0/1805642551 wait complete. 2024-03-18T14:44:12.305 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T14:44:12.305 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout disabled 2024-03-18T14:44:12.312 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T14:44:12.312 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-03-18T14:44:12.312 INFO:teuthology.orchestra.run.smithi076.stdout:firewalld.service is not enabled 2024-03-18T14:44:12.312 INFO:teuthology.orchestra.run.smithi076.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-18T14:44:12.314 INFO:teuthology.orchestra.run.smithi076.stdout:Ceph Dashboard is now available at: 2024-03-18T14:44:12.314 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:44:12.314 INFO:teuthology.orchestra.run.smithi076.stdout: URL: https://smithi076.front.sepia.ceph.com:8443/ 2024-03-18T14:44:12.314 INFO:teuthology.orchestra.run.smithi076.stdout: User: admin 2024-03-18T14:44:12.314 INFO:teuthology.orchestra.run.smithi076.stdout: Password: 2tiwmc26h4 2024-03-18T14:44:12.314 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:44:12.314 INFO:teuthology.orchestra.run.smithi076.stdout:Saving cluster configuration to /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config directory 2024-03-18T14:44:12.315 INFO:teuthology.orchestra.run.smithi076.stdout:Enabling autotune for osd_memory_target 2024-03-18T14:44:12.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:12 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1805642551' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-18T14:44:13.643 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:13 smithi076 ceph-mon[32205]: mgrmap e13: x(active, since 6s) 2024-03-18T14:44:13.650 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.744+0000 7f40c2778700 1 Processor -- start 2024-03-18T14:44:13.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.744+0000 7f40c2778700 1 -- start start 2024-03-18T14:44:13.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.744+0000 7f40c2778700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40bc105e80 0x7f40bc1062a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.744+0000 7f40c2778700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f40bc106870 con 0x7f40bc105e80 2024-03-18T14:44:13.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.744+0000 7f40bbfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40bc105e80 0x7f40bc1062a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.744+0000 7f40bbfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40bc105e80 0x7f40bc1062a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47928/0 (socket says 172.21.15.76:47928) 2024-03-18T14:44:13.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.744+0000 7f40bbfff700 1 -- 172.21.15.76:0/2447270043 learned_addr learned my addr 172.21.15.76:0/2447270043 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:13.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.745+0000 7f40bbfff700 1 -- 172.21.15.76:0/2447270043 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40bc108e10 con 0x7f40bc105e80 2024-03-18T14:44:13.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.745+0000 7f40bbfff700 1 --2- 172.21.15.76:0/2447270043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40bc105e80 0x7f40bc1062a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0098a0 tx=0x7f40ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f672f714146b50 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:13.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.745+0000 7f40baffd700 1 -- 172.21.15.76:0/2447270043 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40ac004160 con 0x7f40bc105e80 2024-03-18T14:44:13.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.745+0000 7f40baffd700 1 -- 172.21.15.76:0/2447270043 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40ac004300 con 0x7f40bc105e80 2024-03-18T14:44:13.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.745+0000 7f40baffd700 1 -- 172.21.15.76:0/2447270043 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40ac011b20 con 0x7f40bc105e80 2024-03-18T14:44:13.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.746+0000 7f40c2778700 1 -- 172.21.15.76:0/2447270043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40bc105e80 msgr2=0x7f40bc1062a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:13.651 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.746+0000 7f40c2778700 1 --2- 172.21.15.76:0/2447270043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40bc105e80 0x7f40bc1062a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0098a0 tx=0x7f40ac009bb0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.746+0000 7f40c2778700 1 -- 172.21.15.76:0/2447270043 shutdown_connections 2024-03-18T14:44:13.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.746+0000 7f40c2778700 1 --2- 172.21.15.76:0/2447270043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40bc105e80 0x7f40bc1062a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.746+0000 7f40c2778700 1 -- 172.21.15.76:0/2447270043 >> 172.21.15.76:0/2447270043 conn(0x7f40bc101790 msgr2=0x7f40bc103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:13.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.746+0000 7f40c2778700 1 -- 172.21.15.76:0/2447270043 shutdown_connections 2024-03-18T14:44:13.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.746+0000 7f40c2778700 1 -- 172.21.15.76:0/2447270043 wait complete. 2024-03-18T14:44:13.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.746+0000 7f40c2778700 1 Processor -- start 2024-03-18T14:44:13.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.746+0000 7f40c2778700 1 -- start start 2024-03-18T14:44:13.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.747+0000 7f40c2778700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40bc105e80 0x7f40bc19bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.747+0000 7f40c2778700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f40bc10c1c0 con 0x7f40bc105e80 2024-03-18T14:44:13.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.747+0000 7f40bbfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40bc105e80 0x7f40bc19bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.747+0000 7f40bbfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40bc105e80 0x7f40bc19bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47930/0 (socket says 172.21.15.76:47930) 2024-03-18T14:44:13.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.747+0000 7f40bbfff700 1 -- 172.21.15.76:0/2303072833 learned_addr learned my addr 172.21.15.76:0/2303072833 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:13.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.747+0000 7f40bbfff700 1 -- 172.21.15.76:0/2303072833 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40bc19c080 con 0x7f40bc105e80 2024-03-18T14:44:13.652 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.747+0000 7f40bbfff700 1 --2- 172.21.15.76:0/2303072833 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40bc105e80 0x7f40bc19bb40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0116a0 tx=0x7f40ac011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:13.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.747+0000 7f40b97fa700 1 -- 172.21.15.76:0/2303072833 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40ac003a40 con 0x7f40bc105e80 2024-03-18T14:44:13.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.747+0000 7f40b97fa700 1 -- 172.21.15.76:0/2303072833 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40ac003be0 con 0x7f40bc105e80 2024-03-18T14:44:13.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.747+0000 7f40c2778700 1 -- 172.21.15.76:0/2303072833 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40bc19c310 con 0x7f40bc105e80 2024-03-18T14:44:13.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.747+0000 7f40b97fa700 1 -- 172.21.15.76:0/2303072833 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40ac01a720 con 0x7f40bc105e80 2024-03-18T14:44:13.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.747+0000 7f40c2778700 1 -- 172.21.15.76:0/2303072833 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40bc19c770 con 0x7f40bc105e80 2024-03-18T14:44:13.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.748+0000 7f40b97fa700 1 -- 172.21.15.76:0/2303072833 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f40ac01aa40 con 0x7f40bc105e80 2024-03-18T14:44:13.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.748+0000 7f40c2778700 1 -- 172.21.15.76:0/2303072833 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40bc063900 con 0x7f40bc105e80 2024-03-18T14:44:13.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.748+0000 7f40b97fa700 1 --2- 172.21.15.76:0/2303072833 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f40a4038fc0 0x7f40a403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.748+0000 7f40b97fa700 1 -- 172.21.15.76:0/2303072833 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f40ac04d8f0 con 0x7f40bc105e80 2024-03-18T14:44:13.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.748+0000 7f40bb7fe700 1 --2- 172.21.15.76:0/2303072833 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f40a4038fc0 0x7f40a403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.749+0000 7f40bb7fe700 1 --2- 172.21.15.76:0/2303072833 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f40a4038fc0 0x7f40a403b480 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f40b0006fd0 tx=0x7f40b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:13.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.753+0000 7f40b97fa700 1 -- 172.21.15.76:0/2303072833 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f40ac011e20 con 0x7f40bc105e80 2024-03-18T14:44:13.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.955+0000 7f40c2778700 1 -- 172.21.15.76:0/2303072833 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f40bc19f200 con 0x7f40bc105e80 2024-03-18T14:44:13.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.955+0000 7f40b97fa700 1 -- 172.21.15.76:0/2303072833 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f40ac02d910 con 0x7f40bc105e80 2024-03-18T14:44:13.653 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.957+0000 7f40c2778700 1 -- 172.21.15.76:0/2303072833 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f40a4038fc0 msgr2=0x7f40a403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:13.654 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.957+0000 7f40c2778700 1 --2- 172.21.15.76:0/2303072833 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f40a4038fc0 0x7f40a403b480 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f40b0006fd0 tx=0x7f40b0006e40 comp rx=0 tx=0).stop 2024-03-18T14:44:13.654 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.957+0000 7f40c2778700 1 -- 172.21.15.76:0/2303072833 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40bc105e80 msgr2=0x7f40bc19bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:13.654 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.957+0000 7f40c2778700 1 --2- 172.21.15.76:0/2303072833 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40bc105e80 0x7f40bc19bb40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0116a0 tx=0x7f40ac011ba0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.654 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.957+0000 7f40c2778700 1 -- 172.21.15.76:0/2303072833 shutdown_connections 2024-03-18T14:44:13.654 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.957+0000 7f40c2778700 1 --2- 172.21.15.76:0/2303072833 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f40a4038fc0 0x7f40a403b480 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.654 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.957+0000 7f40c2778700 1 --2- 172.21.15.76:0/2303072833 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40bc105e80 0x7f40bc19bb40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.654 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.957+0000 7f40c2778700 1 -- 172.21.15.76:0/2303072833 >> 172.21.15.76:0/2303072833 conn(0x7f40bc101790 msgr2=0x7f40bc102280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:13.654 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.957+0000 7f40c2778700 1 -- 172.21.15.76:0/2303072833 shutdown_connections 2024-03-18T14:44:13.654 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:12.957+0000 7f40c2778700 1 -- 172.21.15.76:0/2303072833 wait complete. 2024-03-18T14:44:15.012 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.057+0000 7f13e0082700 1 Processor -- start 2024-03-18T14:44:15.012 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.057+0000 7f13e0082700 1 -- start start 2024-03-18T14:44:15.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.057+0000 7f13e0082700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d8106490 0x7f13d81068b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:15.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.058+0000 7f13e0082700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f13d807bdf0 con 0x7f13d8106490 2024-03-18T14:44:15.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.058+0000 7f13dde1e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d8106490 0x7f13d81068b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:15.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.058+0000 7f13dde1e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d8106490 0x7f13d81068b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47940/0 (socket says 172.21.15.76:47940) 2024-03-18T14:44:15.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.058+0000 7f13dde1e700 1 -- 172.21.15.76:0/2453454857 learned_addr learned my addr 172.21.15.76:0/2453454857 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:15.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.058+0000 7f13dde1e700 1 -- 172.21.15.76:0/2453454857 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d8106df0 con 0x7f13d8106490 2024-03-18T14:44:15.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.058+0000 7f13dde1e700 1 --2- 172.21.15.76:0/2453454857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d8106490 0x7f13d81068b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f13c80098a0 tx=0x7f13c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a4b6bbcd207220 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:15.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.058+0000 7f13dce1c700 1 -- 172.21.15.76:0/2453454857 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13c8004160 con 0x7f13d8106490 2024-03-18T14:44:15.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.058+0000 7f13dce1c700 1 -- 172.21.15.76:0/2453454857 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13c8004300 con 0x7f13d8106490 2024-03-18T14:44:15.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.058+0000 7f13dce1c700 1 -- 172.21.15.76:0/2453454857 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13c8011b20 con 0x7f13d8106490 2024-03-18T14:44:15.013 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.059+0000 7f13e0082700 1 -- 172.21.15.76:0/2453454857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d8106490 msgr2=0x7f13d81068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:15.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.059+0000 7f13e0082700 1 --2- 172.21.15.76:0/2453454857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d8106490 0x7f13d81068b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f13c80098a0 tx=0x7f13c8009bb0 comp rx=0 tx=0).stop 2024-03-18T14:44:15.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.059+0000 7f13e0082700 1 -- 172.21.15.76:0/2453454857 shutdown_connections 2024-03-18T14:44:15.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.059+0000 7f13e0082700 1 --2- 172.21.15.76:0/2453454857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d8106490 0x7f13d81068b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:15.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.059+0000 7f13e0082700 1 -- 172.21.15.76:0/2453454857 >> 172.21.15.76:0/2453454857 conn(0x7f13d8102090 msgr2=0x7f13d81044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:15.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.059+0000 7f13e0082700 1 -- 172.21.15.76:0/2453454857 shutdown_connections 2024-03-18T14:44:15.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.059+0000 7f13e0082700 1 -- 172.21.15.76:0/2453454857 wait complete. 2024-03-18T14:44:15.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.060+0000 7f13e0082700 1 Processor -- start 2024-03-18T14:44:15.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.060+0000 7f13e0082700 1 -- start start 2024-03-18T14:44:15.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.060+0000 7f13e0082700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d8106490 0x7f13d819dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:15.014 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.060+0000 7f13e0082700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f13d8107ec0 con 0x7f13d8106490 2024-03-18T14:44:15.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.060+0000 7f13dde1e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d8106490 0x7f13d819dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:15.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.060+0000 7f13dde1e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d8106490 0x7f13d819dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47950/0 (socket says 172.21.15.76:47950) 2024-03-18T14:44:15.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.060+0000 7f13dde1e700 1 -- 172.21.15.76:0/961295728 learned_addr learned my addr 172.21.15.76:0/961295728 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:15.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.060+0000 7f13dde1e700 1 -- 172.21.15.76:0/961295728 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d819e1b0 con 0x7f13d8106490 2024-03-18T14:44:15.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.060+0000 7f13dde1e700 1 --2- 172.21.15.76:0/961295728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d8106490 0x7f13d819dc70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f13c80117e0 tx=0x7f13c8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:15.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.060+0000 7f13d6ffd700 1 -- 172.21.15.76:0/961295728 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13c8003ba0 con 0x7f13d8106490 2024-03-18T14:44:15.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.060+0000 7f13d6ffd700 1 -- 172.21.15.76:0/961295728 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13c801a910 con 0x7f13d8106490 2024-03-18T14:44:15.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.060+0000 7f13e0082700 1 -- 172.21.15.76:0/961295728 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d819e440 con 0x7f13d8106490 2024-03-18T14:44:15.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.060+0000 7f13d6ffd700 1 -- 172.21.15.76:0/961295728 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13c80248f0 con 0x7f13d8106490 2024-03-18T14:44:15.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.060+0000 7f13e0082700 1 -- 172.21.15.76:0/961295728 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d819e8a0 con 0x7f13d8106490 2024-03-18T14:44:15.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.061+0000 7f13d6ffd700 1 -- 172.21.15.76:0/961295728 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f13c8022020 con 0x7f13d8106490 2024-03-18T14:44:15.015 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.061+0000 7f13d6ffd700 1 --2- 172.21.15.76:0/961295728 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f13c0039070 0x7f13c003b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:15.016 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.062+0000 7f13d6ffd700 1 -- 172.21.15.76:0/961295728 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f13c804d440 con 0x7f13d8106490 2024-03-18T14:44:15.016 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.062+0000 7f13e0082700 1 -- 172.21.15.76:0/961295728 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13d8063900 con 0x7f13d8106490 2024-03-18T14:44:15.016 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.062+0000 7f13dd61d700 1 --2- 172.21.15.76:0/961295728 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f13c0039070 0x7f13c003b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:15.016 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.062+0000 7f13dd61d700 1 --2- 172.21.15.76:0/961295728 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f13c0039070 0x7f13c003b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f13cc006fd0 tx=0x7f13cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:15.016 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.067+0000 7f13d6ffd700 1 -- 172.21.15.76:0/961295728 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f13c80172b0 con 0x7f13d8106490 2024-03-18T14:44:15.016 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.375+0000 7f13e0082700 1 -- 172.21.15.76:0/961295728 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f13d8062ee0 con 0x7f13d8106490 2024-03-18T14:44:15.016 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.377+0000 7f13d6ffd700 1 -- 172.21.15.76:0/961295728 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f13c8032030 con 0x7f13d8106490 2024-03-18T14:44:15.016 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.380+0000 7f13e0082700 1 -- 172.21.15.76:0/961295728 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f13c0039070 msgr2=0x7f13c003b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:15.016 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.380+0000 7f13e0082700 1 --2- 172.21.15.76:0/961295728 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f13c0039070 0x7f13c003b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f13cc006fd0 tx=0x7f13cc006e40 comp rx=0 tx=0).stop 2024-03-18T14:44:15.016 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.380+0000 7f13e0082700 1 -- 172.21.15.76:0/961295728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d8106490 msgr2=0x7f13d819dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:15.016 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.380+0000 7f13e0082700 1 --2- 172.21.15.76:0/961295728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d8106490 0x7f13d819dc70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f13c80117e0 tx=0x7f13c8011ba0 comp rx=0 tx=0).stop 2024-03-18T14:44:15.017 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.380+0000 7f13e0082700 1 -- 172.21.15.76:0/961295728 shutdown_connections 2024-03-18T14:44:15.017 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.380+0000 7f13e0082700 1 --2- 172.21.15.76:0/961295728 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f13c0039070 0x7f13c003b530 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:15.017 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.380+0000 7f13e0082700 1 --2- 172.21.15.76:0/961295728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d8106490 0x7f13d819dc70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:15.017 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.380+0000 7f13e0082700 1 -- 172.21.15.76:0/961295728 >> 172.21.15.76:0/961295728 conn(0x7f13d8102090 msgr2=0x7f13d81948b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:15.017 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.380+0000 7f13e0082700 1 -- 172.21.15.76:0/961295728 shutdown_connections 2024-03-18T14:44:15.017 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-18T14:44:14.380+0000 7f13e0082700 1 -- 172.21.15.76:0/961295728 wait complete. 2024-03-18T14:44:15.017 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-18T14:44:15.017 INFO:teuthology.orchestra.run.smithi076.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-18T14:44:15.017 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:44:15.017 INFO:teuthology.orchestra.run.smithi076.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-18T14:44:15.017 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:44:15.017 INFO:teuthology.orchestra.run.smithi076.stdout:Or, if you are only running a single cluster on this host: 2024-03-18T14:44:15.017 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:44:15.018 INFO:teuthology.orchestra.run.smithi076.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-18T14:44:15.018 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:44:15.018 INFO:teuthology.orchestra.run.smithi076.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-18T14:44:15.018 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:44:15.018 INFO:teuthology.orchestra.run.smithi076.stdout: ceph telemetry on 2024-03-18T14:44:15.018 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:44:15.018 INFO:teuthology.orchestra.run.smithi076.stdout:For more information see: 2024-03-18T14:44:15.018 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:44:15.018 INFO:teuthology.orchestra.run.smithi076.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-18T14:44:15.018 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:44:15.018 INFO:teuthology.orchestra.run.smithi076.stdout:Bootstrap complete. 2024-03-18T14:44:15.051 INFO:tasks.cephadm:Fetching config... 2024-03-18T14:44:15.051 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T14:44:15.051 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-18T14:44:15.066 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-18T14:44:15.066 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T14:44:15.066 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-18T14:44:15.122 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-18T14:44:15.122 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T14:44:15.122 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-18T14:44:15.190 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-18T14:44:15.190 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T14:44:15.190 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-18T14:44:15.245 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-18T14:44:15.245 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRnyV/3Ny3N7Baa21i7Z5gwNkbknoGqURhlqBmX1Q61L/ivn6U9pSk8IDaaqbdLbCw0FQzJAGhWAJC+COQVzWX4BzJbhtDBwHZjcHa06pfXtzBfWinXCXYy3iF6hQ9FXWIWtvraBUBb0akuPiyk2k6sSHTnxwPGhw5661FFbllcefPimf16C02MYx3EYQe6zP2vXJ2h7x1zHe8JT/kmokWBEHBIR8Jmhyytqp/kMAAVE5kg9rIbNSwR7kP30TyZR7TVDxK7o6SnBIwpArHx5K3QcSfetQHBbzlEEaqEjxZI6vZCqzmvzJ+/hnjZGeuPDptQiTZIYC8JubkTNpYE9jt5razrffXsp3EV0lW8kAbLLUcRSyTIKj8Gw7zjPGbpGVfhWtkylcIKYIhiaLBgr4FfCySd05Yux/pHx7Zhbss//m1lPtiRHtauXU1yyrV7wzmGdl7IX42EL0aoLQEaTQSZcCdg1g1IvkHvu1J3gHZ+2Q/fvKf2c2LZpzAZ62kdRU= ceph-9cc83b72-e535-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T14:44:15.323 INFO:teuthology.orchestra.run.smithi076.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRnyV/3Ny3N7Baa21i7Z5gwNkbknoGqURhlqBmX1Q61L/ivn6U9pSk8IDaaqbdLbCw0FQzJAGhWAJC+COQVzWX4BzJbhtDBwHZjcHa06pfXtzBfWinXCXYy3iF6hQ9FXWIWtvraBUBb0akuPiyk2k6sSHTnxwPGhw5661FFbllcefPimf16C02MYx3EYQe6zP2vXJ2h7x1zHe8JT/kmokWBEHBIR8Jmhyytqp/kMAAVE5kg9rIbNSwR7kP30TyZR7TVDxK7o6SnBIwpArHx5K3QcSfetQHBbzlEEaqEjxZI6vZCqzmvzJ+/hnjZGeuPDptQiTZIYC8JubkTNpYE9jt5razrffXsp3EV0lW8kAbLLUcRSyTIKj8Gw7zjPGbpGVfhWtkylcIKYIhiaLBgr4FfCySd05Yux/pHx7Zhbss//m1lPtiRHtauXU1yyrV7wzmGdl7IX42EL0aoLQEaTQSZcCdg1g1IvkHvu1J3gHZ+2Q/fvKf2c2LZpzAZ62kdRU= ceph-9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:44:15.337 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRnyV/3Ny3N7Baa21i7Z5gwNkbknoGqURhlqBmX1Q61L/ivn6U9pSk8IDaaqbdLbCw0FQzJAGhWAJC+COQVzWX4BzJbhtDBwHZjcHa06pfXtzBfWinXCXYy3iF6hQ9FXWIWtvraBUBb0akuPiyk2k6sSHTnxwPGhw5661FFbllcefPimf16C02MYx3EYQe6zP2vXJ2h7x1zHe8JT/kmokWBEHBIR8Jmhyytqp/kMAAVE5kg9rIbNSwR7kP30TyZR7TVDxK7o6SnBIwpArHx5K3QcSfetQHBbzlEEaqEjxZI6vZCqzmvzJ+/hnjZGeuPDptQiTZIYC8JubkTNpYE9jt5razrffXsp3EV0lW8kAbLLUcRSyTIKj8Gw7zjPGbpGVfhWtkylcIKYIhiaLBgr4FfCySd05Yux/pHx7Zhbss//m1lPtiRHtauXU1yyrV7wzmGdl7IX42EL0aoLQEaTQSZcCdg1g1IvkHvu1J3gHZ+2Q/fvKf2c2LZpzAZ62kdRU= ceph-9cc83b72-e535-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T14:44:15.383 INFO:teuthology.orchestra.run.smithi077.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRnyV/3Ny3N7Baa21i7Z5gwNkbknoGqURhlqBmX1Q61L/ivn6U9pSk8IDaaqbdLbCw0FQzJAGhWAJC+COQVzWX4BzJbhtDBwHZjcHa06pfXtzBfWinXCXYy3iF6hQ9FXWIWtvraBUBb0akuPiyk2k6sSHTnxwPGhw5661FFbllcefPimf16C02MYx3EYQe6zP2vXJ2h7x1zHe8JT/kmokWBEHBIR8Jmhyytqp/kMAAVE5kg9rIbNSwR7kP30TyZR7TVDxK7o6SnBIwpArHx5K3QcSfetQHBbzlEEaqEjxZI6vZCqzmvzJ+/hnjZGeuPDptQiTZIYC8JubkTNpYE9jt5razrffXsp3EV0lW8kAbLLUcRSyTIKj8Gw7zjPGbpGVfhWtkylcIKYIhiaLBgr4FfCySd05Yux/pHx7Zhbss//m1lPtiRHtauXU1yyrV7wzmGdl7IX42EL0aoLQEaTQSZcCdg1g1IvkHvu1J3gHZ+2Q/fvKf2c2LZpzAZ62kdRU= ceph-9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:44:15.399 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRnyV/3Ny3N7Baa21i7Z5gwNkbknoGqURhlqBmX1Q61L/ivn6U9pSk8IDaaqbdLbCw0FQzJAGhWAJC+COQVzWX4BzJbhtDBwHZjcHa06pfXtzBfWinXCXYy3iF6hQ9FXWIWtvraBUBb0akuPiyk2k6sSHTnxwPGhw5661FFbllcefPimf16C02MYx3EYQe6zP2vXJ2h7x1zHe8JT/kmokWBEHBIR8Jmhyytqp/kMAAVE5kg9rIbNSwR7kP30TyZR7TVDxK7o6SnBIwpArHx5K3QcSfetQHBbzlEEaqEjxZI6vZCqzmvzJ+/hnjZGeuPDptQiTZIYC8JubkTNpYE9jt5razrffXsp3EV0lW8kAbLLUcRSyTIKj8Gw7zjPGbpGVfhWtkylcIKYIhiaLBgr4FfCySd05Yux/pHx7Zhbss//m1lPtiRHtauXU1yyrV7wzmGdl7IX42EL0aoLQEaTQSZcCdg1g1IvkHvu1J3gHZ+2Q/fvKf2c2LZpzAZ62kdRU= ceph-9cc83b72-e535-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T14:44:15.444 INFO:teuthology.orchestra.run.smithi174.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRnyV/3Ny3N7Baa21i7Z5gwNkbknoGqURhlqBmX1Q61L/ivn6U9pSk8IDaaqbdLbCw0FQzJAGhWAJC+COQVzWX4BzJbhtDBwHZjcHa06pfXtzBfWinXCXYy3iF6hQ9FXWIWtvraBUBb0akuPiyk2k6sSHTnxwPGhw5661FFbllcefPimf16C02MYx3EYQe6zP2vXJ2h7x1zHe8JT/kmokWBEHBIR8Jmhyytqp/kMAAVE5kg9rIbNSwR7kP30TyZR7TVDxK7o6SnBIwpArHx5K3QcSfetQHBbzlEEaqEjxZI6vZCqzmvzJ+/hnjZGeuPDptQiTZIYC8JubkTNpYE9jt5razrffXsp3EV0lW8kAbLLUcRSyTIKj8Gw7zjPGbpGVfhWtkylcIKYIhiaLBgr4FfCySd05Yux/pHx7Zhbss//m1lPtiRHtauXU1yyrV7wzmGdl7IX42EL0aoLQEaTQSZcCdg1g1IvkHvu1J3gHZ+2Q/fvKf2c2LZpzAZ62kdRU= ceph-9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:44:15.459 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-18T14:44:15.635 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:15 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/961295728' entity='client.admin' 2024-03-18T14:44:15.709 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:44:18.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.195+0000 7f68bc4f3700 1 -- 172.21.15.76:0/609437042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f68b4102790 msgr2=0x7f68b4104b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:18.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.195+0000 7f68bc4f3700 1 --2- 172.21.15.76:0/609437042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f68b4102790 0x7f68b4104b80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f68b0009910 tx=0x7f68b0009c20 comp rx=0 tx=0).stop 2024-03-18T14:44:18.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.195+0000 7f68bc4f3700 1 -- 172.21.15.76:0/609437042 shutdown_connections 2024-03-18T14:44:18.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.195+0000 7f68bc4f3700 1 --2- 172.21.15.76:0/609437042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f68b4102790 0x7f68b4104b80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:18.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.195+0000 7f68bc4f3700 1 -- 172.21.15.76:0/609437042 >> 172.21.15.76:0/609437042 conn(0x7f68b40fc1a0 msgr2=0x7f68b40fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:18.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.195+0000 7f68bc4f3700 1 -- 172.21.15.76:0/609437042 shutdown_connections 2024-03-18T14:44:18.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.195+0000 7f68bc4f3700 1 -- 172.21.15.76:0/609437042 wait complete. 2024-03-18T14:44:18.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.196+0000 7f68bc4f3700 1 Processor -- start 2024-03-18T14:44:18.199 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:17 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:18.199 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:17 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:18.199 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:17 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:44:18.199 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:17 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:18.199 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:17 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:44:18.199 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:17 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:18.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.196+0000 7f68bc4f3700 1 -- start start 2024-03-18T14:44:18.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.196+0000 7f68bc4f3700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f68b4102790 0x7f68b4197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:18.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.196+0000 7f68bc4f3700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f68b4107d30 con 0x7f68b4102790 2024-03-18T14:44:18.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.196+0000 7f68ba28f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f68b4102790 0x7f68b4197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:18.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.196+0000 7f68ba28f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f68b4102790 0x7f68b4197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47962/0 (socket says 172.21.15.76:47962) 2024-03-18T14:44:18.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.196+0000 7f68ba28f700 1 -- 172.21.15.76:0/1979861335 learned_addr learned my addr 172.21.15.76:0/1979861335 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:18.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.197+0000 7f68ba28f700 1 -- 172.21.15.76:0/1979861335 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68b41978b0 con 0x7f68b4102790 2024-03-18T14:44:18.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.197+0000 7f68ba28f700 1 --2- 172.21.15.76:0/1979861335 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f68b4102790 0x7f68b4197370 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f68b0004fa0 tx=0x7f68b0005080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:18.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.197+0000 7f68ab7fe700 1 -- 172.21.15.76:0/1979861335 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68b0027070 con 0x7f68b4102790 2024-03-18T14:44:18.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.197+0000 7f68ab7fe700 1 -- 172.21.15.76:0/1979861335 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68b000be40 con 0x7f68b4102790 2024-03-18T14:44:18.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.197+0000 7f68bc4f3700 1 -- 172.21.15.76:0/1979861335 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68b4197ba0 con 0x7f68b4102790 2024-03-18T14:44:18.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.197+0000 7f68bc4f3700 1 -- 172.21.15.76:0/1979861335 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68b419a830 con 0x7f68b4102790 2024-03-18T14:44:18.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.197+0000 7f68ab7fe700 1 -- 172.21.15.76:0/1979861335 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68b000f460 con 0x7f68b4102790 2024-03-18T14:44:18.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.198+0000 7f68ab7fe700 1 -- 172.21.15.76:0/1979861335 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f68b002a070 con 0x7f68b4102790 2024-03-18T14:44:18.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.199+0000 7f68bc4f3700 1 -- 172.21.15.76:0/1979861335 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68b41084b0 con 0x7f68b4102790 2024-03-18T14:44:18.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.199+0000 7f68ab7fe700 1 --2- 172.21.15.76:0/1979861335 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f68a0039020 0x7f68a003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:18.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.199+0000 7f68ab7fe700 1 -- 172.21.15.76:0/1979861335 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f68b001e030 con 0x7f68b4102790 2024-03-18T14:44:18.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.199+0000 7f68b9a8e700 1 --2- 172.21.15.76:0/1979861335 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f68a0039020 0x7f68a003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:18.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.199+0000 7f68b9a8e700 1 --2- 172.21.15.76:0/1979861335 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f68a0039020 0x7f68a003b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f68a4006fd0 tx=0x7f68a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:18.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.204+0000 7f68ab7fe700 1 -- 172.21.15.76:0/1979861335 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f68b0013070 con 0x7f68b4102790 2024-03-18T14:44:18.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.430+0000 7f68bc4f3700 1 -- 172.21.15.76:0/1979861335 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f68b402dd60 con 0x7f68b4102790 2024-03-18T14:44:18.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.432+0000 7f68ab7fe700 1 -- 172.21.15.76:0/1979861335 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f68b002b3b0 con 0x7f68b4102790 2024-03-18T14:44:18.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.435+0000 7f68bc4f3700 1 -- 172.21.15.76:0/1979861335 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f68a0039020 msgr2=0x7f68a003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:18.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.435+0000 7f68bc4f3700 1 --2- 172.21.15.76:0/1979861335 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f68a0039020 0x7f68a003b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f68a4006fd0 tx=0x7f68a4006e40 comp rx=0 tx=0).stop 2024-03-18T14:44:18.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.435+0000 7f68bc4f3700 1 -- 172.21.15.76:0/1979861335 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f68b4102790 msgr2=0x7f68b4197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:18.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.435+0000 7f68bc4f3700 1 --2- 172.21.15.76:0/1979861335 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f68b4102790 0x7f68b4197370 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f68b0004fa0 tx=0x7f68b0005080 comp rx=0 tx=0).stop 2024-03-18T14:44:18.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.435+0000 7f68bc4f3700 1 -- 172.21.15.76:0/1979861335 shutdown_connections 2024-03-18T14:44:18.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.435+0000 7f68bc4f3700 1 --2- 172.21.15.76:0/1979861335 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f68a0039020 0x7f68a003b4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:18.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.435+0000 7f68bc4f3700 1 --2- 172.21.15.76:0/1979861335 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f68b4102790 0x7f68b4197370 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:18.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.435+0000 7f68bc4f3700 1 -- 172.21.15.76:0/1979861335 >> 172.21.15.76:0/1979861335 conn(0x7f68b40fc1a0 msgr2=0x7f68b40fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:18.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.435+0000 7f68bc4f3700 1 -- 172.21.15.76:0/1979861335 shutdown_connections 2024-03-18T14:44:18.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:18.435+0000 7f68bc4f3700 1 -- 172.21.15.76:0/1979861335 wait complete. 2024-03-18T14:44:19.152 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-18T14:44:19.152 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-18T14:44:19.619 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:44:19.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:19 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1979861335' entity='client.admin' 2024-03-18T14:44:19.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:44:19.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:21.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.010+0000 7fb401d71700 1 -- 172.21.15.76:0/2302502262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc101d20 msgr2=0x7fb3fc102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.010+0000 7fb401d71700 1 --2- 172.21.15.76:0/2302502262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc101d20 0x7fb3fc102140 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009910 tx=0x7fb3ec009c20 comp rx=0 tx=0).stop 2024-03-18T14:44:21.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.010+0000 7fb401d71700 1 -- 172.21.15.76:0/2302502262 shutdown_connections 2024-03-18T14:44:21.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.010+0000 7fb401d71700 1 --2- 172.21.15.76:0/2302502262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc101d20 0x7fb3fc102140 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.010+0000 7fb401d71700 1 -- 172.21.15.76:0/2302502262 >> 172.21.15.76:0/2302502262 conn(0x7fb3fc0fd2c0 msgr2=0x7fb3fc0ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:21.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.010+0000 7fb401d71700 1 -- 172.21.15.76:0/2302502262 shutdown_connections 2024-03-18T14:44:21.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.010+0000 7fb401d71700 1 -- 172.21.15.76:0/2302502262 wait complete. 2024-03-18T14:44:21.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.010+0000 7fb401d71700 1 Processor -- start 2024-03-18T14:44:21.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.011+0000 7fb401d71700 1 -- start start 2024-03-18T14:44:21.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.011+0000 7fb401d71700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc101d20 0x7fb3fc1974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.011+0000 7fb401d71700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb3fc107db0 con 0x7fb3fc101d20 2024-03-18T14:44:21.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.011+0000 7fb3fb7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc101d20 0x7fb3fc1974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.011+0000 7fb3fb7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc101d20 0x7fb3fc1974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60116/0 (socket says 172.21.15.76:60116) 2024-03-18T14:44:21.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.011+0000 7fb3fb7fe700 1 -- 172.21.15.76:0/2050478174 learned_addr learned my addr 172.21.15.76:0/2050478174 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:21.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.011+0000 7fb3fb7fe700 1 -- 172.21.15.76:0/2050478174 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3fc1979f0 con 0x7fb3fc101d20 2024-03-18T14:44:21.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.011+0000 7fb3fb7fe700 1 --2- 172.21.15.76:0/2050478174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc101d20 0x7fb3fc1974b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec006010 tx=0x7fb3ec005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:21.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.012+0000 7fb3f8ff9700 1 -- 172.21.15.76:0/2050478174 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3ec027070 con 0x7fb3fc101d20 2024-03-18T14:44:21.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.012+0000 7fb3f8ff9700 1 -- 172.21.15.76:0/2050478174 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3ec00be40 con 0x7fb3fc101d20 2024-03-18T14:44:21.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.012+0000 7fb401d71700 1 -- 172.21.15.76:0/2050478174 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3fc197c80 con 0x7fb3fc101d20 2024-03-18T14:44:21.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.012+0000 7fb3f8ff9700 1 -- 172.21.15.76:0/2050478174 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3ec00f460 con 0x7fb3fc101d20 2024-03-18T14:44:21.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.012+0000 7fb401d71700 1 -- 172.21.15.76:0/2050478174 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3fc19a920 con 0x7fb3fc101d20 2024-03-18T14:44:21.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.012+0000 7fb3f8ff9700 1 -- 172.21.15.76:0/2050478174 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fb3ec02a070 con 0x7fb3fc101d20 2024-03-18T14:44:21.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.013+0000 7fb3f8ff9700 1 --2- 172.21.15.76:0/2050478174 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb3e4038fc0 0x7fb3e403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.013+0000 7fb401d71700 1 -- 172.21.15.76:0/2050478174 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3fc108530 con 0x7fb3fc101d20 2024-03-18T14:44:21.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.013+0000 7fb3f8ff9700 1 -- 172.21.15.76:0/2050478174 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb3ec01e030 con 0x7fb3fc101d20 2024-03-18T14:44:21.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.013+0000 7fb3faffd700 1 --2- 172.21.15.76:0/2050478174 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb3e4038fc0 0x7fb3e403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.013+0000 7fb3faffd700 1 --2- 172.21.15.76:0/2050478174 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb3e4038fc0 0x7fb3e403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0006fd0 tx=0x7fb3f0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:21.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.018+0000 7fb3f8ff9700 1 -- 172.21.15.76:0/2050478174 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb3ec013070 con 0x7fb3fc101d20 2024-03-18T14:44:21.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.219+0000 7fb401d71700 1 -- 172.21.15.76:0/2050478174 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fb3fc02dc10 con 0x7fb3e4038fc0 2024-03-18T14:44:21.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.223+0000 7fb3f8ff9700 1 -- 172.21.15.76:0/2050478174 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb3fc02dc10 con 0x7fb3e4038fc0 2024-03-18T14:44:21.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.225+0000 7fb401d71700 1 -- 172.21.15.76:0/2050478174 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb3e4038fc0 msgr2=0x7fb3e403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.225+0000 7fb401d71700 1 --2- 172.21.15.76:0/2050478174 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb3e4038fc0 0x7fb3e403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0006fd0 tx=0x7fb3f0006e40 comp rx=0 tx=0).stop 2024-03-18T14:44:21.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.225+0000 7fb401d71700 1 -- 172.21.15.76:0/2050478174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc101d20 msgr2=0x7fb3fc1974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.225+0000 7fb401d71700 1 --2- 172.21.15.76:0/2050478174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc101d20 0x7fb3fc1974b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec006010 tx=0x7fb3ec005050 comp rx=0 tx=0).stop 2024-03-18T14:44:21.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.225+0000 7fb401d71700 1 -- 172.21.15.76:0/2050478174 shutdown_connections 2024-03-18T14:44:21.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.225+0000 7fb401d71700 1 --2- 172.21.15.76:0/2050478174 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb3e4038fc0 0x7fb3e403b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.225+0000 7fb401d71700 1 --2- 172.21.15.76:0/2050478174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc101d20 0x7fb3fc1974b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.225+0000 7fb401d71700 1 -- 172.21.15.76:0/2050478174 >> 172.21.15.76:0/2050478174 conn(0x7fb3fc0fd2c0 msgr2=0x7fb3fc0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:21.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.225+0000 7fb401d71700 1 -- 172.21.15.76:0/2050478174 shutdown_connections 2024-03-18T14:44:21.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:21.225+0000 7fb401d71700 1 -- 172.21.15.76:0/2050478174 wait complete. 2024-03-18T14:44:22.062 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi077 2024-03-18T14:44:22.062 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-18T14:44:22.062 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/etc/ceph/ceph.conf 2024-03-18T14:44:22.081 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-18T14:44:22.081 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:44:22.138 INFO:tasks.cephadm:Adding host smithi077 to orchestrator... 2024-03-18T14:44:22.138 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch host add smithi077 2024-03-18T14:44:22.374 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:44:22.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:22 smithi076 ceph-mon[32205]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:44:22.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:22.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:44:22.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:44:22.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:44:22.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:22 smithi076 ceph-mon[32205]: Updating smithi076:/etc/ceph/ceph.conf 2024-03-18T14:44:22.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:22.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:22.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:23.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:23 smithi076 ceph-mon[32205]: Updating smithi076:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:44:23.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:23 smithi076 ceph-mon[32205]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:44:23.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:23 smithi076 ceph-mon[32205]: Updating smithi076:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T14:44:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.819+0000 7f7724ca9700 1 -- 172.21.15.76:0/1597764968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7720106ce0 msgr2=0x7f77201070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.819+0000 7f7724ca9700 1 --2- 172.21.15.76:0/1597764968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7720106ce0 0x7f77201070c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7710009910 tx=0x7f7710009c20 comp rx=0 tx=0).stop 2024-03-18T14:44:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.819+0000 7f7724ca9700 1 -- 172.21.15.76:0/1597764968 shutdown_connections 2024-03-18T14:44:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.819+0000 7f7724ca9700 1 --2- 172.21.15.76:0/1597764968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7720106ce0 0x7f77201070c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.819+0000 7f7724ca9700 1 -- 172.21.15.76:0/1597764968 >> 172.21.15.76:0/1597764968 conn(0x7f77200fbe20 msgr2=0x7f77200fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.819+0000 7f7724ca9700 1 -- 172.21.15.76:0/1597764968 shutdown_connections 2024-03-18T14:44:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.819+0000 7f7724ca9700 1 -- 172.21.15.76:0/1597764968 wait complete. 2024-03-18T14:44:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.819+0000 7f7724ca9700 1 Processor -- start 2024-03-18T14:44:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.820+0000 7f7724ca9700 1 -- start start 2024-03-18T14:44:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.820+0000 7f7724ca9700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7720106ce0 0x7f772019d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.820+0000 7f7724ca9700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f772010c440 con 0x7f7720106ce0 2024-03-18T14:44:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.820+0000 7f771e59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7720106ce0 0x7f772019d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.820+0000 7f771e59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7720106ce0 0x7f772019d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60132/0 (socket says 172.21.15.76:60132) 2024-03-18T14:44:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.820+0000 7f771e59c700 1 -- 172.21.15.76:0/3085936690 learned_addr learned my addr 172.21.15.76:0/3085936690 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.820+0000 7f771e59c700 1 -- 172.21.15.76:0/3085936690 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f772019dad0 con 0x7f7720106ce0 2024-03-18T14:44:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.820+0000 7f771e59c700 1 --2- 172.21.15.76:0/3085936690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7720106ce0 0x7f772019d590 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7710005850 tx=0x7f7710005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.821+0000 7f7717fff700 1 -- 172.21.15.76:0/3085936690 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7710026070 con 0x7f7720106ce0 2024-03-18T14:44:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.821+0000 7f7717fff700 1 -- 172.21.15.76:0/3085936690 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f771000bc30 con 0x7f7720106ce0 2024-03-18T14:44:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.821+0000 7f7724ca9700 1 -- 172.21.15.76:0/3085936690 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f772019dd60 con 0x7f7720106ce0 2024-03-18T14:44:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.821+0000 7f7717fff700 1 -- 172.21.15.76:0/3085936690 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7710003a20 con 0x7f7720106ce0 2024-03-18T14:44:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.821+0000 7f7724ca9700 1 -- 172.21.15.76:0/3085936690 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7720197660 con 0x7f7720106ce0 2024-03-18T14:44:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.822+0000 7f7717fff700 1 -- 172.21.15.76:0/3085936690 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f7710029070 con 0x7f7720106ce0 2024-03-18T14:44:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.822+0000 7f7724ca9700 1 -- 172.21.15.76:0/3085936690 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7720103420 con 0x7f7720106ce0 2024-03-18T14:44:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.822+0000 7f7717fff700 1 --2- 172.21.15.76:0/3085936690 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f770c038fc0 0x7f770c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.822+0000 7f7717fff700 1 -- 172.21.15.76:0/3085936690 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f771001e070 con 0x7f7720106ce0 2024-03-18T14:44:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.822+0000 7f7715d9b700 1 --2- 172.21.15.76:0/3085936690 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f770c038fc0 0x7f770c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.822+0000 7f7715d9b700 1 --2- 172.21.15.76:0/3085936690 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f770c038fc0 0x7f770c03b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7708006fd0 tx=0x7f7708006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:23.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:23.827+0000 7f7717fff700 1 -- 172.21.15.76:0/3085936690 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7710013070 con 0x7f7720106ce0 2024-03-18T14:44:24.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:24.025+0000 7f7724ca9700 1 -- 172.21.15.76:0/3085936690 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi077", "target": ["mon-mgr", ""]}) -- 0x7f7720198b90 con 0x7f770c038fc0 2024-03-18T14:44:24.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:24 smithi076 ceph-mon[32205]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi077", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:44:25.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:25 smithi076 ceph-mon[32205]: Deploying cephadm binary to smithi077 2024-03-18T14:44:27.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:27 smithi076 ceph-mon[32205]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:28.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:28.219+0000 7f7717fff700 1 -- 172.21.15.76:0/3085936690 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f7720198b90 con 0x7f770c038fc0 2024-03-18T14:44:28.222 INFO:teuthology.orchestra.run.smithi076.stdout:Added host 'smithi077' with addr '172.21.15.77' 2024-03-18T14:44:28.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:28.222+0000 7f7724ca9700 1 -- 172.21.15.76:0/3085936690 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f770c038fc0 msgr2=0x7f770c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:28.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:28.222+0000 7f7724ca9700 1 --2- 172.21.15.76:0/3085936690 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f770c038fc0 0x7f770c03b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7708006fd0 tx=0x7f7708006e40 comp rx=0 tx=0).stop 2024-03-18T14:44:28.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:28.222+0000 7f7724ca9700 1 -- 172.21.15.76:0/3085936690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7720106ce0 msgr2=0x7f772019d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:28.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:28.222+0000 7f7724ca9700 1 --2- 172.21.15.76:0/3085936690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7720106ce0 0x7f772019d590 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7710005850 tx=0x7f7710005030 comp rx=0 tx=0).stop 2024-03-18T14:44:28.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:28.222+0000 7f7724ca9700 1 -- 172.21.15.76:0/3085936690 shutdown_connections 2024-03-18T14:44:28.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:28.222+0000 7f7724ca9700 1 --2- 172.21.15.76:0/3085936690 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f770c038fc0 0x7f770c03b480 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:28.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:28.222+0000 7f7724ca9700 1 --2- 172.21.15.76:0/3085936690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7720106ce0 0x7f772019d590 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:28.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:28.222+0000 7f7724ca9700 1 -- 172.21.15.76:0/3085936690 >> 172.21.15.76:0/3085936690 conn(0x7f77200fbe20 msgr2=0x7f77201097e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:28.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:28.223+0000 7f7724ca9700 1 -- 172.21.15.76:0/3085936690 shutdown_connections 2024-03-18T14:44:28.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:28.223+0000 7f7724ca9700 1 -- 172.21.15.76:0/3085936690 wait complete. 2024-03-18T14:44:28.770 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-18T14:44:29.021 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:44:29.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:29 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:29.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:29 smithi076 ceph-mon[32205]: Added host smithi077 2024-03-18T14:44:29.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:29 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:44:29.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:29 smithi076 ceph-mon[32205]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:29.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:29 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:30.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.373+0000 7f1565904700 1 -- 172.21.15.76:0/554874603 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1560101f10 msgr2=0x7f15601022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:30.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.373+0000 7f1565904700 1 --2- 172.21.15.76:0/554874603 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1560101f10 0x7f15601022f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f155800b0a0 tx=0x7f155800b3b0 comp rx=0 tx=0).stop 2024-03-18T14:44:30.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.373+0000 7f1565904700 1 -- 172.21.15.76:0/554874603 shutdown_connections 2024-03-18T14:44:30.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.373+0000 7f1565904700 1 --2- 172.21.15.76:0/554874603 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1560101f10 0x7f15601022f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:30.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.373+0000 7f1565904700 1 -- 172.21.15.76:0/554874603 >> 172.21.15.76:0/554874603 conn(0x7f15600fd7e0 msgr2=0x7f15600ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:30.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.373+0000 7f1565904700 1 -- 172.21.15.76:0/554874603 shutdown_connections 2024-03-18T14:44:30.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.373+0000 7f1565904700 1 -- 172.21.15.76:0/554874603 wait complete. 2024-03-18T14:44:30.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.374+0000 7f1565904700 1 Processor -- start 2024-03-18T14:44:30.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.374+0000 7f1565904700 1 -- start start 2024-03-18T14:44:30.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.374+0000 7f1565904700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1560101f10 0x7f1560197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:30.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.374+0000 7f1565904700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f156010c2e0 con 0x7f1560101f10 2024-03-18T14:44:30.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.374+0000 7f155effd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1560101f10 0x7f1560197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:30.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.375+0000 7f155effd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1560101f10 0x7f1560197710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45496/0 (socket says 172.21.15.76:45496) 2024-03-18T14:44:30.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.375+0000 7f155effd700 1 -- 172.21.15.76:0/810048561 learned_addr learned my addr 172.21.15.76:0/810048561 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:30.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.375+0000 7f155effd700 1 -- 172.21.15.76:0/810048561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1560197c50 con 0x7f1560101f10 2024-03-18T14:44:30.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.375+0000 7f155effd700 1 --2- 172.21.15.76:0/810048561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1560101f10 0x7f1560197710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1558015040 tx=0x7f1558003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.375+0000 7f1564902700 1 -- 172.21.15.76:0/810048561 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1558017040 con 0x7f1560101f10 2024-03-18T14:44:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.375+0000 7f1565904700 1 -- 172.21.15.76:0/810048561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1560197ee0 con 0x7f1560101f10 2024-03-18T14:44:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.375+0000 7f1564902700 1 -- 172.21.15.76:0/810048561 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1558022070 con 0x7f1560101f10 2024-03-18T14:44:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.375+0000 7f1565904700 1 -- 172.21.15.76:0/810048561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f156019bfe0 con 0x7f1560101f10 2024-03-18T14:44:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.375+0000 7f1564902700 1 -- 172.21.15.76:0/810048561 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f155801d5d0 con 0x7f1560101f10 2024-03-18T14:44:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.376+0000 7f1564902700 1 -- 172.21.15.76:0/810048561 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f155801d8d0 con 0x7f1560101f10 2024-03-18T14:44:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.376+0000 7f1565904700 1 -- 172.21.15.76:0/810048561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1540005330 con 0x7f1560101f10 2024-03-18T14:44:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.376+0000 7f1564902700 1 --2- 172.21.15.76:0/810048561 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f154c038fc0 0x7f154c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.376+0000 7f1564902700 1 -- 172.21.15.76:0/810048561 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f155800f050 con 0x7f1560101f10 2024-03-18T14:44:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.376+0000 7f155e7fc700 1 --2- 172.21.15.76:0/810048561 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f154c038fc0 0x7f154c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.377+0000 7f155e7fc700 1 --2- 172.21.15.76:0/810048561 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f154c038fc0 0x7f154c03b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1548006fd0 tx=0x7f1548006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:30.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.381+0000 7f1564902700 1 -- 172.21.15.76:0/810048561 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1558007e20 con 0x7f1560101f10 2024-03-18T14:44:30.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.584+0000 7f1565904700 1 -- 172.21.15.76:0/810048561 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1540000bc0 con 0x7f154c038fc0 2024-03-18T14:44:30.586 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:30 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:30.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.585+0000 7f1564902700 1 -- 172.21.15.76:0/810048561 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f1540000bc0 con 0x7f154c038fc0 2024-03-18T14:44:30.587 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:44:30.587 INFO:teuthology.orchestra.run.smithi076.stdout:[{"addr": "172.21.15.76", "hostname": "smithi076", "labels": [], "status": ""}, {"addr": "172.21.15.77", "hostname": "smithi077", "labels": [], "status": ""}] 2024-03-18T14:44:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.587+0000 7f1565904700 1 -- 172.21.15.76:0/810048561 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f154c038fc0 msgr2=0x7f154c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.587+0000 7f1565904700 1 --2- 172.21.15.76:0/810048561 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f154c038fc0 0x7f154c03b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1548006fd0 tx=0x7f1548006e40 comp rx=0 tx=0).stop 2024-03-18T14:44:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.587+0000 7f1565904700 1 -- 172.21.15.76:0/810048561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1560101f10 msgr2=0x7f1560197710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.587+0000 7f1565904700 1 --2- 172.21.15.76:0/810048561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1560101f10 0x7f1560197710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1558015040 tx=0x7f1558003d50 comp rx=0 tx=0).stop 2024-03-18T14:44:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.587+0000 7f1565904700 1 -- 172.21.15.76:0/810048561 shutdown_connections 2024-03-18T14:44:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.587+0000 7f1565904700 1 --2- 172.21.15.76:0/810048561 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f154c038fc0 0x7f154c03b480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.587+0000 7f1565904700 1 --2- 172.21.15.76:0/810048561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1560101f10 0x7f1560197710 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.587+0000 7f1565904700 1 -- 172.21.15.76:0/810048561 >> 172.21.15.76:0/810048561 conn(0x7f15600fd7e0 msgr2=0x7f15601038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.587+0000 7f1565904700 1 -- 172.21.15.76:0/810048561 shutdown_connections 2024-03-18T14:44:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:30.588+0000 7f1565904700 1 -- 172.21.15.76:0/810048561 wait complete. 2024-03-18T14:44:31.179 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi174 2024-03-18T14:44:31.179 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-18T14:44:31.179 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.conf 2024-03-18T14:44:31.197 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-18T14:44:31.197 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:44:31.254 INFO:tasks.cephadm:Adding host smithi174 to orchestrator... 2024-03-18T14:44:31.254 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch host add smithi174 2024-03-18T14:44:31.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:31 smithi076 ceph-mon[32205]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:31.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:31 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:31.488 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:44:32.500 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:32 smithi076 ceph-mon[32205]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:44:32.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.969+0000 7f9853562700 1 -- 172.21.15.76:0/2701878989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c101f10 msgr2=0x7f984c1022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:32.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.969+0000 7f9853562700 1 --2- 172.21.15.76:0/2701878989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c101f10 0x7f984c1022f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f983c009910 tx=0x7f983c009c20 comp rx=0 tx=0).stop 2024-03-18T14:44:32.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.969+0000 7f9853562700 1 -- 172.21.15.76:0/2701878989 shutdown_connections 2024-03-18T14:44:32.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.969+0000 7f9853562700 1 --2- 172.21.15.76:0/2701878989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c101f10 0x7f984c1022f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:32.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.969+0000 7f9853562700 1 -- 172.21.15.76:0/2701878989 >> 172.21.15.76:0/2701878989 conn(0x7f984c0fd7e0 msgr2=0x7f984c0ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:32.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.970+0000 7f9853562700 1 -- 172.21.15.76:0/2701878989 shutdown_connections 2024-03-18T14:44:32.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.970+0000 7f9853562700 1 -- 172.21.15.76:0/2701878989 wait complete. 2024-03-18T14:44:32.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.970+0000 7f9853562700 1 Processor -- start 2024-03-18T14:44:32.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.970+0000 7f9853562700 1 -- start start 2024-03-18T14:44:32.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.970+0000 7f9853562700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c101f10 0x7f984c19d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:32.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.970+0000 7f9853562700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f984c10c2e0 con 0x7f984c101f10 2024-03-18T14:44:32.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.971+0000 7f98512fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c101f10 0x7f984c19d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:32.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.971+0000 7f98512fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c101f10 0x7f984c19d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45514/0 (socket says 172.21.15.76:45514) 2024-03-18T14:44:32.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.971+0000 7f98512fe700 1 -- 172.21.15.76:0/3143577572 learned_addr learned my addr 172.21.15.76:0/3143577572 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:32.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.971+0000 7f98512fe700 1 -- 172.21.15.76:0/3143577572 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f984c19db20 con 0x7f984c101f10 2024-03-18T14:44:32.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.971+0000 7f98512fe700 1 --2- 172.21.15.76:0/3143577572 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c101f10 0x7f984c19d5e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f983c005850 tx=0x7f983c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:32.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.971+0000 7f98427fc700 1 -- 172.21.15.76:0/3143577572 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f983c026070 con 0x7f984c101f10 2024-03-18T14:44:32.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.971+0000 7f98427fc700 1 -- 172.21.15.76:0/3143577572 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f983c00bbd0 con 0x7f984c101f10 2024-03-18T14:44:32.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.971+0000 7f98427fc700 1 -- 172.21.15.76:0/3143577572 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f983c003970 con 0x7f984c101f10 2024-03-18T14:44:32.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.971+0000 7f9853562700 1 -- 172.21.15.76:0/3143577572 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f984c19de10 con 0x7f984c101f10 2024-03-18T14:44:32.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.971+0000 7f9853562700 1 -- 172.21.15.76:0/3143577572 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f984c197880 con 0x7f984c101f10 2024-03-18T14:44:32.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.972+0000 7f98427fc700 1 -- 172.21.15.76:0/3143577572 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f983c029070 con 0x7f984c101f10 2024-03-18T14:44:32.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.972+0000 7f9853562700 1 -- 172.21.15.76:0/3143577572 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f984c1033f0 con 0x7f984c101f10 2024-03-18T14:44:32.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.972+0000 7f98427fc700 1 --2- 172.21.15.76:0/3143577572 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9838039020 0x7f983803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:32.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.972+0000 7f98427fc700 1 -- 172.21.15.76:0/3143577572 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f983c01e070 con 0x7f984c101f10 2024-03-18T14:44:32.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.972+0000 7f9850afd700 1 --2- 172.21.15.76:0/3143577572 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9838039020 0x7f983803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:32.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.973+0000 7f9850afd700 1 --2- 172.21.15.76:0/3143577572 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9838039020 0x7f983803b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9848006fd0 tx=0x7f9848006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:32.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:32.977+0000 7f98427fc700 1 -- 172.21.15.76:0/3143577572 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f983c013070 con 0x7f984c101f10 2024-03-18T14:44:33.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:33.180+0000 7f9853562700 1 -- 172.21.15.76:0/3143577572 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}) -- 0x7f984c198be0 con 0x7f9838039020 2024-03-18T14:44:33.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:33 smithi076 ceph-mon[32205]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:34.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:34 smithi076 ceph-mon[32205]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:44:34.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:34 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:34.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:34 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:34.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:34 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:34.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:34 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:34.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:34 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:44:34.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:34 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:44:34.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:34 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:44:35.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:35 smithi076 ceph-mon[32205]: Deploying cephadm binary to smithi174 2024-03-18T14:44:35.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:35 smithi076 ceph-mon[32205]: Updating smithi077:/etc/ceph/ceph.conf 2024-03-18T14:44:35.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:35 smithi076 ceph-mon[32205]: Updating smithi077:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:44:35.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:35 smithi076 ceph-mon[32205]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:37.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:37.246+0000 7f98427fc700 1 -- 172.21.15.76:0/3143577572 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f984c198be0 con 0x7f9838039020 2024-03-18T14:44:37.248 INFO:teuthology.orchestra.run.smithi076.stdout:Added host 'smithi174' with addr '172.21.15.174' 2024-03-18T14:44:37.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:37.248+0000 7f9853562700 1 -- 172.21.15.76:0/3143577572 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9838039020 msgr2=0x7f983803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:37.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:37.248+0000 7f9853562700 1 --2- 172.21.15.76:0/3143577572 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9838039020 0x7f983803b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9848006fd0 tx=0x7f9848006e40 comp rx=0 tx=0).stop 2024-03-18T14:44:37.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:37.248+0000 7f9853562700 1 -- 172.21.15.76:0/3143577572 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c101f10 msgr2=0x7f984c19d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:37.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:37.248+0000 7f9853562700 1 --2- 172.21.15.76:0/3143577572 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c101f10 0x7f984c19d5e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f983c005850 tx=0x7f983c005030 comp rx=0 tx=0).stop 2024-03-18T14:44:37.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:37.248+0000 7f9853562700 1 -- 172.21.15.76:0/3143577572 shutdown_connections 2024-03-18T14:44:37.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:37.248+0000 7f9853562700 1 --2- 172.21.15.76:0/3143577572 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9838039020 0x7f983803b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:37.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:37.248+0000 7f9853562700 1 --2- 172.21.15.76:0/3143577572 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c101f10 0x7f984c19d5e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:37.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:37.248+0000 7f9853562700 1 -- 172.21.15.76:0/3143577572 >> 172.21.15.76:0/3143577572 conn(0x7f984c0fd7e0 msgr2=0x7f984c104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:37.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:37.248+0000 7f9853562700 1 -- 172.21.15.76:0/3143577572 shutdown_connections 2024-03-18T14:44:37.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:37.248+0000 7f9853562700 1 -- 172.21.15.76:0/3143577572 wait complete. 2024-03-18T14:44:37.565 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:37 smithi076 ceph-mon[32205]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:37.565 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:37 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:37.565 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:37 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:37.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:37 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:37.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:37 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:37.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:37 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:44:37.841 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-18T14:44:38.074 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:44:38.494 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:38 smithi076 ceph-mon[32205]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:44:38.495 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:38 smithi076 ceph-mon[32205]: Updating smithi077:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T14:44:38.495 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:38 smithi076 ceph-mon[32205]: Added host smithi174 2024-03-18T14:44:38.495 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:38 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:38.495 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:38 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:39.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:39 smithi076 ceph-mon[32205]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:39.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.610+0000 7fa67a1a9700 1 -- 172.21.15.76:0/1353333760 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa674101f10 msgr2=0x7fa6741022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:39.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.610+0000 7fa67a1a9700 1 --2- 172.21.15.76:0/1353333760 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa674101f10 0x7fa6741022f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa664009910 tx=0x7fa664009c20 comp rx=0 tx=0).stop 2024-03-18T14:44:39.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.610+0000 7fa67a1a9700 1 -- 172.21.15.76:0/1353333760 shutdown_connections 2024-03-18T14:44:39.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.610+0000 7fa67a1a9700 1 --2- 172.21.15.76:0/1353333760 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa674101f10 0x7fa6741022f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:39.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.610+0000 7fa67a1a9700 1 -- 172.21.15.76:0/1353333760 >> 172.21.15.76:0/1353333760 conn(0x7fa6740fd7e0 msgr2=0x7fa6740ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:39.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.610+0000 7fa67a1a9700 1 -- 172.21.15.76:0/1353333760 shutdown_connections 2024-03-18T14:44:39.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.610+0000 7fa67a1a9700 1 -- 172.21.15.76:0/1353333760 wait complete. 2024-03-18T14:44:39.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.611+0000 7fa67a1a9700 1 Processor -- start 2024-03-18T14:44:39.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.611+0000 7fa67a1a9700 1 -- start start 2024-03-18T14:44:39.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.612+0000 7fa67a1a9700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa674101f10 0x7fa6740745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:39.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.612+0000 7fa67a1a9700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa67410c2e0 con 0x7fa674101f10 2024-03-18T14:44:39.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.612+0000 7fa6737fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa674101f10 0x7fa6740745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:39.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.612+0000 7fa6737fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa674101f10 0x7fa6740745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:32846/0 (socket says 172.21.15.76:32846) 2024-03-18T14:44:39.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.612+0000 7fa6737fe700 1 -- 172.21.15.76:0/2638363711 learned_addr learned my addr 172.21.15.76:0/2638363711 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:39.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.612+0000 7fa6737fe700 1 -- 172.21.15.76:0/2638363711 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6740764c0 con 0x7fa674101f10 2024-03-18T14:44:39.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.612+0000 7fa6737fe700 1 --2- 172.21.15.76:0/2638363711 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa674101f10 0x7fa6740745d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa664005850 tx=0x7fa664005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:39.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.612+0000 7fa670ff9700 1 -- 172.21.15.76:0/2638363711 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa664026070 con 0x7fa674101f10 2024-03-18T14:44:39.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.612+0000 7fa670ff9700 1 -- 172.21.15.76:0/2638363711 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa66400bc30 con 0x7fa674101f10 2024-03-18T14:44:39.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.612+0000 7fa67a1a9700 1 -- 172.21.15.76:0/2638363711 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa674074b70 con 0x7fa674101f10 2024-03-18T14:44:39.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.612+0000 7fa670ff9700 1 -- 172.21.15.76:0/2638363711 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa664003a20 con 0x7fa674101f10 2024-03-18T14:44:39.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.612+0000 7fa67a1a9700 1 -- 172.21.15.76:0/2638363711 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa674075030 con 0x7fa674101f10 2024-03-18T14:44:39.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.613+0000 7fa670ff9700 1 -- 172.21.15.76:0/2638363711 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fa664029070 con 0x7fa674101f10 2024-03-18T14:44:39.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.613+0000 7fa670ff9700 1 --2- 172.21.15.76:0/2638363711 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa65c039020 0x7fa65c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:39.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.613+0000 7fa670ff9700 1 -- 172.21.15.76:0/2638363711 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa66401e070 con 0x7fa674101f10 2024-03-18T14:44:39.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.613+0000 7fa67a1a9700 1 -- 172.21.15.76:0/2638363711 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6741033f0 con 0x7fa674101f10 2024-03-18T14:44:39.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.613+0000 7fa672ffd700 1 --2- 172.21.15.76:0/2638363711 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa65c039020 0x7fa65c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:39.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.614+0000 7fa672ffd700 1 --2- 172.21.15.76:0/2638363711 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa65c039020 0x7fa65c03b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa668006fd0 tx=0x7fa668006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:39.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.618+0000 7fa670ff9700 1 -- 172.21.15.76:0/2638363711 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa664030070 con 0x7fa674101f10 2024-03-18T14:44:39.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.819+0000 7fa67a1a9700 1 -- 172.21.15.76:0/2638363711 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa674075b00 con 0x7fa65c039020 2024-03-18T14:44:39.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.820+0000 7fa670ff9700 1 -- 172.21.15.76:0/2638363711 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7fa674075b00 con 0x7fa65c039020 2024-03-18T14:44:39.821 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:44:39.821 INFO:teuthology.orchestra.run.smithi076.stdout:[{"addr": "172.21.15.76", "hostname": "smithi076", "labels": [], "status": ""}, {"addr": "172.21.15.77", "hostname": "smithi077", "labels": [], "status": ""}, {"addr": "172.21.15.174", "hostname": "smithi174", "labels": [], "status": ""}] 2024-03-18T14:44:39.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.822+0000 7fa67a1a9700 1 -- 172.21.15.76:0/2638363711 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa65c039020 msgr2=0x7fa65c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:39.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.822+0000 7fa67a1a9700 1 --2- 172.21.15.76:0/2638363711 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa65c039020 0x7fa65c03b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa668006fd0 tx=0x7fa668006e40 comp rx=0 tx=0).stop 2024-03-18T14:44:39.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.822+0000 7fa67a1a9700 1 -- 172.21.15.76:0/2638363711 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa674101f10 msgr2=0x7fa6740745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:39.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.822+0000 7fa67a1a9700 1 --2- 172.21.15.76:0/2638363711 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa674101f10 0x7fa6740745d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa664005850 tx=0x7fa664005030 comp rx=0 tx=0).stop 2024-03-18T14:44:39.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.822+0000 7fa67a1a9700 1 -- 172.21.15.76:0/2638363711 shutdown_connections 2024-03-18T14:44:39.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.822+0000 7fa67a1a9700 1 --2- 172.21.15.76:0/2638363711 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa65c039020 0x7fa65c03b4e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:39.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.822+0000 7fa67a1a9700 1 --2- 172.21.15.76:0/2638363711 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa674101f10 0x7fa6740745d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:39.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.822+0000 7fa67a1a9700 1 -- 172.21.15.76:0/2638363711 >> 172.21.15.76:0/2638363711 conn(0x7fa6740fd7e0 msgr2=0x7fa674104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:39.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.822+0000 7fa67a1a9700 1 -- 172.21.15.76:0/2638363711 shutdown_connections 2024-03-18T14:44:39.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:39.822+0000 7fa67a1a9700 1 -- 172.21.15.76:0/2638363711 wait complete. 2024-03-18T14:44:40.376 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:40.377 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-18T14:44:40.377 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-18T14:44:40.614 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:44:41.638 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:41 smithi076 ceph-mon[32205]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:44:41.638 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:41 smithi076 ceph-mon[32205]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:42.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.115+0000 7ffb6ffaf700 1 -- 172.21.15.76:0/1033758903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb68101ee0 msgr2=0x7ffb681022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:42.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.115+0000 7ffb6ffaf700 1 --2- 172.21.15.76:0/1033758903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb68101ee0 0x7ffb681022c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffb64009910 tx=0x7ffb64009c20 comp rx=0 tx=0).stop 2024-03-18T14:44:42.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.115+0000 7ffb6ffaf700 1 -- 172.21.15.76:0/1033758903 shutdown_connections 2024-03-18T14:44:42.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.115+0000 7ffb6ffaf700 1 --2- 172.21.15.76:0/1033758903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb68101ee0 0x7ffb681022c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:42.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.115+0000 7ffb6ffaf700 1 -- 172.21.15.76:0/1033758903 >> 172.21.15.76:0/1033758903 conn(0x7ffb680fd7b0 msgr2=0x7ffb680ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:42.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.115+0000 7ffb6ffaf700 1 -- 172.21.15.76:0/1033758903 shutdown_connections 2024-03-18T14:44:42.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.115+0000 7ffb6ffaf700 1 -- 172.21.15.76:0/1033758903 wait complete. 2024-03-18T14:44:42.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.115+0000 7ffb6ffaf700 1 Processor -- start 2024-03-18T14:44:42.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.115+0000 7ffb6ffaf700 1 -- start start 2024-03-18T14:44:42.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.116+0000 7ffb6ffaf700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb68101ee0 0x7ffb6819d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:42.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.116+0000 7ffb6ffaf700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6810c2b0 con 0x7ffb68101ee0 2024-03-18T14:44:42.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.116+0000 7ffb6dd4b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb68101ee0 0x7ffb6819d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:42.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.116+0000 7ffb6dd4b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb68101ee0 0x7ffb6819d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:32864/0 (socket says 172.21.15.76:32864) 2024-03-18T14:44:42.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.116+0000 7ffb6dd4b700 1 -- 172.21.15.76:0/3945439726 learned_addr learned my addr 172.21.15.76:0/3945439726 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:44:42.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.116+0000 7ffb6dd4b700 1 -- 172.21.15.76:0/3945439726 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6819daf0 con 0x7ffb68101ee0 2024-03-18T14:44:42.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.116+0000 7ffb6dd4b700 1 --2- 172.21.15.76:0/3945439726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb68101ee0 0x7ffb6819d5b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffb64005850 tx=0x7ffb64005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:42.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.116+0000 7ffb5effd700 1 -- 172.21.15.76:0/3945439726 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb64026070 con 0x7ffb68101ee0 2024-03-18T14:44:42.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.116+0000 7ffb6ffaf700 1 -- 172.21.15.76:0/3945439726 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb6819dde0 con 0x7ffb68101ee0 2024-03-18T14:44:42.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.116+0000 7ffb5effd700 1 -- 172.21.15.76:0/3945439726 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb6400bbd0 con 0x7ffb68101ee0 2024-03-18T14:44:42.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.116+0000 7ffb5effd700 1 -- 172.21.15.76:0/3945439726 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb64003970 con 0x7ffb68101ee0 2024-03-18T14:44:42.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.116+0000 7ffb6ffaf700 1 -- 172.21.15.76:0/3945439726 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb68197850 con 0x7ffb68101ee0 2024-03-18T14:44:42.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.117+0000 7ffb5effd700 1 -- 172.21.15.76:0/3945439726 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7ffb64029070 con 0x7ffb68101ee0 2024-03-18T14:44:42.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.117+0000 7ffb6ffaf700 1 -- 172.21.15.76:0/3945439726 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb681033c0 con 0x7ffb68101ee0 2024-03-18T14:44:42.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.117+0000 7ffb5effd700 1 --2- 172.21.15.76:0/3945439726 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffb54038fc0 0x7ffb5403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:42.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.117+0000 7ffb5effd700 1 -- 172.21.15.76:0/3945439726 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ffb6401e070 con 0x7ffb68101ee0 2024-03-18T14:44:42.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.117+0000 7ffb6d54a700 1 --2- 172.21.15.76:0/3945439726 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffb54038fc0 0x7ffb5403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:42.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.118+0000 7ffb6d54a700 1 --2- 172.21.15.76:0/3945439726 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffb54038fc0 0x7ffb5403b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ffb58006fd0 tx=0x7ffb58006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:42.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.122+0000 7ffb5effd700 1 -- 172.21.15.76:0/3945439726 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffb64013070 con 0x7ffb68101ee0 2024-03-18T14:44:42.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.321+0000 7ffb6ffaf700 1 -- 172.21.15.76:0/3945439726 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7ffb68198bb0 con 0x7ffb68101ee0 2024-03-18T14:44:42.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.332+0000 7ffb5effd700 1 -- 172.21.15.76:0/3945439726 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7ffb6400fe20 con 0x7ffb68101ee0 2024-03-18T14:44:42.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.334+0000 7ffb6ffaf700 1 -- 172.21.15.76:0/3945439726 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffb54038fc0 msgr2=0x7ffb5403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:42.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.334+0000 7ffb6ffaf700 1 --2- 172.21.15.76:0/3945439726 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffb54038fc0 0x7ffb5403b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ffb58006fd0 tx=0x7ffb58006e40 comp rx=0 tx=0).stop 2024-03-18T14:44:42.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.334+0000 7ffb6ffaf700 1 -- 172.21.15.76:0/3945439726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb68101ee0 msgr2=0x7ffb6819d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:42.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.334+0000 7ffb6ffaf700 1 --2- 172.21.15.76:0/3945439726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb68101ee0 0x7ffb6819d5b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffb64005850 tx=0x7ffb64005030 comp rx=0 tx=0).stop 2024-03-18T14:44:42.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.334+0000 7ffb6ffaf700 1 -- 172.21.15.76:0/3945439726 shutdown_connections 2024-03-18T14:44:42.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.334+0000 7ffb6ffaf700 1 --2- 172.21.15.76:0/3945439726 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffb54038fc0 0x7ffb5403b480 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:42.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.334+0000 7ffb6ffaf700 1 --2- 172.21.15.76:0/3945439726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb68101ee0 0x7ffb6819d5b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:42.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.334+0000 7ffb6ffaf700 1 -- 172.21.15.76:0/3945439726 >> 172.21.15.76:0/3945439726 conn(0x7ffb680fd7b0 msgr2=0x7ffb68104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:42.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.334+0000 7ffb6ffaf700 1 -- 172.21.15.76:0/3945439726 shutdown_connections 2024-03-18T14:44:42.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:44:42.334+0000 7ffb6ffaf700 1 -- 172.21.15.76:0/3945439726 wait complete. 2024-03-18T14:44:42.337 INFO:teuthology.orchestra.run.smithi076.stderr:adjusted tunables profile to default 2024-03-18T14:44:42.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:42 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3945439726' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T14:44:42.955 INFO:tasks.cephadm:Adding mon.a on smithi076 2024-03-18T14:44:42.955 INFO:tasks.cephadm:Adding mon.b on smithi077 2024-03-18T14:44:42.955 INFO:tasks.cephadm:Adding mon.c on smithi174 2024-03-18T14:44:42.955 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi076:172.21.15.76=a;smithi077:172.21.15.77=b;smithi174:172.21.15.174=c' 2024-03-18T14:44:43.188 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:44:43.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:43 smithi076 ceph-mon[32205]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:43.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:43 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3945439726' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T14:44:43.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:43 smithi076 ceph-mon[32205]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T14:44:43.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:43 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:43.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:43 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:43.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:43 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:43.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:43 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:43.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:43 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:44:43.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:43 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:44:43.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:43 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:44:44.630 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.628+0000 7f098238d700 1 -- 172.21.15.174:0/3913828143 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f097c102e70 msgr2=0x7f097c103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:44.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.628+0000 7f098238d700 1 --2- 172.21.15.174:0/3913828143 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f097c102e70 0x7f097c103250 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f096c009910 tx=0x7f096c009c20 comp rx=0 tx=0).stop 2024-03-18T14:44:44.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.628+0000 7f098238d700 1 -- 172.21.15.174:0/3913828143 shutdown_connections 2024-03-18T14:44:44.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.628+0000 7f098238d700 1 --2- 172.21.15.174:0/3913828143 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f097c102e70 0x7f097c103250 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:44.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.628+0000 7f098238d700 1 -- 172.21.15.174:0/3913828143 >> 172.21.15.174:0/3913828143 conn(0x7f097c0fc6c0 msgr2=0x7f097c0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:44.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.628+0000 7f098238d700 1 -- 172.21.15.174:0/3913828143 shutdown_connections 2024-03-18T14:44:44.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.628+0000 7f098238d700 1 -- 172.21.15.174:0/3913828143 wait complete. 2024-03-18T14:44:44.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.629+0000 7f098238d700 1 Processor -- start 2024-03-18T14:44:44.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.629+0000 7f098238d700 1 -- start start 2024-03-18T14:44:44.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.629+0000 7f098238d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f097c102e70 0x7f097c19d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:44.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.629+0000 7f098238d700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f097c10c360 con 0x7f097c102e70 2024-03-18T14:44:44.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.630+0000 7f097bfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f097c102e70 0x7f097c19d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:44.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.630+0000 7f097bfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f097c102e70 0x7f097c19d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.174:41148/0 (socket says 172.21.15.174:41148) 2024-03-18T14:44:44.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.630+0000 7f097bfff700 1 -- 172.21.15.174:0/1494315837 learned_addr learned my addr 172.21.15.174:0/1494315837 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-18T14:44:44.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.630+0000 7f097bfff700 1 -- 172.21.15.174:0/1494315837 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097c19dac0 con 0x7f097c102e70 2024-03-18T14:44:44.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.630+0000 7f097bfff700 1 --2- 172.21.15.174:0/1494315837 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f097c102e70 0x7f097c19d580 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f096c006010 tx=0x7f096c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:44.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.630+0000 7f09797fa700 1 -- 172.21.15.174:0/1494315837 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f096c026070 con 0x7f097c102e70 2024-03-18T14:44:44.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.630+0000 7f09797fa700 1 -- 172.21.15.174:0/1494315837 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f096c00bc30 con 0x7f097c102e70 2024-03-18T14:44:44.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.630+0000 7f098238d700 1 -- 172.21.15.174:0/1494315837 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f097c19dd50 con 0x7f097c102e70 2024-03-18T14:44:44.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.630+0000 7f09797fa700 1 -- 172.21.15.174:0/1494315837 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f096c003ab0 con 0x7f097c102e70 2024-03-18T14:44:44.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.630+0000 7f098238d700 1 -- 172.21.15.174:0/1494315837 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f097c197650 con 0x7f097c102e70 2024-03-18T14:44:44.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.631+0000 7f098238d700 1 -- 172.21.15.174:0/1494315837 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f097c06aa30 con 0x7f097c102e70 2024-03-18T14:44:44.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.631+0000 7f09797fa700 1 -- 172.21.15.174:0/1494315837 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f096c029070 con 0x7f097c102e70 2024-03-18T14:44:44.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.631+0000 7f09797fa700 1 --2- 172.21.15.174:0/1494315837 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0968038fc0 0x7f096803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:44.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.632+0000 7f09797fa700 1 -- 172.21.15.174:0/1494315837 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f096c01e070 con 0x7f097c102e70 2024-03-18T14:44:44.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.632+0000 7f097b7fe700 1 --2- 172.21.15.174:0/1494315837 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0968038fc0 0x7f096803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:44.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.632+0000 7f097b7fe700 1 --2- 172.21.15.174:0/1494315837 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0968038fc0 0x7f096803b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0964006fd0 tx=0x7f0964006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:44.638 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.637+0000 7f09797fa700 1 -- 172.21.15.174:0/1494315837 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f096c013070 con 0x7f097c102e70 2024-03-18T14:44:44.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:44 smithi076 ceph-mon[32205]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-18T14:44:44.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:44 smithi076 ceph-mon[32205]: Updating smithi174:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:44:44.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:44 smithi076 ceph-mon[32205]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:44:44.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:44 smithi076 ceph-mon[32205]: Updating smithi174:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T14:44:44.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:44 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:44.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:44 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:44.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:44 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:44.847 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.845+0000 7f098238d700 1 -- 172.21.15.174:0/1494315837 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi076:172.21.15.76=a;smithi077:172.21.15.77=b;smithi174:172.21.15.174=c", "target": ["mon-mgr", ""]}) -- 0x7f097c198b80 con 0x7f0968038fc0 2024-03-18T14:44:44.853 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.852+0000 7f09797fa700 1 -- 172.21.15.174:0/1494315837 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f097c198b80 con 0x7f0968038fc0 2024-03-18T14:44:44.853 INFO:teuthology.orchestra.run.smithi174.stdout:Scheduled mon update... 2024-03-18T14:44:44.855 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.854+0000 7f098238d700 1 -- 172.21.15.174:0/1494315837 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0968038fc0 msgr2=0x7f096803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:44.855 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.854+0000 7f098238d700 1 --2- 172.21.15.174:0/1494315837 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0968038fc0 0x7f096803b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0964006fd0 tx=0x7f0964006e40 comp rx=0 tx=0).stop 2024-03-18T14:44:44.855 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.854+0000 7f098238d700 1 -- 172.21.15.174:0/1494315837 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f097c102e70 msgr2=0x7f097c19d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:44.856 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.854+0000 7f098238d700 1 --2- 172.21.15.174:0/1494315837 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f097c102e70 0x7f097c19d580 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f096c006010 tx=0x7f096c005050 comp rx=0 tx=0).stop 2024-03-18T14:44:44.856 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.854+0000 7f098238d700 1 -- 172.21.15.174:0/1494315837 shutdown_connections 2024-03-18T14:44:44.856 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.854+0000 7f098238d700 1 --2- 172.21.15.174:0/1494315837 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0968038fc0 0x7f096803b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:44.856 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.854+0000 7f098238d700 1 --2- 172.21.15.174:0/1494315837 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f097c102e70 0x7f097c19d580 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:44.856 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.854+0000 7f098238d700 1 -- 172.21.15.174:0/1494315837 >> 172.21.15.174:0/1494315837 conn(0x7f097c0fc6c0 msgr2=0x7f097c103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:44.856 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.854+0000 7f098238d700 1 -- 172.21.15.174:0/1494315837 shutdown_connections 2024-03-18T14:44:44.856 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:44.854+0000 7f098238d700 1 -- 172.21.15.174:0/1494315837 wait complete. 2024-03-18T14:44:45.440 DEBUG:teuthology.orchestra.run.smithi077:mon.b> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mon.b.service 2024-03-18T14:44:45.443 DEBUG:teuthology.orchestra.run.smithi174:mon.c> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mon.c.service 2024-03-18T14:44:45.446 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T14:44:45.446 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T14:44:45.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:45 smithi076 ceph-mon[32205]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:45.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:45 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:45.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:45 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:44:45.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:45 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:44:45.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:45 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:44:45.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:45 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:45.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:45 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T14:44:45.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:45 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:44:46.568 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:44:46.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:46 smithi076 ceph-mon[32205]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi076:172.21.15.76=a;smithi077:172.21.15.77=b;smithi174:172.21.15.174=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:44:46.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:46 smithi076 ceph-mon[32205]: Saving service mon spec with placement smithi076:172.21.15.76=a;smithi077:172.21.15.77=b;smithi174:172.21.15.174=c;count:3 2024-03-18T14:44:46.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:46 smithi076 ceph-mon[32205]: Deploying daemon mon.c on smithi174 2024-03-18T14:44:47.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:47 smithi076 ceph-mon[32205]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:48.520 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.518+0000 7f95c0efe700 1 -- 172.21.15.174:0/3728290479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95bc100e40 msgr2=0x7f95bc101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:48.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.518+0000 7f95c0efe700 1 --2- 172.21.15.174:0/3728290479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95bc100e40 0x7f95bc101220 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009910 tx=0x7f95ac009c20 comp rx=0 tx=0).stop 2024-03-18T14:44:48.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.518+0000 7f95c0efe700 1 -- 172.21.15.174:0/3728290479 shutdown_connections 2024-03-18T14:44:48.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.518+0000 7f95c0efe700 1 --2- 172.21.15.174:0/3728290479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95bc100e40 0x7f95bc101220 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:48.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.518+0000 7f95c0efe700 1 -- 172.21.15.174:0/3728290479 >> 172.21.15.174:0/3728290479 conn(0x7f95bc0fc6d0 msgr2=0x7f95bc0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:48.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.518+0000 7f95c0efe700 1 -- 172.21.15.174:0/3728290479 shutdown_connections 2024-03-18T14:44:48.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.518+0000 7f95c0efe700 1 -- 172.21.15.174:0/3728290479 wait complete. 2024-03-18T14:44:48.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.519+0000 7f95c0efe700 1 Processor -- start 2024-03-18T14:44:48.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.519+0000 7f95c0efe700 1 -- start start 2024-03-18T14:44:48.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.519+0000 7f95c0efe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95bc100e40 0x7f95bc198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:48.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.519+0000 7f95c0efe700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc10b210 con 0x7f95bc100e40 2024-03-18T14:44:48.523 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.519+0000 7f95ba59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95bc100e40 0x7f95bc198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:48.523 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.519+0000 7f95ba59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95bc100e40 0x7f95bc198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.174:41166/0 (socket says 172.21.15.174:41166) 2024-03-18T14:44:48.523 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.519+0000 7f95ba59c700 1 -- 172.21.15.174:0/2764648823 learned_addr learned my addr 172.21.15.174:0/2764648823 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-18T14:44:48.524 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.520+0000 7f95ba59c700 1 -- 172.21.15.174:0/2764648823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95bc198ba0 con 0x7f95bc100e40 2024-03-18T14:44:48.524 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.520+0000 7f95ba59c700 1 --2- 172.21.15.174:0/2764648823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95bc100e40 0x7f95bc198660 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f95ac005850 tx=0x7f95ac005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:48.524 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.520+0000 7f95b3fff700 1 -- 172.21.15.174:0/2764648823 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95ac026070 con 0x7f95bc100e40 2024-03-18T14:44:48.524 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.520+0000 7f95c0efe700 1 -- 172.21.15.174:0/2764648823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95bc198e90 con 0x7f95bc100e40 2024-03-18T14:44:48.524 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.520+0000 7f95b3fff700 1 -- 172.21.15.174:0/2764648823 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95ac00bbd0 con 0x7f95bc100e40 2024-03-18T14:44:48.524 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.520+0000 7f95b3fff700 1 -- 172.21.15.174:0/2764648823 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95ac003970 con 0x7f95bc100e40 2024-03-18T14:44:48.524 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.520+0000 7f95c0efe700 1 -- 172.21.15.174:0/2764648823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95bc199350 con 0x7f95bc100e40 2024-03-18T14:44:48.524 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.521+0000 7f95b3fff700 1 -- 172.21.15.174:0/2764648823 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f95ac029070 con 0x7f95bc100e40 2024-03-18T14:44:48.524 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.521+0000 7f95b3fff700 1 --2- 172.21.15.174:0/2764648823 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f95a8038fc0 0x7f95a803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:48.524 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.521+0000 7f95b3fff700 1 -- 172.21.15.174:0/2764648823 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f95ac01e070 con 0x7f95bc100e40 2024-03-18T14:44:48.524 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.521+0000 7f95c0efe700 1 -- 172.21.15.174:0/2764648823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95bc102320 con 0x7f95bc100e40 2024-03-18T14:44:48.524 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.521+0000 7f95b1d9b700 1 --2- 172.21.15.174:0/2764648823 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f95a8038fc0 0x7f95a803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:48.524 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.522+0000 7f95b1d9b700 1 --2- 172.21.15.174:0/2764648823 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f95a8038fc0 0x7f95a803b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f95a4006fd0 tx=0x7f95a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:48.528 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.527+0000 7f95b3fff700 1 -- 172.21.15.174:0/2764648823 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f95ac030070 con 0x7f95bc100e40 2024-03-18T14:44:48.807 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.805+0000 7f95c0efe700 1 -- 172.21.15.174:0/2764648823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f95bc063900 con 0x7f95bc100e40 2024-03-18T14:44:48.807 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.806+0000 7f95b3fff700 1 -- 172.21.15.174:0/2764648823 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f95ac013070 con 0x7f95bc100e40 2024-03-18T14:44:48.807 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:44:48.807 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","modified":"2024-03-18T14:43:08.949643Z","created":"2024-03-18T14:43:08.949643Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3300","nonce":0},{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T14:44:48.809 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.808+0000 7f95c0efe700 1 -- 172.21.15.174:0/2764648823 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f95a8038fc0 msgr2=0x7f95a803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:48.810 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.808+0000 7f95c0efe700 1 --2- 172.21.15.174:0/2764648823 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f95a8038fc0 0x7f95a803b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f95a4006fd0 tx=0x7f95a4006e40 comp rx=0 tx=0).stop 2024-03-18T14:44:48.810 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.808+0000 7f95c0efe700 1 -- 172.21.15.174:0/2764648823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95bc100e40 msgr2=0x7f95bc198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:48.810 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.808+0000 7f95c0efe700 1 --2- 172.21.15.174:0/2764648823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95bc100e40 0x7f95bc198660 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f95ac005850 tx=0x7f95ac005030 comp rx=0 tx=0).stop 2024-03-18T14:44:48.810 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.808+0000 7f95c0efe700 1 -- 172.21.15.174:0/2764648823 shutdown_connections 2024-03-18T14:44:48.810 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.808+0000 7f95c0efe700 1 --2- 172.21.15.174:0/2764648823 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f95a8038fc0 0x7f95a803b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:48.810 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.808+0000 7f95c0efe700 1 --2- 172.21.15.174:0/2764648823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95bc100e40 0x7f95bc198660 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:48.810 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.808+0000 7f95c0efe700 1 -- 172.21.15.174:0/2764648823 >> 172.21.15.174:0/2764648823 conn(0x7f95bc0fc6d0 msgr2=0x7f95bc102810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:48.810 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.808+0000 7f95c0efe700 1 -- 172.21.15.174:0/2764648823 shutdown_connections 2024-03-18T14:44:48.810 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:48.808+0000 7f95c0efe700 1 -- 172.21.15.174:0/2764648823 wait complete. 2024-03-18T14:44:48.811 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-03-18T14:44:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:49 smithi076 ceph-mon[32205]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:49.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:49 smithi076 ceph-mon[32205]: from='client.? 172.21.15.174:0/2764648823' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T14:44:50.358 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:50 smithi174 ceph-mon[36787]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-18T14:44:50.499 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T14:44:50.499 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T14:44:50.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:50 smithi174 bash[36768]: 5e438e85c48fd81a25099c8a0f90b6831032cb49884ff491891529a6900f7355 2024-03-18T14:44:50.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:50 smithi174 systemd[1]: Started Ceph mon.c for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:44:50.732 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T14:44:55.463 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.458+0000 7efd26c67700 1 -- 172.21.15.174:0/1914240544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd100035c0 msgr2=0x7efd10005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:55.463 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.458+0000 7efd26c67700 1 --2- 172.21.15.174:0/1914240544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd100035c0 0x7efd10005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efd20079560 tx=0x7efd14013e90 comp rx=0 tx=0).stop 2024-03-18T14:44:55.463 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.458+0000 7efd26c67700 1 -- 172.21.15.174:0/1914240544 shutdown_connections 2024-03-18T14:44:55.463 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.458+0000 7efd26c67700 1 --2- 172.21.15.174:0/1914240544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd100035c0 0x7efd10005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:55.463 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.458+0000 7efd26c67700 1 -- 172.21.15.174:0/1914240544 >> 172.21.15.174:0/1914240544 conn(0x7efd20077290 msgr2=0x7efd200776a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:55.463 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.458+0000 7efd26c67700 1 -- 172.21.15.174:0/1914240544 shutdown_connections 2024-03-18T14:44:55.463 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.458+0000 7efd26c67700 1 -- 172.21.15.174:0/1914240544 wait complete. 2024-03-18T14:44:55.463 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.459+0000 7efd26c67700 1 Processor -- start 2024-03-18T14:44:55.464 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.459+0000 7efd26c67700 1 -- start start 2024-03-18T14:44:55.464 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.459+0000 7efd26c67700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd100035c0 0x7efd2019d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:55.464 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.459+0000 7efd26c67700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd2019db40 0x7efd201976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:55.464 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.459+0000 7efd26c67700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efd20109e40 con 0x7efd2019db40 2024-03-18T14:44:55.464 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.459+0000 7efd26c67700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efd20109cc0 con 0x7efd100035c0 2024-03-18T14:44:55.464 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.459+0000 7efd24a03700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd100035c0 0x7efd2019d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:55.464 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.459+0000 7efd24a03700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd100035c0 0x7efd2019d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:47730/0 (socket says 172.21.15.174:47730) 2024-03-18T14:44:55.464 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.459+0000 7efd24a03700 1 -- 172.21.15.174:0/3594535122 learned_addr learned my addr 172.21.15.174:0/3594535122 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-18T14:44:55.464 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.459+0000 7efd1ffff700 1 --2- 172.21.15.174:0/3594535122 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd2019db40 0x7efd201976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:55.464 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.460+0000 7efd24a03700 1 -- 172.21.15.174:0/3594535122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd100035c0 msgr2=0x7efd2019d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-18T14:44:55.464 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.460+0000 7efd24a03700 1 -- 172.21.15.174:0/3594535122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd100035c0 msgr2=0x7efd2019d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T14:44:55.464 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.460+0000 7efd24a03700 1 --2- 172.21.15.174:0/3594535122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd100035c0 0x7efd2019d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T14:44:55.464 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.460+0000 7efd24a03700 1 --2- 172.21.15.174:0/3594535122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd100035c0 0x7efd2019d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T14:44:55.464 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.460+0000 7efd1ffff700 1 -- 172.21.15.174:0/3594535122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd100035c0 msgr2=0x7efd2019d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T14:44:55.464 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.460+0000 7efd1ffff700 1 --2- 172.21.15.174:0/3594535122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd100035c0 0x7efd2019d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:55.464 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.460+0000 7efd1ffff700 1 -- 172.21.15.174:0/3594535122 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2019e0c0 con 0x7efd2019db40 2024-03-18T14:44:55.465 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.460+0000 7efd1ffff700 1 --2- 172.21.15.174:0/3594535122 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd2019db40 0x7efd201976d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efd0c00b4d0 tx=0x7efd0c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:55.465 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.460+0000 7efd1dffb700 1 -- 172.21.15.174:0/3594535122 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7efd0c011630 con 0x7efd2019db40 2024-03-18T14:44:55.465 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.460+0000 7efd1dffb700 1 -- 172.21.15.174:0/3594535122 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd0c011cb0 con 0x7efd2019db40 2024-03-18T14:44:55.465 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.460+0000 7efd26c67700 1 -- 172.21.15.174:0/3594535122 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd20197c70 con 0x7efd2019db40 2024-03-18T14:44:55.465 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.460+0000 7efd1dffb700 1 -- 172.21.15.174:0/3594535122 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7efd0c010e60 con 0x7efd2019db40 2024-03-18T14:44:55.465 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.461+0000 7efd26c67700 1 -- 172.21.15.174:0/3594535122 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd20198200 con 0x7efd2019db40 2024-03-18T14:44:55.465 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.461+0000 7efd1dffb700 1 -- 172.21.15.174:0/3594535122 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7efd0c018020 con 0x7efd2019db40 2024-03-18T14:44:55.465 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.461+0000 7efd1dffb700 1 --2- 172.21.15.174:0/3594535122 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efd04038d60 0x7efd0403b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:55.465 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.462+0000 7efd1dffb700 1 -- 172.21.15.174:0/3594535122 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7efd0c04ae30 con 0x7efd2019db40 2024-03-18T14:44:55.465 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.462+0000 7efd24a03700 1 --2- 172.21.15.174:0/3594535122 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efd04038d60 0x7efd0403b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:55.466 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.462+0000 7efd26c67700 1 -- 172.21.15.174:0/3594535122 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd2006a200 con 0x7efd2019db40 2024-03-18T14:44:55.466 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.462+0000 7efd24a03700 1 --2- 172.21.15.174:0/3594535122 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efd04038d60 0x7efd0403b220 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efd14013a40 tx=0x7efd1400f3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:55.468 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.467+0000 7efd1dffb700 1 -- 172.21.15.174:0/3594535122 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efd0c01f020 con 0x7efd2019db40 2024-03-18T14:44:55.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:55.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:44:55.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:55.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: mon.a calling monitor election 2024-03-18T14:44:55.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:55.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:55.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:55.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: mon.c calling monitor election 2024-03-18T14:44:55.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:55.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:55.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:55.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:55.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T14:44:55.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: monmap epoch 2 2024-03-18T14:44:55.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: fsid 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:44:55.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: last_changed 2024-03-18T14:44:50.340639+0000 2024-03-18T14:44:55.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: created 2024-03-18T14:43:08.949643+0000 2024-03-18T14:44:55.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: min_mon_release 19 (squid) 2024-03-18T14:44:55.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: election_strategy: 1 2024-03-18T14:44:55.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-03-18T14:44:55.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-03-18T14:44:55.664 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: fsmap 2024-03-18T14:44:55.664 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T14:44:55.664 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: mgrmap e13: x(active, since 49s) 2024-03-18T14:44:55.664 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: overall HEALTH_OK 2024-03-18T14:44:55.664 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:55.664 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:55.664 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:55.664 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T14:44:55.664 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:55 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:44:55.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:55.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:44:55.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:55.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: mon.a calling monitor election 2024-03-18T14:44:55.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:55.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:55.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:55.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: mon.c calling monitor election 2024-03-18T14:44:55.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:55.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:55.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:55.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:55.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T14:44:55.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: monmap epoch 2 2024-03-18T14:44:55.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: fsid 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:44:55.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: last_changed 2024-03-18T14:44:50.340639+0000 2024-03-18T14:44:55.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: created 2024-03-18T14:43:08.949643+0000 2024-03-18T14:44:55.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: min_mon_release 19 (squid) 2024-03-18T14:44:55.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: election_strategy: 1 2024-03-18T14:44:55.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-03-18T14:44:55.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-03-18T14:44:55.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: fsmap 2024-03-18T14:44:55.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T14:44:55.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: mgrmap e13: x(active, since 49s) 2024-03-18T14:44:55.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: overall HEALTH_OK 2024-03-18T14:44:55.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:55.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:55.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:55.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T14:44:55.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:55 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:44:55.750 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.749+0000 7efd26c67700 1 -- 172.21.15.174:0/3594535122 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7efd20198c00 con 0x7efd2019db40 2024-03-18T14:44:55.751 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.749+0000 7efd1dffb700 1 -- 172.21.15.174:0/3594535122 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7efd0c01d120 con 0x7efd2019db40 2024-03-18T14:44:55.751 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:44:55.751 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":2,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","modified":"2024-03-18T14:44:50.340639Z","created":"2024-03-18T14:43:08.949643Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3300","nonce":0},{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-18T14:44:55.753 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.751+0000 7efd26c67700 1 -- 172.21.15.174:0/3594535122 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efd04038d60 msgr2=0x7efd0403b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:55.753 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.751+0000 7efd26c67700 1 --2- 172.21.15.174:0/3594535122 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efd04038d60 0x7efd0403b220 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efd14013a40 tx=0x7efd1400f3c0 comp rx=0 tx=0).stop 2024-03-18T14:44:55.753 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.751+0000 7efd26c67700 1 -- 172.21.15.174:0/3594535122 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd2019db40 msgr2=0x7efd201976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:55.753 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.751+0000 7efd26c67700 1 --2- 172.21.15.174:0/3594535122 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd2019db40 0x7efd201976d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efd0c00b4d0 tx=0x7efd0c00b890 comp rx=0 tx=0).stop 2024-03-18T14:44:55.753 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.752+0000 7efd26c67700 1 -- 172.21.15.174:0/3594535122 shutdown_connections 2024-03-18T14:44:55.754 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.752+0000 7efd26c67700 1 --2- 172.21.15.174:0/3594535122 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efd04038d60 0x7efd0403b220 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:55.754 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.752+0000 7efd26c67700 1 --2- 172.21.15.174:0/3594535122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd100035c0 0x7efd2019d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:55.754 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.752+0000 7efd26c67700 1 --2- 172.21.15.174:0/3594535122 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd2019db40 0x7efd201976d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:55.754 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.752+0000 7efd26c67700 1 -- 172.21.15.174:0/3594535122 >> 172.21.15.174:0/3594535122 conn(0x7efd20077290 msgr2=0x7efd2018f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:55.754 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.752+0000 7efd26c67700 1 -- 172.21.15.174:0/3594535122 shutdown_connections 2024-03-18T14:44:55.754 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:55.752+0000 7efd26c67700 1 -- 172.21.15.174:0/3594535122 wait complete. 2024-03-18T14:44:55.754 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 2 2024-03-18T14:44:56.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:56 smithi174 ceph-mon[36787]: Deploying daemon mon.b on smithi077 2024-03-18T14:44:56.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:56 smithi174 ceph-mon[36787]: from='client.? 172.21.15.174:0/3594535122' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T14:44:56.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:56 smithi174 ceph-mon[36787]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:56.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:56 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:56.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:56 smithi076 ceph-mon[32205]: Deploying daemon mon.b on smithi077 2024-03-18T14:44:56.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:56 smithi076 ceph-mon[32205]: from='client.? 172.21.15.174:0/3594535122' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T14:44:56.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:56 smithi076 ceph-mon[32205]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:56.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:56 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:57.380 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T14:44:57.380 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T14:44:57.614 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T14:44:57.661 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:44:57 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:44:57.340+0000 7fa6733bf700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-18T14:44:59.120 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.117+0000 7fed576c8700 1 -- 172.21.15.174:0/4200654024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed50074670 msgr2=0x7fed50074a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:59.120 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.117+0000 7fed576c8700 1 --2- 172.21.15.174:0/4200654024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed50074670 0x7fed50074a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fed40009910 tx=0x7fed40009c20 comp rx=0 tx=0).stop 2024-03-18T14:44:59.120 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.117+0000 7fed576c8700 1 -- 172.21.15.174:0/4200654024 shutdown_connections 2024-03-18T14:44:59.120 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.117+0000 7fed576c8700 1 --2- 172.21.15.174:0/4200654024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed50074670 0x7fed50074a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:59.120 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.117+0000 7fed576c8700 1 -- 172.21.15.174:0/4200654024 >> 172.21.15.174:0/4200654024 conn(0x7fed500fd8a0 msgr2=0x7fed500ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:59.120 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.117+0000 7fed576c8700 1 -- 172.21.15.174:0/4200654024 shutdown_connections 2024-03-18T14:44:59.120 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.117+0000 7fed576c8700 1 -- 172.21.15.174:0/4200654024 wait complete. 2024-03-18T14:44:59.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.118+0000 7fed576c8700 1 Processor -- start 2024-03-18T14:44:59.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.118+0000 7fed576c8700 1 -- start start 2024-03-18T14:44:59.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.118+0000 7fed576c8700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed50074670 0x7fed5019e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:59.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.118+0000 7fed576c8700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed5019e5d0 0x7fed501a2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:59.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.118+0000 7fed576c8700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fed50112a50 con 0x7fed50074670 2024-03-18T14:44:59.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.118+0000 7fed576c8700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fed501128d0 con 0x7fed5019e5d0 2024-03-18T14:44:59.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.118+0000 7fed54c63700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed5019e5d0 0x7fed501a2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:59.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.118+0000 7fed54c63700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed5019e5d0 0x7fed501a2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:47736/0 (socket says 172.21.15.174:47736) 2024-03-18T14:44:59.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.118+0000 7fed54c63700 1 -- 172.21.15.174:0/2602667620 learned_addr learned my addr 172.21.15.174:0/2602667620 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-18T14:44:59.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.118+0000 7fed54c63700 1 -- 172.21.15.174:0/2602667620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed5019e5d0 msgr2=0x7fed501a2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-18T14:44:59.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.118+0000 7fed54c63700 1 -- 172.21.15.174:0/2602667620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed5019e5d0 msgr2=0x7fed501a2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T14:44:59.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.118+0000 7fed54c63700 1 --2- 172.21.15.174:0/2602667620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed5019e5d0 0x7fed501a2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T14:44:59.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.118+0000 7fed55464700 1 --2- 172.21.15.174:0/2602667620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed50074670 0x7fed5019e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:59.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.118+0000 7fed54c63700 1 --2- 172.21.15.174:0/2602667620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed5019e5d0 0x7fed501a2960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T14:44:59.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.119+0000 7fed55464700 1 -- 172.21.15.174:0/2602667620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed5019e5d0 msgr2=0x7fed501a2960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T14:44:59.122 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.119+0000 7fed55464700 1 --2- 172.21.15.174:0/2602667620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed5019e5d0 0x7fed501a2960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:59.122 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.119+0000 7fed55464700 1 -- 172.21.15.174:0/2602667620 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5019ec10 con 0x7fed50074670 2024-03-18T14:44:59.122 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.119+0000 7fed55464700 1 --2- 172.21.15.174:0/2602667620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed50074670 0x7fed5019e090 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fed40014000 tx=0x7fed40005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:59.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.119+0000 7fed4e7fc700 1 -- 172.21.15.174:0/2602667620 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fed40005dc0 con 0x7fed50074670 2024-03-18T14:44:59.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.119+0000 7fed4e7fc700 1 -- 172.21.15.174:0/2602667620 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed4000be40 con 0x7fed50074670 2024-03-18T14:44:59.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.119+0000 7fed576c8700 1 -- 172.21.15.174:0/2602667620 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed501a2ea0 con 0x7fed50074670 2024-03-18T14:44:59.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.119+0000 7fed4e7fc700 1 -- 172.21.15.174:0/2602667620 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fed4000f460 con 0x7fed50074670 2024-03-18T14:44:59.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.119+0000 7fed576c8700 1 -- 172.21.15.174:0/2602667620 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed501a33a0 con 0x7fed50074670 2024-03-18T14:44:59.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.120+0000 7fed576c8700 1 -- 172.21.15.174:0/2602667620 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed50075db0 con 0x7fed50074670 2024-03-18T14:44:59.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.120+0000 7fed4e7fc700 1 -- 172.21.15.174:0/2602667620 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fed4002a070 con 0x7fed50074670 2024-03-18T14:44:59.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.120+0000 7fed4e7fc700 1 --2- 172.21.15.174:0/2602667620 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fed38038d60 0x7fed3803b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:59.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.120+0000 7fed4e7fc700 1 -- 172.21.15.174:0/2602667620 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fed4001e030 con 0x7fed50074670 2024-03-18T14:44:59.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.121+0000 7fed54c63700 1 --2- 172.21.15.174:0/2602667620 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fed38038d60 0x7fed3803b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:59.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.121+0000 7fed54c63700 1 --2- 172.21.15.174:0/2602667620 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fed38038d60 0x7fed3803b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fed44009180 tx=0x7fed44008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:59.127 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.126+0000 7fed4e7fc700 1 -- 172.21.15.174:0/2602667620 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fed40013070 con 0x7fed50074670 2024-03-18T14:44:59.347 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 podman[35846]: 2024-03-18 14:44:59.323320611 +0000 UTC m=+0.344029954 container start a1709407081a240278764b7663ece9ff8127269e6128b30870b305dc4ee40af1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-18T14:44:59.376 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.375+0000 7fed4e7fc700 1 -- 172.21.15.174:0/2602667620 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed400227c0 con 0x7fed50074670 2024-03-18T14:44:59.402 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:44:59.401+0000 7fed576c8700 1 -- 172.21.15.174:0/2602667620 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fed501a3650 con 0x7fed50074670 2024-03-18T14:44:59.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:44:59 smithi076 ceph-mon[32205]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:59.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:44:59 smithi174 ceph-mon[36787]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:59.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: set uid:gid to 167:167 (ceph:ceph) 2024-03-18T14:44:59.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-18T14:44:59.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: pidfile_write: ignore empty --pid-file 2024-03-18T14:44:59.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: load: jerasure load: lrc 2024-03-18T14:44:59.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: RocksDB version: 7.9.2 2024-03-18T14:44:59.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Git sha 0 2024-03-18T14:44:59.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-18T14:44:59.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: DB SUMMARY 2024-03-18T14:44:59.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: DB Session ID: BQOKGDJKKXY46CP5BEPP 2024-03-18T14:44:59.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: CURRENT file: CURRENT 2024-03-18T14:44:59.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: IDENTITY file: IDENTITY 2024-03-18T14:44:59.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-18T14:44:59.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-18T14:44:59.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-18T14:44:59.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.error_if_exists: 0 2024-03-18T14:44:59.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.create_if_missing: 0 2024-03-18T14:44:59.730 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.paranoid_checks: 1 2024-03-18T14:44:59.730 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-18T14:44:59.730 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-18T14:44:59.730 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-18T14:44:59.730 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.env: 0x55a1977ccc60 2024-03-18T14:44:59.730 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.fs: PosixFileSystem 2024-03-18T14:44:59.730 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.info_log: 0x55a1997ffac0 2024-03-18T14:44:59.730 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_file_opening_threads: 16 2024-03-18T14:44:59.730 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.statistics: (nil) 2024-03-18T14:44:59.730 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.use_fsync: 0 2024-03-18T14:44:59.730 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_log_file_size: 0 2024-03-18T14:44:59.730 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-18T14:44:59.730 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-18T14:44:59.730 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.keep_log_file_num: 1000 2024-03-18T14:44:59.730 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.recycle_log_file_num: 0 2024-03-18T14:44:59.731 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.allow_fallocate: 1 2024-03-18T14:44:59.731 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.allow_mmap_reads: 0 2024-03-18T14:44:59.731 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.allow_mmap_writes: 0 2024-03-18T14:44:59.731 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.use_direct_reads: 0 2024-03-18T14:44:59.731 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-18T14:44:59.731 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.create_missing_column_families: 0 2024-03-18T14:44:59.731 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.db_log_dir: 2024-03-18T14:44:59.731 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.wal_dir: 2024-03-18T14:44:59.731 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-18T14:44:59.732 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-18T14:44:59.733 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-18T14:44:59.733 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-18T14:44:59.733 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-18T14:44:59.733 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-18T14:44:59.733 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.advise_random_on_open: 1 2024-03-18T14:44:59.733 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.db_write_buffer_size: 0 2024-03-18T14:44:59.733 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.write_buffer_manager: 0x55a1997d21e0 2024-03-18T14:44:59.733 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-18T14:44:59.733 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-18T14:44:59.733 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-18T14:44:59.733 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.rate_limiter: (nil) 2024-03-18T14:44:59.733 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-18T14:44:59.733 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.wal_recovery_mode: 2 2024-03-18T14:44:59.733 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.enable_thread_tracking: 0 2024-03-18T14:44:59.733 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.enable_pipelined_write: 0 2024-03-18T14:44:59.734 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.unordered_write: 0 2024-03-18T14:44:59.734 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-18T14:44:59.734 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-18T14:44:59.734 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-18T14:44:59.734 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-18T14:44:59.734 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.row_cache: None 2024-03-18T14:44:59.734 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.wal_filter: None 2024-03-18T14:44:59.734 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-18T14:44:59.734 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.allow_ingest_behind: 0 2024-03-18T14:44:59.734 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.two_write_queues: 0 2024-03-18T14:44:59.734 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.manual_wal_flush: 0 2024-03-18T14:44:59.734 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.wal_compression: 0 2024-03-18T14:44:59.734 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.atomic_flush: 0 2024-03-18T14:44:59.734 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-18T14:44:59.734 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-18T14:44:59.735 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-18T14:44:59.735 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.log_readahead_size: 0 2024-03-18T14:44:59.735 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-18T14:44:59.735 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.best_efforts_recovery: 0 2024-03-18T14:44:59.735 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-18T14:44:59.735 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-18T14:44:59.735 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.allow_data_in_errors: 0 2024-03-18T14:44:59.735 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.db_host_id: __hostname__ 2024-03-18T14:44:59.735 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-18T14:44:59.735 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_background_jobs: 2 2024-03-18T14:44:59.735 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_background_compactions: -1 2024-03-18T14:44:59.735 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_subcompactions: 1 2024-03-18T14:44:59.735 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-18T14:44:59.735 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-18T14:44:59.735 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-18T14:44:59.735 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_total_wal_size: 0 2024-03-18T14:44:59.735 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-18T14:44:59.736 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-18T14:44:59.736 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-18T14:44:59.736 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-18T14:44:59.736 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_open_files: -1 2024-03-18T14:44:59.736 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.bytes_per_sync: 0 2024-03-18T14:44:59.736 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-18T14:44:59.736 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-18T14:44:59.736 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compaction_readahead_size: 0 2024-03-18T14:44:59.736 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_background_flushes: -1 2024-03-18T14:44:59.736 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Compression algorithms supported: 2024-03-18T14:44:59.736 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-18T14:44:59.736 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: kZSTD supported: 0 2024-03-18T14:44:59.736 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: kXpressCompression supported: 0 2024-03-18T14:44:59.736 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: kLZ4HCCompression supported: 1 2024-03-18T14:44:59.736 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: kZlibCompression supported: 1 2024-03-18T14:44:59.736 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: kSnappyCompression supported: 1 2024-03-18T14:44:59.736 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: kLZ4Compression supported: 1 2024-03-18T14:44:59.737 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: kBZip2Compression supported: 0 2024-03-18T14:44:59.737 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-18T14:44:59.737 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-18T14:44:59.737 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-18T14:44:59.737 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-18T14:44:59.737 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-18T14:44:59.737 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.merge_operator: 2024-03-18T14:44:59.737 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compaction_filter: None 2024-03-18T14:44:59.737 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compaction_filter_factory: None 2024-03-18T14:44:59.737 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.sst_partitioner_factory: None 2024-03-18T14:44:59.737 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-18T14:44:59.737 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-18T14:44:59.737 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a1997ffbc0) 2024-03-18T14:44:59.737 INFO:journalctl@ceph.mon.b.smithi077.stdout: cache_index_and_filter_blocks: 1 2024-03-18T14:44:59.737 INFO:journalctl@ceph.mon.b.smithi077.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-18T14:44:59.737 INFO:journalctl@ceph.mon.b.smithi077.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-18T14:44:59.737 INFO:journalctl@ceph.mon.b.smithi077.stdout: pin_top_level_index_and_filter: 1 2024-03-18T14:44:59.738 INFO:journalctl@ceph.mon.b.smithi077.stdout: index_type: 0 2024-03-18T14:44:59.738 INFO:journalctl@ceph.mon.b.smithi077.stdout: data_block_index_type: 0 2024-03-18T14:44:59.738 INFO:journalctl@ceph.mon.b.smithi077.stdout: index_shortening: 1 2024-03-18T14:44:59.738 INFO:journalctl@ceph.mon.b.smithi077.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-18T14:44:59.738 INFO:journalctl@ceph.mon.b.smithi077.stdout: checksum: 4 2024-03-18T14:44:59.738 INFO:journalctl@ceph.mon.b.smithi077.stdout: no_block_cache: 0 2024-03-18T14:44:59.738 INFO:journalctl@ceph.mon.b.smithi077.stdout: block_cache: 0x55a199805090 2024-03-18T14:44:59.738 INFO:journalctl@ceph.mon.b.smithi077.stdout: block_cache_name: BinnedLRUCache 2024-03-18T14:44:59.738 INFO:journalctl@ceph.mon.b.smithi077.stdout: block_cache_options: 2024-03-18T14:44:59.738 INFO:journalctl@ceph.mon.b.smithi077.stdout: capacity : 536870912 2024-03-18T14:44:59.738 INFO:journalctl@ceph.mon.b.smithi077.stdout: num_shard_bits : 4 2024-03-18T14:44:59.738 INFO:journalctl@ceph.mon.b.smithi077.stdout: strict_capacity_limit : 0 2024-03-18T14:44:59.738 INFO:journalctl@ceph.mon.b.smithi077.stdout: high_pri_pool_ratio: 0.000 2024-03-18T14:44:59.738 INFO:journalctl@ceph.mon.b.smithi077.stdout: block_cache_compressed: (nil) 2024-03-18T14:44:59.738 INFO:journalctl@ceph.mon.b.smithi077.stdout: persistent_cache: (nil) 2024-03-18T14:44:59.738 INFO:journalctl@ceph.mon.b.smithi077.stdout: block_size: 4096 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout: block_size_deviation: 10 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout: block_restart_interval: 16 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout: index_block_restart_interval: 1 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout: metadata_block_size: 4096 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout: partition_filters: 0 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout: use_delta_encoding: 1 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout: filter_policy: bloomfilter 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout: whole_key_filtering: 1 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout: verify_compression: 0 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout: read_amp_bytes_per_bit: 0 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout: format_version: 5 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout: enable_index_compression: 1 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout: block_align: 0 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout: max_auto_readahead_size: 262144 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout: prepopulate_block_cache: 0 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout: initial_auto_readahead_size: 8192 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout: num_file_reads_for_auto_readahead: 2 2024-03-18T14:44:59.739 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.write_buffer_size: 33554432 2024-03-18T14:44:59.740 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_write_buffer_number: 2 2024-03-18T14:44:59.740 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compression: NoCompression 2024-03-18T14:44:59.740 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.bottommost_compression: Disabled 2024-03-18T14:44:59.740 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T14:44:59.740 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T14:44:59.740 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.num_levels: 7 2024-03-18T14:44:59.740 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T14:44:59.740 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T14:44:59.740 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T14:44:59.740 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T14:44:59.740 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T14:44:59.740 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T14:44:59.740 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T14:44:59.740 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T14:44:59.740 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T14:44:59.740 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T14:44:59.740 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-18T14:44:59.741 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-18T14:44:59.741 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T14:44:59.741 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T14:44:59.741 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T14:44:59.741 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T14:44:59.741 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T14:44:59.741 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-18T14:44:59.741 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T14:44:59.741 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T14:44:59.741 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-18T14:44:59.741 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T14:44:59.741 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T14:44:59.741 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T14:44:59.741 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T14:44:59.741 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T14:44:59.741 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T14:44:59.742 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T14:44:59.742 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T14:44:59.742 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T14:44:59.742 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T14:44:59.742 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T14:44:59.742 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T14:44:59.742 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T14:44:59.742 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T14:44:59.742 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T14:44:59.742 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T14:44:59.742 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T14:44:59.742 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-18T14:44:59.742 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.arena_block_size: 1048576 2024-03-18T14:44:59.742 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T14:44:59.742 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T14:44:59.742 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T14:44:59.743 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T14:44:59.743 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T14:44:59.743 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T14:44:59.743 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T14:44:59.743 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T14:44:59.743 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T14:44:59.743 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T14:44:59.743 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T14:44:59.743 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T14:44:59.743 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T14:44:59.743 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.table_properties_collectors: 2024-03-18T14:44:59.743 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.inplace_update_support: 0 2024-03-18T14:44:59.743 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T14:44:59.743 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T14:44:59.743 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T14:44:59.744 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T14:44:59.744 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.bloom_locality: 0 2024-03-18T14:44:59.744 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.max_successive_merges: 0 2024-03-18T14:44:59.744 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T14:44:59.744 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T14:44:59.744 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T14:44:59.744 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T14:44:59.744 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.ttl: 2592000 2024-03-18T14:44:59.744 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T14:44:59.744 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-18T14:44:59.744 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-18T14:44:59.744 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.enable_blob_files: false 2024-03-18T14:44:59.744 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.min_blob_size: 0 2024-03-18T14:44:59.744 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T14:44:59.744 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T14:44:59.744 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T14:44:59.744 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T14:44:59.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-18T14:44:59.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-18T14:44:59.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.blob_file_starting_level: 0 2024-03-18T14:44:59.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-18T14:44:59.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-18T14:44:59.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-18T14:44:59.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0233fea2-797c-4547-83a9-371910092338 2024-03-18T14:44:59.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710773099352975, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-18T14:44:59.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-18T14:44:59.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710773099353374, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710773099, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0233fea2-797c-4547-83a9-371910092338", "db_session_id": "BQOKGDJKKXY46CP5BEPP", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-18T14:44:59.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710773099353452, "job": 1, "event": "recovery_finished"} 2024-03-18T14:44:59.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-18T14:44:59.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T14:44:59.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a1998f8000 2024-03-18T14:44:59.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: DB pointer 0x55a1998e2000 2024-03-18T14:44:59.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-18T14:44:59.746 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-18T14:44:59.746 INFO:journalctl@ceph.mon.b.smithi077.stdout: ** DB Stats ** 2024-03-18T14:44:59.746 INFO:journalctl@ceph.mon.b.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T14:44:59.746 INFO:journalctl@ceph.mon.b.smithi077.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T14:44:59.746 INFO:journalctl@ceph.mon.b.smithi077.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T14:44:59.746 INFO:journalctl@ceph.mon.b.smithi077.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T14:44:59.746 INFO:journalctl@ceph.mon.b.smithi077.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T14:44:59.746 INFO:journalctl@ceph.mon.b.smithi077.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T14:44:59.746 INFO:journalctl@ceph.mon.b.smithi077.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T14:44:59.746 INFO:journalctl@ceph.mon.b.smithi077.stdout: 2024-03-18T14:44:59.746 INFO:journalctl@ceph.mon.b.smithi077.stdout: ** Compaction Stats [default] ** 2024-03-18T14:44:59.746 INFO:journalctl@ceph.mon.b.smithi077.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T14:44:59.746 INFO:journalctl@ceph.mon.b.smithi077.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-18T14:44:59.746 INFO:journalctl@ceph.mon.b.smithi077.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T14:44:59.746 INFO:journalctl@ceph.mon.b.smithi077.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T14:44:59.747 INFO:journalctl@ceph.mon.b.smithi077.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T14:44:59.747 INFO:journalctl@ceph.mon.b.smithi077.stdout: 2024-03-18T14:44:59.747 INFO:journalctl@ceph.mon.b.smithi077.stdout: ** Compaction Stats [default] ** 2024-03-18T14:44:59.747 INFO:journalctl@ceph.mon.b.smithi077.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T14:44:59.747 INFO:journalctl@ceph.mon.b.smithi077.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T14:44:59.747 INFO:journalctl@ceph.mon.b.smithi077.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T14:44:59.747 INFO:journalctl@ceph.mon.b.smithi077.stdout: 2024-03-18T14:44:59.747 INFO:journalctl@ceph.mon.b.smithi077.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-18T14:44:59.747 INFO:journalctl@ceph.mon.b.smithi077.stdout: 2024-03-18T14:44:59.747 INFO:journalctl@ceph.mon.b.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T14:44:59.747 INFO:journalctl@ceph.mon.b.smithi077.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T14:44:59.747 INFO:journalctl@ceph.mon.b.smithi077.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T14:44:59.747 INFO:journalctl@ceph.mon.b.smithi077.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T14:44:59.747 INFO:journalctl@ceph.mon.b.smithi077.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T14:44:59.747 INFO:journalctl@ceph.mon.b.smithi077.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T14:44:59.747 INFO:journalctl@ceph.mon.b.smithi077.stdout: Cumulative compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T14:44:59.748 INFO:journalctl@ceph.mon.b.smithi077.stdout: Interval compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T14:44:59.748 INFO:journalctl@ceph.mon.b.smithi077.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T14:44:59.748 INFO:journalctl@ceph.mon.b.smithi077.stdout: Block cache BinnedLRUCache@0x55a199805090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-18T14:44:59.748 INFO:journalctl@ceph.mon.b.smithi077.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-18T14:44:59.748 INFO:journalctl@ceph.mon.b.smithi077.stdout: 2024-03-18T14:44:59.748 INFO:journalctl@ceph.mon.b.smithi077.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T14:44:59.748 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-18T14:44:59.748 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: using public_addr v2:172.21.15.77:0/0 -> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] 2024-03-18T14:44:59.748 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: starting mon.b rank -1 at public addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] at bind addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:44:59.748 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.b@-1(???) e0 preinit fsid 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:44:59.748 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.b@-1(synchronizing).mds e1 new map 2024-03-18T14:44:59.748 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-18T14:44:59.748 INFO:journalctl@ceph.mon.b.smithi077.stdout: e1 2024-03-18T14:44:59.748 INFO:journalctl@ceph.mon.b.smithi077.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T14:44:59.748 INFO:journalctl@ceph.mon.b.smithi077.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T14:44:59.748 INFO:journalctl@ceph.mon.b.smithi077.stdout: legacy client fscid: -1 2024-03-18T14:44:59.748 INFO:journalctl@ceph.mon.b.smithi077.stdout: 2024-03-18T14:44:59.749 INFO:journalctl@ceph.mon.b.smithi077.stdout: No filesystems configured 2024-03-18T14:44:59.749 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-18T14:44:59.749 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-18T14:44:59.749 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-18T14:44:59.749 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-18T14:44:59.749 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-18T14:44:59.749 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-18T14:44:59.749 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T14:44:59.749 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T14:44:59.749 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T14:44:59.749 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T14:44:59.749 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.749 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:59.749 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.749 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:44:59.749 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:59.749 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:44:59.750 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.750 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.750 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.750 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.750 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:44:59.750 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:44:59.750 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:44:59.750 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: Deploying cephadm binary to smithi174 2024-03-18T14:44:59.750 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: Updating smithi077:/etc/ceph/ceph.conf 2024-03-18T14:44:59.750 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: Updating smithi077:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:44:59.750 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:59.751 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:59.751 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.751 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.751 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.751 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.751 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:44:59.751 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:44:59.751 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: Updating smithi077:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T14:44:59.751 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: Added host smithi174 2024-03-18T14:44:59.751 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.751 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.751 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:59.752 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.752 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:44:59.752 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:59.752 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3945439726' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T14:44:59.752 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:59.752 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3945439726' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T14:44:59.752 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T14:44:59.752 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.752 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.752 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.752 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.752 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:44:59.752 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:44:59.752 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:44:59.752 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-18T14:44:59.752 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: Updating smithi174:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:44:59.752 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:44:59.753 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: Updating smithi174:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T14:44:59.753 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.753 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.753 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.753 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:59.753 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.753 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:44:59.753 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:44:59.753 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:44:59.753 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.753 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T14:44:59.753 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:44:59.753 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi076:172.21.15.76=a;smithi077:172.21.15.77=b;smithi174:172.21.15.174=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:44:59.753 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: Saving service mon spec with placement smithi076:172.21.15.76=a;smithi077:172.21.15.77=b;smithi174:172.21.15.174=c;count:3 2024-03-18T14:44:59.753 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: Deploying daemon mon.c on smithi174 2024-03-18T14:44:59.754 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:59.754 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:59.754 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='client.? 172.21.15.174:0/2764648823' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T14:44:59.754 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:59.754 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:44:59.754 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:59.754 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.a calling monitor election 2024-03-18T14:44:59.754 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:59.754 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:59.754 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:59.754 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.c calling monitor election 2024-03-18T14:44:59.754 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:59.754 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:59.754 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:59.754 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:59.754 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T14:44:59.755 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: monmap epoch 2 2024-03-18T14:44:59.755 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: fsid 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:44:59.755 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: last_changed 2024-03-18T14:44:50.340639+0000 2024-03-18T14:44:59.755 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: created 2024-03-18T14:43:08.949643+0000 2024-03-18T14:44:59.755 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: min_mon_release 19 (squid) 2024-03-18T14:44:59.755 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: election_strategy: 1 2024-03-18T14:44:59.755 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-03-18T14:44:59.755 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-03-18T14:44:59.755 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: fsmap 2024-03-18T14:44:59.755 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T14:44:59.755 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mgrmap e13: x(active, since 49s) 2024-03-18T14:44:59.755 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: overall HEALTH_OK 2024-03-18T14:44:59.755 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.755 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.755 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:44:59.755 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T14:44:59.756 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:44:59.756 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: Deploying daemon mon.b on smithi077 2024-03-18T14:44:59.756 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='client.? 172.21.15.174:0/3594535122' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T14:44:59.756 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:59.756 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:44:59.756 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:44:59.756 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 ceph-mon[35865]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-18T14:44:59.756 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 bash[35846]: a1709407081a240278764b7663ece9ff8127269e6128b30870b305dc4ee40af1 2024-03-18T14:44:59.756 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:44:59 smithi077 systemd[1]: Started Ceph mon.b for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:45:04.433 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:04.432+0000 7fed4e7fc700 1 -- 172.21.15.174:0/2602667620 <== mon.0 v2:172.21.15.76:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1281 (secure 0 0 0) 0x7fed40028260 con 0x7fed50074670 2024-03-18T14:45:04.434 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-18T14:45:04.434 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":3,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","modified":"2024-03-18T14:44:59.375289Z","created":"2024-03-18T14:43:08.949643Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3300","nonce":0},{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-18T14:45:04.436 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:04.435+0000 7fed576c8700 1 -- 172.21.15.174:0/2602667620 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fed38038d60 msgr2=0x7fed3803b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:04.436 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:04.435+0000 7fed576c8700 1 --2- 172.21.15.174:0/2602667620 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fed38038d60 0x7fed3803b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fed44009180 tx=0x7fed44008040 comp rx=0 tx=0).stop 2024-03-18T14:45:04.436 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:04.435+0000 7fed576c8700 1 -- 172.21.15.174:0/2602667620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed50074670 msgr2=0x7fed5019e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:04.436 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:04.435+0000 7fed576c8700 1 --2- 172.21.15.174:0/2602667620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed50074670 0x7fed5019e090 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fed40014000 tx=0x7fed40005030 comp rx=0 tx=0).stop 2024-03-18T14:45:04.437 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:04.435+0000 7fed576c8700 1 -- 172.21.15.174:0/2602667620 shutdown_connections 2024-03-18T14:45:04.437 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:04.435+0000 7fed576c8700 1 --2- 172.21.15.174:0/2602667620 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fed38038d60 0x7fed3803b220 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:04.437 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:04.435+0000 7fed576c8700 1 --2- 172.21.15.174:0/2602667620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed50074670 0x7fed5019e090 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:04.437 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:04.435+0000 7fed576c8700 1 --2- 172.21.15.174:0/2602667620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed5019e5d0 0x7fed501a2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:04.437 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:04.435+0000 7fed576c8700 1 -- 172.21.15.174:0/2602667620 >> 172.21.15.174:0/2602667620 conn(0x7fed500fd8a0 msgr2=0x7fed50103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:45:04.437 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:04.435+0000 7fed576c8700 1 -- 172.21.15.174:0/2602667620 shutdown_connections 2024-03-18T14:45:04.437 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:04.435+0000 7fed576c8700 1 -- 172.21.15.174:0/2602667620 wait complete. 2024-03-18T14:45:04.437 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 3 2024-03-18T14:45:04.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:45:04.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: mon.a calling monitor election 2024-03-18T14:45:04.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: mon.c calling monitor election 2024-03-18T14:45:04.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:45:04.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: from='client.? 172.21.15.174:0/2602667620' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T14:45:04.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:04.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: mon.b calling monitor election 2024-03-18T14:45:04.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:04.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:04.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T14:45:04.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: monmap epoch 3 2024-03-18T14:45:04.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: fsid 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:45:04.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: last_changed 2024-03-18T14:44:59.375289+0000 2024-03-18T14:45:04.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: created 2024-03-18T14:43:08.949643+0000 2024-03-18T14:45:04.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: min_mon_release 19 (squid) 2024-03-18T14:45:04.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: election_strategy: 1 2024-03-18T14:45:04.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-03-18T14:45:04.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-03-18T14:45:04.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: 2: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.b 2024-03-18T14:45:04.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: fsmap 2024-03-18T14:45:04.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T14:45:04.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: mgrmap e13: x(active, since 58s) 2024-03-18T14:45:04.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: overall HEALTH_OK 2024-03-18T14:45:04.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:04.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:04.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:04.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:04 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:04.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:45:04.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: mon.a calling monitor election 2024-03-18T14:45:04.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: mon.c calling monitor election 2024-03-18T14:45:04.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:45:04.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: from='client.? 172.21.15.174:0/2602667620' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T14:45:04.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:04.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: mon.b calling monitor election 2024-03-18T14:45:04.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:04.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:04.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T14:45:04.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: monmap epoch 3 2024-03-18T14:45:04.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: fsid 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:45:04.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: last_changed 2024-03-18T14:44:59.375289+0000 2024-03-18T14:45:04.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: created 2024-03-18T14:43:08.949643+0000 2024-03-18T14:45:04.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: min_mon_release 19 (squid) 2024-03-18T14:45:04.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: election_strategy: 1 2024-03-18T14:45:04.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-03-18T14:45:04.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-03-18T14:45:04.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: 2: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.b 2024-03-18T14:45:04.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: fsmap 2024-03-18T14:45:04.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T14:45:04.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: mgrmap e13: x(active, since 58s) 2024-03-18T14:45:04.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: overall HEALTH_OK 2024-03-18T14:45:04.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:04.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:04.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:04.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:04 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:04.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:45:04.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: mon.a calling monitor election 2024-03-18T14:45:04.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: mon.c calling monitor election 2024-03-18T14:45:04.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:45:04.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: from='client.? 172.21.15.174:0/2602667620' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T14:45:04.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:04.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: mon.b calling monitor election 2024-03-18T14:45:04.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:04.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:04.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:04.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T14:45:04.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: monmap epoch 3 2024-03-18T14:45:04.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: fsid 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:45:04.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: last_changed 2024-03-18T14:44:59.375289+0000 2024-03-18T14:45:04.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: created 2024-03-18T14:43:08.949643+0000 2024-03-18T14:45:04.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: min_mon_release 19 (squid) 2024-03-18T14:45:04.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: election_strategy: 1 2024-03-18T14:45:04.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-03-18T14:45:04.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-03-18T14:45:04.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: 2: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.b 2024-03-18T14:45:04.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: fsmap 2024-03-18T14:45:04.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T14:45:04.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: mgrmap e13: x(active, since 58s) 2024-03-18T14:45:04.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: overall HEALTH_OK 2024-03-18T14:45:04.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:04.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:04.981 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:04.981 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:04 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:05.024 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-18T14:45:05.025 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-18T14:45:05.271 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:45:05.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:05 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:05.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:05 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:05.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:05 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:05.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:05 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:05.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:05 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:05.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:05 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:45:06.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.838+0000 7fb6164ce700 1 -- 172.21.15.76:0/3029743286 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb610100e00 msgr2=0x7fb6101011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:06.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.838+0000 7fb6164ce700 1 --2- 172.21.15.76:0/3029743286 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb610100e00 0x7fb6101011e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb600009910 tx=0x7fb600009c20 comp rx=0 tx=0).stop 2024-03-18T14:45:06.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.838+0000 7fb6164ce700 1 -- 172.21.15.76:0/3029743286 shutdown_connections 2024-03-18T14:45:06.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.838+0000 7fb6164ce700 1 --2- 172.21.15.76:0/3029743286 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb610100e00 0x7fb6101011e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:06.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.838+0000 7fb6164ce700 1 -- 172.21.15.76:0/3029743286 >> 172.21.15.76:0/3029743286 conn(0x7fb610077810 msgr2=0x7fb6100feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:45:06.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.839+0000 7fb6164ce700 1 -- 172.21.15.76:0/3029743286 shutdown_connections 2024-03-18T14:45:06.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.839+0000 7fb6164ce700 1 -- 172.21.15.76:0/3029743286 wait complete. 2024-03-18T14:45:06.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.839+0000 7fb6164ce700 1 Processor -- start 2024-03-18T14:45:06.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.839+0000 7fb6164ce700 1 -- start start 2024-03-18T14:45:06.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb6164ce700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb610100e00 0x7fb6101988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:45:06.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb6164ce700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb610198e10 0x7fb610199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:45:06.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb6164ce700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb6101a1940 0x7fb6101a3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:45:06.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb6164ce700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb61010b1d0 con 0x7fb610100e00 2024-03-18T14:45:06.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb6164ce700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb61010b050 con 0x7fb610198e10 2024-03-18T14:45:06.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb6164ce700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb61010b350 con 0x7fb6101a1940 2024-03-18T14:45:06.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb60ffff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb610100e00 0x7fb6101988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:45:06.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb60ffff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb610100e00 0x7fb6101988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51544/0 (socket says 172.21.15.76:51544) 2024-03-18T14:45:06.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb60f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb610198e10 0x7fb610199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:45:06.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb60ffff700 1 -- 172.21.15.76:0/3262999910 learned_addr learned my addr 172.21.15.76:0/3262999910 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:45:06.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb614a6b700 1 --2- 172.21.15.76:0/3262999910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb6101a1940 0x7fb6101a3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:45:06.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb60ffff700 1 -- 172.21.15.76:0/3262999910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb6101a1940 msgr2=0x7fb6101a3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:06.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb60ffff700 1 --2- 172.21.15.76:0/3262999910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb6101a1940 0x7fb6101a3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:06.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb60ffff700 1 -- 172.21.15.76:0/3262999910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb610198e10 msgr2=0x7fb610199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:06.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb60ffff700 1 --2- 172.21.15.76:0/3262999910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb610198e10 0x7fb610199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:06.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb60ffff700 1 -- 172.21.15.76:0/3262999910 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6101a44f0 con 0x7fb610100e00 2024-03-18T14:45:06.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb60ffff700 1 --2- 172.21.15.76:0/3262999910 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb610100e00 0x7fb6101988d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb600005850 tx=0x7fb600004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:45:06.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb60d7fa700 1 -- 172.21.15.76:0/3262999910 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb600026070 con 0x7fb610100e00 2024-03-18T14:45:06.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb60d7fa700 1 -- 172.21.15.76:0/3262999910 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb60000b6f0 con 0x7fb610100e00 2024-03-18T14:45:06.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb6164ce700 1 -- 172.21.15.76:0/3262999910 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6101a47e0 con 0x7fb610100e00 2024-03-18T14:45:06.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.840+0000 7fb60d7fa700 1 -- 172.21.15.76:0/3262999910 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6000038e0 con 0x7fb610100e00 2024-03-18T14:45:06.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.841+0000 7fb6164ce700 1 -- 172.21.15.76:0/3262999910 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6101a4d20 con 0x7fb610100e00 2024-03-18T14:45:06.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.841+0000 7fb60d7fa700 1 -- 172.21.15.76:0/3262999910 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fb600029070 con 0x7fb610100e00 2024-03-18T14:45:06.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.841+0000 7fb60d7fa700 1 --2- 172.21.15.76:0/3262999910 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb5f8039450 0x7fb5f803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:45:06.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.841+0000 7fb60d7fa700 1 -- 172.21.15.76:0/3262999910 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb60004dc80 con 0x7fb610100e00 2024-03-18T14:45:06.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.841+0000 7fb6164ce700 1 -- 172.21.15.76:0/3262999910 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6101022e0 con 0x7fb610100e00 2024-03-18T14:45:06.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.842+0000 7fb60f7fe700 1 --2- 172.21.15.76:0/3262999910 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb5f8039450 0x7fb5f803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:45:06.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.842+0000 7fb60f7fe700 1 --2- 172.21.15.76:0/3262999910 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb5f8039450 0x7fb5f803b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb604009ad0 tx=0x7fb604009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:45:06.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:06.847+0000 7fb60d7fa700 1 -- 172.21.15.76:0/3262999910 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb600030070 con 0x7fb610100e00 2024-03-18T14:45:06.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:06.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:06.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:06.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:06.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:06.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:06.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: Updating smithi076:/etc/ceph/ceph.conf 2024-03-18T14:45:06.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: Updating smithi077:/etc/ceph/ceph.conf 2024-03-18T14:45:06.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-18T14:45:06.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: Updating smithi076:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:45:06.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: Updating smithi077:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:45:06.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: Updating smithi174:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:45:06.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:06.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:06.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:06.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:06.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:06.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:06.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:06.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:06.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:06.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:06.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:06.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:06.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:06.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T14:45:06.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T14:45:06.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:07.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:07.052+0000 7fb6164ce700 1 -- 172.21.15.76:0/3262999910 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fb6101a4f80 con 0x7fb610100e00 2024-03-18T14:45:07.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:07.053+0000 7fb60d7fa700 1 -- 172.21.15.76:0/3262999910 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7fb600013070 con 0x7fb610100e00 2024-03-18T14:45:07.054 INFO:teuthology.orchestra.run.smithi076.stdout:# minimal ceph.conf for 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:45:07.054 INFO:teuthology.orchestra.run.smithi076.stdout:[global] 2024-03-18T14:45:07.055 INFO:teuthology.orchestra.run.smithi076.stdout: fsid = 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:45:07.055 INFO:teuthology.orchestra.run.smithi076.stdout: mon_host = [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-03-18T14:45:07.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:07.055+0000 7fb6164ce700 1 -- 172.21.15.76:0/3262999910 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb5f8039450 msgr2=0x7fb5f803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:07.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:07.055+0000 7fb6164ce700 1 --2- 172.21.15.76:0/3262999910 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb5f8039450 0x7fb5f803b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb604009ad0 tx=0x7fb604009210 comp rx=0 tx=0).stop 2024-03-18T14:45:07.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:07.055+0000 7fb6164ce700 1 -- 172.21.15.76:0/3262999910 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb610100e00 msgr2=0x7fb6101988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:07.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:07.055+0000 7fb6164ce700 1 --2- 172.21.15.76:0/3262999910 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb610100e00 0x7fb6101988d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb600005850 tx=0x7fb600004f80 comp rx=0 tx=0).stop 2024-03-18T14:45:07.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:07.055+0000 7fb6164ce700 1 -- 172.21.15.76:0/3262999910 shutdown_connections 2024-03-18T14:45:07.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:07.055+0000 7fb6164ce700 1 --2- 172.21.15.76:0/3262999910 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb5f8039450 0x7fb5f803b910 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:07.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:07.055+0000 7fb6164ce700 1 --2- 172.21.15.76:0/3262999910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb6101a1940 0x7fb6101a3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:07.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:07.055+0000 7fb6164ce700 1 --2- 172.21.15.76:0/3262999910 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb610100e00 0x7fb6101988d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:07.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:07.055+0000 7fb6164ce700 1 --2- 172.21.15.76:0/3262999910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb610198e10 0x7fb610199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:07.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:07.055+0000 7fb6164ce700 1 -- 172.21.15.76:0/3262999910 >> 172.21.15.76:0/3262999910 conn(0x7fb610077810 msgr2=0x7fb6101027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:45:07.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:07.056+0000 7fb6164ce700 1 -- 172.21.15.76:0/3262999910 shutdown_connections 2024-03-18T14:45:07.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:07.056+0000 7fb6164ce700 1 -- 172.21.15.76:0/3262999910 wait complete. 2024-03-18T14:45:07.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:07.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:07.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: Updating smithi076:/etc/ceph/ceph.conf 2024-03-18T14:45:07.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: Updating smithi077:/etc/ceph/ceph.conf 2024-03-18T14:45:07.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-18T14:45:07.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: Updating smithi076:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:45:07.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: Updating smithi077:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:45:07.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: Updating smithi174:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:45:07.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:07.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T14:45:07.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T14:45:07.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:07.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:07.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:07.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: Updating smithi076:/etc/ceph/ceph.conf 2024-03-18T14:45:07.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: Updating smithi077:/etc/ceph/ceph.conf 2024-03-18T14:45:07.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-18T14:45:07.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: Updating smithi076:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:45:07.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: Updating smithi077:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:45:07.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: Updating smithi174:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:45:07.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:07.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:07.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T14:45:07.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T14:45:07.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:07.616 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-18T14:45:07.616 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T14:45:07.616 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T14:45:07.645 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T14:45:07.645 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:45:07.713 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-18T14:45:07.713 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T14:45:07.748 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-18T14:45:07.748 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:45:07.818 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-18T14:45:07.819 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T14:45:07.853 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-18T14:45:07.853 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:45:07.923 INFO:tasks.cephadm:Adding mgr.x on smithi076 2024-03-18T14:45:07.924 INFO:tasks.cephadm:Adding mgr.y on smithi077 2024-03-18T14:45:07.924 INFO:tasks.cephadm:Adding mgr.z on smithi174 2024-03-18T14:45:07.924 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi076=x;smithi077=y;smithi174=z' 2024-03-18T14:45:08.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:07 smithi076 ceph-mon[32205]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T14:45:08.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:07 smithi076 ceph-mon[32205]: Reconfiguring daemon mon.a on smithi076 2024-03-18T14:45:08.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:07 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3262999910' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:08.198 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T14:45:08.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:07 smithi174 ceph-mon[36787]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T14:45:08.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:07 smithi174 ceph-mon[36787]: Reconfiguring daemon mon.a on smithi076 2024-03-18T14:45:08.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:07 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3262999910' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:08.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:07 smithi077 ceph-mon[35865]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T14:45:08.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:07 smithi077 ceph-mon[35865]: Reconfiguring daemon mon.a on smithi076 2024-03-18T14:45:08.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:07 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3262999910' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:09.153 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:08 smithi077 ceph-mon[35865]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:09.154 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:08 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:09.154 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:08 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:09.154 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:08 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T14:45:09.154 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:08 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T14:45:09.154 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:08 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:09.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:08 smithi076 ceph-mon[32205]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:09.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:08 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:09.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:08 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:09.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:08 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T14:45:09.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:08 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T14:45:09.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:08 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:09.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:08 smithi174 ceph-mon[36787]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:09.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:08 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:09.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:08 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:09.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:08 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T14:45:09.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:08 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T14:45:09.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:08 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:09.563 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.559+0000 7f3559834700 1 -- 172.21.15.174:0/2874745890 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3554100d40 msgr2=0x7f3554101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:09.563 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.559+0000 7f3559834700 1 --2- 172.21.15.174:0/2874745890 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3554100d40 0x7f3554101160 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3544009910 tx=0x7f3544009c20 comp rx=0 tx=0).stop 2024-03-18T14:45:09.563 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.559+0000 7f3559834700 1 -- 172.21.15.174:0/2874745890 shutdown_connections 2024-03-18T14:45:09.563 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.559+0000 7f3559834700 1 --2- 172.21.15.174:0/2874745890 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3554100d40 0x7f3554101160 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:09.563 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.559+0000 7f3559834700 1 -- 172.21.15.174:0/2874745890 >> 172.21.15.174:0/2874745890 conn(0x7f35540fc2e0 msgr2=0x7f35540fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:45:09.564 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.559+0000 7f3559834700 1 -- 172.21.15.174:0/2874745890 shutdown_connections 2024-03-18T14:45:09.564 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.559+0000 7f3559834700 1 -- 172.21.15.174:0/2874745890 wait complete. 2024-03-18T14:45:09.564 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.560+0000 7f3559834700 1 Processor -- start 2024-03-18T14:45:09.564 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.560+0000 7f3559834700 1 -- start start 2024-03-18T14:45:09.564 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.560+0000 7f3559834700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3554100d40 0x7f35540fe280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:45:09.564 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.560+0000 7f3559834700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35540fc8d0 0x7f35540fcd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:45:09.564 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.560+0000 7f3559834700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f35540fd290 0x7f35541a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:45:09.564 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.560+0000 7f3559834700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3554106d80 con 0x7f35540fd290 2024-03-18T14:45:09.564 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.560+0000 7f3559834700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3554106c00 con 0x7f35540fc8d0 2024-03-18T14:45:09.564 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.560+0000 7f3559834700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3554106f00 con 0x7f3554100d40 2024-03-18T14:45:09.564 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.560+0000 7f3552ffd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3554100d40 0x7f35540fe280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:45:09.564 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.560+0000 7f3552ffd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3554100d40 0x7f35540fe280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:38048/0 (socket says 172.21.15.174:38048) 2024-03-18T14:45:09.564 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.560+0000 7f3552ffd700 1 -- 172.21.15.174:0/1855045849 learned_addr learned my addr 172.21.15.174:0/1855045849 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-18T14:45:09.564 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.560+0000 7f354a7fc700 1 --2- 172.21.15.174:0/1855045849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35540fc8d0 0x7f35540fcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:45:09.565 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.561+0000 7f35537fe700 1 --2- 172.21.15.174:0/1855045849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f35540fd290 0x7f35541a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:45:09.565 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.561+0000 7f3552ffd700 1 -- 172.21.15.174:0/1855045849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35540fc8d0 msgr2=0x7f35540fcd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:09.565 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.561+0000 7f3552ffd700 1 --2- 172.21.15.174:0/1855045849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35540fc8d0 0x7f35540fcd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:09.565 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.561+0000 7f3552ffd700 1 -- 172.21.15.174:0/1855045849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f35540fd290 msgr2=0x7f35541a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:09.565 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.561+0000 7f3552ffd700 1 --2- 172.21.15.174:0/1855045849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f35540fd290 0x7f35541a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:09.565 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.561+0000 7f3552ffd700 1 -- 172.21.15.174:0/1855045849 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35541a1c60 con 0x7f3554100d40 2024-03-18T14:45:09.565 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.561+0000 7f3552ffd700 1 --2- 172.21.15.174:0/1855045849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3554100d40 0x7f35540fe280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3544000c00 tx=0x7f3544005000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:45:09.565 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.561+0000 7f3558832700 1 -- 172.21.15.174:0/1855045849 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3544027070 con 0x7f3554100d40 2024-03-18T14:45:09.565 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.561+0000 7f3558832700 1 -- 172.21.15.174:0/1855045849 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3544022470 con 0x7f3554100d40 2024-03-18T14:45:09.565 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.561+0000 7f3559834700 1 -- 172.21.15.174:0/1855045849 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35541a1ef0 con 0x7f3554100d40 2024-03-18T14:45:09.565 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.561+0000 7f3558832700 1 -- 172.21.15.174:0/1855045849 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354400f6c0 con 0x7f3554100d40 2024-03-18T14:45:09.565 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.561+0000 7f3559834700 1 -- 172.21.15.174:0/1855045849 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35541a2360 con 0x7f3554100d40 2024-03-18T14:45:09.566 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.562+0000 7f3558832700 1 -- 172.21.15.174:0/1855045849 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f354402a070 con 0x7f3554100d40 2024-03-18T14:45:09.566 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.562+0000 7f3559834700 1 -- 172.21.15.174:0/1855045849 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3554063900 con 0x7f3554100d40 2024-03-18T14:45:09.566 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.562+0000 7f3558832700 1 --2- 172.21.15.174:0/1855045849 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3540039450 0x7f354003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:45:09.567 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.562+0000 7f3558832700 1 -- 172.21.15.174:0/1855045849 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f354404d1c0 con 0x7f3554100d40 2024-03-18T14:45:09.567 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.563+0000 7f354a7fc700 1 --2- 172.21.15.174:0/1855045849 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3540039450 0x7f354003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:45:09.567 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.563+0000 7f354a7fc700 1 --2- 172.21.15.174:0/1855045849 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3540039450 0x7f354003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f353c009ad0 tx=0x7f353c009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:45:09.569 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.567+0000 7f3558832700 1 -- 172.21.15.174:0/1855045849 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3544013070 con 0x7f3554100d40 2024-03-18T14:45:09.773 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.772+0000 7f3559834700 1 -- 172.21.15.174:0/1855045849 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi076=x;smithi077=y;smithi174=z", "target": ["mon-mgr", ""]}) -- 0x7f35540627a0 con 0x7f3540039450 2024-03-18T14:45:09.779 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.778+0000 7f3558832700 1 -- 172.21.15.174:0/1855045849 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f35540627a0 con 0x7f3540039450 2024-03-18T14:45:09.779 INFO:teuthology.orchestra.run.smithi174.stdout:Scheduled mgr update... 2024-03-18T14:45:09.782 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.780+0000 7f3559834700 1 -- 172.21.15.174:0/1855045849 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3540039450 msgr2=0x7f354003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:09.782 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.780+0000 7f3559834700 1 --2- 172.21.15.174:0/1855045849 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3540039450 0x7f354003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f353c009ad0 tx=0x7f353c009210 comp rx=0 tx=0).stop 2024-03-18T14:45:09.782 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.780+0000 7f3559834700 1 -- 172.21.15.174:0/1855045849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3554100d40 msgr2=0x7f35540fe280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:09.782 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.780+0000 7f3559834700 1 --2- 172.21.15.174:0/1855045849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3554100d40 0x7f35540fe280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3544000c00 tx=0x7f3544005000 comp rx=0 tx=0).stop 2024-03-18T14:45:09.782 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.780+0000 7f3559834700 1 -- 172.21.15.174:0/1855045849 shutdown_connections 2024-03-18T14:45:09.782 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.780+0000 7f3559834700 1 --2- 172.21.15.174:0/1855045849 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3540039450 0x7f354003b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:09.782 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.780+0000 7f3559834700 1 --2- 172.21.15.174:0/1855045849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f35540fd290 0x7f35541a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:09.782 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.780+0000 7f3559834700 1 --2- 172.21.15.174:0/1855045849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3554100d40 0x7f35540fe280 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:09.782 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.780+0000 7f3559834700 1 --2- 172.21.15.174:0/1855045849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35540fc8d0 0x7f35540fcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:09.782 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.780+0000 7f3559834700 1 -- 172.21.15.174:0/1855045849 >> 172.21.15.174:0/1855045849 conn(0x7f35540fc2e0 msgr2=0x7f355406aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:45:09.782 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.780+0000 7f3559834700 1 -- 172.21.15.174:0/1855045849 shutdown_connections 2024-03-18T14:45:09.782 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:45:09.780+0000 7f3559834700 1 -- 172.21.15.174:0/1855045849 wait complete. 2024-03-18T14:45:10.095 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:09 smithi174 ceph-mon[36787]: Reconfiguring mon.b (monmap changed)... 2024-03-18T14:45:10.095 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:09 smithi174 ceph-mon[36787]: Reconfiguring daemon mon.b on smithi077 2024-03-18T14:45:10.095 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:09 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:10.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:09 smithi076 ceph-mon[32205]: Reconfiguring mon.b (monmap changed)... 2024-03-18T14:45:10.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:09 smithi076 ceph-mon[32205]: Reconfiguring daemon mon.b on smithi077 2024-03-18T14:45:10.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:09 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:10.169 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:09 smithi077 ceph-mon[35865]: Reconfiguring mon.b (monmap changed)... 2024-03-18T14:45:10.169 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:09 smithi077 ceph-mon[35865]: Reconfiguring daemon mon.b on smithi077 2024-03-18T14:45:10.169 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:09 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:10.374 DEBUG:teuthology.orchestra.run.smithi077:mgr.y> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mgr.y.service 2024-03-18T14:45:10.377 DEBUG:teuthology.orchestra.run.smithi174:mgr.z> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mgr.z.service 2024-03-18T14:45:10.379 INFO:tasks.cephadm:Deploying OSDs... 2024-03-18T14:45:10.379 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T14:45:10.379 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T14:45:10.398 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T14:45:10.399 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_1 2024-03-18T14:45:10.456 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T14:45:10.456 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:45:10.456 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-18T14:45:10.456 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:45:10.456 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:45:10.456 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-03-18 14:44:18.124928246 +0000 2024-03-18T14:45:10.456 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-03-18 14:37:26.507309318 +0000 2024-03-18T14:45:10.456 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-03-18 14:37:26.507309318 +0000 2024-03-18T14:45:10.457 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-03-18 14:37:26.507309318 +0000 2024-03-18T14:45:10.457 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T14:45:10.525 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-03-18T14:45:10.526 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-03-18T14:45:10.526 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000220683 s, 2.3 MB/s 2024-03-18T14:45:10.527 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T14:45:10.584 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_2 2024-03-18T14:45:10.640 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T14:45:10.640 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:45:10.640 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-18T14:45:10.640 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:45:10.640 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:45:10.640 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-03-18 14:44:18.153927309 +0000 2024-03-18T14:45:10.640 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-03-18 14:37:26.874297305 +0000 2024-03-18T14:45:10.641 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-03-18 14:37:26.874297305 +0000 2024-03-18T14:45:10.641 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-03-18 14:37:26.874297305 +0000 2024-03-18T14:45:10.641 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T14:45:10.707 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-03-18T14:45:10.707 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-03-18T14:45:10.708 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000248673 s, 2.1 MB/s 2024-03-18T14:45:10.709 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T14:45:10.765 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_3 2024-03-18T14:45:10.822 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T14:45:10.822 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:45:10.822 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-18T14:45:10.822 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:45:10.822 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:45:10.822 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-03-18 14:44:18.179926468 +0000 2024-03-18T14:45:10.822 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-03-18 14:37:27.239285359 +0000 2024-03-18T14:45:10.822 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-03-18 14:37:27.239285359 +0000 2024-03-18T14:45:10.823 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-03-18 14:37:27.239285359 +0000 2024-03-18T14:45:10.823 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T14:45:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-03-18T14:45:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-03-18T14:45:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000198038 s, 2.6 MB/s 2024-03-18T14:45:10.889 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T14:45:10.945 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_4 2024-03-18T14:45:11.000 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T14:45:11.000 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:45:11.000 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-18T14:45:11.000 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:45:11.000 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:45:11.000 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-03-18 14:44:18.208925531 +0000 2024-03-18T14:45:11.000 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-03-18 14:37:27.600273542 +0000 2024-03-18T14:45:11.000 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-03-18 14:37:27.600273542 +0000 2024-03-18T14:45:11.001 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-03-18 14:37:27.600273542 +0000 2024-03-18T14:45:11.001 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T14:45:11.065 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-03-18T14:45:11.065 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-03-18T14:45:11.065 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000233483 s, 2.2 MB/s 2024-03-18T14:45:11.066 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T14:45:11.123 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-18T14:45:11.123 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T14:45:11.141 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T14:45:11.142 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2024-03-18T14:45:11.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:10 smithi076 ceph-mon[32205]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi076=x;smithi077=y;smithi174=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:45:11.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:10 smithi076 ceph-mon[32205]: Saving service mgr spec with placement smithi076=x;smithi077=y;smithi174=z;count:3 2024-03-18T14:45:11.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:10 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:11.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:10 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:11.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:10 smithi076 ceph-mon[32205]: Reconfiguring mon.c (monmap changed)... 2024-03-18T14:45:11.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:10 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T14:45:11.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:10 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T14:45:11.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:10 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:11.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:10 smithi076 ceph-mon[32205]: Reconfiguring daemon mon.c on smithi174 2024-03-18T14:45:11.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:10 smithi076 ceph-mon[32205]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:11.199 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T14:45:11.199 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:45:11.199 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-18T14:45:11.200 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:45:11.200 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:45:11.200 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-18 14:44:58.808848444 +0000 2024-03-18T14:45:11.200 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-18 14:37:58.525831306 +0000 2024-03-18T14:45:11.200 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-18 14:37:58.525831306 +0000 2024-03-18T14:45:11.200 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-03-18 14:37:58.525831306 +0000 2024-03-18T14:45:11.200 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T14:45:11.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:10 smithi174 ceph-mon[36787]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi076=x;smithi077=y;smithi174=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:45:11.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:10 smithi174 ceph-mon[36787]: Saving service mgr spec with placement smithi076=x;smithi077=y;smithi174=z;count:3 2024-03-18T14:45:11.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:10 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:11.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:10 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:11.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:10 smithi174 ceph-mon[36787]: Reconfiguring mon.c (monmap changed)... 2024-03-18T14:45:11.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:10 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T14:45:11.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:10 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T14:45:11.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:10 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:11.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:10 smithi174 ceph-mon[36787]: Reconfiguring daemon mon.c on smithi174 2024-03-18T14:45:11.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:10 smithi174 ceph-mon[36787]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:11.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:10 smithi077 ceph-mon[35865]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi076=x;smithi077=y;smithi174=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:45:11.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:10 smithi077 ceph-mon[35865]: Saving service mgr spec with placement smithi076=x;smithi077=y;smithi174=z;count:3 2024-03-18T14:45:11.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:10 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:11.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:10 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:11.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:10 smithi077 ceph-mon[35865]: Reconfiguring mon.c (monmap changed)... 2024-03-18T14:45:11.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:10 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T14:45:11.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:10 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T14:45:11.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:10 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:11.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:10 smithi077 ceph-mon[35865]: Reconfiguring daemon mon.c on smithi174 2024-03-18T14:45:11.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:10 smithi077 ceph-mon[35865]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:11.256 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-03-18T14:45:11.256 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-03-18T14:45:11.257 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000229715 s, 2.2 MB/s 2024-03-18T14:45:11.258 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T14:45:11.316 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2024-03-18T14:45:11.372 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T14:45:11.372 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:45:11.372 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-03-18T14:45:11.372 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:45:11.373 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:45:11.373 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-18 14:44:58.809848422 +0000 2024-03-18T14:45:11.373 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-18 14:37:58.866823270 +0000 2024-03-18T14:45:11.373 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-18 14:37:58.866823270 +0000 2024-03-18T14:45:11.373 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-03-18 14:37:58.866823270 +0000 2024-03-18T14:45:11.373 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T14:45:11.439 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-03-18T14:45:11.440 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-03-18T14:45:11.440 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000211947 s, 2.4 MB/s 2024-03-18T14:45:11.441 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T14:45:11.497 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2024-03-18T14:45:11.552 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T14:45:11.552 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:45:11.553 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-18T14:45:11.553 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:45:11.553 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:45:11.553 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-18 14:44:58.809848422 +0000 2024-03-18T14:45:11.553 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-18 14:37:59.213815093 +0000 2024-03-18T14:45:11.553 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-18 14:37:59.213815093 +0000 2024-03-18T14:45:11.553 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-03-18 14:37:59.213815093 +0000 2024-03-18T14:45:11.553 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T14:45:11.619 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-03-18T14:45:11.619 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-03-18T14:45:11.619 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000216214 s, 2.4 MB/s 2024-03-18T14:45:11.620 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T14:45:11.675 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2024-03-18T14:45:11.731 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T14:45:11.731 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:45:11.731 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-18T14:45:11.731 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:45:11.731 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:45:11.731 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-18 14:44:58.810848399 +0000 2024-03-18T14:45:11.731 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-18 14:37:59.566806775 +0000 2024-03-18T14:45:11.731 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-18 14:37:59.566806775 +0000 2024-03-18T14:45:11.731 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-03-18 14:37:59.566806775 +0000 2024-03-18T14:45:11.732 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T14:45:11.797 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-03-18T14:45:11.797 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-03-18T14:45:11.797 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000206495 s, 2.5 MB/s 2024-03-18T14:45:11.798 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T14:45:11.854 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-18T14:45:11.854 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T14:45:11.870 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T14:45:11.870 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_1 2024-03-18T14:45:11.926 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T14:45:11.926 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:45:11.926 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-18T14:45:11.927 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:45:11.927 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:45:11.927 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-18 14:44:49.823724784 +0000 2024-03-18T14:45:11.927 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-18 14:37:51.181195914 +0000 2024-03-18T14:45:11.927 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-18 14:37:51.181195914 +0000 2024-03-18T14:45:11.927 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-18 14:37:51.181195914 +0000 2024-03-18T14:45:11.927 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T14:45:11.993 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-18T14:45:11.994 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-18T14:45:11.994 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000245166 s, 2.1 MB/s 2024-03-18T14:45:11.995 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T14:45:12.050 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_2 2024-03-18T14:45:12.106 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T14:45:12.106 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:45:12.106 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-18T14:45:12.106 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:45:12.106 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:45:12.106 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-18 14:44:49.824724771 +0000 2024-03-18T14:45:12.106 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-18 14:37:51.544190889 +0000 2024-03-18T14:45:12.106 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-18 14:37:51.544190889 +0000 2024-03-18T14:45:12.107 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-18 14:37:51.544190889 +0000 2024-03-18T14:45:12.107 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T14:45:12.172 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-18T14:45:12.173 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-18T14:45:12.173 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000209427 s, 2.4 MB/s 2024-03-18T14:45:12.174 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T14:45:12.230 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_3 2024-03-18T14:45:12.285 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T14:45:12.285 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:45:12.285 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-18T14:45:12.285 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:45:12.285 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:45:12.285 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-18 14:44:49.824724771 +0000 2024-03-18T14:45:12.285 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-18 14:37:51.897186003 +0000 2024-03-18T14:45:12.285 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-18 14:37:51.897186003 +0000 2024-03-18T14:45:12.285 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-18 14:37:51.897186003 +0000 2024-03-18T14:45:12.285 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T14:45:12.351 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-18T14:45:12.351 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-18T14:45:12.351 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000219126 s, 2.3 MB/s 2024-03-18T14:45:12.352 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T14:45:12.408 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_4 2024-03-18T14:45:12.463 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T14:45:12.463 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:45:12.463 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-18T14:45:12.463 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:45:12.463 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:45:12.463 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-18 14:44:49.824724771 +0000 2024-03-18T14:45:12.463 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-18 14:37:52.256181033 +0000 2024-03-18T14:45:12.463 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-18 14:37:52.256181033 +0000 2024-03-18T14:45:12.463 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-18 14:37:52.256181033 +0000 2024-03-18T14:45:12.463 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T14:45:12.529 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-18T14:45:12.529 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-18T14:45:12.529 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000179439 s, 2.9 MB/s 2024-03-18T14:45:12.531 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T14:45:12.586 INFO:tasks.cephadm:Deploying osd.0 on smithi076 with /dev/vg_nvme/lv_4... 2024-03-18T14:45:12.587 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T14:45:12.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:12 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:12.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:12 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:12.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:12 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:12.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:12 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:12.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:12 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:12.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:12 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:12.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:12 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T14:45:12.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:12 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T14:45:12.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:12 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T14:45:12.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:12 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:12.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:12 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:12.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:12 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:12.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:12 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:12.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:12 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:12.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:12 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:12.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:12 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:12.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:12 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T14:45:12.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:12 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T14:45:12.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:12 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T14:45:12.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:12 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:12.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:12 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:12.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:12 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:12.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:12 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:12.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:12 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:12.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:12 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:12.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:12 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:12.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:12 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T14:45:12.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:12 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T14:45:12.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:12 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T14:45:12.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:12 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:12.819 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:45:13.478 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:13 smithi076 ceph-mon[32205]: Deploying daemon mgr.z on smithi174 2024-03-18T14:45:13.478 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:13 smithi076 ceph-mon[32205]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:13.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:13 smithi174 ceph-mon[36787]: Deploying daemon mgr.z on smithi174 2024-03-18T14:45:13.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:13 smithi174 ceph-mon[36787]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:13.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:13 smithi077 ceph-mon[35865]: Deploying daemon mgr.z on smithi174 2024-03-18T14:45:13.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:13 smithi077 ceph-mon[35865]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:14.163 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:13 smithi174 systemd[1]: Starting Ceph mgr.z for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T14:45:14.413 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:14 smithi174 podman[38627]: 2024-03-18 14:45:14.06516377 +0000 UTC m=+0.017627834 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T14:45:14.414 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:14 smithi174 podman[38627]: 2024-03-18 14:45:14.174957863 +0000 UTC m=+0.127421911 container create 2a8836b2bbaa56a2957195e22418704495d6a582757a97994d846384a2cbb930 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T14:45:14.414 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:14 smithi174 podman[38627]: 2024-03-18 14:45:14.385520048 +0000 UTC m=+0.337984092 container init 2a8836b2bbaa56a2957195e22418704495d6a582757a97994d846384a2cbb930 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-18T14:45:14.414 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:14 smithi174 podman[38627]: 2024-03-18 14:45:14.388131191 +0000 UTC m=+0.340595241 container start 2a8836b2bbaa56a2957195e22418704495d6a582757a97994d846384a2cbb930 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T14:45:14.582 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T14:45:14.582 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T14:45:14.582 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-03-18T14:45:14.582 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-03-18T14:45:14.582 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117312 s, 894 MB/s 2024-03-18T14:45:14.582 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-03-18T14:45:14.717 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:14 smithi174 ceph-mgr[38646]: -- 172.21.15.174:0/4114461535 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56151faaa5a0 con 0x56151fa7f000 2024-03-18T14:45:14.717 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:14 smithi174 bash[38627]: 2a8836b2bbaa56a2957195e22418704495d6a582757a97994d846384a2cbb930 2024-03-18T14:45:14.718 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:14 smithi174 systemd[1]: Started Ceph mgr.z for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:45:15.186 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi076:vg_nvme/lv_4 2024-03-18T14:45:15.217 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:14.922+0000 7f69733b8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T14:45:15.217 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:15 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:15.013+0000 7f69733b8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T14:45:15.421 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:45:15.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:15 smithi077 ceph-mon[35865]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:15.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:15.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:15.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:15.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T14:45:15.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T14:45:15.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T14:45:15.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:15.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:15 smithi076 ceph-mon[32205]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:15.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:15.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:15.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:15.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T14:45:15.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T14:45:15.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T14:45:15.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:15.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:15 smithi174 ceph-mon[36787]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:15.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:15.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:15.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:15.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T14:45:15.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T14:45:15.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T14:45:15.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:15.820 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:15 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:15.646+0000 7f69733b8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T14:45:15.821 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:15 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:15.739+0000 7f69733b8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T14:45:16.073 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:15 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:15.886+0000 7f69733b8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T14:45:16.073 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:15 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:15.986+0000 7f69733b8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T14:45:16.073 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:16 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:16.071+0000 7f69733b8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T14:45:16.629 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:16 smithi174 ceph-mon[36787]: Deploying daemon mgr.y on smithi077 2024-03-18T14:45:16.629 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:16 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:16.336+0000 7f69733b8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T14:45:16.629 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:16 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:16.447+0000 7f69733b8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T14:45:16.629 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:16 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:16.532+0000 7f69733b8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T14:45:16.629 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:16 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:16.627+0000 7f69733b8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T14:45:16.650 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:16 smithi076 ceph-mon[32205]: Deploying daemon mgr.y on smithi077 2024-03-18T14:45:16.696 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:16 smithi077 ceph-mon[35865]: Deploying daemon mgr.y on smithi077 2024-03-18T14:45:16.696 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:16 smithi077 systemd[1]: Starting Ceph mgr.y for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T14:45:16.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.866+0000 7f6e08270700 1 -- 172.21.15.76:0/2020715548 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00102300 msgr2=0x7f6e00102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:16.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.866+0000 7f6e08270700 1 --2- 172.21.15.76:0/2020715548 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00102300 0x7f6e00102720 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc0099c0 tx=0x7f6dfc009cd0 comp rx=0 tx=0).stop 2024-03-18T14:45:16.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.867+0000 7f6e08270700 1 -- 172.21.15.76:0/2020715548 shutdown_connections 2024-03-18T14:45:16.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.867+0000 7f6e08270700 1 --2- 172.21.15.76:0/2020715548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e00103ea0 0x7f6e0010a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:16.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.867+0000 7f6e08270700 1 --2- 172.21.15.76:0/2020715548 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00102300 0x7f6e00102720 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:16.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.867+0000 7f6e08270700 1 --2- 172.21.15.76:0/2020715548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e00103500 0x7f6e00103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:16.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.867+0000 7f6e08270700 1 -- 172.21.15.76:0/2020715548 >> 172.21.15.76:0/2020715548 conn(0x7f6e000fd8e0 msgr2=0x7f6e000ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:45:16.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.867+0000 7f6e08270700 1 -- 172.21.15.76:0/2020715548 shutdown_connections 2024-03-18T14:45:16.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.867+0000 7f6e08270700 1 -- 172.21.15.76:0/2020715548 wait complete. 2024-03-18T14:45:16.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.867+0000 7f6e08270700 1 Processor -- start 2024-03-18T14:45:16.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.867+0000 7f6e08270700 1 -- start start 2024-03-18T14:45:16.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.867+0000 7f6e08270700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00102300 0x7f6e00198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:45:16.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.867+0000 7f6e08270700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e00103500 0x7f6e00198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:45:16.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6e08270700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e00103ea0 0x7f6e0019fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:45:16.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6e08270700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0010def0 con 0x7f6e00102300 2024-03-18T14:45:16.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6e08270700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0010dd70 con 0x7f6e00103ea0 2024-03-18T14:45:16.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6e08270700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0010e070 con 0x7f6e00103500 2024-03-18T14:45:16.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6e0600c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00102300 0x7f6e00198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:45:16.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6e0600c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00102300 0x7f6e00198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51744/0 (socket says 172.21.15.76:51744) 2024-03-18T14:45:16.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6e0600c700 1 -- 172.21.15.76:0/284710424 learned_addr learned my addr 172.21.15.76:0/284710424 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:45:16.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6e0580b700 1 --2- 172.21.15.76:0/284710424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e00103500 0x7f6e00198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:45:16.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6e0680d700 1 --2- 172.21.15.76:0/284710424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e00103ea0 0x7f6e0019fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:45:16.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6e0600c700 1 -- 172.21.15.76:0/284710424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e00103500 msgr2=0x7f6e00198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:16.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6e0600c700 1 --2- 172.21.15.76:0/284710424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e00103500 0x7f6e00198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:16.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6e0600c700 1 -- 172.21.15.76:0/284710424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e00103ea0 msgr2=0x7f6e0019fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:16.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6e0600c700 1 --2- 172.21.15.76:0/284710424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e00103ea0 0x7f6e0019fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:16.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6e0600c700 1 -- 172.21.15.76:0/284710424 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e001a01d0 con 0x7f6e00102300 2024-03-18T14:45:16.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6e0600c700 1 --2- 172.21.15.76:0/284710424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00102300 0x7f6e00198570 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc009510 tx=0x7f6dfc009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:45:16.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6df77fe700 1 -- 172.21.15.76:0/284710424 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc026070 con 0x7f6e00102300 2024-03-18T14:45:16.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6df77fe700 1 -- 172.21.15.76:0/284710424 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6dfc005940 con 0x7f6e00102300 2024-03-18T14:45:16.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6e08270700 1 -- 172.21.15.76:0/284710424 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e001a0460 con 0x7f6e00102300 2024-03-18T14:45:16.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6df77fe700 1 -- 172.21.15.76:0/284710424 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc00fdc0 con 0x7f6e00102300 2024-03-18T14:45:16.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.868+0000 7f6e08270700 1 -- 172.21.15.76:0/284710424 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e001a0990 con 0x7f6e00102300 2024-03-18T14:45:16.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.869+0000 7f6df77fe700 1 -- 172.21.15.76:0/284710424 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f6dfc029070 con 0x7f6e00102300 2024-03-18T14:45:16.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.869+0000 7f6df77fe700 1 --2- 172.21.15.76:0/284710424 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6dec039200 0x7f6dec03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:45:16.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.869+0000 7f6e08270700 1 -- 172.21.15.76:0/284710424 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e00068f90 con 0x7f6e00102300 2024-03-18T14:45:16.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.869+0000 7f6df77fe700 1 -- 172.21.15.76:0/284710424 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6dfc04cb90 con 0x7f6e00102300 2024-03-18T14:45:16.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.869+0000 7f6e0580b700 1 --2- 172.21.15.76:0/284710424 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6dec039200 0x7f6dec03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:45:16.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.870+0000 7f6e0580b700 1 --2- 172.21.15.76:0/284710424 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6dec039200 0x7f6dec03b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6df0004630 tx=0x7f6df000b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:45:16.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:16.874+0000 7f6df77fe700 1 -- 172.21.15.76:0/284710424 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6dfc013070 con 0x7f6e00102300 2024-03-18T14:45:16.948 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:16 smithi077 podman[37237]: 2024-03-18 14:45:16.695372436 +0000 UTC m=+0.112022993 container create c90a05c8a55e874f979a92f5048e289551dae1179aba5ecd9b4dfc151bf42853 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T14:45:16.948 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:16 smithi077 podman[37237]: 2024-03-18 14:45:16.600826277 +0000 UTC m=+0.017476844 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T14:45:16.948 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:16 smithi077 podman[37237]: 2024-03-18 14:45:16.864471976 +0000 UTC m=+0.281122543 container init c90a05c8a55e874f979a92f5048e289551dae1179aba5ecd9b4dfc151bf42853 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-18T14:45:16.948 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:16 smithi077 podman[37237]: 2024-03-18 14:45:16.867082015 +0000 UTC m=+0.283732568 container start c90a05c8a55e874f979a92f5048e289551dae1179aba5ecd9b4dfc151bf42853 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T14:45:16.949 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:16 smithi077 ceph-mgr[37256]: -- 172.21.15.77:0/222396097 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5586faca45a0 con 0x5586fac78800 2024-03-18T14:45:16.949 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:16 smithi077 bash[37237]: c90a05c8a55e874f979a92f5048e289551dae1179aba5ecd9b4dfc151bf42853 2024-03-18T14:45:16.967 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:16 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:16.805+0000 7f69733b8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T14:45:17.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:17.085+0000 7f6e08270700 1 -- 172.21.15.76:0/284710424 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6e00062710 con 0x7f6dec039200 2024-03-18T14:45:17.228 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:16 smithi077 systemd[1]: Started Ceph mgr.y for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:45:17.536 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:17 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:17.435+0000 7f75382d1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T14:45:17.536 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:17 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:17.534+0000 7f75382d1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T14:45:17.536 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:17 smithi077 ceph-mon[35865]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:17.536 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:17 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:17.536 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:17 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:17.536 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:17 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:17.537 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:17 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:17.537 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:17 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:45:17.537 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:17 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:45:17.537 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:17 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:17.537 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:17 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:17.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:17 smithi076 ceph-mon[32205]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:17.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:17 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:17.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:17 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:17.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:17 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:17.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:17 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:17.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:17 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:45:17.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:17 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:45:17.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:17 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:17.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:17 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:17.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:17 smithi174 ceph-mon[36787]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:17.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:17 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:17.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:17 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:17.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:17 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:17.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:17 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:17.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:17 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:45:17.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:17 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:45:17.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:17 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:17.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:17 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:18.164 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:17 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:17.817+0000 7f69733b8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T14:45:18.164 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:17 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:17.951+0000 7f69733b8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T14:45:18.276 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:18 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:18.224+0000 7f75382d1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T14:45:18.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:18 smithi174 ceph-mon[36787]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:45:18.467 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:18 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:18.294+0000 7f69733b8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T14:45:18.467 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:18 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:18.386+0000 7f69733b8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T14:45:18.574 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:18 smithi077 ceph-mon[35865]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:45:18.574 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:18 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:18.331+0000 7f75382d1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T14:45:18.574 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:18 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:18.478+0000 7f75382d1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T14:45:18.575 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:18 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:18.573+0000 7f75382d1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T14:45:18.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:18 smithi076 ceph-mon[32205]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:45:18.920 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:18 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:18.656+0000 7f75382d1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T14:45:18.921 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:18 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:18.919+0000 7f75382d1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T14:45:18.967 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:18 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:18.589+0000 7f69733b8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:45:18.967 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:18 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:18.775+0000 7f69733b8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T14:45:19.202 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:19 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:19.026+0000 7f75382d1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T14:45:19.202 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:19 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:19.109+0000 7f75382d1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T14:45:19.202 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:19 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:19.200+0000 7f75382d1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T14:45:19.317 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:19 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:19.055+0000 7f69733b8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:45:19.317 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:19 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:19.315+0000 7f69733b8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T14:45:19.478 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:19 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:19.373+0000 7f75382d1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T14:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:19 smithi077 ceph-mon[35865]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:19 smithi077 ceph-mon[35865]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T14:45:19.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T14:45:19.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T14:45:19.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:19.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:19 smithi077 ceph-mon[35865]: Reconfiguring daemon mgr.x on smithi076 2024-03-18T14:45:19.717 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:19 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:19.551+0000 7f69733b8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T14:45:19.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:19 smithi174 ceph-mon[36787]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:19.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:19.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:19.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:19.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:19.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:19.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:19.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:19.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:19 smithi174 ceph-mon[36787]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T14:45:19.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T14:45:19.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T14:45:19.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:19.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:19 smithi174 ceph-mon[36787]: Reconfiguring daemon mgr.x on smithi076 2024-03-18T14:45:19.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:19 smithi076 ceph-mon[32205]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:19.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:19.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:19.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:19.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:19.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:19.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:19.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:19.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:19 smithi076 ceph-mon[32205]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T14:45:19.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T14:45:19.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T14:45:19.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:19.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:19 smithi076 ceph-mon[32205]: Reconfiguring daemon mgr.x on smithi076 2024-03-18T14:45:20.717 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:20 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:20.349+0000 7f69733b8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T14:45:20.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:20 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3360767482' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b8ee0a0-29f7-4333-978a-08c4db54db0d"}]: dispatch 2024-03-18T14:45:20.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:20 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3360767482' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b8ee0a0-29f7-4333-978a-08c4db54db0d"}]': finished 2024-03-18T14:45:20.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:20 smithi174 ceph-mon[36787]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T14:45:20.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:20 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:20.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:20 smithi174 ceph-mon[36787]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:20.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:20 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1671319186' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:45:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:20 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3360767482' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b8ee0a0-29f7-4333-978a-08c4db54db0d"}]: dispatch 2024-03-18T14:45:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:20 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3360767482' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b8ee0a0-29f7-4333-978a-08c4db54db0d"}]': finished 2024-03-18T14:45:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:20 smithi077 ceph-mon[35865]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T14:45:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:20 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:20 smithi077 ceph-mon[35865]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:20 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1671319186' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:45:20.728 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:20 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:20.328+0000 7f75382d1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T14:45:20.728 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:20 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:20.457+0000 7f75382d1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T14:45:20.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:20 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3360767482' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b8ee0a0-29f7-4333-978a-08c4db54db0d"}]: dispatch 2024-03-18T14:45:20.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:20 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3360767482' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b8ee0a0-29f7-4333-978a-08c4db54db0d"}]': finished 2024-03-18T14:45:20.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:20 smithi076 ceph-mon[32205]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T14:45:20.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:20 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:20.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:20 smithi076 ceph-mon[32205]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:20.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:20 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1671319186' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:45:21.066 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:20 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:20.786+0000 7f75382d1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T14:45:21.066 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:20 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:20.873+0000 7f75382d1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T14:45:21.066 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:21 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:21.065+0000 7f75382d1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:45:21.217 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:20 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:20.917+0000 7f69733b8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T14:45:21.217 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:21 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:21.014+0000 7f69733b8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T14:45:21.477 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:21 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:21.246+0000 7f75382d1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T14:45:21.717 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:45:21 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:45:21.228+0000 7f69733b8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T14:45:21.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:21.724+0000 7f6df77fe700 1 -- 172.21.15.76:0/284710424 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 14) ==== 91698+0+0 (secure 0 0 0) 0x7f6dfc003c70 con 0x7f6e00102300 2024-03-18T14:45:21.786 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:21 smithi077 ceph-mon[35865]: Standby manager daemon z started 2024-03-18T14:45:21.786 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:21 smithi077 ceph-mon[35865]: from='mgr.? 172.21.15.174:0/1245848881' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T14:45:21.786 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:21 smithi077 ceph-mon[35865]: from='mgr.? 172.21.15.174:0/1245848881' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T14:45:21.786 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:21 smithi077 ceph-mon[35865]: from='mgr.? 172.21.15.174:0/1245848881' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T14:45:21.786 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:21 smithi077 ceph-mon[35865]: from='mgr.? 172.21.15.174:0/1245848881' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T14:45:21.787 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:21 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:21.787 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:21 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:21.525+0000 7f75382d1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:45:21.787 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:21 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:21.784+0000 7f75382d1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T14:45:22.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:21 smithi076 ceph-mon[32205]: Standby manager daemon z started 2024-03-18T14:45:22.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:21 smithi076 ceph-mon[32205]: from='mgr.? 172.21.15.174:0/1245848881' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T14:45:22.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:21 smithi076 ceph-mon[32205]: from='mgr.? 172.21.15.174:0/1245848881' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T14:45:22.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:21 smithi076 ceph-mon[32205]: from='mgr.? 172.21.15.174:0/1245848881' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T14:45:22.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:21 smithi076 ceph-mon[32205]: from='mgr.? 172.21.15.174:0/1245848881' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T14:45:22.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:21 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:22.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:21 smithi174 ceph-mon[36787]: Standby manager daemon z started 2024-03-18T14:45:22.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:21 smithi174 ceph-mon[36787]: from='mgr.? 172.21.15.174:0/1245848881' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T14:45:22.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:21 smithi174 ceph-mon[36787]: from='mgr.? 172.21.15.174:0/1245848881' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T14:45:22.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:21 smithi174 ceph-mon[36787]: from='mgr.? 172.21.15.174:0/1245848881' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T14:45:22.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:21 smithi174 ceph-mon[36787]: from='mgr.? 172.21.15.174:0/1245848881' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T14:45:22.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:21 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:22.228 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:22 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:22.018+0000 7f75382d1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T14:45:23.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:22 smithi076 ceph-mon[32205]: mgrmap e14: x(active, since 75s), standbys: z 2024-03-18T14:45:23.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T14:45:23.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:22 smithi076 ceph-mon[32205]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:23.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:22 smithi174 ceph-mon[36787]: mgrmap e14: x(active, since 75s), standbys: z 2024-03-18T14:45:23.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:22 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T14:45:23.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:22 smithi174 ceph-mon[36787]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:23.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:22 smithi077 ceph-mon[35865]: mgrmap e14: x(active, since 75s), standbys: z 2024-03-18T14:45:23.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:22 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T14:45:23.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:22 smithi077 ceph-mon[35865]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:23.228 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:22 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:22.803+0000 7f75382d1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T14:45:23.672 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:23 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:23.364+0000 7f75382d1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T14:45:23.672 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:23 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:23.460+0000 7f75382d1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T14:45:23.672 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:45:23 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:45:23.670+0000 7f75382d1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T14:45:23.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:23.736+0000 7f6df77fe700 1 -- 172.21.15.76:0/284710424 <== mon.0 v2:172.21.15.76:3300/0 8 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f6dfc00f460 con 0x7f6e00102300 2024-03-18T14:45:23.905 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:23 smithi076 ceph-mon[32205]: Standby manager daemon y started 2024-03-18T14:45:23.905 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:23 smithi076 ceph-mon[32205]: from='mgr.? 172.21.15.77:0/1611850228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T14:45:23.905 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:23 smithi076 ceph-mon[32205]: from='mgr.? 172.21.15.77:0/1611850228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T14:45:23.905 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:23 smithi076 ceph-mon[32205]: from='mgr.? 172.21.15.77:0/1611850228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T14:45:23.905 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:23 smithi076 ceph-mon[32205]: from='mgr.? 172.21.15.77:0/1611850228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T14:45:23.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:23 smithi077 ceph-mon[35865]: Standby manager daemon y started 2024-03-18T14:45:23.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:23 smithi077 ceph-mon[35865]: from='mgr.? 172.21.15.77:0/1611850228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T14:45:23.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:23 smithi077 ceph-mon[35865]: from='mgr.? 172.21.15.77:0/1611850228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T14:45:23.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:23 smithi077 ceph-mon[35865]: from='mgr.? 172.21.15.77:0/1611850228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T14:45:23.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:23 smithi077 ceph-mon[35865]: from='mgr.? 172.21.15.77:0/1611850228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T14:45:24.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:23 smithi174 ceph-mon[36787]: Standby manager daemon y started 2024-03-18T14:45:24.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:23 smithi174 ceph-mon[36787]: from='mgr.? 172.21.15.77:0/1611850228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T14:45:24.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:23 smithi174 ceph-mon[36787]: from='mgr.? 172.21.15.77:0/1611850228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T14:45:24.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:23 smithi174 ceph-mon[36787]: from='mgr.? 172.21.15.77:0/1611850228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T14:45:24.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:23 smithi174 ceph-mon[36787]: from='mgr.? 172.21.15.77:0/1611850228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T14:45:24.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:24 smithi076 ceph-mon[32205]: mgrmap e15: x(active, since 77s), standbys: z, y 2024-03-18T14:45:24.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T14:45:24.988 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:24 smithi076 ceph-mon[32205]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:25.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:24 smithi174 ceph-mon[36787]: mgrmap e15: x(active, since 77s), standbys: z, y 2024-03-18T14:45:25.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T14:45:25.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:24 smithi174 ceph-mon[36787]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:25.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:24 smithi077 ceph-mon[35865]: mgrmap e15: x(active, since 77s), standbys: z, y 2024-03-18T14:45:25.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T14:45:25.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:24 smithi077 ceph-mon[35865]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:26.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:26.285 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:26.285 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:26.285 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:26.285 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:26.285 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:26.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:26.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:26.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:26.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:26.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:26.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:26.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:26.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:26.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:26.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:26.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:26.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:27.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:27 smithi174 ceph-mon[36787]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:27.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:27 smithi076 ceph-mon[32205]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:27.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:27 smithi077 ceph-mon[35865]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:28.106 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:28 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T14:45:28.106 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:28 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:28.106 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:28 smithi076 ceph-mon[32205]: Deploying daemon osd.0 on smithi076 2024-03-18T14:45:28.354 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:28 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T14:45:28.354 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:28 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:28.355 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:28 smithi077 ceph-mon[35865]: Deploying daemon osd.0 on smithi076 2024-03-18T14:45:28.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:28 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T14:45:28.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:28 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:28.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:28 smithi174 ceph-mon[36787]: Deploying daemon osd.0 on smithi076 2024-03-18T14:45:29.146 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:29 smithi076 ceph-mon[32205]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:29.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:29 smithi174 ceph-mon[36787]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:29.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:29 smithi077 ceph-mon[35865]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:31.502 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:31 smithi076 ceph-mon[32205]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:31.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:31 smithi174 ceph-mon[36787]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:31.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:31 smithi077 ceph-mon[35865]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:33.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:33 smithi076 ceph-mon[32205]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:33.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:33 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:33.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:33 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:33.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:33 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:33.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:33 smithi174 ceph-mon[36787]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:33.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:33 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:33.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:33 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:33.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:33 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:33.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:33 smithi077 ceph-mon[35865]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:33.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:33 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:33.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:33 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:33.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:33 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:35.583 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:35 smithi076 ceph-mon[32205]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:35.583 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:35 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:35.583 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:35 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:35.583 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:35 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:35.583 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:35 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:35.583 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:35 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:35.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:35 smithi174 ceph-mon[36787]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:35.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:35 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:35.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:35 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:35.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:35 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:35.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:35 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:35.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:35 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:35.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:35 smithi077 ceph-mon[35865]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:35.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:35 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:35.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:35 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:35.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:35 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:35.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:35 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:35.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:35 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:35.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:35.942+0000 7f6df77fe700 1 -- 172.21.15.76:0/284710424 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6e00062710 con 0x7f6dec039200 2024-03-18T14:45:35.944 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 0 on host 'smithi076' 2024-03-18T14:45:35.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:35.944+0000 7f6e08270700 1 -- 172.21.15.76:0/284710424 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6dec039200 msgr2=0x7f6dec03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:35.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:35.944+0000 7f6e08270700 1 --2- 172.21.15.76:0/284710424 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6dec039200 0x7f6dec03b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6df0004630 tx=0x7f6df000b3c0 comp rx=0 tx=0).stop 2024-03-18T14:45:35.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:35.945+0000 7f6e08270700 1 -- 172.21.15.76:0/284710424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00102300 msgr2=0x7f6e00198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:35.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:35.945+0000 7f6e08270700 1 --2- 172.21.15.76:0/284710424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00102300 0x7f6e00198570 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc009510 tx=0x7f6dfc009540 comp rx=0 tx=0).stop 2024-03-18T14:45:35.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:35.945+0000 7f6e08270700 1 -- 172.21.15.76:0/284710424 shutdown_connections 2024-03-18T14:45:35.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:35.945+0000 7f6e08270700 1 --2- 172.21.15.76:0/284710424 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6dec039200 0x7f6dec03b6c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:35.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:35.945+0000 7f6e08270700 1 --2- 172.21.15.76:0/284710424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e00103ea0 0x7f6e0019fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:35.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:35.945+0000 7f6e08270700 1 --2- 172.21.15.76:0/284710424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00102300 0x7f6e00198570 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:35.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:35.945+0000 7f6e08270700 1 --2- 172.21.15.76:0/284710424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e00103500 0x7f6e00198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:35.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:35.945+0000 7f6e08270700 1 -- 172.21.15.76:0/284710424 >> 172.21.15.76:0/284710424 conn(0x7f6e000fd8e0 msgr2=0x7f6e00106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:45:35.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:35.945+0000 7f6e08270700 1 -- 172.21.15.76:0/284710424 shutdown_connections 2024-03-18T14:45:35.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:45:35.945+0000 7f6e08270700 1 -- 172.21.15.76:0/284710424 wait complete. 2024-03-18T14:45:36.558 DEBUG:teuthology.orchestra.run.smithi076:osd.0> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.0.service 2024-03-18T14:45:36.560 INFO:tasks.cephadm:Deploying osd.1 on smithi077 with /dev/vg_nvme/lv_4... 2024-03-18T14:45:36.560 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T14:45:36.802 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T14:45:37.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:36 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:37.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:36 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:37.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:36 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:37.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:36 smithi076 ceph-mon[32205]: from='osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T14:45:37.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:36 smithi076 ceph-mon[32205]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:37.186 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:36 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:37.186 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:36 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:37.186 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:36 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:37.186 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:36 smithi077 ceph-mon[35865]: from='osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T14:45:37.186 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:36 smithi077 ceph-mon[35865]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:37.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:36 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:37.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:36 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:37.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:36 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:37.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:36 smithi174 ceph-mon[36787]: from='osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T14:45:37.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:36 smithi174 ceph-mon[36787]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:38.062 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:37 smithi077 ceph-mon[35865]: from='osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T14:45:38.062 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:37 smithi077 ceph-mon[35865]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T14:45:38.062 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:37 smithi077 ceph-mon[35865]: from='osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-18T14:45:38.062 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:37 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:38.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:37 smithi076 ceph-mon[32205]: from='osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T14:45:38.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:37 smithi076 ceph-mon[32205]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T14:45:38.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:37 smithi076 ceph-mon[32205]: from='osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-18T14:45:38.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:37 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:38.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:37 smithi174 ceph-mon[36787]: from='osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T14:45:38.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:37 smithi174 ceph-mon[36787]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T14:45:38.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:37 smithi174 ceph-mon[36787]: from='osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-18T14:45:38.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:37 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:38.627 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T14:45:38.627 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T14:45:38.627 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-03-18T14:45:38.627 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-03-18T14:45:38.627 INFO:teuthology.orchestra.run.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112998 s, 928 MB/s 2024-03-18T14:45:38.627 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 2024-03-18T14:45:38.627 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-03-18T14:45:38.662 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 14:45:38 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-0[46643]: 2024-03-18T14:45:38.325+0000 7f18c60f7700 -1 osd.0 0 waiting for initial osdmap 2024-03-18T14:45:38.662 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 14:45:38 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-0[46643]: 2024-03-18T14:45:38.331+0000 7f18bdee2700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:45:39.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:38 smithi076 ceph-mon[32205]: from='osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-18T14:45:39.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:38 smithi076 ceph-mon[32205]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T14:45:39.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:38 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:39.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:38 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:39.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:38 smithi076 ceph-mon[32205]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:39.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:38 smithi076 ceph-mon[32205]: from='osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861]' entity='osd.0' 2024-03-18T14:45:39.204 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:38 smithi077 ceph-mon[35865]: from='osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-18T14:45:39.204 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:38 smithi077 ceph-mon[35865]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T14:45:39.204 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:38 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:39.204 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:38 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:39.204 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:38 smithi077 ceph-mon[35865]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:39.204 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:38 smithi077 ceph-mon[35865]: from='osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861]' entity='osd.0' 2024-03-18T14:45:39.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:38 smithi174 ceph-mon[36787]: from='osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-18T14:45:39.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:38 smithi174 ceph-mon[36787]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T14:45:39.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:38 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:39.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:38 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:39.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:38 smithi174 ceph-mon[36787]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T14:45:39.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:38 smithi174 ceph-mon[36787]: from='osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861]' entity='osd.0' 2024-03-18T14:45:39.230 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi077:vg_nvme/lv_4 2024-03-18T14:45:39.480 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T14:45:40.151 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:39 smithi077 ceph-mon[35865]: purged_snaps scrub starts 2024-03-18T14:45:40.152 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:39 smithi077 ceph-mon[35865]: purged_snaps scrub ok 2024-03-18T14:45:40.152 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:39 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:40.152 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:39 smithi077 ceph-mon[35865]: osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861] boot 2024-03-18T14:45:40.152 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:39 smithi077 ceph-mon[35865]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T14:45:40.152 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:39 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:40.152 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:39 smithi077 ceph-mon[35865]: Detected new or changed devices on smithi076 2024-03-18T14:45:40.152 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:39 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:40.152 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:39 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:40.152 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:39 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:45:40.152 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:39 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi076 to 17088M 2024-03-18T14:45:40.152 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:39 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:40.152 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:39 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:40.152 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:39 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:40.152 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:39 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:40.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:39 smithi174 ceph-mon[36787]: purged_snaps scrub starts 2024-03-18T14:45:40.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:39 smithi174 ceph-mon[36787]: purged_snaps scrub ok 2024-03-18T14:45:40.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:39 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:40.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:39 smithi174 ceph-mon[36787]: osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861] boot 2024-03-18T14:45:40.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:39 smithi174 ceph-mon[36787]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T14:45:40.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:39 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:40.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:39 smithi174 ceph-mon[36787]: Detected new or changed devices on smithi076 2024-03-18T14:45:40.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:39 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:40.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:39 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:40.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:39 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:45:40.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:39 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi076 to 17088M 2024-03-18T14:45:40.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:39 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:40.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:39 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:40.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:39 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:40.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:39 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:40.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:39 smithi076 ceph-mon[32205]: purged_snaps scrub starts 2024-03-18T14:45:40.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:39 smithi076 ceph-mon[32205]: purged_snaps scrub ok 2024-03-18T14:45:40.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:39 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:40.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:39 smithi076 ceph-mon[32205]: osd.0 [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861] boot 2024-03-18T14:45:40.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:39 smithi076 ceph-mon[32205]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T14:45:40.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:39 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:45:40.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:39 smithi076 ceph-mon[32205]: Detected new or changed devices on smithi076 2024-03-18T14:45:40.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:39 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:40.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:39 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:40.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:39 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:45:40.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:39 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi076 to 17088M 2024-03-18T14:45:40.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:39 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:40.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:39 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:40.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:39 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:40.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:39 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:40.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.930+0000 7fe052614700 1 -- 172.21.15.77:0/2565034335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe04c1023f0 msgr2=0x7fe04c102850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:40.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.930+0000 7fe052614700 1 --2- 172.21.15.77:0/2565034335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe04c1023f0 0x7fe04c102850 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe03c0099c0 tx=0x7fe03c009cd0 comp rx=0 tx=0).stop 2024-03-18T14:45:40.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.930+0000 7fe052614700 1 -- 172.21.15.77:0/2565034335 shutdown_connections 2024-03-18T14:45:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.930+0000 7fe052614700 1 --2- 172.21.15.77:0/2565034335 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe04c102d90 0x7fe04c109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.930+0000 7fe052614700 1 --2- 172.21.15.77:0/2565034335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe04c1011f0 0x7fe04c101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.930+0000 7fe052614700 1 --2- 172.21.15.77:0/2565034335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe04c1023f0 0x7fe04c102850 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.930+0000 7fe052614700 1 -- 172.21.15.77:0/2565034335 >> 172.21.15.77:0/2565034335 conn(0x7fe04c0fc790 msgr2=0x7fe04c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:45:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.930+0000 7fe052614700 1 -- 172.21.15.77:0/2565034335 shutdown_connections 2024-03-18T14:45:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.930+0000 7fe052614700 1 -- 172.21.15.77:0/2565034335 wait complete. 2024-03-18T14:45:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe052614700 1 Processor -- start 2024-03-18T14:45:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe052614700 1 -- start start 2024-03-18T14:45:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe052614700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe04c1011f0 0x7fe04c105180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:45:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe052614700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe04c1023f0 0x7fe04c1037d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:45:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe052614700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe04c102d90 0x7fe04c103d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:45:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe052614700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe04c10cdd0 con 0x7fe04c102d90 2024-03-18T14:45:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe052614700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe04c10cc50 con 0x7fe04c1023f0 2024-03-18T14:45:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe052614700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe04c10cf50 con 0x7fe04c1011f0 2024-03-18T14:45:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe04b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe04c1023f0 0x7fe04c1037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:45:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe04b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe04c1023f0 0x7fe04c1037d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47518/0 (socket says 172.21.15.77:47518) 2024-03-18T14:45:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe04b7fe700 1 -- 172.21.15.77:0/1776697513 learned_addr learned my addr 172.21.15.77:0/1776697513 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-18T14:45:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe050bb1700 1 --2- 172.21.15.77:0/1776697513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe04c102d90 0x7fe04c103d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:45:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe04bfff700 1 --2- 172.21.15.77:0/1776697513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe04c1011f0 0x7fe04c105180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:45:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe04b7fe700 1 -- 172.21.15.77:0/1776697513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe04c1011f0 msgr2=0x7fe04c105180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe04b7fe700 1 --2- 172.21.15.77:0/1776697513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe04c1011f0 0x7fe04c105180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe04b7fe700 1 -- 172.21.15.77:0/1776697513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe04c102d90 msgr2=0x7fe04c103d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe04b7fe700 1 --2- 172.21.15.77:0/1776697513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe04c102d90 0x7fe04c103d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe04b7fe700 1 -- 172.21.15.77:0/1776697513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe04c1045c0 con 0x7fe04c1023f0 2024-03-18T14:45:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe04bfff700 1 --2- 172.21.15.77:0/1776697513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe04c1011f0 0x7fe04c105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:45:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.931+0000 7fe050bb1700 1 --2- 172.21.15.77:0/1776697513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe04c102d90 0x7fe04c103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:45:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.932+0000 7fe04b7fe700 1 --2- 172.21.15.77:0/1776697513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe04c1023f0 0x7fe04c1037d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe03c005910 tx=0x7fe03c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:45:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.932+0000 7fe0497fa700 1 -- 172.21.15.77:0/1776697513 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c027070 con 0x7fe04c1023f0 2024-03-18T14:45:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.932+0000 7fe0497fa700 1 -- 172.21.15.77:0/1776697513 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe03c004150 con 0x7fe04c1023f0 2024-03-18T14:45:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.932+0000 7fe052614700 1 -- 172.21.15.77:0/1776697513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe04c1a2640 con 0x7fe04c1023f0 2024-03-18T14:45:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.932+0000 7fe0497fa700 1 -- 172.21.15.77:0/1776697513 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c00f460 con 0x7fe04c1023f0 2024-03-18T14:45:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.932+0000 7fe052614700 1 -- 172.21.15.77:0/1776697513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe04c1a2a90 con 0x7fe04c1023f0 2024-03-18T14:45:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.933+0000 7fe052614700 1 -- 172.21.15.77:0/1776697513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe04c068f90 con 0x7fe04c1023f0 2024-03-18T14:45:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.934+0000 7fe0497fa700 1 -- 172.21.15.77:0/1776697513 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7fe03c02a070 con 0x7fe04c1023f0 2024-03-18T14:45:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.934+0000 7fe0497fa700 1 --2- 172.21.15.77:0/1776697513 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe0340a2800 0x7fe0340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:45:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.935+0000 7fe0497fa700 1 -- 172.21.15.77:0/1776697513 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fe03c0cdf50 con 0x7fe04c1023f0 2024-03-18T14:45:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.935+0000 7fe04bfff700 1 --2- 172.21.15.77:0/1776697513 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe0340a2800 0x7fe0340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:45:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.935+0000 7fe04bfff700 1 --2- 172.21.15.77:0/1776697513 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe0340a2800 0x7fe0340a4cc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe040005fd0 tx=0x7fe040005de0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:45:40.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:40.940+0000 7fe0497fa700 1 -- 172.21.15.77:0/1776697513 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe03c05e740 con 0x7fe04c1023f0 2024-03-18T14:45:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:40 smithi077 ceph-mon[35865]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:41.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:41.144+0000 7fe052614700 1 -- 172.21.15.77:0/1776697513 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fe04c0fe930 con 0x7fe0340a2800 2024-03-18T14:45:41.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:40 smithi174 ceph-mon[36787]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:41.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:40 smithi076 ceph-mon[32205]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:42.041 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:41 smithi077 ceph-mon[35865]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T14:45:42.041 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:41 smithi077 ceph-mon[35865]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:45:42.041 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:41 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:45:42.041 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:41 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:45:42.041 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:41 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:42.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:41 smithi174 ceph-mon[36787]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T14:45:42.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:41 smithi174 ceph-mon[36787]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:45:42.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:41 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:45:42.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:41 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:45:42.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:41 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:42.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:41 smithi076 ceph-mon[32205]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T14:45:42.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:41 smithi076 ceph-mon[32205]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:45:42.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:41 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:45:42.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:41 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:45:42.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:41 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:43.046 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:42 smithi077 ceph-mon[35865]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:43.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:42 smithi174 ceph-mon[36787]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:43.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:42 smithi076 ceph-mon[32205]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:44.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:43 smithi174 ceph-mon[36787]: from='client.? 172.21.15.77:0/1172880301' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71422220-775b-4b49-a9a4-ae0a7a16c2e5"}]: dispatch 2024-03-18T14:45:44.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:43 smithi174 ceph-mon[36787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71422220-775b-4b49-a9a4-ae0a7a16c2e5"}]: dispatch 2024-03-18T14:45:44.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:43 smithi174 ceph-mon[36787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71422220-775b-4b49-a9a4-ae0a7a16c2e5"}]': finished 2024-03-18T14:45:44.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:43 smithi174 ceph-mon[36787]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T14:45:44.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:43 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:45:44.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:43 smithi077 ceph-mon[35865]: from='client.? 172.21.15.77:0/1172880301' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71422220-775b-4b49-a9a4-ae0a7a16c2e5"}]: dispatch 2024-03-18T14:45:44.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:43 smithi077 ceph-mon[35865]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71422220-775b-4b49-a9a4-ae0a7a16c2e5"}]: dispatch 2024-03-18T14:45:44.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:43 smithi077 ceph-mon[35865]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71422220-775b-4b49-a9a4-ae0a7a16c2e5"}]': finished 2024-03-18T14:45:44.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:43 smithi077 ceph-mon[35865]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T14:45:44.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:43 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:45:44.313 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:43 smithi076 ceph-mon[32205]: from='client.? 172.21.15.77:0/1172880301' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71422220-775b-4b49-a9a4-ae0a7a16c2e5"}]: dispatch 2024-03-18T14:45:44.313 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:43 smithi076 ceph-mon[32205]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71422220-775b-4b49-a9a4-ae0a7a16c2e5"}]: dispatch 2024-03-18T14:45:44.313 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:43 smithi076 ceph-mon[32205]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71422220-775b-4b49-a9a4-ae0a7a16c2e5"}]': finished 2024-03-18T14:45:44.313 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:43 smithi076 ceph-mon[32205]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T14:45:44.313 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:43 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:45:45.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:44 smithi174 ceph-mon[36787]: from='client.? 172.21.15.77:0/1760691474' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:45:45.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:44 smithi174 ceph-mon[36787]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:45.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:44 smithi077 ceph-mon[35865]: from='client.? 172.21.15.77:0/1760691474' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:45:45.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:44 smithi077 ceph-mon[35865]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:45.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:44 smithi076 ceph-mon[32205]: from='client.? 172.21.15.77:0/1760691474' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:45:45.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:44 smithi076 ceph-mon[32205]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:47.543 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:47 smithi077 ceph-mon[35865]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:47.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:47 smithi076 ceph-mon[32205]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:47.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:47 smithi174 ceph-mon[36787]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:49.580 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:49 smithi077 ceph-mon[35865]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:49 smithi076 ceph-mon[32205]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:49 smithi174 ceph-mon[36787]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:51.660 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:51 smithi077 ceph-mon[35865]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:51.661 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:51 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T14:45:51.661 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:51 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:51.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:51 smithi076 ceph-mon[32205]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:51.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:51 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T14:45:51.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:51 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:51.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:51 smithi174 ceph-mon[36787]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:51.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:51 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T14:45:51.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:51 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:52.599 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:52 smithi077 ceph-mon[35865]: Deploying daemon osd.1 on smithi077 2024-03-18T14:45:52.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:52 smithi076 ceph-mon[32205]: Deploying daemon osd.1 on smithi077 2024-03-18T14:45:52.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:52 smithi174 ceph-mon[36787]: Deploying daemon osd.1 on smithi077 2024-03-18T14:45:53.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:53 smithi076 ceph-mon[32205]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:53.668 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:53 smithi077 ceph-mon[35865]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:53.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:53 smithi174 ceph-mon[36787]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:55.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:55 smithi077 ceph-mon[35865]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:55.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:55 smithi076 ceph-mon[32205]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:55.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:55 smithi174 ceph-mon[36787]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:56.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:56 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:56.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:56 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:56.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:56 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:56.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:56 smithi076 ceph-mon[32205]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:56.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:56 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:56.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:56 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:56.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:56 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:56.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:56 smithi174 ceph-mon[36787]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:56.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:56 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:56.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:56 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:56.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:56 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:56.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:56 smithi077 ceph-mon[35865]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:58.815+0000 7fe0497fa700 1 -- 172.21.15.77:0/1776697513 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe04c0fe930 con 0x7fe0340a2800 2024-03-18T14:45:58.818 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 1 on host 'smithi077' 2024-03-18T14:45:58.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:58.818+0000 7fe052614700 1 -- 172.21.15.77:0/1776697513 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe0340a2800 msgr2=0x7fe0340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:58.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:58.818+0000 7fe052614700 1 --2- 172.21.15.77:0/1776697513 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe0340a2800 0x7fe0340a4cc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe040005fd0 tx=0x7fe040005de0 comp rx=0 tx=0).stop 2024-03-18T14:45:58.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:58.818+0000 7fe052614700 1 -- 172.21.15.77:0/1776697513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe04c1023f0 msgr2=0x7fe04c1037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:45:58.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:58.818+0000 7fe052614700 1 --2- 172.21.15.77:0/1776697513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe04c1023f0 0x7fe04c1037d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe03c005910 tx=0x7fe03c009580 comp rx=0 tx=0).stop 2024-03-18T14:45:58.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:58.818+0000 7fe052614700 1 -- 172.21.15.77:0/1776697513 shutdown_connections 2024-03-18T14:45:58.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:58.818+0000 7fe052614700 1 --2- 172.21.15.77:0/1776697513 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe0340a2800 0x7fe0340a4cc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:58.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:58.818+0000 7fe052614700 1 --2- 172.21.15.77:0/1776697513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe04c102d90 0x7fe04c103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:58.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:58.818+0000 7fe052614700 1 --2- 172.21.15.77:0/1776697513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe04c1011f0 0x7fe04c105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:58.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:58.818+0000 7fe052614700 1 --2- 172.21.15.77:0/1776697513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe04c1023f0 0x7fe04c1037d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:45:58.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:58.818+0000 7fe052614700 1 -- 172.21.15.77:0/1776697513 >> 172.21.15.77:0/1776697513 conn(0x7fe04c0fc790 msgr2=0x7fe04c0fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:45:58.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:58.818+0000 7fe052614700 1 -- 172.21.15.77:0/1776697513 shutdown_connections 2024-03-18T14:45:58.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:45:58.818+0000 7fe052614700 1 -- 172.21.15.77:0/1776697513 wait complete. 2024-03-18T14:45:59.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:59.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:59.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:59.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:59.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:59.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:59 smithi077 ceph-mon[35865]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:59.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:59.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:59.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:45:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:59.471 DEBUG:teuthology.orchestra.run.smithi077:osd.1> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.1.service 2024-03-18T14:45:59.473 INFO:tasks.cephadm:Deploying osd.2 on smithi174 with /dev/vg_nvme/lv_4... 2024-03-18T14:45:59.473 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T14:45:59.503 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:59 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:59.503 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:59 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:59.503 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:59 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:59.503 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:59 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:59.503 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:59 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:59.504 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:59 smithi174 ceph-mon[36787]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:59.504 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:59 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:59.504 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:59 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:59.504 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:45:59 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:59.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:59 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:59.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:59 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:59.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:59 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:45:59.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:59 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:45:59.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:59 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:59.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:59 smithi076 ceph-mon[32205]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:45:59.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:59 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:59.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:59 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:45:59.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:45:59 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:45:59.715 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T14:45:59.727 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 14:45:59 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-1[40462]: 2024-03-18T14:45:59.563+0000 7f97a0de27c0 -1 osd.1 0 log_to_monitors true 2024-03-18T14:46:00.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:00 smithi174 ceph-mon[36787]: from='osd.1 [v2:172.21.15.77:6800/3243872836,v1:172.21.15.77:6801/3243872836]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T14:46:00.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:00 smithi174 ceph-mon[36787]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T14:46:00.630 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:00 smithi077 ceph-mon[35865]: from='osd.1 [v2:172.21.15.77:6800/3243872836,v1:172.21.15.77:6801/3243872836]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T14:46:00.630 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:00 smithi077 ceph-mon[35865]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T14:46:00.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:00 smithi076 ceph-mon[32205]: from='osd.1 [v2:172.21.15.77:6800/3243872836,v1:172.21.15.77:6801/3243872836]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T14:46:00.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:00 smithi076 ceph-mon[32205]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T14:46:01.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:01 smithi174 ceph-mon[36787]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T14:46:01.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:01 smithi174 ceph-mon[36787]: from='osd.1 [v2:172.21.15.77:6800/3243872836,v1:172.21.15.77:6801/3243872836]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:46:01.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:01 smithi174 ceph-mon[36787]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T14:46:01.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:01 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:46:01.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:01 smithi174 ceph-mon[36787]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:46:01.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:01 smithi174 ceph-mon[36787]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:46:01.534 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 14:46:01 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-1[40462]: 2024-03-18T14:46:01.272+0000 7f9796447700 -1 osd.1 0 waiting for initial osdmap 2024-03-18T14:46:01.534 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 14:46:01 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-1[40462]: 2024-03-18T14:46:01.280+0000 7f979223a700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:46:01.535 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:01 smithi077 ceph-mon[35865]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T14:46:01.535 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:01 smithi077 ceph-mon[35865]: from='osd.1 [v2:172.21.15.77:6800/3243872836,v1:172.21.15.77:6801/3243872836]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:46:01.535 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:01 smithi077 ceph-mon[35865]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T14:46:01.535 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:01 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:46:01.535 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:01 smithi077 ceph-mon[35865]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:46:01.535 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:01 smithi077 ceph-mon[35865]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:46:01.656 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T14:46:01.656 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T14:46:01.656 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-03-18T14:46:01.657 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-03-18T14:46:01.657 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0157728 s, 665 MB/s 2024-03-18T14:46:01.657 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-03-18T14:46:01.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:01 smithi076 ceph-mon[32205]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T14:46:01.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:01 smithi076 ceph-mon[32205]: from='osd.1 [v2:172.21.15.77:6800/3243872836,v1:172.21.15.77:6801/3243872836]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:46:01.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:01 smithi076 ceph-mon[32205]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T14:46:01.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:01 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:46:01.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:01 smithi076 ceph-mon[32205]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:46:01.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:01 smithi076 ceph-mon[32205]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:46:02.270 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi174:vg_nvme/lv_4 2024-03-18T14:46:02.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:02 smithi077 ceph-mon[35865]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-18T14:46:02.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:02 smithi077 ceph-mon[35865]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T14:46:02.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:02 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:46:02.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:02 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:46:02.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:02 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:02.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:02 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:02.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:02 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:46:02.480 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:02 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:02.480 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:02 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:02.480 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:02 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:46:02.480 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:02 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:02.537 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:02 smithi174 ceph-mon[36787]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-18T14:46:02.537 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:02 smithi174 ceph-mon[36787]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T14:46:02.537 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:02 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:46:02.537 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:02 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:46:02.537 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:02 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:02.537 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:02 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:02.537 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:02 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:46:02.538 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:02 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:02.538 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:02 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:02.538 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:02 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:46:02.538 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:02 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:02.552 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T14:46:02.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:02 smithi076 ceph-mon[32205]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-18T14:46:02.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:02 smithi076 ceph-mon[32205]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T14:46:02.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:02 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:46:02.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:02 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:46:02.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:02 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:02.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:02 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:02.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:02 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:46:02.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:02 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:02.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:02 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:02.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:02 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:46:02.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:02 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:03.579 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:03 smithi174 ceph-mon[36787]: purged_snaps scrub starts 2024-03-18T14:46:03.579 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:03 smithi174 ceph-mon[36787]: purged_snaps scrub ok 2024-03-18T14:46:03.579 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:03 smithi174 ceph-mon[36787]: Detected new or changed devices on smithi077 2024-03-18T14:46:03.579 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:03 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi077 to 17080M 2024-03-18T14:46:03.579 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:03 smithi174 ceph-mon[36787]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:46:03.579 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:03 smithi174 ceph-mon[36787]: osd.1 [v2:172.21.15.77:6800/3243872836,v1:172.21.15.77:6801/3243872836] boot 2024-03-18T14:46:03.579 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:03 smithi174 ceph-mon[36787]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T14:46:03.579 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:03 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:46:03.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:03 smithi076 ceph-mon[32205]: purged_snaps scrub starts 2024-03-18T14:46:03.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:03 smithi076 ceph-mon[32205]: purged_snaps scrub ok 2024-03-18T14:46:03.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:03 smithi076 ceph-mon[32205]: Detected new or changed devices on smithi077 2024-03-18T14:46:03.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:03 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi077 to 17080M 2024-03-18T14:46:03.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:03 smithi076 ceph-mon[32205]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:46:03.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:03 smithi076 ceph-mon[32205]: osd.1 [v2:172.21.15.77:6800/3243872836,v1:172.21.15.77:6801/3243872836] boot 2024-03-18T14:46:03.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:03 smithi076 ceph-mon[32205]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T14:46:03.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:03 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:46:03.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:03 smithi077 ceph-mon[35865]: purged_snaps scrub starts 2024-03-18T14:46:03.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:03 smithi077 ceph-mon[35865]: purged_snaps scrub ok 2024-03-18T14:46:03.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:03 smithi077 ceph-mon[35865]: Detected new or changed devices on smithi077 2024-03-18T14:46:03.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:03 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi077 to 17080M 2024-03-18T14:46:03.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:03 smithi077 ceph-mon[35865]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T14:46:03.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:03 smithi077 ceph-mon[35865]: osd.1 [v2:172.21.15.77:6800/3243872836,v1:172.21.15.77:6801/3243872836] boot 2024-03-18T14:46:03.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:03 smithi077 ceph-mon[35865]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T14:46:03.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:03 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:46:04.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.041+0000 7f59e3c17700 1 -- 172.21.15.174:0/1411759798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc1056f0 msgr2=0x7f59dc107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:46:04.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.041+0000 7f59e3c17700 1 --2- 172.21.15.174:0/1411759798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc1056f0 0x7f59dc107ae0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f59cc009990 tx=0x7f59cc009ca0 comp rx=0 tx=0).stop 2024-03-18T14:46:04.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.041+0000 7f59e3c17700 1 -- 172.21.15.174:0/1411759798 shutdown_connections 2024-03-18T14:46:04.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.041+0000 7f59e3c17700 1 --2- 172.21.15.174:0/1411759798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59dc108020 0x7f59dc10a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:04.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.041+0000 7f59e3c17700 1 --2- 172.21.15.174:0/1411759798 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59dc102dc0 0x7f59dc1051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:04.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.041+0000 7f59e3c17700 1 --2- 172.21.15.174:0/1411759798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc1056f0 0x7f59dc107ae0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:04.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.041+0000 7f59e3c17700 1 -- 172.21.15.174:0/1411759798 >> 172.21.15.174:0/1411759798 conn(0x7f59dc0fc7f0 msgr2=0x7f59dc0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:46:04.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.041+0000 7f59e3c17700 1 -- 172.21.15.174:0/1411759798 shutdown_connections 2024-03-18T14:46:04.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.041+0000 7f59e3c17700 1 -- 172.21.15.174:0/1411759798 wait complete. 2024-03-18T14:46:04.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.041+0000 7f59e3c17700 1 Processor -- start 2024-03-18T14:46:04.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.041+0000 7f59e3c17700 1 -- start start 2024-03-18T14:46:04.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e3c17700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59dc102dc0 0x7f59dc102720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:46:04.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e3c17700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59dc1056f0 0x7f59dc100d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:46:04.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e3c17700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc108020 0x7f59dc1012b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:46:04.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e3c17700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc10df60 con 0x7f59dc102dc0 2024-03-18T14:46:04.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e3c17700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc10dde0 con 0x7f59dc1056f0 2024-03-18T14:46:04.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e3c17700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc10e0e0 con 0x7f59dc108020 2024-03-18T14:46:04.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e21b4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc108020 0x7f59dc1012b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:46:04.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e21b4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc108020 0x7f59dc1012b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:37630/0 (socket says 172.21.15.174:37630) 2024-03-18T14:46:04.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e21b4700 1 -- 172.21.15.174:0/3261113657 learned_addr learned my addr 172.21.15.174:0/3261113657 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-18T14:46:04.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e11b2700 1 --2- 172.21.15.174:0/3261113657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59dc1056f0 0x7f59dc100d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:46:04.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e19b3700 1 --2- 172.21.15.174:0/3261113657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59dc102dc0 0x7f59dc102720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:46:04.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e21b4700 1 -- 172.21.15.174:0/3261113657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59dc1056f0 msgr2=0x7f59dc100d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:46:04.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e21b4700 1 --2- 172.21.15.174:0/3261113657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59dc1056f0 0x7f59dc100d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:04.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e21b4700 1 -- 172.21.15.174:0/3261113657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59dc102dc0 msgr2=0x7f59dc102720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:46:04.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e21b4700 1 --2- 172.21.15.174:0/3261113657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59dc102dc0 0x7f59dc102720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:04.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e21b4700 1 -- 172.21.15.174:0/3261113657 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59dc101b60 con 0x7f59dc108020 2024-03-18T14:46:04.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e11b2700 1 --2- 172.21.15.174:0/3261113657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59dc1056f0 0x7f59dc100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:46:04.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e19b3700 1 --2- 172.21.15.174:0/3261113657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59dc102dc0 0x7f59dc102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:46:04.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e21b4700 1 --2- 172.21.15.174:0/3261113657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc108020 0x7f59dc1012b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f59d40094f0 tx=0x7f59d4007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:46:04.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59d2ffd700 1 -- 172.21.15.174:0/3261113657 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4004db0 con 0x7f59dc108020 2024-03-18T14:46:04.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59d2ffd700 1 -- 172.21.15.174:0/3261113657 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59d400db60 con 0x7f59dc108020 2024-03-18T14:46:04.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59d2ffd700 1 -- 172.21.15.174:0/3261113657 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4015660 con 0x7f59dc108020 2024-03-18T14:46:04.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.042+0000 7f59e3c17700 1 -- 172.21.15.174:0/3261113657 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59dc1a26c0 con 0x7f59dc108020 2024-03-18T14:46:04.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.043+0000 7f59e3c17700 1 -- 172.21.15.174:0/3261113657 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59dc1a2b80 con 0x7f59dc108020 2024-03-18T14:46:04.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.043+0000 7f59e3c17700 1 -- 172.21.15.174:0/3261113657 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59dc068f90 con 0x7f59dc108020 2024-03-18T14:46:04.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.044+0000 7f59d2ffd700 1 -- 172.21.15.174:0/3261113657 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f59d4005020 con 0x7f59dc108020 2024-03-18T14:46:04.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.045+0000 7f59d2ffd700 1 --2- 172.21.15.174:0/3261113657 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f59c80a2800 0x7f59c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:46:04.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.045+0000 7f59d2ffd700 1 -- 172.21.15.174:0/3261113657 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f59d40ccd80 con 0x7f59dc108020 2024-03-18T14:46:04.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.046+0000 7f59e19b3700 1 --2- 172.21.15.174:0/3261113657 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f59c80a2800 0x7f59c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:46:04.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.046+0000 7f59e19b3700 1 --2- 172.21.15.174:0/3261113657 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f59c80a2800 0x7f59c80a4cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f59d8005fd0 tx=0x7f59d8005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:46:04.051 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.050+0000 7f59d2ffd700 1 -- 172.21.15.174:0/3261113657 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f59d4010070 con 0x7f59dc108020 2024-03-18T14:46:04.254 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:04.253+0000 7f59e3c17700 1 -- 172.21.15.174:0/3261113657 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f59dc0627d0 con 0x7f59c80a2800 2024-03-18T14:46:04.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:04 smithi076 ceph-mon[32205]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T14:46:04.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:04 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:46:04.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:04 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:46:04.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:04 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:04.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:04 smithi174 ceph-mon[36787]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T14:46:04.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:04 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:46:04.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:04 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:46:04.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:04 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:04.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:04 smithi077 ceph-mon[35865]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T14:46:04.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:04 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:46:04.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:04 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:46:04.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:04 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:05.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:05 smithi174 ceph-mon[36787]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:46:05.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:05 smithi174 ceph-mon[36787]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:05.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:05 smithi076 ceph-mon[32205]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:46:05.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:05 smithi076 ceph-mon[32205]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:05.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:05 smithi077 ceph-mon[35865]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:46:05.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:05 smithi077 ceph-mon[35865]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:07.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:07 smithi174 ceph-mon[36787]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:07 smithi174 ceph-mon[36787]: from='client.? 172.21.15.174:0/1349460388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a5bf599-bc62-4986-9bf3-6486a08857a2"}]: dispatch 2024-03-18T14:46:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:07 smithi174 ceph-mon[36787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a5bf599-bc62-4986-9bf3-6486a08857a2"}]: dispatch 2024-03-18T14:46:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:07 smithi174 ceph-mon[36787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a5bf599-bc62-4986-9bf3-6486a08857a2"}]': finished 2024-03-18T14:46:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:07 smithi174 ceph-mon[36787]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T14:46:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:07 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:07.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:07 smithi076 ceph-mon[32205]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:07.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:07 smithi076 ceph-mon[32205]: from='client.? 172.21.15.174:0/1349460388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a5bf599-bc62-4986-9bf3-6486a08857a2"}]: dispatch 2024-03-18T14:46:07.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:07 smithi076 ceph-mon[32205]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a5bf599-bc62-4986-9bf3-6486a08857a2"}]: dispatch 2024-03-18T14:46:07.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:07 smithi076 ceph-mon[32205]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a5bf599-bc62-4986-9bf3-6486a08857a2"}]': finished 2024-03-18T14:46:07.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:07 smithi076 ceph-mon[32205]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T14:46:07.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:07 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:07 smithi077 ceph-mon[35865]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:07.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:07 smithi077 ceph-mon[35865]: from='client.? 172.21.15.174:0/1349460388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a5bf599-bc62-4986-9bf3-6486a08857a2"}]: dispatch 2024-03-18T14:46:07.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:07 smithi077 ceph-mon[35865]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a5bf599-bc62-4986-9bf3-6486a08857a2"}]: dispatch 2024-03-18T14:46:07.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:07 smithi077 ceph-mon[35865]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a5bf599-bc62-4986-9bf3-6486a08857a2"}]': finished 2024-03-18T14:46:07.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:07 smithi077 ceph-mon[35865]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T14:46:07.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:07 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:08.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:08 smithi076 ceph-mon[32205]: from='client.? 172.21.15.174:0/3781815539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:46:08.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:08 smithi174 ceph-mon[36787]: from='client.? 172.21.15.174:0/3781815539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:46:08.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:08 smithi077 ceph-mon[35865]: from='client.? 172.21.15.174:0/3781815539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:46:09.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:09 smithi076 ceph-mon[32205]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:09.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:09 smithi174 ceph-mon[36787]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:09.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:09 smithi077 ceph-mon[35865]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:11.598 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:11 smithi174 ceph-mon[36787]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:11.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:11 smithi076 ceph-mon[32205]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:11.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:11 smithi077 ceph-mon[35865]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:13.601 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:13 smithi174 ceph-mon[36787]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:13.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:13 smithi076 ceph-mon[32205]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:13.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:13 smithi077 ceph-mon[35865]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:14.583 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:14 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T14:46:14.584 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:14 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:14.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:14 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T14:46:14.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:14 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:14.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:14 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T14:46:14.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:14 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:15.523 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:15 smithi174 ceph-mon[36787]: Deploying daemon osd.2 on smithi174 2024-03-18T14:46:15.523 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:15 smithi174 ceph-mon[36787]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:15.629 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:15 smithi077 ceph-mon[35865]: Deploying daemon osd.2 on smithi174 2024-03-18T14:46:15.629 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:15 smithi077 ceph-mon[35865]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:15.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:15 smithi076 ceph-mon[32205]: Deploying daemon osd.2 on smithi174 2024-03-18T14:46:15.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:15 smithi076 ceph-mon[32205]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:17.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:17 smithi174 ceph-mon[36787]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:17.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:17 smithi076 ceph-mon[32205]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:17.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:17 smithi077 ceph-mon[35865]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:19.581 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:19 smithi174 ceph-mon[36787]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:19.581 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:19.581 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:19.581 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:46:19.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:19 smithi076 ceph-mon[32205]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:19.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:19.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:19.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:46:19.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:19 smithi077 ceph-mon[35865]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:19.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:19.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:19.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:46:21.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:21 smithi174 ceph-mon[36787]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:21.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:21 smithi076 ceph-mon[32205]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:21.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:21 smithi077 ceph-mon[35865]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:22.217 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:22.216+0000 7f59d2ffd700 1 -- 172.21.15.174:0/3261113657 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f59dc0627d0 con 0x7f59c80a2800 2024-03-18T14:46:22.218 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 2 on host 'smithi174' 2024-03-18T14:46:22.219 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:22.218+0000 7f59e3c17700 1 -- 172.21.15.174:0/3261113657 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f59c80a2800 msgr2=0x7f59c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:46:22.219 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:22.218+0000 7f59e3c17700 1 --2- 172.21.15.174:0/3261113657 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f59c80a2800 0x7f59c80a4cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f59d8005fd0 tx=0x7f59d8005ee0 comp rx=0 tx=0).stop 2024-03-18T14:46:22.219 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:22.218+0000 7f59e3c17700 1 -- 172.21.15.174:0/3261113657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc108020 msgr2=0x7f59dc1012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:46:22.219 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:22.218+0000 7f59e3c17700 1 --2- 172.21.15.174:0/3261113657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc108020 0x7f59dc1012b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f59d40094f0 tx=0x7f59d4007f90 comp rx=0 tx=0).stop 2024-03-18T14:46:22.219 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:22.218+0000 7f59e3c17700 1 -- 172.21.15.174:0/3261113657 shutdown_connections 2024-03-18T14:46:22.220 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:22.218+0000 7f59e3c17700 1 --2- 172.21.15.174:0/3261113657 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f59c80a2800 0x7f59c80a4cc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:22.220 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:22.218+0000 7f59e3c17700 1 --2- 172.21.15.174:0/3261113657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc108020 0x7f59dc1012b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:22.220 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:22.218+0000 7f59e3c17700 1 --2- 172.21.15.174:0/3261113657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59dc102dc0 0x7f59dc102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:22.220 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:22.218+0000 7f59e3c17700 1 --2- 172.21.15.174:0/3261113657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59dc1056f0 0x7f59dc100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:22.220 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:22.218+0000 7f59e3c17700 1 -- 172.21.15.174:0/3261113657 >> 172.21.15.174:0/3261113657 conn(0x7f59dc0fc7f0 msgr2=0x7f59dc103700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:46:22.220 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:22.218+0000 7f59e3c17700 1 -- 172.21.15.174:0/3261113657 shutdown_connections 2024-03-18T14:46:22.220 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:46:22.218+0000 7f59e3c17700 1 -- 172.21.15.174:0/3261113657 wait complete. 2024-03-18T14:46:22.874 DEBUG:teuthology.orchestra.run.smithi174:osd.2> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.2.service 2024-03-18T14:46:22.877 INFO:tasks.cephadm:Deploying osd.3 on smithi076 with /dev/vg_nvme/lv_3... 2024-03-18T14:46:22.877 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T14:46:23.099 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:22 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:23.099 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:22 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:23.099 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:22 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:23.099 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:22 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:46:23.099 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:22 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:23.099 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:22 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:23.099 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:22 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:23.099 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:22 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:46:23.099 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:22 smithi174 ceph-mon[36787]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:23.099 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:22 smithi174 ceph-mon[36787]: from='osd.2 [v2:172.21.15.174:6800/3812090370,v1:172.21.15.174:6801/3812090370]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T14:46:23.100 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:22 smithi174 ceph-mon[36787]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T14:46:23.126 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:46:23.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:23.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:23.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:23.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:46:23.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:23.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:23.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:23.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:46:23.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:22 smithi076 ceph-mon[32205]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:23.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:22 smithi076 ceph-mon[32205]: from='osd.2 [v2:172.21.15.174:6800/3812090370,v1:172.21.15.174:6801/3812090370]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T14:46:23.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:22 smithi076 ceph-mon[32205]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T14:46:23.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:22 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:23.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:22 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:23.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:22 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:23.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:22 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:46:23.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:22 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:23.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:22 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:23.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:22 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:23.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:22 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:46:23.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:22 smithi077 ceph-mon[35865]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:23.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:22 smithi077 ceph-mon[35865]: from='osd.2 [v2:172.21.15.174:6800/3812090370,v1:172.21.15.174:6801/3812090370]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T14:46:23.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:22 smithi077 ceph-mon[35865]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T14:46:24.355 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:24 smithi174 ceph-mon[36787]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T14:46:24.355 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:24 smithi174 ceph-mon[36787]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T14:46:24.355 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:24.355 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:24 smithi174 ceph-mon[36787]: from='osd.2 [v2:172.21.15.174:6800/3812090370,v1:172.21.15.174:6801/3812090370]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:46:24.355 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:24 smithi174 ceph-mon[36787]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:46:24.355 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:24 smithi076 ceph-mon[32205]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T14:46:24.355 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:24 smithi076 ceph-mon[32205]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T14:46:24.356 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:24.356 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:24 smithi076 ceph-mon[32205]: from='osd.2 [v2:172.21.15.174:6800/3812090370,v1:172.21.15.174:6801/3812090370]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:46:24.356 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:24 smithi076 ceph-mon[32205]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:46:24.356 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 14:46:24 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-2[41937]: 2024-03-18T14:46:24.196+0000 7f86b0f4a700 -1 osd.2 0 waiting for initial osdmap 2024-03-18T14:46:24.356 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 14:46:24 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-2[41937]: 2024-03-18T14:46:24.200+0000 7f86acd3d700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:46:24.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:24 smithi077 ceph-mon[35865]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T14:46:24.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:24 smithi077 ceph-mon[35865]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T14:46:24.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:24.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:24 smithi077 ceph-mon[35865]: from='osd.2 [v2:172.21.15.174:6800/3812090370,v1:172.21.15.174:6801/3812090370]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:46:24.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:24 smithi077 ceph-mon[35865]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:46:24.916 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T14:46:24.916 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T14:46:24.916 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-03-18T14:46:24.916 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-03-18T14:46:24.916 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127991 s, 819 MB/s 2024-03-18T14:46:24.916 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-03-18T14:46:25.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:25 smithi174 ceph-mon[36787]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-18T14:46:25.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:25 smithi174 ceph-mon[36787]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T14:46:25.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:25 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:25.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:25 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:25.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:25 smithi174 ceph-mon[36787]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:25.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:25 smithi077 ceph-mon[35865]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-18T14:46:25.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:25 smithi077 ceph-mon[35865]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T14:46:25.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:25 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:25.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:25 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:25.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:25 smithi077 ceph-mon[35865]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:25.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:25 smithi076 ceph-mon[32205]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-18T14:46:25.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:25 smithi076 ceph-mon[32205]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T14:46:25.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:25 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:25.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:25 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:25.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:25 smithi076 ceph-mon[32205]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T14:46:25.571 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi076:vg_nvme/lv_3 2024-03-18T14:46:25.824 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:46:26.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:26 smithi174 ceph-mon[36787]: purged_snaps scrub starts 2024-03-18T14:46:26.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:26 smithi174 ceph-mon[36787]: purged_snaps scrub ok 2024-03-18T14:46:26.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:26.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:26 smithi174 ceph-mon[36787]: osd.2 [v2:172.21.15.174:6800/3812090370,v1:172.21.15.174:6801/3812090370] boot 2024-03-18T14:46:26.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:26 smithi174 ceph-mon[36787]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T14:46:26.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:26.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:26.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:26.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:46:26.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:26.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:26.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:46:26.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:26.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:26 smithi077 ceph-mon[35865]: purged_snaps scrub starts 2024-03-18T14:46:26.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:26 smithi077 ceph-mon[35865]: purged_snaps scrub ok 2024-03-18T14:46:26.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:26.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:26 smithi077 ceph-mon[35865]: osd.2 [v2:172.21.15.174:6800/3812090370,v1:172.21.15.174:6801/3812090370] boot 2024-03-18T14:46:26.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:26 smithi077 ceph-mon[35865]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T14:46:26.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:26.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:26.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:26.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:46:26.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:26.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:26.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:46:26.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:26.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:26 smithi076 ceph-mon[32205]: purged_snaps scrub starts 2024-03-18T14:46:26.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:26 smithi076 ceph-mon[32205]: purged_snaps scrub ok 2024-03-18T14:46:26.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:26.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:26 smithi076 ceph-mon[32205]: osd.2 [v2:172.21.15.174:6800/3812090370,v1:172.21.15.174:6801/3812090370] boot 2024-03-18T14:46:26.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:26 smithi076 ceph-mon[32205]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T14:46:26.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:46:26.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:26.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:26.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:46:26.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:26.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:26.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:46:26.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:27.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:27 smithi077 ceph-mon[35865]: Detected new or changed devices on smithi174 2024-03-18T14:46:27.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:27 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi174 to 17080M 2024-03-18T14:46:27.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:27 smithi077 ceph-mon[35865]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T14:46:27.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:27 smithi077 ceph-mon[35865]: pgmap v79: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:27.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:27 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T14:46:27.481 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:27 smithi076 ceph-mon[32205]: Detected new or changed devices on smithi174 2024-03-18T14:46:27.481 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:27 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi174 to 17080M 2024-03-18T14:46:27.482 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:27 smithi076 ceph-mon[32205]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T14:46:27.482 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:27 smithi076 ceph-mon[32205]: pgmap v79: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:27.482 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:27 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T14:46:27.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.492+0000 7fb2097b1700 1 -- 172.21.15.76:0/3369130984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb204102400 msgr2=0x7fb204102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:46:27.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.492+0000 7fb2097b1700 1 --2- 172.21.15.76:0/3369130984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb204102400 0x7fb204102860 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb1f80099c0 tx=0x7fb1f8009cd0 comp rx=0 tx=0).stop 2024-03-18T14:46:27.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.492+0000 7fb2097b1700 1 -- 172.21.15.76:0/3369130984 shutdown_connections 2024-03-18T14:46:27.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.492+0000 7fb2097b1700 1 --2- 172.21.15.76:0/3369130984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb204102da0 0x7fb204109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:27.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.492+0000 7fb2097b1700 1 --2- 172.21.15.76:0/3369130984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb204101200 0x7fb204101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:27.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.492+0000 7fb2097b1700 1 --2- 172.21.15.76:0/3369130984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb204102400 0x7fb204102860 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:27.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.492+0000 7fb2097b1700 1 -- 172.21.15.76:0/3369130984 >> 172.21.15.76:0/3369130984 conn(0x7fb2040fc7a0 msgr2=0x7fb2040febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:46:27.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.492+0000 7fb2097b1700 1 -- 172.21.15.76:0/3369130984 shutdown_connections 2024-03-18T14:46:27.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.492+0000 7fb2097b1700 1 -- 172.21.15.76:0/3369130984 wait complete. 2024-03-18T14:46:27.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.492+0000 7fb2097b1700 1 Processor -- start 2024-03-18T14:46:27.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb2097b1700 1 -- start start 2024-03-18T14:46:27.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb2097b1700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb204101200 0x7fb20419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:46:27.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb2097b1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb204102400 0x7fb20419aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:46:27.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb2097b1700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb204102da0 0x7fb2041a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:46:27.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb2097b1700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb20410cdc0 con 0x7fb204101200 2024-03-18T14:46:27.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb2097b1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb20410cc40 con 0x7fb204102400 2024-03-18T14:46:27.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb2097b1700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb20410cf40 con 0x7fb204102da0 2024-03-18T14:46:27.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb202ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb204101200 0x7fb20419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:46:27.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb202ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb204101200 0x7fb20419a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37404/0 (socket says 172.21.15.76:37404) 2024-03-18T14:46:27.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb202ffd700 1 -- 172.21.15.76:0/3517034394 learned_addr learned my addr 172.21.15.76:0/3517034394 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:46:27.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb2027fc700 1 --2- 172.21.15.76:0/3517034394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb204102400 0x7fb20419aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:46:27.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb2037fe700 1 --2- 172.21.15.76:0/3517034394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb204102da0 0x7fb2041a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:46:27.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb202ffd700 1 -- 172.21.15.76:0/3517034394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb204102da0 msgr2=0x7fb2041a1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:46:27.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb202ffd700 1 --2- 172.21.15.76:0/3517034394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb204102da0 0x7fb2041a1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:27.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb202ffd700 1 -- 172.21.15.76:0/3517034394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb204102400 msgr2=0x7fb20419aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:46:27.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb202ffd700 1 --2- 172.21.15.76:0/3517034394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb204102400 0x7fb20419aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:27.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb202ffd700 1 -- 172.21.15.76:0/3517034394 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2041a23c0 con 0x7fb204101200 2024-03-18T14:46:27.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb2027fc700 1 --2- 172.21.15.76:0/3517034394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb204102400 0x7fb20419aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:46:27.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.493+0000 7fb202ffd700 1 --2- 172.21.15.76:0/3517034394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb204101200 0x7fb20419a760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb1f400c9a0 tx=0x7fb1f400cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:46:27.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.494+0000 7fb1ebfff700 1 -- 172.21.15.76:0/3517034394 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4004220 con 0x7fb204101200 2024-03-18T14:46:27.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.494+0000 7fb1ebfff700 1 -- 172.21.15.76:0/3517034394 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1f4007d40 con 0x7fb204101200 2024-03-18T14:46:27.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.494+0000 7fb2097b1700 1 -- 172.21.15.76:0/3517034394 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2041a26b0 con 0x7fb204101200 2024-03-18T14:46:27.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.494+0000 7fb1ebfff700 1 -- 172.21.15.76:0/3517034394 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f400f450 con 0x7fb204101200 2024-03-18T14:46:27.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.494+0000 7fb2097b1700 1 -- 172.21.15.76:0/3517034394 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2041a2bf0 con 0x7fb204101200 2024-03-18T14:46:27.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.494+0000 7fb2097b1700 1 -- 172.21.15.76:0/3517034394 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb204068f90 con 0x7fb204101200 2024-03-18T14:46:27.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.496+0000 7fb1ebfff700 1 -- 172.21.15.76:0/3517034394 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7fb1f4007860 con 0x7fb204101200 2024-03-18T14:46:27.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.496+0000 7fb1ebfff700 1 --2- 172.21.15.76:0/3517034394 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb1ec0a2800 0x7fb1ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:46:27.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.496+0000 7fb2027fc700 1 --2- 172.21.15.76:0/3517034394 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb1ec0a2800 0x7fb1ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:46:27.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.496+0000 7fb1ebfff700 1 -- 172.21.15.76:0/3517034394 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3226+0+0 (secure 0 0 0) 0x7fb1f40cd540 con 0x7fb204101200 2024-03-18T14:46:27.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.497+0000 7fb2027fc700 1 --2- 172.21.15.76:0/3517034394 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb1ec0a2800 0x7fb1ec0a4cc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb1f80039c0 tx=0x7fb1f8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:46:27.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.500+0000 7fb1ebfff700 1 -- 172.21.15.76:0/3517034394 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb1f4014030 con 0x7fb204101200 2024-03-18T14:46:27.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:27.705+0000 7fb2097b1700 1 -- 172.21.15.76:0/3517034394 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb2040fe9a0 con 0x7fb1ec0a2800 2024-03-18T14:46:27.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:27 smithi174 ceph-mon[36787]: Detected new or changed devices on smithi174 2024-03-18T14:46:27.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:27 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi174 to 17080M 2024-03-18T14:46:27.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:27 smithi174 ceph-mon[36787]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T14:46:27.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:27 smithi174 ceph-mon[36787]: pgmap v79: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:27.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:27 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T14:46:28.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:28 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T14:46:28.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:28 smithi077 ceph-mon[35865]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T14:46:28.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:28 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T14:46:28.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:28 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:46:28.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:28 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:46:28.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:28 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:28.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:28 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T14:46:28.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:28 smithi076 ceph-mon[32205]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T14:46:28.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:28 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T14:46:28.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:28 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:46:28.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:28 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:46:28.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:28 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:28.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:28 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T14:46:28.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:28 smithi174 ceph-mon[36787]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T14:46:28.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:28 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T14:46:28.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:28 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:46:28.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:28 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:46:28.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:28 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:28.828 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 14:46:28 smithi076 sudo[51157]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T14:46:28.828 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 14:46:28 smithi076 sudo[51157]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T14:46:28.828 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 14:46:28 smithi076 sudo[51157]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T14:46:28.828 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 14:46:28 smithi076 sudo[51157]: pam_unix(sudo:session): session closed for user root 2024-03-18T14:46:29.161 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 14:46:28 smithi076 sudo[51172]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T14:46:29.161 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 14:46:28 smithi076 sudo[51172]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T14:46:29.161 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 14:46:28 smithi076 sudo[51172]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T14:46:29.162 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 14:46:28 smithi076 sudo[51172]: pam_unix(sudo:session): session closed for user root 2024-03-18T14:46:29.354 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:29 smithi077 ceph-mon[35865]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:46:29.354 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:29 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T14:46:29.354 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:29 smithi077 ceph-mon[35865]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T14:46:29.354 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:29 smithi077 ceph-mon[35865]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:29.354 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:29 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:29.354 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 14:46:29 smithi077 sudo[44306]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T14:46:29.354 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 14:46:29 smithi077 sudo[44306]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T14:46:29.354 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 14:46:29 smithi077 sudo[44306]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T14:46:29.355 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 14:46:29 smithi077 sudo[44306]: pam_unix(sudo:session): session closed for user root 2024-03-18T14:46:29.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:29 smithi076 ceph-mon[32205]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:46:29.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:29 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T14:46:29.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:29 smithi076 ceph-mon[32205]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T14:46:29.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:29 smithi076 ceph-mon[32205]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:29.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:29 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:29.623 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:29 smithi174 ceph-mon[36787]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:46:29.623 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:29 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T14:46:29.623 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:29 smithi174 ceph-mon[36787]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T14:46:29.623 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:29 smithi174 ceph-mon[36787]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:29.623 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:29 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:29.727 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 14:46:29 smithi077 sudo[44309]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T14:46:29.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 14:46:29 smithi077 sudo[44309]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T14:46:29.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 14:46:29 smithi077 sudo[44309]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T14:46:29.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 14:46:29 smithi077 sudo[44309]: pam_unix(sudo:session): session closed for user root 2024-03-18T14:46:29.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:29.860+0000 7fb1ebfff700 1 -- 172.21.15.76:0/3517034394 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb1f4097a10 con 0x7fb204101200 2024-03-18T14:46:29.880 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 14:46:29 smithi174 sudo[46016]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T14:46:29.880 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 14:46:29 smithi174 sudo[46016]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T14:46:29.880 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 14:46:29 smithi174 sudo[46016]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T14:46:29.880 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 14:46:29 smithi174 sudo[46016]: pam_unix(sudo:session): session closed for user root 2024-03-18T14:46:30.217 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 14:46:29 smithi174 sudo[46019]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T14:46:30.217 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 14:46:29 smithi174 sudo[46019]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T14:46:30.218 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 14:46:29 smithi174 sudo[46019]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T14:46:30.218 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 14:46:29 smithi174 sudo[46019]: pam_unix(sudo:session): session closed for user root 2024-03-18T14:46:30.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:30 smithi076 sudo[51237]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T14:46:30.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:30 smithi076 sudo[51237]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T14:46:30.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:30 smithi076 sudo[51237]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T14:46:30.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:30 smithi076 sudo[51237]: pam_unix(sudo:session): session closed for user root 2024-03-18T14:46:30.676 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:30 smithi174 ceph-mon[36787]: osdmap e22: 3 total, 3 up, 3 in 2024-03-18T14:46:30.676 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:30 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:30.676 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:30 smithi174 ceph-mon[36787]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T14:46:30.676 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:30 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:30.676 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:30 smithi174 ceph-mon[36787]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T14:46:30.676 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:30 smithi174 sudo[46022]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T14:46:30.676 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:30 smithi174 sudo[46022]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T14:46:30.676 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:30 smithi174 sudo[46022]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T14:46:30.676 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:30 smithi174 sudo[46022]: pam_unix(sudo:session): session closed for user root 2024-03-18T14:46:30.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:30 smithi077 ceph-mon[35865]: osdmap e22: 3 total, 3 up, 3 in 2024-03-18T14:46:30.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:30 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:30.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:30 smithi077 ceph-mon[35865]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T14:46:30.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:30 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:30.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:30 smithi077 ceph-mon[35865]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T14:46:30.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:30 smithi076 sudo[51240]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T14:46:30.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:30 smithi076 sudo[51240]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T14:46:30.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:30 smithi076 sudo[51240]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T14:46:30.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:30 smithi076 sudo[51240]: pam_unix(sudo:session): session closed for user root 2024-03-18T14:46:30.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:30 smithi076 ceph-mon[32205]: osdmap e22: 3 total, 3 up, 3 in 2024-03-18T14:46:30.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:30 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:30.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:30 smithi076 ceph-mon[32205]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T14:46:30.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:30 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:30.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:30 smithi076 ceph-mon[32205]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T14:46:30.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:30 smithi174 sudo[46025]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T14:46:30.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:30 smithi174 sudo[46025]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T14:46:30.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:30 smithi174 sudo[46025]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T14:46:30.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:30 smithi174 sudo[46025]: pam_unix(sudo:session): session closed for user root 2024-03-18T14:46:31.413 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 sudo[44312]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T14:46:31.414 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 sudo[44312]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T14:46:31.414 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 sudo[44312]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T14:46:31.414 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 sudo[44312]: pam_unix(sudo:session): session closed for user root 2024-03-18T14:46:31.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:31 smithi076 ceph-mon[32205]: pgmap v84: 1 pgs: 1 active+clean; 321 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:31.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:31 smithi076 ceph-mon[32205]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T14:46:31.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:31 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:46:31.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:31 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:46:31.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:31 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:46:31.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:31 smithi076 ceph-mon[32205]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T14:46:31.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:31 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:46:31.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:31 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:46:31.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:31 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:46:31.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:31 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/464660568' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5060b19a-4b1f-4a15-bd07-a920a267bcfd"}]: dispatch 2024-03-18T14:46:31.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:31 smithi076 ceph-mon[32205]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T14:46:31.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:31 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/464660568' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5060b19a-4b1f-4a15-bd07-a920a267bcfd"}]': finished 2024-03-18T14:46:31.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:31 smithi076 ceph-mon[32205]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T14:46:31.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:31 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:46:31.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:31 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:46:31.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:31 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:46:31.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:31 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:46:31.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:31 smithi174 ceph-mon[36787]: pgmap v84: 1 pgs: 1 active+clean; 321 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:31.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:31 smithi174 ceph-mon[36787]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T14:46:31.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:31 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:46:31.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:31 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:46:31.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:31 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:46:31.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:31 smithi174 ceph-mon[36787]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T14:46:31.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:31 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:46:31.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:31 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:46:31.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:31 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:46:31.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:31 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/464660568' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5060b19a-4b1f-4a15-bd07-a920a267bcfd"}]: dispatch 2024-03-18T14:46:31.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:31 smithi174 ceph-mon[36787]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T14:46:31.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:31 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/464660568' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5060b19a-4b1f-4a15-bd07-a920a267bcfd"}]': finished 2024-03-18T14:46:31.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:31 smithi174 ceph-mon[36787]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T14:46:31.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:31 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:46:31.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:31 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:46:31.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:31 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:46:31.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:31 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:46:31.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 sudo[44315]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T14:46:31.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 sudo[44315]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T14:46:31.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 sudo[44315]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T14:46:31.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 sudo[44315]: pam_unix(sudo:session): session closed for user root 2024-03-18T14:46:31.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 ceph-mon[35865]: pgmap v84: 1 pgs: 1 active+clean; 321 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:31.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 ceph-mon[35865]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T14:46:31.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:46:31.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:46:31.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:46:31.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 ceph-mon[35865]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T14:46:31.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:46:31.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:46:31.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:46:31.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/464660568' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5060b19a-4b1f-4a15-bd07-a920a267bcfd"}]: dispatch 2024-03-18T14:46:31.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 ceph-mon[35865]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T14:46:31.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/464660568' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5060b19a-4b1f-4a15-bd07-a920a267bcfd"}]': finished 2024-03-18T14:46:31.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 ceph-mon[35865]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T14:46:31.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:46:31.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:46:31.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:46:31.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:31 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:46:32.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:32 smithi174 ceph-mon[36787]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T14:46:32.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:32 smithi174 ceph-mon[36787]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T14:46:32.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:32 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3348283525' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:46:32.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:32 smithi174 ceph-mon[36787]: pgmap v86: 1 pgs: 1 active+clean; 321 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:32.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:32 smithi077 ceph-mon[35865]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T14:46:32.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:32 smithi077 ceph-mon[35865]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T14:46:32.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:32 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3348283525' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:46:32.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:32 smithi077 ceph-mon[35865]: pgmap v86: 1 pgs: 1 active+clean; 321 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:32.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:32 smithi076 ceph-mon[32205]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T14:46:32.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:32 smithi076 ceph-mon[32205]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T14:46:32.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:32 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3348283525' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:46:32.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:32 smithi076 ceph-mon[32205]: pgmap v86: 1 pgs: 1 active+clean; 321 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:35.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:35 smithi076 ceph-mon[32205]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 12 KiB/s rd, 452 KiB/s wr, 34 op/s 2024-03-18T14:46:35.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:35 smithi174 ceph-mon[36787]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 12 KiB/s rd, 452 KiB/s wr, 34 op/s 2024-03-18T14:46:35.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:35 smithi077 ceph-mon[35865]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 12 KiB/s rd, 452 KiB/s wr, 34 op/s 2024-03-18T14:46:37.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:37 smithi076 ceph-mon[32205]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 8.8 KiB/s rd, 324 KiB/s wr, 24 op/s 2024-03-18T14:46:37.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:37 smithi174 ceph-mon[36787]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 8.8 KiB/s rd, 324 KiB/s wr, 24 op/s 2024-03-18T14:46:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:37 smithi077 ceph-mon[35865]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 8.8 KiB/s rd, 324 KiB/s wr, 24 op/s 2024-03-18T14:46:38.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:38 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T14:46:38.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:38 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:38.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:38 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T14:46:38.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:38 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:38.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:38 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T14:46:38.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:38 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:39.640 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:39 smithi076 ceph-mon[32205]: Deploying daemon osd.3 on smithi076 2024-03-18T14:46:39.641 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:39 smithi076 ceph-mon[32205]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 6.8 KiB/s rd, 252 KiB/s wr, 19 op/s 2024-03-18T14:46:39.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:39 smithi174 ceph-mon[36787]: Deploying daemon osd.3 on smithi076 2024-03-18T14:46:39.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:39 smithi174 ceph-mon[36787]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 6.8 KiB/s rd, 252 KiB/s wr, 19 op/s 2024-03-18T14:46:39.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:39 smithi077 ceph-mon[35865]: Deploying daemon osd.3 on smithi076 2024-03-18T14:46:39.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:39 smithi077 ceph-mon[35865]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 6.8 KiB/s rd, 252 KiB/s wr, 19 op/s 2024-03-18T14:46:41.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:41 smithi076 ceph-mon[32205]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 6.2 KiB/s rd, 228 KiB/s wr, 17 op/s 2024-03-18T14:46:41.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:41 smithi174 ceph-mon[36787]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 6.2 KiB/s rd, 228 KiB/s wr, 17 op/s 2024-03-18T14:46:41.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:41 smithi077 ceph-mon[35865]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 6.2 KiB/s rd, 228 KiB/s wr, 17 op/s 2024-03-18T14:46:43.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:43 smithi076 ceph-mon[32205]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 201 KiB/s wr, 15 op/s 2024-03-18T14:46:43.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:43 smithi174 ceph-mon[36787]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 201 KiB/s wr, 15 op/s 2024-03-18T14:46:43.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:43 smithi077 ceph-mon[35865]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 201 KiB/s wr, 15 op/s 2024-03-18T14:46:44.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:44 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:44.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:44 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:44.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:44 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:46:44.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:44 smithi076 ceph-mon[32205]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.2 KiB/s rd, 190 KiB/s wr, 14 op/s 2024-03-18T14:46:44.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:44 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:44.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:44 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:44.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:44 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:46:44.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:44 smithi174 ceph-mon[36787]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.2 KiB/s rd, 190 KiB/s wr, 14 op/s 2024-03-18T14:46:44.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:44 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:44.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:44 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:44.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:44 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:46:44.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:44 smithi077 ceph-mon[35865]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.2 KiB/s rd, 190 KiB/s wr, 14 op/s 2024-03-18T14:46:47.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:47.412+0000 7fb1ebfff700 1 -- 172.21.15.76:0/3517034394 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb2040fe9a0 con 0x7fb1ec0a2800 2024-03-18T14:46:47.414 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 3 on host 'smithi076' 2024-03-18T14:46:47.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:47.414+0000 7fb2097b1700 1 -- 172.21.15.76:0/3517034394 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb1ec0a2800 msgr2=0x7fb1ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:46:47.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:47.414+0000 7fb2097b1700 1 --2- 172.21.15.76:0/3517034394 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb1ec0a2800 0x7fb1ec0a4cc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb1f80039c0 tx=0x7fb1f8005fd0 comp rx=0 tx=0).stop 2024-03-18T14:46:47.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:47.415+0000 7fb2097b1700 1 -- 172.21.15.76:0/3517034394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb204101200 msgr2=0x7fb20419a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:46:47.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:47.415+0000 7fb2097b1700 1 --2- 172.21.15.76:0/3517034394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb204101200 0x7fb20419a760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb1f400c9a0 tx=0x7fb1f400cd60 comp rx=0 tx=0).stop 2024-03-18T14:46:47.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:47.415+0000 7fb2097b1700 1 -- 172.21.15.76:0/3517034394 shutdown_connections 2024-03-18T14:46:47.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:47.415+0000 7fb2097b1700 1 --2- 172.21.15.76:0/3517034394 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb1ec0a2800 0x7fb1ec0a4cc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:47.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:47.415+0000 7fb2097b1700 1 --2- 172.21.15.76:0/3517034394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb204102da0 0x7fb2041a1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:47.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:47.415+0000 7fb2097b1700 1 --2- 172.21.15.76:0/3517034394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb204101200 0x7fb20419a760 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:47.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:47.415+0000 7fb2097b1700 1 --2- 172.21.15.76:0/3517034394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb204102400 0x7fb20419aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:47.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:47.415+0000 7fb2097b1700 1 -- 172.21.15.76:0/3517034394 >> 172.21.15.76:0/3517034394 conn(0x7fb2040fc7a0 msgr2=0x7fb2040fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:46:47.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:47.415+0000 7fb2097b1700 1 -- 172.21.15.76:0/3517034394 shutdown_connections 2024-03-18T14:46:47.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:46:47.415+0000 7fb2097b1700 1 -- 172.21.15.76:0/3517034394 wait complete. 2024-03-18T14:46:47.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:47 smithi076 ceph-mon[32205]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:47.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:47 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:47.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:47 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:47.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:47 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:47.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:47 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:46:47.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:47 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:47.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:47 smithi174 ceph-mon[36787]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:47.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:47 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:47.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:47 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:47.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:47 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:47.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:47 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:46:47.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:47 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:47.681 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:47 smithi077 ceph-mon[35865]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:47.681 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:47 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:47.681 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:47 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:47.681 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:47 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:47.681 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:47 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:46:47.681 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:47 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:48.043 DEBUG:teuthology.orchestra.run.smithi076:osd.3> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.3.service 2024-03-18T14:46:48.045 INFO:tasks.cephadm:Deploying osd.4 on smithi077 with /dev/vg_nvme/lv_3... 2024-03-18T14:46:48.045 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T14:46:48.284 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T14:46:48.626 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:48 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:46:48.627 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:48 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:48.627 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:48 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:48.627 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:48 smithi077 ceph-mon[35865]: from='osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T14:46:48.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:48 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:46:48.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:48 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:48.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:48 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:48.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:48 smithi076 ceph-mon[32205]: from='osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T14:46:48.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:48 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:46:48.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:48 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:48.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:48 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:48.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:48 smithi174 ceph-mon[36787]: from='osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T14:46:49.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:49 smithi076 ceph-mon[32205]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:49.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:49 smithi076 ceph-mon[32205]: from='osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T14:46:49.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:49 smithi076 ceph-mon[32205]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T14:46:49.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:49 smithi076 ceph-mon[32205]: from='osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-18T14:46:49.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:46:49.633 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:49 smithi077 ceph-mon[35865]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:49.634 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:49 smithi077 ceph-mon[35865]: from='osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T14:46:49.634 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:49 smithi077 ceph-mon[35865]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T14:46:49.634 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:49 smithi077 ceph-mon[35865]: from='osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-18T14:46:49.634 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:46:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:49 smithi174 ceph-mon[36787]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:49 smithi174 ceph-mon[36787]: from='osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T14:46:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:49 smithi174 ceph-mon[36787]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T14:46:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:49 smithi174 ceph-mon[36787]: from='osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-18T14:46:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:46:50.161 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 14:46:49 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3[53298]: 2024-03-18T14:46:49.782+0000 7fef579b0700 -1 osd.3 0 waiting for initial osdmap 2024-03-18T14:46:50.162 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 14:46:49 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3[53298]: 2024-03-18T14:46:49.787+0000 7fef4f79b700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:46:50.186 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T14:46:50.186 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T14:46:50.186 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-03-18T14:46:50.186 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-03-18T14:46:50.186 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127423 s, 823 MB/s 2024-03-18T14:46:50.186 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-03-18T14:46:50.520 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:50 smithi077 ceph-mon[35865]: from='osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-18T14:46:50.520 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:50 smithi077 ceph-mon[35865]: osdmap e25: 4 total, 3 up, 4 in 2024-03-18T14:46:50.520 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:50 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:46:50.520 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:50 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:46:50.521 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:50 smithi077 ceph-mon[35865]: from='osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439]' entity='osd.3' 2024-03-18T14:46:50.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:50 smithi076 ceph-mon[32205]: from='osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-18T14:46:50.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:50 smithi076 ceph-mon[32205]: osdmap e25: 4 total, 3 up, 4 in 2024-03-18T14:46:50.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:50 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:46:50.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:50 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:46:50.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:50 smithi076 ceph-mon[32205]: from='osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439]' entity='osd.3' 2024-03-18T14:46:50.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:50 smithi174 ceph-mon[36787]: from='osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-18T14:46:50.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:50 smithi174 ceph-mon[36787]: osdmap e25: 4 total, 3 up, 4 in 2024-03-18T14:46:50.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:50 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:46:50.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:50 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:46:50.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:50 smithi174 ceph-mon[36787]: from='osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439]' entity='osd.3' 2024-03-18T14:46:50.854 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi077:vg_nvme/lv_3 2024-03-18T14:46:51.087 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T14:46:51.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:51 smithi077 ceph-mon[35865]: purged_snaps scrub starts 2024-03-18T14:46:51.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:51 smithi077 ceph-mon[35865]: purged_snaps scrub ok 2024-03-18T14:46:51.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:51 smithi077 ceph-mon[35865]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:51.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:51 smithi077 ceph-mon[35865]: osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439] boot 2024-03-18T14:46:51.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:51 smithi077 ceph-mon[35865]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T14:46:51.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:51 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:46:51.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:51 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:51.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:51 smithi076 ceph-mon[32205]: purged_snaps scrub starts 2024-03-18T14:46:51.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:51 smithi076 ceph-mon[32205]: purged_snaps scrub ok 2024-03-18T14:46:51.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:51 smithi076 ceph-mon[32205]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:51.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:51 smithi076 ceph-mon[32205]: osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439] boot 2024-03-18T14:46:51.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:51 smithi076 ceph-mon[32205]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T14:46:51.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:51 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:46:51.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:51 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:51.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:51 smithi174 ceph-mon[36787]: purged_snaps scrub starts 2024-03-18T14:46:51.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:51 smithi174 ceph-mon[36787]: purged_snaps scrub ok 2024-03-18T14:46:51.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:51 smithi174 ceph-mon[36787]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T14:46:51.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:51 smithi174 ceph-mon[36787]: osd.3 [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439] boot 2024-03-18T14:46:51.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:51 smithi174 ceph-mon[36787]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T14:46:51.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:51 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:46:51.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:51 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:52.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:52 smithi076 ceph-mon[32205]: Detected new or changed devices on smithi076 2024-03-18T14:46:52.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:52 smithi076 ceph-mon[32205]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T14:46:52.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:52 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:52.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:52 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:46:52.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:52 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:46:52.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:52 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi076 to 8544M 2024-03-18T14:46:52.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:52 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:52.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:52 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:52.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:52 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:46:52.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:52 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:52.663 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:52 smithi077 ceph-mon[35865]: Detected new or changed devices on smithi076 2024-03-18T14:46:52.663 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:52 smithi077 ceph-mon[35865]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T14:46:52.663 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:52 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:52.663 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:52 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:46:52.663 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:52 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:46:52.664 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:52 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi076 to 8544M 2024-03-18T14:46:52.664 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:52 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:52.664 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:52 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:52.664 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:52 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:46:52.664 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:52 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:52.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.673+0000 7ff10642d700 1 -- 172.21.15.77:0/4077345809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff100102da0 msgr2=0x7ff100109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:46:52.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.673+0000 7ff10642d700 1 --2- 172.21.15.77:0/4077345809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff100102da0 0x7ff100109650 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800b0d0 tx=0x7ff0f800b3e0 comp rx=0 tx=0).stop 2024-03-18T14:46:52.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.673+0000 7ff10642d700 1 -- 172.21.15.77:0/4077345809 shutdown_connections 2024-03-18T14:46:52.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.673+0000 7ff10642d700 1 --2- 172.21.15.77:0/4077345809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff100102da0 0x7ff100109650 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:52.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.673+0000 7ff10642d700 1 --2- 172.21.15.77:0/4077345809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff100101200 0x7ff100101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:52.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.673+0000 7ff10642d700 1 --2- 172.21.15.77:0/4077345809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff100102400 0x7ff100102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:52.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.673+0000 7ff10642d700 1 -- 172.21.15.77:0/4077345809 >> 172.21.15.77:0/4077345809 conn(0x7ff1000fc7a0 msgr2=0x7ff1000febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:46:52.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.673+0000 7ff10642d700 1 -- 172.21.15.77:0/4077345809 shutdown_connections 2024-03-18T14:46:52.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.673+0000 7ff10642d700 1 -- 172.21.15.77:0/4077345809 wait complete. 2024-03-18T14:46:52.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff10642d700 1 Processor -- start 2024-03-18T14:46:52.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff10642d700 1 -- start start 2024-03-18T14:46:52.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff10642d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff100101200 0x7ff100197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:46:52.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff10642d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff100102400 0x7ff100197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:46:52.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff10642d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff100102da0 0x7ff10019ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:46:52.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff10642d700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff10010ce00 con 0x7ff100102400 2024-03-18T14:46:52.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff10642d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff10010cc80 con 0x7ff100101200 2024-03-18T14:46:52.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff10642d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff10010cf80 con 0x7ff100102da0 2024-03-18T14:46:52.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff0fffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff100101200 0x7ff100197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:46:52.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff0fffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff100101200 0x7ff100197440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45854/0 (socket says 172.21.15.77:45854) 2024-03-18T14:46:52.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff0fffff700 1 -- 172.21.15.77:0/2954794858 learned_addr learned my addr 172.21.15.77:0/2954794858 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-18T14:46:52.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff0ff7fe700 1 --2- 172.21.15.77:0/2954794858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff100102400 0x7ff100197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:46:52.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff1049ca700 1 --2- 172.21.15.77:0/2954794858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff100102da0 0x7ff10019ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:46:52.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff0fffff700 1 -- 172.21.15.77:0/2954794858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff100102da0 msgr2=0x7ff10019ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:46:52.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff0fffff700 1 --2- 172.21.15.77:0/2954794858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff100102da0 0x7ff10019ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:52.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff0fffff700 1 -- 172.21.15.77:0/2954794858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff100102400 msgr2=0x7ff100197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:46:52.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff0fffff700 1 --2- 172.21.15.77:0/2954794858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff100102400 0x7ff100197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:46:52.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff0fffff700 1 -- 172.21.15.77:0/2954794858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10019f0a0 con 0x7ff100101200 2024-03-18T14:46:52.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.674+0000 7ff1049ca700 1 --2- 172.21.15.77:0/2954794858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff100102da0 0x7ff10019ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:46:52.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.675+0000 7ff0fffff700 1 --2- 172.21.15.77:0/2954794858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff100101200 0x7ff100197440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0007b50 tx=0x7ff0f0007f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:46:52.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.675+0000 7ff0fd7fa700 1 -- 172.21.15.77:0/2954794858 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0019020 con 0x7ff100101200 2024-03-18T14:46:52.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.675+0000 7ff0fd7fa700 1 -- 172.21.15.77:0/2954794858 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0f0004500 con 0x7ff100101200 2024-03-18T14:46:52.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.675+0000 7ff10642d700 1 -- 172.21.15.77:0/2954794858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff100069fe0 con 0x7ff100101200 2024-03-18T14:46:52.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.675+0000 7ff0fd7fa700 1 -- 172.21.15.77:0/2954794858 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f000e760 con 0x7ff100101200 2024-03-18T14:46:52.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.675+0000 7ff10642d700 1 -- 172.21.15.77:0/2954794858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff10006a4d0 con 0x7ff100101200 2024-03-18T14:46:52.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.676+0000 7ff10642d700 1 -- 172.21.15.77:0/2954794858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff100068f90 con 0x7ff100101200 2024-03-18T14:46:52.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.677+0000 7ff0fd7fa700 1 -- 172.21.15.77:0/2954794858 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff0f0003fc0 con 0x7ff100101200 2024-03-18T14:46:52.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.678+0000 7ff0fd7fa700 1 --2- 172.21.15.77:0/2954794858 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff0e80a2890 0x7ff0e80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:46:52.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.678+0000 7ff0fd7fa700 1 -- 172.21.15.77:0/2954794858 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3698+0+0 (secure 0 0 0) 0x7ff0f00cedd0 con 0x7ff100101200 2024-03-18T14:46:52.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.678+0000 7ff0ff7fe700 1 --2- 172.21.15.77:0/2954794858 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff0e80a2890 0x7ff0e80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:46:52.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.678+0000 7ff0ff7fe700 1 --2- 172.21.15.77:0/2954794858 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff0e80a2890 0x7ff0e80a4d50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4005fd0 tx=0x7ff0f4005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:46:52.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.682+0000 7ff0fd7fa700 1 -- 172.21.15.77:0/2954794858 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff0f0023020 con 0x7ff100101200 2024-03-18T14:46:52.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:52 smithi174 ceph-mon[36787]: Detected new or changed devices on smithi076 2024-03-18T14:46:52.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:52 smithi174 ceph-mon[36787]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T14:46:52.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:52 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:52.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:52 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:46:52.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:52 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:46:52.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:52 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi076 to 8544M 2024-03-18T14:46:52.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:52 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:52.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:52 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:52.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:52 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:46:52.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:52 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:46:52.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:46:52.886+0000 7ff10642d700 1 -- 172.21.15.77:0/2954794858 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ff10006a730 con 0x7ff0e80a2890 2024-03-18T14:46:53.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:53 smithi076 ceph-mon[32205]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:46:53.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:53 smithi076 ceph-mon[32205]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T14:46:53.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:53 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:46:53.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:53 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:46:53.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:53 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:53.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:53 smithi076 ceph-mon[32205]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T14:46:53.691 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:53 smithi077 ceph-mon[35865]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:46:53.691 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:53 smithi077 ceph-mon[35865]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T14:46:53.691 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:53 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:46:53.691 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:53 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:46:53.691 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:53 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:53.691 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:53 smithi077 ceph-mon[35865]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T14:46:53.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:53 smithi174 ceph-mon[36787]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:46:53.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:53 smithi174 ceph-mon[36787]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T14:46:53.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:53 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:46:53.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:53 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:46:53.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:53 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:46:53.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:53 smithi174 ceph-mon[36787]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T14:46:54.553 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:54 smithi077 ceph-mon[35865]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:46:54.553 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:54 smithi077 ceph-mon[35865]: osdmap e30: 4 total, 4 up, 4 in 2024-03-18T14:46:54.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:54 smithi076 ceph-mon[32205]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:46:54.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:54 smithi076 ceph-mon[32205]: osdmap e30: 4 total, 4 up, 4 in 2024-03-18T14:46:54.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:54 smithi174 ceph-mon[36787]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:46:54.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:54 smithi174 ceph-mon[36787]: osdmap e30: 4 total, 4 up, 4 in 2024-03-18T14:46:55.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:55 smithi076 ceph-mon[32205]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-03-18T14:46:55.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:55 smithi174 ceph-mon[36787]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-03-18T14:46:55.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:55 smithi077 ceph-mon[35865]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-03-18T14:46:56.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:56 smithi076 ceph-mon[32205]: from='client.? 172.21.15.77:0/3495013928' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b88f6022-4690-4c54-9072-8a1f0febddd3"}]: dispatch 2024-03-18T14:46:56.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:56 smithi076 ceph-mon[32205]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b88f6022-4690-4c54-9072-8a1f0febddd3"}]: dispatch 2024-03-18T14:46:56.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:56 smithi076 ceph-mon[32205]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b88f6022-4690-4c54-9072-8a1f0febddd3"}]': finished 2024-03-18T14:46:56.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:56 smithi076 ceph-mon[32205]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T14:46:56.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:56 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:46:56.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:56 smithi076 ceph-mon[32205]: from='client.? 172.21.15.77:0/43082049' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:46:56.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:56 smithi174 ceph-mon[36787]: from='client.? 172.21.15.77:0/3495013928' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b88f6022-4690-4c54-9072-8a1f0febddd3"}]: dispatch 2024-03-18T14:46:56.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:56 smithi174 ceph-mon[36787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b88f6022-4690-4c54-9072-8a1f0febddd3"}]: dispatch 2024-03-18T14:46:56.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:56 smithi174 ceph-mon[36787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b88f6022-4690-4c54-9072-8a1f0febddd3"}]': finished 2024-03-18T14:46:56.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:56 smithi174 ceph-mon[36787]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T14:46:56.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:56 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:46:56.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:56 smithi174 ceph-mon[36787]: from='client.? 172.21.15.77:0/43082049' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:46:56.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:56 smithi077 ceph-mon[35865]: from='client.? 172.21.15.77:0/3495013928' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b88f6022-4690-4c54-9072-8a1f0febddd3"}]: dispatch 2024-03-18T14:46:56.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:56 smithi077 ceph-mon[35865]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b88f6022-4690-4c54-9072-8a1f0febddd3"}]: dispatch 2024-03-18T14:46:56.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:56 smithi077 ceph-mon[35865]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b88f6022-4690-4c54-9072-8a1f0febddd3"}]': finished 2024-03-18T14:46:56.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:56 smithi077 ceph-mon[35865]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T14:46:56.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:56 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:46:56.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:56 smithi077 ceph-mon[35865]: from='client.? 172.21.15.77:0/43082049' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:46:57.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:57 smithi174 ceph-mon[36787]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T14:46:57.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:57 smithi077 ceph-mon[35865]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T14:46:57.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:57 smithi076 ceph-mon[32205]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T14:46:59.669 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:46:59 smithi077 ceph-mon[35865]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-18T14:46:59.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:46:59 smithi174 ceph-mon[36787]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-18T14:46:59.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:46:59 smithi076 ceph-mon[32205]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-18T14:47:01.579 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:01 smithi077 ceph-mon[35865]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T14:47:01.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:01 smithi174 ceph-mon[36787]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T14:47:01.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:01 smithi076 ceph-mon[32205]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T14:47:02.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:02 smithi174 ceph-mon[36787]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:02.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:02 smithi077 ceph-mon[35865]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:02.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:02 smithi076 ceph-mon[32205]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:03.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:03 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T14:47:03.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:03 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:03.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:03 smithi174 ceph-mon[36787]: Deploying daemon osd.4 on smithi077 2024-03-18T14:47:03.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:03 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T14:47:03.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:03 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:03.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:03 smithi077 ceph-mon[35865]: Deploying daemon osd.4 on smithi077 2024-03-18T14:47:03.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:03 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T14:47:03.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:03 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:03.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:03 smithi076 ceph-mon[32205]: Deploying daemon osd.4 on smithi077 2024-03-18T14:47:04.692 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:04 smithi077 ceph-mon[35865]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:04.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:04 smithi174 ceph-mon[36787]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:04.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:04 smithi076 ceph-mon[32205]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:07.585 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:07 smithi077 ceph-mon[35865]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:07.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:07 smithi076 ceph-mon[32205]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:07.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:07 smithi174 ceph-mon[36787]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:08.523 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:08 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:47:08.523 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:08 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:08.523 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:08 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:08.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:08 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:47:08.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:08 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:08.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:08 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:08.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:08 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:47:08.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:08 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:08.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:08 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:09.615 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:09 smithi077 ceph-mon[35865]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:09.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:09 smithi076 ceph-mon[32205]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:09.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:09 smithi174 ceph-mon[36787]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:11.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:47:11.398+0000 7ff0fd7fa700 1 -- 172.21.15.77:0/2954794858 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff10006a730 con 0x7ff0e80a2890 2024-03-18T14:47:11.402 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 4 on host 'smithi077' 2024-03-18T14:47:11.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:47:11.401+0000 7ff10642d700 1 -- 172.21.15.77:0/2954794858 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff0e80a2890 msgr2=0x7ff0e80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:47:11.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:47:11.401+0000 7ff10642d700 1 --2- 172.21.15.77:0/2954794858 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff0e80a2890 0x7ff0e80a4d50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4005fd0 tx=0x7ff0f4005f60 comp rx=0 tx=0).stop 2024-03-18T14:47:11.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:47:11.401+0000 7ff10642d700 1 -- 172.21.15.77:0/2954794858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff100101200 msgr2=0x7ff100197440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:47:11.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:47:11.401+0000 7ff10642d700 1 --2- 172.21.15.77:0/2954794858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff100101200 0x7ff100197440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0007b50 tx=0x7ff0f0007f10 comp rx=0 tx=0).stop 2024-03-18T14:47:11.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:47:11.401+0000 7ff10642d700 1 -- 172.21.15.77:0/2954794858 shutdown_connections 2024-03-18T14:47:11.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:47:11.401+0000 7ff10642d700 1 --2- 172.21.15.77:0/2954794858 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff0e80a2890 0x7ff0e80a4d50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:11.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:47:11.401+0000 7ff10642d700 1 --2- 172.21.15.77:0/2954794858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff100102da0 0x7ff10019ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:11.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:47:11.401+0000 7ff10642d700 1 --2- 172.21.15.77:0/2954794858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff100101200 0x7ff100197440 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:11.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:47:11.401+0000 7ff10642d700 1 --2- 172.21.15.77:0/2954794858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff100102400 0x7ff100197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:11.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:47:11.401+0000 7ff10642d700 1 -- 172.21.15.77:0/2954794858 >> 172.21.15.77:0/2954794858 conn(0x7ff1000fc7a0 msgr2=0x7ff100105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:47:11.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:47:11.401+0000 7ff10642d700 1 -- 172.21.15.77:0/2954794858 shutdown_connections 2024-03-18T14:47:11.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:47:11.401+0000 7ff10642d700 1 -- 172.21.15.77:0/2954794858 wait complete. 2024-03-18T14:47:11.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:11 smithi076 ceph-mon[32205]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:11.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:11 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:11.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:11 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:11.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:11 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:11.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:11 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:47:11.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:11 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:11.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:11 smithi174 ceph-mon[36787]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:11.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:11 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:11.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:11 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:11.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:11 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:11.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:11 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:47:11.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:11 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:11.722 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:11 smithi077 ceph-mon[35865]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:11.722 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:11 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:11.722 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:11 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:11.722 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:11 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:11.722 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:11 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:47:11.722 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:11 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:12.102 DEBUG:teuthology.orchestra.run.smithi077:osd.4> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.4.service 2024-03-18T14:47:12.104 INFO:tasks.cephadm:Deploying osd.5 on smithi174 with /dev/vg_nvme/lv_3... 2024-03-18T14:47:12.105 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T14:47:12.349 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T14:47:12.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:12 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:12.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:12 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:12.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:12 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:47:12.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:12 smithi174 ceph-mon[36787]: from='osd.4 [v2:172.21.15.77:6808/1533069359,v1:172.21.15.77:6809/1533069359]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T14:47:12.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:12 smithi174 ceph-mon[36787]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T14:47:12.654 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:12 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:12.654 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:12 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:12.654 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:12 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:47:12.655 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:12 smithi077 ceph-mon[35865]: from='osd.4 [v2:172.21.15.77:6808/1533069359,v1:172.21.15.77:6809/1533069359]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T14:47:12.655 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:12 smithi077 ceph-mon[35865]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T14:47:12.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:12 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:12.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:12 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:12.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:12 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:47:12.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:12 smithi076 ceph-mon[32205]: from='osd.4 [v2:172.21.15.77:6808/1533069359,v1:172.21.15.77:6809/1533069359]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T14:47:12.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:12 smithi076 ceph-mon[32205]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T14:47:13.624 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:13 smithi077 ceph-mon[35865]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:13.624 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:13 smithi077 ceph-mon[35865]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T14:47:13.624 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:13 smithi077 ceph-mon[35865]: from='osd.4 [v2:172.21.15.77:6808/1533069359,v1:172.21.15.77:6809/1533069359]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:47:13.624 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:13 smithi077 ceph-mon[35865]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T14:47:13.624 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:13 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:47:13.625 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:13 smithi077 ceph-mon[35865]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:47:13.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:13 smithi076 ceph-mon[32205]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:13.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:13 smithi076 ceph-mon[32205]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T14:47:13.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:13 smithi076 ceph-mon[32205]: from='osd.4 [v2:172.21.15.77:6808/1533069359,v1:172.21.15.77:6809/1533069359]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:47:13.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:13 smithi076 ceph-mon[32205]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T14:47:13.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:13 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:47:13.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:13 smithi076 ceph-mon[32205]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:47:13.673 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:13 smithi174 ceph-mon[36787]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:13.673 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:13 smithi174 ceph-mon[36787]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T14:47:13.674 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:13 smithi174 ceph-mon[36787]: from='osd.4 [v2:172.21.15.77:6808/1533069359,v1:172.21.15.77:6809/1533069359]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:47:13.674 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:13 smithi174 ceph-mon[36787]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T14:47:13.674 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:13 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:47:13.674 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:13 smithi174 ceph-mon[36787]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:47:13.942 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 14:47:13 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-4[47126]: 2024-03-18T14:47:13.814+0000 7f4f14f54700 -1 osd.4 0 waiting for initial osdmap 2024-03-18T14:47:13.942 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 14:47:13 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-4[47126]: 2024-03-18T14:47:13.820+0000 7f4f0cd3f700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:47:14.376 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T14:47:14.376 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T14:47:14.376 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-03-18T14:47:14.376 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-03-18T14:47:14.376 INFO:teuthology.orchestra.run.smithi174.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011542 s, 908 MB/s 2024-03-18T14:47:14.376 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-03-18T14:47:14.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:14 smithi076 ceph-mon[32205]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-18T14:47:14.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:14 smithi076 ceph-mon[32205]: osdmap e33: 5 total, 4 up, 5 in 2024-03-18T14:47:14.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:14 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:47:14.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:14 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:47:14.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:14 smithi076 ceph-mon[32205]: from='osd.4 ' entity='osd.4' 2024-03-18T14:47:14.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:14 smithi174 ceph-mon[36787]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-18T14:47:14.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:14 smithi174 ceph-mon[36787]: osdmap e33: 5 total, 4 up, 5 in 2024-03-18T14:47:14.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:14 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:47:14.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:14 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:47:14.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:14 smithi174 ceph-mon[36787]: from='osd.4 ' entity='osd.4' 2024-03-18T14:47:14.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:14 smithi077 ceph-mon[35865]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-18T14:47:14.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:14 smithi077 ceph-mon[35865]: osdmap e33: 5 total, 4 up, 5 in 2024-03-18T14:47:14.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:14 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:47:14.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:14 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:47:14.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:14 smithi077 ceph-mon[35865]: from='osd.4 ' entity='osd.4' 2024-03-18T14:47:15.115 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi174:vg_nvme/lv_3 2024-03-18T14:47:15.350 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T14:47:15.467 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:15 smithi076 ceph-mon[32205]: purged_snaps scrub starts 2024-03-18T14:47:15.467 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:15 smithi076 ceph-mon[32205]: purged_snaps scrub ok 2024-03-18T14:47:15.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:15 smithi076 ceph-mon[32205]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:15.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:47:15.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:15 smithi076 ceph-mon[32205]: osd.4 [v2:172.21.15.77:6808/1533069359,v1:172.21.15.77:6809/1533069359] boot 2024-03-18T14:47:15.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:15 smithi076 ceph-mon[32205]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T14:47:15.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:47:15.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:15.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:15.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:47:15.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:47:15.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:15.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:15.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:47:15.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:15.650 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:15 smithi174 ceph-mon[36787]: purged_snaps scrub starts 2024-03-18T14:47:15.650 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:15 smithi174 ceph-mon[36787]: purged_snaps scrub ok 2024-03-18T14:47:15.650 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:15 smithi174 ceph-mon[36787]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:15.650 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:47:15.651 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:15 smithi174 ceph-mon[36787]: osd.4 [v2:172.21.15.77:6808/1533069359,v1:172.21.15.77:6809/1533069359] boot 2024-03-18T14:47:15.651 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:15 smithi174 ceph-mon[36787]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T14:47:15.651 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:47:15.651 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:15.651 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:15.651 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:47:15.651 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:47:15.651 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:15.651 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:15.651 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:47:15.651 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:15.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:15 smithi077 ceph-mon[35865]: purged_snaps scrub starts 2024-03-18T14:47:15.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:15 smithi077 ceph-mon[35865]: purged_snaps scrub ok 2024-03-18T14:47:15.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:15 smithi077 ceph-mon[35865]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-18T14:47:15.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:47:15.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:15 smithi077 ceph-mon[35865]: osd.4 [v2:172.21.15.77:6808/1533069359,v1:172.21.15.77:6809/1533069359] boot 2024-03-18T14:47:15.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:15 smithi077 ceph-mon[35865]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T14:47:15.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:47:15.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:15.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:15.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:47:15.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:47:15.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:15.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:15.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:47:15.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:16.527 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:16 smithi174 ceph-mon[36787]: Detected new or changed devices on smithi077 2024-03-18T14:47:16.527 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:16 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi077 to 8540M 2024-03-18T14:47:16.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:16 smithi076 ceph-mon[32205]: Detected new or changed devices on smithi077 2024-03-18T14:47:16.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:16 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi077 to 8540M 2024-03-18T14:47:16.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:16 smithi077 ceph-mon[35865]: Detected new or changed devices on smithi077 2024-03-18T14:47:16.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:16 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi077 to 8540M 2024-03-18T14:47:17.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.041+0000 7f6772566700 1 -- 172.21.15.174:0/1793527507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f676c103ec0 msgr2=0x7f676c10a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:47:17.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.041+0000 7f6772566700 1 --2- 172.21.15.174:0/1793527507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f676c103ec0 0x7f676c10a770 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f676400b0a0 tx=0x7f676400b3b0 comp rx=0 tx=0).stop 2024-03-18T14:47:17.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.041+0000 7f6772566700 1 -- 172.21.15.174:0/1793527507 shutdown_connections 2024-03-18T14:47:17.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.041+0000 7f6772566700 1 --2- 172.21.15.174:0/1793527507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f676c103ec0 0x7f676c10a770 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:17.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.041+0000 7f6772566700 1 --2- 172.21.15.174:0/1793527507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f676c102320 0x7f676c102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:17.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.041+0000 7f6772566700 1 --2- 172.21.15.174:0/1793527507 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f676c103520 0x7f676c103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:17.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.041+0000 7f6772566700 1 -- 172.21.15.174:0/1793527507 >> 172.21.15.174:0/1793527507 conn(0x7f676c0fd880 msgr2=0x7f676c0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:47:17.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.041+0000 7f6772566700 1 -- 172.21.15.174:0/1793527507 shutdown_connections 2024-03-18T14:47:17.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.041+0000 7f6772566700 1 -- 172.21.15.174:0/1793527507 wait complete. 2024-03-18T14:47:17.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.042+0000 7f6772566700 1 Processor -- start 2024-03-18T14:47:17.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.042+0000 7f6772566700 1 -- start start 2024-03-18T14:47:17.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.042+0000 7f6772566700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f676c102320 0x7f676c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:47:17.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.042+0000 7f6772566700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f676c103520 0x7f676c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:47:17.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.042+0000 7f6772566700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f676c103ec0 0x7f676c19fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:47:17.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.042+0000 7f6772566700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f676c10df90 con 0x7f676c103ec0 2024-03-18T14:47:17.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.042+0000 7f6772566700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f676c10de10 con 0x7f676c103520 2024-03-18T14:47:17.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.042+0000 7f6772566700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f676c10e110 con 0x7f676c102320 2024-03-18T14:47:17.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.042+0000 7f676bfff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f676c102320 0x7f676c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:47:17.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.042+0000 7f676bfff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f676c102320 0x7f676c198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:51878/0 (socket says 172.21.15.174:51878) 2024-03-18T14:47:17.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.042+0000 7f676bfff700 1 -- 172.21.15.174:0/2606863051 learned_addr learned my addr 172.21.15.174:0/2606863051 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-18T14:47:17.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.042+0000 7f6770b03700 1 --2- 172.21.15.174:0/2606863051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f676c103ec0 0x7f676c19fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:47:17.049 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.043+0000 7f676bfff700 1 -- 172.21.15.174:0/2606863051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f676c103520 msgr2=0x7f676c198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:47:17.049 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.043+0000 7f676bfff700 1 --2- 172.21.15.174:0/2606863051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f676c103520 0x7f676c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:17.049 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.043+0000 7f676bfff700 1 -- 172.21.15.174:0/2606863051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f676c103ec0 msgr2=0x7f676c19fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:47:17.049 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.043+0000 7f676bfff700 1 --2- 172.21.15.174:0/2606863051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f676c103ec0 0x7f676c19fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:17.049 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.043+0000 7f676bfff700 1 -- 172.21.15.174:0/2606863051 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f676c1a0170 con 0x7f676c102320 2024-03-18T14:47:17.049 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.043+0000 7f6770b03700 1 --2- 172.21.15.174:0/2606863051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f676c103ec0 0x7f676c19fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:47:17.049 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.043+0000 7f676bfff700 1 --2- 172.21.15.174:0/2606863051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f676c102320 0x7f676c198510 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f675c00b530 tx=0x7f675c00b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:47:17.049 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.043+0000 7f67697fa700 1 -- 172.21.15.174:0/2606863051 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675c015020 con 0x7f676c102320 2024-03-18T14:47:17.049 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.043+0000 7f67697fa700 1 -- 172.21.15.174:0/2606863051 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f675c011d10 con 0x7f676c102320 2024-03-18T14:47:17.049 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.043+0000 7f6772566700 1 -- 172.21.15.174:0/2606863051 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f676c1a0460 con 0x7f676c102320 2024-03-18T14:47:17.049 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.043+0000 7f67697fa700 1 -- 172.21.15.174:0/2606863051 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675c00f2e0 con 0x7f676c102320 2024-03-18T14:47:17.049 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.043+0000 7f6772566700 1 -- 172.21.15.174:0/2606863051 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f676c1a09a0 con 0x7f676c102320 2024-03-18T14:47:17.049 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.044+0000 7f6772566700 1 -- 172.21.15.174:0/2606863051 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f676c068f90 con 0x7f676c102320 2024-03-18T14:47:17.049 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.045+0000 7f67697fa700 1 -- 172.21.15.174:0/2606863051 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f675c018020 con 0x7f676c102320 2024-03-18T14:47:17.049 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.045+0000 7f67697fa700 1 --2- 172.21.15.174:0/2606863051 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f67540a2910 0x7f67540a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:47:17.050 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.046+0000 7f67697fa700 1 -- 172.21.15.174:0/2606863051 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7f675c0cd190 con 0x7f676c102320 2024-03-18T14:47:17.050 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.046+0000 7f676b7fe700 1 --2- 172.21.15.174:0/2606863051 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f67540a2910 0x7f67540a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:47:17.050 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.046+0000 7f676b7fe700 1 --2- 172.21.15.174:0/2606863051 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f67540a2910 0x7f67540a4dd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6760005d90 tx=0x7f6760005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:47:17.051 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.050+0000 7f67697fa700 1 -- 172.21.15.174:0/2606863051 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f675c01f020 con 0x7f676c102320 2024-03-18T14:47:17.256 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:17.255+0000 7f6772566700 1 -- 172.21.15.174:0/2606863051 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f676c062710 con 0x7f67540a2910 2024-03-18T14:47:17.541 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:17 smithi174 ceph-mon[36787]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:17.542 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:17 smithi174 ceph-mon[36787]: osdmap e35: 5 total, 5 up, 5 in 2024-03-18T14:47:17.542 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:17 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:47:17.542 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:17 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:47:17.542 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:17 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:17.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:17 smithi076 ceph-mon[32205]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:17.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:17 smithi076 ceph-mon[32205]: osdmap e35: 5 total, 5 up, 5 in 2024-03-18T14:47:17.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:17 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:47:17.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:17 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:47:17.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:17 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:17.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:17 smithi077 ceph-mon[35865]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:17.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:17 smithi077 ceph-mon[35865]: osdmap e35: 5 total, 5 up, 5 in 2024-03-18T14:47:17.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:17 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:47:17.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:17 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:47:17.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:17 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:18.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:18 smithi174 ceph-mon[36787]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:47:18.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:18 smithi076 ceph-mon[32205]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:47:18.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:18 smithi077 ceph-mon[35865]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:47:19.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:19 smithi076 ceph-mon[32205]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:19.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:19 smithi174 ceph-mon[36787]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:19.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:19 smithi077 ceph-mon[35865]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:20.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:20 smithi076 ceph-mon[32205]: from='client.? 172.21.15.174:0/2878913436' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53063e2c-b521-41d8-859d-7a4bcf470fa2"}]: dispatch 2024-03-18T14:47:20.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:20 smithi076 ceph-mon[32205]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53063e2c-b521-41d8-859d-7a4bcf470fa2"}]: dispatch 2024-03-18T14:47:20.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:20 smithi076 ceph-mon[32205]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53063e2c-b521-41d8-859d-7a4bcf470fa2"}]': finished 2024-03-18T14:47:20.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:20 smithi076 ceph-mon[32205]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T14:47:20.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:20 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:47:20.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:20 smithi174 ceph-mon[36787]: from='client.? 172.21.15.174:0/2878913436' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53063e2c-b521-41d8-859d-7a4bcf470fa2"}]: dispatch 2024-03-18T14:47:20.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:20 smithi174 ceph-mon[36787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53063e2c-b521-41d8-859d-7a4bcf470fa2"}]: dispatch 2024-03-18T14:47:20.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:20 smithi174 ceph-mon[36787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53063e2c-b521-41d8-859d-7a4bcf470fa2"}]': finished 2024-03-18T14:47:20.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:20 smithi174 ceph-mon[36787]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T14:47:20.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:20 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:47:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:20 smithi077 ceph-mon[35865]: from='client.? 172.21.15.174:0/2878913436' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53063e2c-b521-41d8-859d-7a4bcf470fa2"}]: dispatch 2024-03-18T14:47:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:20 smithi077 ceph-mon[35865]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53063e2c-b521-41d8-859d-7a4bcf470fa2"}]: dispatch 2024-03-18T14:47:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:20 smithi077 ceph-mon[35865]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53063e2c-b521-41d8-859d-7a4bcf470fa2"}]': finished 2024-03-18T14:47:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:20 smithi077 ceph-mon[35865]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T14:47:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:20 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:47:21.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:21 smithi076 ceph-mon[32205]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:21.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:21 smithi076 ceph-mon[32205]: from='client.? 172.21.15.174:0/3102977584' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:47:21.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:21 smithi174 ceph-mon[36787]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:21.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:21 smithi174 ceph-mon[36787]: from='client.? 172.21.15.174:0/3102977584' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:47:21.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:21 smithi077 ceph-mon[35865]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:21.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:21 smithi077 ceph-mon[35865]: from='client.? 172.21.15.174:0/3102977584' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:47:23.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:23 smithi076 ceph-mon[32205]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:23.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:23 smithi174 ceph-mon[36787]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:23.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:23 smithi077 ceph-mon[35865]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:25.609 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:25 smithi174 ceph-mon[36787]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:25.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:25 smithi076 ceph-mon[32205]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:25.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:25 smithi077 ceph-mon[35865]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:27.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:27 smithi076 ceph-mon[32205]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:27.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:27 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T14:47:27.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:27 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:27.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:27 smithi174 ceph-mon[36787]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:27.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:27 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T14:47:27.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:27 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:27.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:27 smithi077 ceph-mon[35865]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:27.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:27 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T14:47:27.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:27 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:28.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:28 smithi174 ceph-mon[36787]: Deploying daemon osd.5 on smithi174 2024-03-18T14:47:28.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:28 smithi077 ceph-mon[35865]: Deploying daemon osd.5 on smithi174 2024-03-18T14:47:28.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:28 smithi076 ceph-mon[32205]: Deploying daemon osd.5 on smithi174 2024-03-18T14:47:29.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:29 smithi174 ceph-mon[36787]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:29.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:29 smithi077 ceph-mon[35865]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:29.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:29 smithi076 ceph-mon[32205]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:31.667 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:31 smithi174 ceph-mon[36787]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:31.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:31 smithi077 ceph-mon[35865]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:31.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:31 smithi076 ceph-mon[32205]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:33.607 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:33 smithi174 ceph-mon[36787]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:33.607 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:33 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:33.607 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:33 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:33.607 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:33 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:47:33.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:33 smithi077 ceph-mon[35865]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:33.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:33 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:33.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:33 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:33.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:33 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:47:33.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:33 smithi076 ceph-mon[32205]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:33.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:33 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:33.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:33 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:33.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:33 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:47:35.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:35 smithi174 ceph-mon[36787]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:35.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:35 smithi077 ceph-mon[35865]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:35.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:35 smithi076 ceph-mon[32205]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:36.370 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:36.369+0000 7f67697fa700 1 -- 172.21.15.174:0/2606863051 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f676c062710 con 0x7f67540a2910 2024-03-18T14:47:36.370 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 5 on host 'smithi174' 2024-03-18T14:47:36.374 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:36.372+0000 7f6772566700 1 -- 172.21.15.174:0/2606863051 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f67540a2910 msgr2=0x7f67540a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:47:36.374 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:36.372+0000 7f6772566700 1 --2- 172.21.15.174:0/2606863051 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f67540a2910 0x7f67540a4dd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6760005d90 tx=0x7f6760005d00 comp rx=0 tx=0).stop 2024-03-18T14:47:36.374 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:36.372+0000 7f6772566700 1 -- 172.21.15.174:0/2606863051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f676c102320 msgr2=0x7f676c198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:47:36.374 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:36.372+0000 7f6772566700 1 --2- 172.21.15.174:0/2606863051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f676c102320 0x7f676c198510 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f675c00b530 tx=0x7f675c00b8f0 comp rx=0 tx=0).stop 2024-03-18T14:47:36.374 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:36.372+0000 7f6772566700 1 -- 172.21.15.174:0/2606863051 shutdown_connections 2024-03-18T14:47:36.375 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:36.372+0000 7f6772566700 1 --2- 172.21.15.174:0/2606863051 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f67540a2910 0x7f67540a4dd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:36.375 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:36.372+0000 7f6772566700 1 --2- 172.21.15.174:0/2606863051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f676c103ec0 0x7f676c19fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:36.375 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:36.372+0000 7f6772566700 1 --2- 172.21.15.174:0/2606863051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f676c102320 0x7f676c198510 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:36.375 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:36.372+0000 7f6772566700 1 --2- 172.21.15.174:0/2606863051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f676c103520 0x7f676c198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:36.375 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:36.372+0000 7f6772566700 1 -- 172.21.15.174:0/2606863051 >> 172.21.15.174:0/2606863051 conn(0x7f676c0fd880 msgr2=0x7f676c1067c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:47:36.375 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:36.373+0000 7f6772566700 1 -- 172.21.15.174:0/2606863051 shutdown_connections 2024-03-18T14:47:36.375 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:47:36.373+0000 7f6772566700 1 -- 172.21.15.174:0/2606863051 wait complete. 2024-03-18T14:47:37.196 DEBUG:teuthology.orchestra.run.smithi174:osd.5> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.5.service 2024-03-18T14:47:37.198 INFO:tasks.cephadm:Deploying osd.6 on smithi076 with /dev/vg_nvme/lv_2... 2024-03-18T14:47:37.199 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T14:47:37.457 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:47:37.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:37 smithi174 ceph-mon[36787]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:37.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:37 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:37.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:37 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:37.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:37 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:37.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:37 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:37.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:37 smithi174 ceph-mon[36787]: from='osd.5 [v2:172.21.15.174:6808/1607563490,v1:172.21.15.174:6809/1607563490]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T14:47:37.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:37 smithi174 ceph-mon[36787]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T14:47:37.578 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:37 smithi076 ceph-mon[32205]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:37.578 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:37 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:37.578 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:37 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:37.579 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:37 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:37.579 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:37 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:37.579 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:37 smithi076 ceph-mon[32205]: from='osd.5 [v2:172.21.15.174:6808/1607563490,v1:172.21.15.174:6809/1607563490]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T14:47:37.579 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:37 smithi076 ceph-mon[32205]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T14:47:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:37 smithi077 ceph-mon[35865]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:37.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:37 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:37.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:37 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:37.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:37 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:37.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:37 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:37.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:37 smithi077 ceph-mon[35865]: from='osd.5 [v2:172.21.15.174:6808/1607563490,v1:172.21.15.174:6809/1607563490]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T14:47:37.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:37 smithi077 ceph-mon[35865]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T14:47:38.892 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:38 smithi076 ceph-mon[32205]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T14:47:38.892 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:38 smithi076 ceph-mon[32205]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T14:47:38.893 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:38 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:47:38.893 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:38 smithi076 ceph-mon[32205]: from='osd.5 [v2:172.21.15.174:6808/1607563490,v1:172.21.15.174:6809/1607563490]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:47:38.893 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:38 smithi076 ceph-mon[32205]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:47:38.893 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:38 smithi076 ceph-mon[32205]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:38.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:38 smithi174 ceph-mon[36787]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T14:47:38.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:38 smithi174 ceph-mon[36787]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T14:47:38.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:38 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:47:38.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:38 smithi174 ceph-mon[36787]: from='osd.5 [v2:172.21.15.174:6808/1607563490,v1:172.21.15.174:6809/1607563490]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:47:38.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:38 smithi174 ceph-mon[36787]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:47:38.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:38 smithi174 ceph-mon[36787]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:38.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:38 smithi077 ceph-mon[35865]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T14:47:38.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:38 smithi077 ceph-mon[35865]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T14:47:38.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:38 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:47:38.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:38 smithi077 ceph-mon[35865]: from='osd.5 [v2:172.21.15.174:6808/1607563490,v1:172.21.15.174:6809/1607563490]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:47:38.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:38 smithi077 ceph-mon[35865]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:47:38.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:38 smithi077 ceph-mon[35865]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T14:47:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T14:47:39.464 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-03-18T14:47:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-03-18T14:47:39.464 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113686 s, 922 MB/s 2024-03-18T14:47:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-03-18T14:47:39.467 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 14:47:39 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T14:47:39.128+0000 7f7a6b77d700 -1 osd.5 0 waiting for initial osdmap 2024-03-18T14:47:39.467 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 14:47:39 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T14:47:39.136+0000 7f7a63568700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:47:39.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:39 smithi174 ceph-mon[36787]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-18T14:47:39.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:39 smithi174 ceph-mon[36787]: osdmap e38: 6 total, 5 up, 6 in 2024-03-18T14:47:39.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:39 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:47:39.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:39 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:47:39.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:39 smithi174 ceph-mon[36787]: from='osd.5 ' entity='osd.5' 2024-03-18T14:47:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:39 smithi077 ceph-mon[35865]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-18T14:47:39.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:39 smithi077 ceph-mon[35865]: osdmap e38: 6 total, 5 up, 6 in 2024-03-18T14:47:39.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:39 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:47:39.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:39 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:47:39.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:39 smithi077 ceph-mon[35865]: from='osd.5 ' entity='osd.5' 2024-03-18T14:47:40.101 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:39 smithi076 ceph-mon[32205]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-18T14:47:40.102 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:39 smithi076 ceph-mon[32205]: osdmap e38: 6 total, 5 up, 6 in 2024-03-18T14:47:40.102 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:39 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:47:40.102 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:39 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:47:40.102 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:39 smithi076 ceph-mon[32205]: from='osd.5 ' entity='osd.5' 2024-03-18T14:47:40.135 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi076:vg_nvme/lv_2 2024-03-18T14:47:40.383 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:47:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: purged_snaps scrub starts 2024-03-18T14:47:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: purged_snaps scrub ok 2024-03-18T14:47:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: osd.5 [v2:172.21.15.174:6808/1607563490,v1:172.21.15.174:6809/1607563490] boot 2024-03-18T14:47:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T14:47:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:47:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: Detected new or changed devices on smithi174 2024-03-18T14:47:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:47:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:47:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi174 to 8540M 2024-03-18T14:47:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:40.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:40.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:47:40.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:40.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:40.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:47:40.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:40.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:47:40.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:41.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: purged_snaps scrub starts 2024-03-18T14:47:41.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: purged_snaps scrub ok 2024-03-18T14:47:41.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: osd.5 [v2:172.21.15.174:6808/1607563490,v1:172.21.15.174:6809/1607563490] boot 2024-03-18T14:47:41.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T14:47:41.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:47:41.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: Detected new or changed devices on smithi174 2024-03-18T14:47:41.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:41.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:41.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:47:41.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:47:41.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi174 to 8540M 2024-03-18T14:47:41.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:41.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:41.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:47:41.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:41.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:41.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:47:41.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:41.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:47:41.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:41.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: purged_snaps scrub starts 2024-03-18T14:47:41.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: purged_snaps scrub ok 2024-03-18T14:47:41.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: osd.5 [v2:172.21.15.174:6808/1607563490,v1:172.21.15.174:6809/1607563490] boot 2024-03-18T14:47:41.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T14:47:41.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:47:41.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: Detected new or changed devices on smithi174 2024-03-18T14:47:41.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:41.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:41.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:47:41.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:47:41.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi174 to 8540M 2024-03-18T14:47:41.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:41.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:41.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:47:41.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:41.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T14:47:41.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:47:41.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:41.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:47:41.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:41.803 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:41 smithi076 ceph-mon[32205]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T14:47:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:41 smithi077 ceph-mon[35865]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T14:47:42.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.015+0000 7fb45c8cf700 1 -- 172.21.15.76:0/2660664152 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb458101510 msgr2=0x7fb458101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:47:42.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.015+0000 7fb45c8cf700 1 --2- 172.21.15.76:0/2660664152 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb458101510 0x7fb458101970 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb44c0099c0 tx=0x7fb44c009cd0 comp rx=0 tx=0).stop 2024-03-18T14:47:42.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.016+0000 7fb45c8cf700 1 -- 172.21.15.76:0/2660664152 shutdown_connections 2024-03-18T14:47:42.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.016+0000 7fb45c8cf700 1 --2- 172.21.15.76:0/2660664152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb458101eb0 0x7fb458109770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:42.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.016+0000 7fb45c8cf700 1 --2- 172.21.15.76:0/2660664152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4581027d0 0x7fb458102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:42.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.016+0000 7fb45c8cf700 1 --2- 172.21.15.76:0/2660664152 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb458101510 0x7fb458101970 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:42.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.016+0000 7fb45c8cf700 1 -- 172.21.15.76:0/2660664152 >> 172.21.15.76:0/2660664152 conn(0x7fb4580fbf60 msgr2=0x7fb4580fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:47:42.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.016+0000 7fb45c8cf700 1 -- 172.21.15.76:0/2660664152 shutdown_connections 2024-03-18T14:47:42.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.016+0000 7fb45c8cf700 1 -- 172.21.15.76:0/2660664152 wait complete. 2024-03-18T14:47:42.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.016+0000 7fb45c8cf700 1 Processor -- start 2024-03-18T14:47:42.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.016+0000 7fb45c8cf700 1 -- start start 2024-03-18T14:47:42.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.016+0000 7fb45c8cf700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb458101510 0x7fb458198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:47:42.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.016+0000 7fb45c8cf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb458101eb0 0x7fb458198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:47:42.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.016+0000 7fb45c8cf700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4581027d0 0x7fb45819faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:47:42.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.016+0000 7fb45c8cf700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb45810ce90 con 0x7fb458101510 2024-03-18T14:47:42.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.016+0000 7fb45c8cf700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb45810cd10 con 0x7fb458101eb0 2024-03-18T14:47:42.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.016+0000 7fb45c8cf700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb45810d010 con 0x7fb4581027d0 2024-03-18T14:47:42.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.017+0000 7fb45659c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb458101510 0x7fb458198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:47:42.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.017+0000 7fb45659c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb458101510 0x7fb458198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56100/0 (socket says 172.21.15.76:56100) 2024-03-18T14:47:42.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.017+0000 7fb45659c700 1 -- 172.21.15.76:0/2042521813 learned_addr learned my addr 172.21.15.76:0/2042521813 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:47:42.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.017+0000 7fb455d9b700 1 --2- 172.21.15.76:0/2042521813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb458101eb0 0x7fb458198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:47:42.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.017+0000 7fb456d9d700 1 --2- 172.21.15.76:0/2042521813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4581027d0 0x7fb45819faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:47:42.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.017+0000 7fb45659c700 1 -- 172.21.15.76:0/2042521813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4581027d0 msgr2=0x7fb45819faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:47:42.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.017+0000 7fb45659c700 1 --2- 172.21.15.76:0/2042521813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4581027d0 0x7fb45819faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:42.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.017+0000 7fb45659c700 1 -- 172.21.15.76:0/2042521813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb458101eb0 msgr2=0x7fb458198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:47:42.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.017+0000 7fb45659c700 1 --2- 172.21.15.76:0/2042521813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb458101eb0 0x7fb458198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:47:42.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.017+0000 7fb45659c700 1 -- 172.21.15.76:0/2042521813 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4581a0190 con 0x7fb458101510 2024-03-18T14:47:42.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.017+0000 7fb45659c700 1 --2- 172.21.15.76:0/2042521813 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb458101510 0x7fb458198530 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb44800d9b0 tx=0x7fb44800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:47:42.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.017+0000 7fb4477fe700 1 -- 172.21.15.76:0/2042521813 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb448017030 con 0x7fb458101510 2024-03-18T14:47:42.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.017+0000 7fb45c8cf700 1 -- 172.21.15.76:0/2042521813 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4581a0480 con 0x7fb458101510 2024-03-18T14:47:42.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.017+0000 7fb4477fe700 1 -- 172.21.15.76:0/2042521813 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb448004500 con 0x7fb458101510 2024-03-18T14:47:42.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.017+0000 7fb4477fe700 1 -- 172.21.15.76:0/2042521813 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb448003aa0 con 0x7fb458101510 2024-03-18T14:47:42.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.017+0000 7fb45c8cf700 1 -- 172.21.15.76:0/2042521813 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4581a0a90 con 0x7fb458101510 2024-03-18T14:47:42.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.018+0000 7fb45c8cf700 1 -- 172.21.15.76:0/2042521813 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb458068f90 con 0x7fb458101510 2024-03-18T14:47:42.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.020+0000 7fb4477fe700 1 -- 172.21.15.76:0/2042521813 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb44801a070 con 0x7fb458101510 2024-03-18T14:47:42.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.021+0000 7fb4477fe700 1 --2- 172.21.15.76:0/2042521813 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb4400a2890 0x7fb4400a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:47:42.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.021+0000 7fb455d9b700 1 --2- 172.21.15.76:0/2042521813 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb4400a2890 0x7fb4400a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:47:42.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.021+0000 7fb4477fe700 1 -- 172.21.15.76:0/2042521813 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4562+0+0 (secure 0 0 0) 0x7fb4480cd870 con 0x7fb458101510 2024-03-18T14:47:42.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.021+0000 7fb455d9b700 1 --2- 172.21.15.76:0/2042521813 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb4400a2890 0x7fb4400a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb44c000c00 tx=0x7fb44c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:47:42.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.024+0000 7fb4477fe700 1 -- 172.21.15.76:0/2042521813 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb44800e350 con 0x7fb458101510 2024-03-18T14:47:42.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:41 smithi174 ceph-mon[36787]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T14:47:42.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:47:42.228+0000 7fb45c8cf700 1 -- 172.21.15.76:0/2042521813 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fb4581a0db0 con 0x7fb4400a2890 2024-03-18T14:47:43.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:42 smithi076 ceph-mon[32205]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T14:47:43.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:42 smithi076 ceph-mon[32205]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:47:43.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:42 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:47:43.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:42 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:47:43.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:42 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:43.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:42 smithi076 ceph-mon[32205]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:47:43.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:42 smithi174 ceph-mon[36787]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T14:47:43.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:42 smithi174 ceph-mon[36787]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:47:43.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:42 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:47:43.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:42 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:47:43.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:42 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:43.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:42 smithi174 ceph-mon[36787]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:47:43.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:42 smithi077 ceph-mon[35865]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T14:47:43.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:42 smithi077 ceph-mon[35865]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:47:43.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:42 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:47:43.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:42 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:47:43.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:42 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:43.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:42 smithi077 ceph-mon[35865]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:47:44.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:43 smithi076 ceph-mon[32205]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T14:47:44.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:43 smithi174 ceph-mon[36787]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T14:47:44.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:43 smithi077 ceph-mon[35865]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T14:47:45.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:44 smithi076 ceph-mon[32205]: osdmap e43: 6 total, 6 up, 6 in 2024-03-18T14:47:45.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:44 smithi076 ceph-mon[32205]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T14:47:45.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:44 smithi174 ceph-mon[36787]: osdmap e43: 6 total, 6 up, 6 in 2024-03-18T14:47:45.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:44 smithi174 ceph-mon[36787]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T14:47:45.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:44 smithi077 ceph-mon[35865]: osdmap e43: 6 total, 6 up, 6 in 2024-03-18T14:47:45.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:44 smithi077 ceph-mon[35865]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T14:47:46.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:45 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/615494964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f008e913-20ac-4df7-8bed-fb345d6d84bb"}]: dispatch 2024-03-18T14:47:46.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:45 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/615494964' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f008e913-20ac-4df7-8bed-fb345d6d84bb"}]': finished 2024-03-18T14:47:46.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:45 smithi076 ceph-mon[32205]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T14:47:46.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:45 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:47:46.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:45 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/442471975' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:47:46.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:45 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/615494964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f008e913-20ac-4df7-8bed-fb345d6d84bb"}]: dispatch 2024-03-18T14:47:46.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:45 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/615494964' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f008e913-20ac-4df7-8bed-fb345d6d84bb"}]': finished 2024-03-18T14:47:46.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:45 smithi174 ceph-mon[36787]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T14:47:46.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:45 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:47:46.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:45 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/442471975' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:47:46.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:45 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/615494964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f008e913-20ac-4df7-8bed-fb345d6d84bb"}]: dispatch 2024-03-18T14:47:46.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:45 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/615494964' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f008e913-20ac-4df7-8bed-fb345d6d84bb"}]': finished 2024-03-18T14:47:46.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:45 smithi077 ceph-mon[35865]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T14:47:46.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:45 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:47:46.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:45 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/442471975' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:47:47.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:46 smithi174 ceph-mon[36787]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-18T14:47:47.128 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:46 smithi077 ceph-mon[35865]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-18T14:47:47.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:46 smithi076 ceph-mon[32205]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-18T14:47:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:49 smithi076 ceph-mon[32205]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T14:47:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:49 smithi174 ceph-mon[36787]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T14:47:49.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:49 smithi077 ceph-mon[35865]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T14:47:51.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:51 smithi076 ceph-mon[32205]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-18T14:47:51.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:51 smithi174 ceph-mon[36787]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-18T14:47:51.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:51 smithi077 ceph-mon[35865]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-18T14:47:53.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:53 smithi076 ceph-mon[32205]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:47:53.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:53 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T14:47:53.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:53 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:53.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:53 smithi076 ceph-mon[32205]: Deploying daemon osd.6 on smithi076 2024-03-18T14:47:53.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:53 smithi174 ceph-mon[36787]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:47:53.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:53 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T14:47:53.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:53 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:53.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:53 smithi174 ceph-mon[36787]: Deploying daemon osd.6 on smithi076 2024-03-18T14:47:53.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:53 smithi077 ceph-mon[35865]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:47:53.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:53 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T14:47:53.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:53 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:47:53.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:53 smithi077 ceph-mon[35865]: Deploying daemon osd.6 on smithi076 2024-03-18T14:47:55.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:55 smithi076 ceph-mon[32205]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:47:55.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:55 smithi174 ceph-mon[36787]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:47:55.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:55 smithi077 ceph-mon[35865]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:47:57.650 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:57 smithi076 ceph-mon[32205]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:47:57.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:57 smithi174 ceph-mon[36787]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:47:57.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:57 smithi077 ceph-mon[35865]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:47:59.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:58 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:59.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:58 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:59.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:58 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:47:59.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:47:58 smithi076 ceph-mon[32205]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:47:59.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:58 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:59.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:58 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:59.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:58 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:47:59.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:47:58 smithi174 ceph-mon[36787]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:47:59.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:58 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:59.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:58 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:47:59.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:58 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:47:59.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:47:58 smithi077 ceph-mon[35865]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:48:01.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:01 smithi076 ceph-mon[32205]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:48:01.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:01 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:01.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:01 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:01.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:01 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:01.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:01 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:01.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:01 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:01.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:01 smithi174 ceph-mon[36787]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:48:01.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:01 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:01.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:01 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:01.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:01 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:01.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:01 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:01.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:01 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:01.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:01 smithi077 ceph-mon[35865]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:48:01.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:01 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:01.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:01 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:01.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:01 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:01.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:01 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:01.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:01 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:01.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:01.733+0000 7fb4477fe700 1 -- 172.21.15.76:0/2042521813 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb4581a0db0 con 0x7fb4400a2890 2024-03-18T14:48:01.734 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 6 on host 'smithi076' 2024-03-18T14:48:01.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:01.735+0000 7fb45c8cf700 1 -- 172.21.15.76:0/2042521813 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb4400a2890 msgr2=0x7fb4400a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:48:01.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:01.735+0000 7fb45c8cf700 1 --2- 172.21.15.76:0/2042521813 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb4400a2890 0x7fb4400a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb44c000c00 tx=0x7fb44c005fb0 comp rx=0 tx=0).stop 2024-03-18T14:48:01.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:01.735+0000 7fb45c8cf700 1 -- 172.21.15.76:0/2042521813 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb458101510 msgr2=0x7fb458198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:48:01.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:01.735+0000 7fb45c8cf700 1 --2- 172.21.15.76:0/2042521813 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb458101510 0x7fb458198530 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb44800d9b0 tx=0x7fb44800dcc0 comp rx=0 tx=0).stop 2024-03-18T14:48:01.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:01.735+0000 7fb45c8cf700 1 -- 172.21.15.76:0/2042521813 shutdown_connections 2024-03-18T14:48:01.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:01.735+0000 7fb45c8cf700 1 --2- 172.21.15.76:0/2042521813 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb4400a2890 0x7fb4400a4d50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:01.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:01.735+0000 7fb45c8cf700 1 --2- 172.21.15.76:0/2042521813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4581027d0 0x7fb45819faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:01.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:01.735+0000 7fb45c8cf700 1 --2- 172.21.15.76:0/2042521813 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb458101510 0x7fb458198530 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:01.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:01.735+0000 7fb45c8cf700 1 --2- 172.21.15.76:0/2042521813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb458101eb0 0x7fb458198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:01.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:01.735+0000 7fb45c8cf700 1 -- 172.21.15.76:0/2042521813 >> 172.21.15.76:0/2042521813 conn(0x7fb4580fbf60 msgr2=0x7fb4580fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:48:01.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:01.735+0000 7fb45c8cf700 1 -- 172.21.15.76:0/2042521813 shutdown_connections 2024-03-18T14:48:01.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:01.735+0000 7fb45c8cf700 1 -- 172.21.15.76:0/2042521813 wait complete. 2024-03-18T14:48:02.471 DEBUG:teuthology.orchestra.run.smithi076:osd.6> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.6.service 2024-03-18T14:48:02.474 INFO:tasks.cephadm:Deploying osd.7 on smithi077 with /dev/vg_nvme/lv_2... 2024-03-18T14:48:02.474 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T14:48:02.720 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T14:48:02.826 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:02 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:02.826 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:02 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:02.826 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:02 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:48:02.826 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:02 smithi077 ceph-mon[35865]: from='osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T14:48:02.826 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:02 smithi077 ceph-mon[35865]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:48:03.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:02 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:03.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:02 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:03.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:02 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:48:03.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:02 smithi076 ceph-mon[32205]: from='osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T14:48:03.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:02 smithi076 ceph-mon[32205]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:48:03.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:02 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:03.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:02 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:03.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:02 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:48:03.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:02 smithi174 ceph-mon[36787]: from='osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T14:48:03.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:02 smithi174 ceph-mon[36787]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:48:04.025 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:03 smithi077 ceph-mon[35865]: from='osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T14:48:04.025 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:03 smithi077 ceph-mon[35865]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T14:48:04.025 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:03 smithi077 ceph-mon[35865]: from='osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-18T14:48:04.025 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:03 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:48:04.051 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:03 smithi076 ceph-mon[32205]: from='osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T14:48:04.051 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:03 smithi076 ceph-mon[32205]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T14:48:04.051 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:03 smithi076 ceph-mon[32205]: from='osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-18T14:48:04.051 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:03 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:48:04.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:03 smithi174 ceph-mon[36787]: from='osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T14:48:04.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:03 smithi174 ceph-mon[36787]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T14:48:04.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:03 smithi174 ceph-mon[36787]: from='osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-18T14:48:04.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:03 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:48:04.412 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 14:48:04 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T14:48:04.175+0000 7f934e137700 -1 osd.6 0 waiting for initial osdmap 2024-03-18T14:48:04.412 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 14:48:04 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T14:48:04.187+0000 7f9345721700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:48:04.706 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T14:48:04.706 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T14:48:04.706 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-03-18T14:48:04.706 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-03-18T14:48:04.706 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112049 s, 936 MB/s 2024-03-18T14:48:04.706 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-03-18T14:48:04.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:04 smithi077 ceph-mon[35865]: from='osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-18T14:48:04.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:04 smithi077 ceph-mon[35865]: osdmap e46: 7 total, 6 up, 7 in 2024-03-18T14:48:04.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:04 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:48:04.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:04 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:48:04.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:04 smithi077 ceph-mon[35865]: from='osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352]' entity='osd.6' 2024-03-18T14:48:04.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:04 smithi077 ceph-mon[35865]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:48:05.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:04 smithi076 ceph-mon[32205]: from='osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-18T14:48:05.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:04 smithi076 ceph-mon[32205]: osdmap e46: 7 total, 6 up, 7 in 2024-03-18T14:48:05.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:04 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:48:05.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:04 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:48:05.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:04 smithi076 ceph-mon[32205]: from='osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352]' entity='osd.6' 2024-03-18T14:48:05.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:04 smithi076 ceph-mon[32205]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:48:05.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:04 smithi174 ceph-mon[36787]: from='osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-18T14:48:05.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:04 smithi174 ceph-mon[36787]: osdmap e46: 7 total, 6 up, 7 in 2024-03-18T14:48:05.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:04 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:48:05.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:04 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:48:05.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:04 smithi174 ceph-mon[36787]: from='osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352]' entity='osd.6' 2024-03-18T14:48:05.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:04 smithi174 ceph-mon[36787]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-18T14:48:05.383 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi077:vg_nvme/lv_2 2024-03-18T14:48:05.629 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T14:48:05.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:05 smithi077 ceph-mon[35865]: purged_snaps scrub starts 2024-03-18T14:48:05.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:05 smithi077 ceph-mon[35865]: purged_snaps scrub ok 2024-03-18T14:48:05.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:05 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:48:05.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:05 smithi077 ceph-mon[35865]: osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352] boot 2024-03-18T14:48:05.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:05 smithi077 ceph-mon[35865]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T14:48:05.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:05 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:48:06.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:05 smithi076 ceph-mon[32205]: purged_snaps scrub starts 2024-03-18T14:48:06.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:05 smithi076 ceph-mon[32205]: purged_snaps scrub ok 2024-03-18T14:48:06.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:05 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:48:06.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:05 smithi076 ceph-mon[32205]: osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352] boot 2024-03-18T14:48:06.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:05 smithi076 ceph-mon[32205]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T14:48:06.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:05 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:48:06.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:05 smithi174 ceph-mon[36787]: purged_snaps scrub starts 2024-03-18T14:48:06.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:05 smithi174 ceph-mon[36787]: purged_snaps scrub ok 2024-03-18T14:48:06.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:05 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:48:06.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:05 smithi174 ceph-mon[36787]: osd.6 [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352] boot 2024-03-18T14:48:06.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:05 smithi174 ceph-mon[36787]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T14:48:06.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:05 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:48:07.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.220+0000 7f36b2201700 1 -- 172.21.15.77:0/2940936237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36ac103520 msgr2=0x7f36ac103980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:48:07.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.220+0000 7f36b2201700 1 --2- 172.21.15.77:0/2940936237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36ac103520 0x7f36ac103980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f36940099c0 tx=0x7f3694009cd0 comp rx=0 tx=0).stop 2024-03-18T14:48:07.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.220+0000 7f36b2201700 1 -- 172.21.15.77:0/2940936237 shutdown_connections 2024-03-18T14:48:07.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.220+0000 7f36b2201700 1 --2- 172.21.15.77:0/2940936237 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36ac103ec0 0x7f36ac10a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:07.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.220+0000 7f36b2201700 1 --2- 172.21.15.77:0/2940936237 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36ac102320 0x7f36ac102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:07.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.220+0000 7f36b2201700 1 --2- 172.21.15.77:0/2940936237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36ac103520 0x7f36ac103980 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:07.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.220+0000 7f36b2201700 1 -- 172.21.15.77:0/2940936237 >> 172.21.15.77:0/2940936237 conn(0x7f36ac0fd8e0 msgr2=0x7f36ac0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:48:07.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.220+0000 7f36b2201700 1 -- 172.21.15.77:0/2940936237 shutdown_connections 2024-03-18T14:48:07.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.220+0000 7f36b2201700 1 -- 172.21.15.77:0/2940936237 wait complete. 2024-03-18T14:48:07.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36b2201700 1 Processor -- start 2024-03-18T14:48:07.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36b2201700 1 -- start start 2024-03-18T14:48:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36b2201700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36ac102320 0x7f36ac198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:48:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36b2201700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36ac103520 0x7f36ac198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:48:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36b2201700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36ac103ec0 0x7f36ac19fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:48:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36b2201700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac10def0 con 0x7f36ac102320 2024-03-18T14:48:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36b2201700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac10dd70 con 0x7f36ac103520 2024-03-18T14:48:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36b2201700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac10e070 con 0x7f36ac103ec0 2024-03-18T14:48:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36a2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36ac103520 0x7f36ac198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:48:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36a2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36ac103520 0x7f36ac198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45252/0 (socket says 172.21.15.77:45252) 2024-03-18T14:48:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36a2ffd700 1 -- 172.21.15.77:0/2914952166 learned_addr learned my addr 172.21.15.77:0/2914952166 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-18T14:48:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36abfff700 1 --2- 172.21.15.77:0/2914952166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36ac103ec0 0x7f36ac19fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:48:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36ab7fe700 1 --2- 172.21.15.77:0/2914952166 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36ac102320 0x7f36ac198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:48:07.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36a2ffd700 1 -- 172.21.15.77:0/2914952166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36ac103ec0 msgr2=0x7f36ac19fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:48:07.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36a2ffd700 1 --2- 172.21.15.77:0/2914952166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36ac103ec0 0x7f36ac19fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:07.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36a2ffd700 1 -- 172.21.15.77:0/2914952166 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36ac102320 msgr2=0x7f36ac198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:48:07.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36a2ffd700 1 --2- 172.21.15.77:0/2914952166 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36ac102320 0x7f36ac198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:07.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36a2ffd700 1 -- 172.21.15.77:0/2914952166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36ac1a0160 con 0x7f36ac103520 2024-03-18T14:48:07.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36abfff700 1 --2- 172.21.15.77:0/2914952166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36ac103ec0 0x7f36ac19fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:48:07.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.221+0000 7f36ab7fe700 1 --2- 172.21.15.77:0/2914952166 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36ac102320 0x7f36ac198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:48:07.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.222+0000 7f36a2ffd700 1 --2- 172.21.15.77:0/2914952166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36ac103520 0x7f36ac198a40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3694003790 tx=0x7f369400f740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:48:07.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.222+0000 7f36a97fa700 1 -- 172.21.15.77:0/2914952166 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3694009e60 con 0x7f36ac103520 2024-03-18T14:48:07.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.222+0000 7f36a97fa700 1 -- 172.21.15.77:0/2914952166 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3694003e40 con 0x7f36ac103520 2024-03-18T14:48:07.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.222+0000 7f36b2201700 1 -- 172.21.15.77:0/2914952166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36ac1a03f0 con 0x7f36ac103520 2024-03-18T14:48:07.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.222+0000 7f36a97fa700 1 -- 172.21.15.77:0/2914952166 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3694017670 con 0x7f36ac103520 2024-03-18T14:48:07.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.222+0000 7f36b2201700 1 -- 172.21.15.77:0/2914952166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36ac1a0840 con 0x7f36ac103520 2024-03-18T14:48:07.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.223+0000 7f36b2201700 1 -- 172.21.15.77:0/2914952166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36ac068f90 con 0x7f36ac103520 2024-03-18T14:48:07.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.224+0000 7f36a97fa700 1 -- 172.21.15.77:0/2914952166 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3694003960 con 0x7f36ac103520 2024-03-18T14:48:07.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.225+0000 7f36a97fa700 1 --2- 172.21.15.77:0/2914952166 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f36980a2890 0x7f36980a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:48:07.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.225+0000 7f36a97fa700 1 -- 172.21.15.77:0/2914952166 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7f36940ced10 con 0x7f36ac103520 2024-03-18T14:48:07.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.225+0000 7f36ab7fe700 1 --2- 172.21.15.77:0/2914952166 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f36980a2890 0x7f36980a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:48:07.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.225+0000 7f36ab7fe700 1 --2- 172.21.15.77:0/2914952166 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f36980a2890 0x7f36980a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f36ac103380 tx=0x7f369c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:48:07.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.228+0000 7f36a97fa700 1 -- 172.21.15.77:0/2914952166 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3694013070 con 0x7f36ac103520 2024-03-18T14:48:07.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:06 smithi076 ceph-mon[32205]: Detected new or changed devices on smithi076 2024-03-18T14:48:07.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:06 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi076 to 5696M 2024-03-18T14:48:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:06 smithi076 ceph-mon[32205]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:06 smithi076 ceph-mon[32205]: osdmap e48: 7 total, 7 up, 7 in 2024-03-18T14:48:07.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:07.429+0000 7f36b2201700 1 -- 172.21.15.77:0/2914952166 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f36ac062710 con 0x7f36980a2890 2024-03-18T14:48:07.430 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:06 smithi077 ceph-mon[35865]: Detected new or changed devices on smithi076 2024-03-18T14:48:07.431 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:07.431 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:07.431 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:07.431 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:07.431 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:07 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:07.431 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:07 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi076 to 5696M 2024-03-18T14:48:07.431 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:07 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:07.431 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:07 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:07.431 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:07 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:07.431 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:07 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:07.431 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:07 smithi077 ceph-mon[35865]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:07.431 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:07 smithi077 ceph-mon[35865]: osdmap e48: 7 total, 7 up, 7 in 2024-03-18T14:48:07.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:06 smithi174 ceph-mon[36787]: Detected new or changed devices on smithi076 2024-03-18T14:48:07.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:07.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:07.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:07.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:07.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:07.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:07 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi076 to 5696M 2024-03-18T14:48:07.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:07 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:07 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:07 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:07 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:07 smithi174 ceph-mon[36787]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:07 smithi174 ceph-mon[36787]: osdmap e48: 7 total, 7 up, 7 in 2024-03-18T14:48:08.338 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:08 smithi077 ceph-mon[35865]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:48:08.339 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:08 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:48:08.339 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:08 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:48:08.339 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:08 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:08.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:08 smithi076 ceph-mon[32205]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:48:08.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:08 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:48:08.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:08 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:48:08.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:08 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:08.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:08 smithi174 ceph-mon[36787]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:48:08.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:08 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:48:08.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:08 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:48:08.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:08 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:09.324 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:09 smithi077 ceph-mon[35865]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:09.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:09 smithi076 ceph-mon[32205]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:09.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:09 smithi174 ceph-mon[36787]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:10.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:10 smithi174 ceph-mon[36787]: from='client.? 172.21.15.77:0/3711480665' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7354ce4c-94d9-4d70-9134-b01836fbfc90"}]: dispatch 2024-03-18T14:48:10.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:10 smithi174 ceph-mon[36787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7354ce4c-94d9-4d70-9134-b01836fbfc90"}]: dispatch 2024-03-18T14:48:10.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:10 smithi174 ceph-mon[36787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7354ce4c-94d9-4d70-9134-b01836fbfc90"}]': finished 2024-03-18T14:48:10.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:10 smithi174 ceph-mon[36787]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T14:48:10.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:10 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:10.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:10 smithi077 ceph-mon[35865]: from='client.? 172.21.15.77:0/3711480665' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7354ce4c-94d9-4d70-9134-b01836fbfc90"}]: dispatch 2024-03-18T14:48:10.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:10 smithi077 ceph-mon[35865]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7354ce4c-94d9-4d70-9134-b01836fbfc90"}]: dispatch 2024-03-18T14:48:10.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:10 smithi077 ceph-mon[35865]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7354ce4c-94d9-4d70-9134-b01836fbfc90"}]': finished 2024-03-18T14:48:10.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:10 smithi077 ceph-mon[35865]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T14:48:10.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:10 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:10.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:10 smithi076 ceph-mon[32205]: from='client.? 172.21.15.77:0/3711480665' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7354ce4c-94d9-4d70-9134-b01836fbfc90"}]: dispatch 2024-03-18T14:48:10.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:10 smithi076 ceph-mon[32205]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7354ce4c-94d9-4d70-9134-b01836fbfc90"}]: dispatch 2024-03-18T14:48:10.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:10 smithi076 ceph-mon[32205]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7354ce4c-94d9-4d70-9134-b01836fbfc90"}]': finished 2024-03-18T14:48:10.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:10 smithi076 ceph-mon[32205]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T14:48:10.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:10 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:11.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:11 smithi174 ceph-mon[36787]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:11.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:11 smithi174 ceph-mon[36787]: from='client.? 172.21.15.77:0/3614572720' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:48:11.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:11 smithi077 ceph-mon[35865]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:11.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:11 smithi077 ceph-mon[35865]: from='client.? 172.21.15.77:0/3614572720' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:48:11.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:11 smithi076 ceph-mon[32205]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:11.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:11 smithi076 ceph-mon[32205]: from='client.? 172.21.15.77:0/3614572720' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:48:13.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:13 smithi076 ceph-mon[32205]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:13.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:13 smithi174 ceph-mon[36787]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:13.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:13 smithi077 ceph-mon[35865]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:15.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:15 smithi076 ceph-mon[32205]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:15.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:15 smithi174 ceph-mon[36787]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:15.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:15 smithi077 ceph-mon[35865]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:17.426 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:17 smithi077 ceph-mon[35865]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:17.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:17 smithi076 ceph-mon[32205]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:17.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:17 smithi174 ceph-mon[36787]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:18.621 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:18 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T14:48:18.621 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:18 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:18.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:18 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T14:48:18.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:18 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:18.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:18 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T14:48:18.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:18 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:19.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:19 smithi077 ceph-mon[35865]: Deploying daemon osd.7 on smithi077 2024-03-18T14:48:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:19 smithi077 ceph-mon[35865]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:19.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:19 smithi076 ceph-mon[32205]: Deploying daemon osd.7 on smithi077 2024-03-18T14:48:19.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:19 smithi076 ceph-mon[32205]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:19.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:19 smithi174 ceph-mon[36787]: Deploying daemon osd.7 on smithi077 2024-03-18T14:48:19.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:19 smithi174 ceph-mon[36787]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:21.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:21 smithi077 ceph-mon[35865]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:21.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:21 smithi076 ceph-mon[32205]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:21.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:21 smithi174 ceph-mon[36787]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:23.532 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:23 smithi077 ceph-mon[35865]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:23.532 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:23 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:23.533 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:23 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:23.533 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:23 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:48:23.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:23 smithi076 ceph-mon[32205]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:23.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:23 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:23.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:23 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:23.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:23 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:48:23.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:23 smithi174 ceph-mon[36787]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:23.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:23 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:23.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:23 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:23.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:23 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:48:25.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:25 smithi076 ceph-mon[32205]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:25.686 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:25 smithi077 ceph-mon[35865]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:25.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:25 smithi174 ceph-mon[36787]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:26.695 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 7 on host 'smithi077' 2024-03-18T14:48:26.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:26.694+0000 7f36a97fa700 1 -- 172.21.15.77:0/2914952166 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f36ac062710 con 0x7f36980a2890 2024-03-18T14:48:26.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:26.696+0000 7f36b2201700 1 -- 172.21.15.77:0/2914952166 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f36980a2890 msgr2=0x7f36980a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:48:26.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:26.696+0000 7f36b2201700 1 --2- 172.21.15.77:0/2914952166 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f36980a2890 0x7f36980a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f36ac103380 tx=0x7f369c009260 comp rx=0 tx=0).stop 2024-03-18T14:48:26.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:26.696+0000 7f36b2201700 1 -- 172.21.15.77:0/2914952166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36ac103520 msgr2=0x7f36ac198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:48:26.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:26.696+0000 7f36b2201700 1 --2- 172.21.15.77:0/2914952166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36ac103520 0x7f36ac198a40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3694003790 tx=0x7f369400f740 comp rx=0 tx=0).stop 2024-03-18T14:48:26.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:26.696+0000 7f36b2201700 1 -- 172.21.15.77:0/2914952166 shutdown_connections 2024-03-18T14:48:26.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:26.696+0000 7f36b2201700 1 --2- 172.21.15.77:0/2914952166 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f36980a2890 0x7f36980a4d50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:26.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:26.696+0000 7f36b2201700 1 --2- 172.21.15.77:0/2914952166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36ac103ec0 0x7f36ac19fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:26.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:26.696+0000 7f36b2201700 1 --2- 172.21.15.77:0/2914952166 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36ac102320 0x7f36ac198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:26.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:26.696+0000 7f36b2201700 1 --2- 172.21.15.77:0/2914952166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36ac103520 0x7f36ac198a40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:26.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:26.696+0000 7f36b2201700 1 -- 172.21.15.77:0/2914952166 >> 172.21.15.77:0/2914952166 conn(0x7f36ac0fd8e0 msgr2=0x7f36ac106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:48:26.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:26.696+0000 7f36b2201700 1 -- 172.21.15.77:0/2914952166 shutdown_connections 2024-03-18T14:48:26.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:48:26.696+0000 7f36b2201700 1 -- 172.21.15.77:0/2914952166 wait complete. 2024-03-18T14:48:26.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:26.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:26.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:26.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:26.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:26.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:26 smithi077 ceph-mon[35865]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:26.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:48:26.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:26.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:26 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:26.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:26 smithi077 ceph-mon[35865]: from='osd.7 [v2:172.21.15.77:6816/3828133263,v1:172.21.15.77:6817/3828133263]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T14:48:26.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:26 smithi077 ceph-mon[35865]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T14:48:27.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:27.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:27.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:27.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:27.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:27.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:26 smithi076 ceph-mon[32205]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:27.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:48:27.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:27.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:26 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:27.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:26 smithi076 ceph-mon[32205]: from='osd.7 [v2:172.21.15.77:6816/3828133263,v1:172.21.15.77:6817/3828133263]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T14:48:27.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:26 smithi076 ceph-mon[32205]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T14:48:27.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:27.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:27.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:27.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:27.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:27.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:26 smithi174 ceph-mon[36787]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:27.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:48:27.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:27.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:26 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:27.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:26 smithi174 ceph-mon[36787]: from='osd.7 [v2:172.21.15.77:6816/3828133263,v1:172.21.15.77:6817/3828133263]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T14:48:27.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:26 smithi174 ceph-mon[36787]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T14:48:27.325 DEBUG:teuthology.orchestra.run.smithi077:osd.7> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.7.service 2024-03-18T14:48:27.328 INFO:tasks.cephadm:Deploying osd.8 on smithi174 with /dev/vg_nvme/lv_2... 2024-03-18T14:48:27.328 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T14:48:27.573 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T14:48:28.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:28 smithi077 ceph-mon[35865]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T14:48:28.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:28 smithi077 ceph-mon[35865]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T14:48:28.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:28 smithi077 ceph-mon[35865]: from='osd.7 [v2:172.21.15.77:6816/3828133263,v1:172.21.15.77:6817/3828133263]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:48:28.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:28 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:28.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:28 smithi077 ceph-mon[35865]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:48:28.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:28 smithi077 ceph-mon[35865]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:28.728 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 14:48:28 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T14:48:28.695+0000 7f869ef31700 -1 osd.7 0 waiting for initial osdmap 2024-03-18T14:48:28.729 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 14:48:28 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T14:48:28.704+0000 7f869ad24700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:48:28.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:28 smithi174 ceph-mon[36787]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T14:48:28.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:28 smithi174 ceph-mon[36787]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T14:48:28.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:28 smithi174 ceph-mon[36787]: from='osd.7 [v2:172.21.15.77:6816/3828133263,v1:172.21.15.77:6817/3828133263]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:48:28.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:28 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:28.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:28 smithi174 ceph-mon[36787]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:48:28.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:28 smithi174 ceph-mon[36787]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:29.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:28 smithi076 ceph-mon[32205]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T14:48:29.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:28 smithi076 ceph-mon[32205]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T14:48:29.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:28 smithi076 ceph-mon[32205]: from='osd.7 [v2:172.21.15.77:6816/3828133263,v1:172.21.15.77:6817/3828133263]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:48:29.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:28 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:29.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:28 smithi076 ceph-mon[32205]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:48:29.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:28 smithi076 ceph-mon[32205]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:29.635 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T14:48:29.635 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T14:48:29.635 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-03-18T14:48:29.635 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-03-18T14:48:29.635 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130521 s, 803 MB/s 2024-03-18T14:48:29.635 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-03-18T14:48:29.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:29 smithi077 ceph-mon[35865]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-18T14:48:29.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:29 smithi077 ceph-mon[35865]: osdmap e51: 8 total, 7 up, 8 in 2024-03-18T14:48:29.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:29 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:29.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:29 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:29.745 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:29 smithi174 ceph-mon[36787]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-18T14:48:29.745 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:29 smithi174 ceph-mon[36787]: osdmap e51: 8 total, 7 up, 8 in 2024-03-18T14:48:29.745 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:29 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:29.745 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:29 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:30.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:29 smithi076 ceph-mon[32205]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-18T14:48:30.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:29 smithi076 ceph-mon[32205]: osdmap e51: 8 total, 7 up, 8 in 2024-03-18T14:48:30.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:29 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:30.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:29 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:30.363 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi174:vg_nvme/lv_2 2024-03-18T14:48:30.603 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T14:48:30.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: purged_snaps scrub starts 2024-03-18T14:48:30.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: purged_snaps scrub ok 2024-03-18T14:48:30.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:30.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: osd.7 [v2:172.21.15.77:6816/3828133263,v1:172.21.15.77:6817/3828133263] boot 2024-03-18T14:48:30.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T14:48:30.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:30.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: Detected new or changed devices on smithi077 2024-03-18T14:48:30.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:30.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:30.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:30.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:30.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:30.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi077 to 5693M 2024-03-18T14:48:30.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:30.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:30.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:30.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:30.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:30 smithi174 ceph-mon[36787]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:30.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: purged_snaps scrub starts 2024-03-18T14:48:30.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: purged_snaps scrub ok 2024-03-18T14:48:30.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:30.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: osd.7 [v2:172.21.15.77:6816/3828133263,v1:172.21.15.77:6817/3828133263] boot 2024-03-18T14:48:30.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T14:48:30.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:30.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: Detected new or changed devices on smithi077 2024-03-18T14:48:30.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:30.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:30.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:30.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:30.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:30.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi077 to 5693M 2024-03-18T14:48:30.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:30.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:30.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:30.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:30.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:30 smithi077 ceph-mon[35865]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:31.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: purged_snaps scrub starts 2024-03-18T14:48:31.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: purged_snaps scrub ok 2024-03-18T14:48:31.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:31.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: osd.7 [v2:172.21.15.77:6816/3828133263,v1:172.21.15.77:6817/3828133263] boot 2024-03-18T14:48:31.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T14:48:31.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:48:31.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: Detected new or changed devices on smithi077 2024-03-18T14:48:31.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:31.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:31.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:31.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:31.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:31.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi077 to 5693M 2024-03-18T14:48:31.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:31.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:31.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:31.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:31.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:30 smithi076 ceph-mon[32205]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-18T14:48:31.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:31 smithi174 ceph-mon[36787]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T14:48:31.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:31 smithi077 ceph-mon[35865]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T14:48:32.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:31 smithi076 ceph-mon[32205]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T14:48:32.348 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.347+0000 7f7d7c38f700 1 -- 172.21.15.174:0/506848127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d74101200 msgr2=0x7f7d74101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:48:32.348 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.347+0000 7f7d7c38f700 1 --2- 172.21.15.174:0/506848127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d74101200 0x7f7d74101620 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7d64009990 tx=0x7f7d64009ca0 comp rx=0 tx=0).stop 2024-03-18T14:48:32.348 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.347+0000 7f7d7c38f700 1 -- 172.21.15.174:0/506848127 shutdown_connections 2024-03-18T14:48:32.348 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.347+0000 7f7d7c38f700 1 --2- 172.21.15.174:0/506848127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d74102da0 0x7f7d74109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:32.349 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.347+0000 7f7d7c38f700 1 --2- 172.21.15.174:0/506848127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d74101200 0x7f7d74101620 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:32.349 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.347+0000 7f7d7c38f700 1 --2- 172.21.15.174:0/506848127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d74102400 0x7f7d74102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:32.349 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.347+0000 7f7d7c38f700 1 -- 172.21.15.174:0/506848127 >> 172.21.15.174:0/506848127 conn(0x7f7d740fc7c0 msgr2=0x7f7d740febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:48:32.349 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.347+0000 7f7d7c38f700 1 -- 172.21.15.174:0/506848127 shutdown_connections 2024-03-18T14:48:32.349 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.347+0000 7f7d7c38f700 1 -- 172.21.15.174:0/506848127 wait complete. 2024-03-18T14:48:32.351 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.348+0000 7f7d7c38f700 1 Processor -- start 2024-03-18T14:48:32.351 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.348+0000 7f7d7c38f700 1 -- start start 2024-03-18T14:48:32.351 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.348+0000 7f7d7c38f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d74101200 0x7f7d74197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:48:32.351 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.348+0000 7f7d7c38f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d74102400 0x7f7d74197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:48:32.351 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.348+0000 7f7d7c38f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d74102da0 0x7f7d7419e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:48:32.351 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.348+0000 7f7d7c38f700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7410ce40 con 0x7f7d74101200 2024-03-18T14:48:32.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.348+0000 7f7d7c38f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7410ccc0 con 0x7f7d74102400 2024-03-18T14:48:32.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.348+0000 7f7d7c38f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7410cfc0 con 0x7f7d74102da0 2024-03-18T14:48:32.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.348+0000 7f7d7a92c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d74102da0 0x7f7d7419e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:48:32.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.348+0000 7f7d7a12b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d74101200 0x7f7d74197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:48:32.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.348+0000 7f7d7a92c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d74102da0 0x7f7d7419e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:35546/0 (socket says 172.21.15.174:35546) 2024-03-18T14:48:32.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.348+0000 7f7d7a92c700 1 -- 172.21.15.174:0/1658184200 learned_addr learned my addr 172.21.15.174:0/1658184200 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-18T14:48:32.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.348+0000 7f7d7992a700 1 --2- 172.21.15.174:0/1658184200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d74102400 0x7f7d74197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:48:32.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.349+0000 7f7d7a92c700 1 -- 172.21.15.174:0/1658184200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d74102400 msgr2=0x7f7d74197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:48:32.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.349+0000 7f7d7a92c700 1 --2- 172.21.15.174:0/1658184200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d74102400 0x7f7d74197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:32.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.349+0000 7f7d7a92c700 1 -- 172.21.15.174:0/1658184200 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d74101200 msgr2=0x7f7d74197430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:48:32.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.349+0000 7f7d7a92c700 1 --2- 172.21.15.174:0/1658184200 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d74101200 0x7f7d74197430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:32.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.349+0000 7f7d7a92c700 1 -- 172.21.15.174:0/1658184200 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d7419f090 con 0x7f7d74102da0 2024-03-18T14:48:32.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.349+0000 7f7d7a92c700 1 --2- 172.21.15.174:0/1658184200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d74102da0 0x7f7d7419e9f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c007d40 tx=0x7f7d6c00a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:48:32.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.349+0000 7f7d6b7fe700 1 -- 172.21.15.174:0/1658184200 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c016020 con 0x7f7d74102da0 2024-03-18T14:48:32.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.349+0000 7f7d7c38f700 1 -- 172.21.15.174:0/1658184200 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d7419f380 con 0x7f7d74102da0 2024-03-18T14:48:32.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.349+0000 7f7d6b7fe700 1 -- 172.21.15.174:0/1658184200 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d6c00ad60 con 0x7f7d74102da0 2024-03-18T14:48:32.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.349+0000 7f7d7c38f700 1 -- 172.21.15.174:0/1658184200 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d7419f940 con 0x7f7d74102da0 2024-03-18T14:48:32.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.349+0000 7f7d6b7fe700 1 -- 172.21.15.174:0/1658184200 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c011600 con 0x7f7d74102da0 2024-03-18T14:48:32.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.350+0000 7f7d7c38f700 1 -- 172.21.15.174:0/1658184200 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d74068f90 con 0x7f7d74102da0 2024-03-18T14:48:32.354 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.351+0000 7f7d6b7fe700 1 -- 172.21.15.174:0/1658184200 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7d6c019020 con 0x7f7d74102da0 2024-03-18T14:48:32.354 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.352+0000 7f7d6b7fe700 1 --2- 172.21.15.174:0/1658184200 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7d600a2910 0x7f7d600a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:48:32.354 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.352+0000 7f7d7a12b700 1 --2- 172.21.15.174:0/1658184200 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7d600a2910 0x7f7d600a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:48:32.354 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.352+0000 7f7d6b7fe700 1 -- 172.21.15.174:0/1658184200 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5426+0+0 (secure 0 0 0) 0x7f7d6c0cdee0 con 0x7f7d74102da0 2024-03-18T14:48:32.354 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.353+0000 7f7d7a12b700 1 --2- 172.21.15.174:0/1658184200 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7d600a2910 0x7f7d600a4dd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7d64003890 tx=0x7f7d64003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:48:32.357 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.357+0000 7f7d6b7fe700 1 -- 172.21.15.174:0/1658184200 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7d6c020020 con 0x7f7d74102da0 2024-03-18T14:48:32.566 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:32.565+0000 7f7d7c38f700 1 -- 172.21.15.174:0/1658184200 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7d7402dfc0 con 0x7f7d600a2910 2024-03-18T14:48:32.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:32 smithi077 ceph-mon[35865]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T14:48:32.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:32 smithi077 ceph-mon[35865]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:32.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:32 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:48:32.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:32 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:48:32.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:32 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:33.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:32 smithi076 ceph-mon[32205]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T14:48:33.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:32 smithi076 ceph-mon[32205]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:33.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:32 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:48:33.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:32 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:48:33.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:32 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:33.167 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:32 smithi174 ceph-mon[36787]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T14:48:33.167 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:32 smithi174 ceph-mon[36787]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:33.167 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:32 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:48:33.167 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:32 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:48:33.167 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:32 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:33.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:33 smithi077 ceph-mon[35865]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:48:33.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:33 smithi077 ceph-mon[35865]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T14:48:34.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:33 smithi076 ceph-mon[32205]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:48:34.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:33 smithi076 ceph-mon[32205]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T14:48:34.183 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:33 smithi174 ceph-mon[36787]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:48:34.183 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:33 smithi174 ceph-mon[36787]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T14:48:34.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:34 smithi174 ceph-mon[36787]: osdmap e56: 8 total, 8 up, 8 in 2024-03-18T14:48:34.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:34 smithi174 ceph-mon[36787]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:35.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:34 smithi076 ceph-mon[32205]: osdmap e56: 8 total, 8 up, 8 in 2024-03-18T14:48:35.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:34 smithi076 ceph-mon[32205]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:35.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:34 smithi077 ceph-mon[35865]: osdmap e56: 8 total, 8 up, 8 in 2024-03-18T14:48:35.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:34 smithi077 ceph-mon[35865]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:36.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:35 smithi076 ceph-mon[32205]: from='client.? 172.21.15.174:0/277581278' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "955ffff4-1c26-4dbd-82f7-aaf6ceb7da21"}]: dispatch 2024-03-18T14:48:36.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:35 smithi076 ceph-mon[32205]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "955ffff4-1c26-4dbd-82f7-aaf6ceb7da21"}]: dispatch 2024-03-18T14:48:36.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:35 smithi076 ceph-mon[32205]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "955ffff4-1c26-4dbd-82f7-aaf6ceb7da21"}]': finished 2024-03-18T14:48:36.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:35 smithi076 ceph-mon[32205]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T14:48:36.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:35 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:48:36.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:35 smithi174 ceph-mon[36787]: from='client.? 172.21.15.174:0/277581278' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "955ffff4-1c26-4dbd-82f7-aaf6ceb7da21"}]: dispatch 2024-03-18T14:48:36.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:35 smithi174 ceph-mon[36787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "955ffff4-1c26-4dbd-82f7-aaf6ceb7da21"}]: dispatch 2024-03-18T14:48:36.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:35 smithi174 ceph-mon[36787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "955ffff4-1c26-4dbd-82f7-aaf6ceb7da21"}]': finished 2024-03-18T14:48:36.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:35 smithi174 ceph-mon[36787]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T14:48:36.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:35 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:48:36.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:35 smithi077 ceph-mon[35865]: from='client.? 172.21.15.174:0/277581278' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "955ffff4-1c26-4dbd-82f7-aaf6ceb7da21"}]: dispatch 2024-03-18T14:48:36.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:35 smithi077 ceph-mon[35865]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "955ffff4-1c26-4dbd-82f7-aaf6ceb7da21"}]: dispatch 2024-03-18T14:48:36.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:35 smithi077 ceph-mon[35865]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "955ffff4-1c26-4dbd-82f7-aaf6ceb7da21"}]': finished 2024-03-18T14:48:36.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:35 smithi077 ceph-mon[35865]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T14:48:36.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:35 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:48:37.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:36 smithi076 ceph-mon[32205]: from='client.? 172.21.15.174:0/526531575' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:48:37.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:36 smithi076 ceph-mon[32205]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:37.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:36 smithi174 ceph-mon[36787]: from='client.? 172.21.15.174:0/526531575' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:48:37.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:36 smithi174 ceph-mon[36787]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:37.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:36 smithi077 ceph-mon[35865]: from='client.? 172.21.15.174:0/526531575' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:48:37.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:36 smithi077 ceph-mon[35865]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:39.637 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:39 smithi174 ceph-mon[36787]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:39.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:39 smithi076 ceph-mon[32205]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:39.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:39 smithi077 ceph-mon[35865]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:41.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:41 smithi174 ceph-mon[36787]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:41.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:41 smithi076 ceph-mon[32205]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:41.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:41 smithi077 ceph-mon[35865]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:43.629 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:43 smithi174 ceph-mon[36787]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:43.629 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:43 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T14:48:43.629 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:43 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:43.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:43 smithi076 ceph-mon[32205]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:43.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:43 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T14:48:43.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:43 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:43.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:43 smithi077 ceph-mon[35865]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:43.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:43 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T14:48:43.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:43 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:44.625 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:44 smithi174 ceph-mon[36787]: Deploying daemon osd.8 on smithi174 2024-03-18T14:48:44.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:44 smithi076 ceph-mon[32205]: Deploying daemon osd.8 on smithi174 2024-03-18T14:48:44.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:44 smithi077 ceph-mon[35865]: Deploying daemon osd.8 on smithi174 2024-03-18T14:48:45.628 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:45 smithi174 ceph-mon[36787]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:45.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:45 smithi076 ceph-mon[32205]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:45.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:45 smithi077 ceph-mon[35865]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:47.614 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:47 smithi174 ceph-mon[36787]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:47.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:47 smithi076 ceph-mon[32205]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:47.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:47 smithi077 ceph-mon[35865]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:49.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:49 smithi174 ceph-mon[36787]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:49.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:49.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:48:49.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:49 smithi076 ceph-mon[32205]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:48:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:49 smithi077 ceph-mon[35865]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:48:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:51.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:51 smithi076 ceph-mon[32205]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:51.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:51 smithi174 ceph-mon[36787]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:51.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:51 smithi077 ceph-mon[35865]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:52.135 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:52.134+0000 7f7d6b7fe700 1 -- 172.21.15.174:0/1658184200 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7d7402dfc0 con 0x7f7d600a2910 2024-03-18T14:48:52.136 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 8 on host 'smithi174' 2024-03-18T14:48:52.138 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:52.137+0000 7f7d7c38f700 1 -- 172.21.15.174:0/1658184200 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7d600a2910 msgr2=0x7f7d600a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:48:52.138 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:52.137+0000 7f7d7c38f700 1 --2- 172.21.15.174:0/1658184200 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7d600a2910 0x7f7d600a4dd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7d64003890 tx=0x7f7d64003960 comp rx=0 tx=0).stop 2024-03-18T14:48:52.138 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:52.137+0000 7f7d7c38f700 1 -- 172.21.15.174:0/1658184200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d74102da0 msgr2=0x7f7d7419e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:48:52.138 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:52.137+0000 7f7d7c38f700 1 --2- 172.21.15.174:0/1658184200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d74102da0 0x7f7d7419e9f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c007d40 tx=0x7f7d6c00a340 comp rx=0 tx=0).stop 2024-03-18T14:48:52.139 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:52.137+0000 7f7d7c38f700 1 -- 172.21.15.174:0/1658184200 shutdown_connections 2024-03-18T14:48:52.139 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:52.137+0000 7f7d7c38f700 1 --2- 172.21.15.174:0/1658184200 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7d600a2910 0x7f7d600a4dd0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:52.139 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:52.137+0000 7f7d7c38f700 1 --2- 172.21.15.174:0/1658184200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d74102da0 0x7f7d7419e9f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:52.139 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:52.137+0000 7f7d7c38f700 1 --2- 172.21.15.174:0/1658184200 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d74101200 0x7f7d74197430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:52.139 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:52.137+0000 7f7d7c38f700 1 --2- 172.21.15.174:0/1658184200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d74102400 0x7f7d74197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:52.139 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:52.137+0000 7f7d7c38f700 1 -- 172.21.15.174:0/1658184200 >> 172.21.15.174:0/1658184200 conn(0x7f7d740fc7c0 msgr2=0x7f7d74105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:48:52.139 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:52.137+0000 7f7d7c38f700 1 -- 172.21.15.174:0/1658184200 shutdown_connections 2024-03-18T14:48:52.139 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:48:52.137+0000 7f7d7c38f700 1 -- 172.21.15.174:0/1658184200 wait complete. 2024-03-18T14:48:52.863 DEBUG:teuthology.orchestra.run.smithi174:osd.8> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.8.service 2024-03-18T14:48:52.866 INFO:tasks.cephadm:Deploying osd.9 on smithi076 with /dev/vg_nvme/lv_1... 2024-03-18T14:48:52.866 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T14:48:53.120 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:48:53.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:53 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:53.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:53 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:53.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:53 smithi076 ceph-mon[32205]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:53.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:53 smithi076 ceph-mon[32205]: from='osd.8 [v2:172.21.15.174:6816/2665844766,v1:172.21.15.174:6817/2665844766]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T14:48:53.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:53 smithi076 ceph-mon[32205]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T14:48:53.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:53 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:53.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:53 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:53.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:53 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:53.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:53 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:53.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:53 smithi174 ceph-mon[36787]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:53.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:53 smithi174 ceph-mon[36787]: from='osd.8 [v2:172.21.15.174:6816/2665844766,v1:172.21.15.174:6817/2665844766]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T14:48:53.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:53 smithi174 ceph-mon[36787]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T14:48:53.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:53 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:53.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:53 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:53.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:53 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:53.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:53 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:53.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:53 smithi077 ceph-mon[35865]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:53.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:53 smithi077 ceph-mon[35865]: from='osd.8 [v2:172.21.15.174:6816/2665844766,v1:172.21.15.174:6817/2665844766]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T14:48:53.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:53 smithi077 ceph-mon[35865]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T14:48:53.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:53 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:53.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:53 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:54.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:54 smithi076 ceph-mon[32205]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T14:48:54.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:54 smithi076 ceph-mon[32205]: from='osd.8 [v2:172.21.15.174:6816/2665844766,v1:172.21.15.174:6817/2665844766]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:48:54.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:54 smithi076 ceph-mon[32205]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T14:48:54.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:54 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:48:54.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:54 smithi076 ceph-mon[32205]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:48:54.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:54 smithi174 ceph-mon[36787]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T14:48:54.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:54 smithi174 ceph-mon[36787]: from='osd.8 [v2:172.21.15.174:6816/2665844766,v1:172.21.15.174:6817/2665844766]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:48:54.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:54 smithi174 ceph-mon[36787]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T14:48:54.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:54 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:48:54.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:54 smithi174 ceph-mon[36787]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:48:54.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:54 smithi077 ceph-mon[35865]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T14:48:54.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:54 smithi077 ceph-mon[35865]: from='osd.8 [v2:172.21.15.174:6816/2665844766,v1:172.21.15.174:6817/2665844766]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:48:54.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:54 smithi077 ceph-mon[35865]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T14:48:54.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:54 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:48:54.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:54 smithi077 ceph-mon[35865]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:48:54.837 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 14:48:54 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T14:48:54.537+0000 7f8a3aa96700 -1 osd.8 0 waiting for initial osdmap 2024-03-18T14:48:54.837 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 14:48:54 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T14:48:54.549+0000 7f8a32881700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:48:55.263 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T14:48:55.263 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T14:48:55.263 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-03-18T14:48:55.264 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-03-18T14:48:55.264 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128045 s, 819 MB/s 2024-03-18T14:48:55.264 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-03-18T14:48:55.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:55 smithi076 ceph-mon[32205]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-18T14:48:55.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:55 smithi076 ceph-mon[32205]: osdmap e59: 9 total, 8 up, 9 in 2024-03-18T14:48:55.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:55 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:48:55.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:55 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:48:55.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:55 smithi076 ceph-mon[32205]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:55.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:55 smithi076 ceph-mon[32205]: from='osd.8 ' entity='osd.8' 2024-03-18T14:48:55.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:55 smithi174 ceph-mon[36787]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-18T14:48:55.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:55 smithi174 ceph-mon[36787]: osdmap e59: 9 total, 8 up, 9 in 2024-03-18T14:48:55.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:55 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:48:55.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:55 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:48:55.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:55 smithi174 ceph-mon[36787]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:55.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:55 smithi174 ceph-mon[36787]: from='osd.8 ' entity='osd.8' 2024-03-18T14:48:55.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:55 smithi077 ceph-mon[35865]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-18T14:48:55.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:55 smithi077 ceph-mon[35865]: osdmap e59: 9 total, 8 up, 9 in 2024-03-18T14:48:55.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:55 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:48:55.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:55 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:48:55.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:55 smithi077 ceph-mon[35865]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-18T14:48:55.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:55 smithi077 ceph-mon[35865]: from='osd.8 ' entity='osd.8' 2024-03-18T14:48:55.979 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi076:vg_nvme/lv_1 2024-03-18T14:48:56.222 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:48:56.343 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:56 smithi076 ceph-mon[32205]: purged_snaps scrub starts 2024-03-18T14:48:56.343 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:56 smithi076 ceph-mon[32205]: purged_snaps scrub ok 2024-03-18T14:48:56.343 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:56 smithi076 ceph-mon[32205]: osd.8 [v2:172.21.15.174:6816/2665844766,v1:172.21.15.174:6817/2665844766] boot 2024-03-18T14:48:56.343 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:56 smithi076 ceph-mon[32205]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T14:48:56.343 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:56 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:48:56.343 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:56 smithi076 ceph-mon[32205]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T14:48:56.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:56 smithi174 ceph-mon[36787]: purged_snaps scrub starts 2024-03-18T14:48:56.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:56 smithi174 ceph-mon[36787]: purged_snaps scrub ok 2024-03-18T14:48:56.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:56 smithi174 ceph-mon[36787]: osd.8 [v2:172.21.15.174:6816/2665844766,v1:172.21.15.174:6817/2665844766] boot 2024-03-18T14:48:56.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:56 smithi174 ceph-mon[36787]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T14:48:56.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:56 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:48:56.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:56 smithi174 ceph-mon[36787]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T14:48:56.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:56 smithi077 ceph-mon[35865]: purged_snaps scrub starts 2024-03-18T14:48:56.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:56 smithi077 ceph-mon[35865]: purged_snaps scrub ok 2024-03-18T14:48:56.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:56 smithi077 ceph-mon[35865]: osd.8 [v2:172.21.15.174:6816/2665844766,v1:172.21.15.174:6817/2665844766] boot 2024-03-18T14:48:56.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:56 smithi077 ceph-mon[35865]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T14:48:56.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:56 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:48:56.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:56 smithi077 ceph-mon[35865]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T14:48:57.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:57 smithi076 ceph-mon[32205]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:48:57.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:57 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:57.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:57 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:57.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:57 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:57.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:57 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:57.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:57 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:57.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:57 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:57.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:57 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:57.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:57 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:57.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:57 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:57.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:57 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:48:57.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:57 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:57.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:57 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:57.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:57 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:57.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:57 smithi076 ceph-mon[32205]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T14:48:57.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:57 smithi174 ceph-mon[36787]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:48:57.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:57 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:57.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:57 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:57.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:57 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:57.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:57 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:57.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:57 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:57.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:57 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:57.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:57 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:57.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:57 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:57.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:57 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:57.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:57 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:48:57.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:57 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:57.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:57 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:57.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:57 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:57.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:57 smithi174 ceph-mon[36787]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T14:48:57.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:57 smithi077 ceph-mon[35865]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:48:57.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:57 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:57.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:57 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:57.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:57 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:57.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:57 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:57.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:57 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:48:57.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:57 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:57.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:57 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:57.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:57 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:57.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:57 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:57.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:57 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:48:57.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:57 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:57.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:57 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:48:57.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:57 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:48:57.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:57 smithi077 ceph-mon[35865]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T14:48:58.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.107+0000 7fe25f935700 1 -- 172.21.15.76:0/3408985250 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe258102300 msgr2=0x7fe258102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:48:58.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.107+0000 7fe25f935700 1 --2- 172.21.15.76:0/3408985250 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe258102300 0x7fe258102720 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe2540099c0 tx=0x7fe254009cd0 comp rx=0 tx=0).stop 2024-03-18T14:48:58.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.108+0000 7fe25f935700 1 -- 172.21.15.76:0/3408985250 shutdown_connections 2024-03-18T14:48:58.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.108+0000 7fe25f935700 1 --2- 172.21.15.76:0/3408985250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe258103ea0 0x7fe25810a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:58.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.108+0000 7fe25f935700 1 --2- 172.21.15.76:0/3408985250 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe258102300 0x7fe258102720 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:58.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.108+0000 7fe25f935700 1 --2- 172.21.15.76:0/3408985250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe258103500 0x7fe258103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:58.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.108+0000 7fe25f935700 1 -- 172.21.15.76:0/3408985250 >> 172.21.15.76:0/3408985250 conn(0x7fe2580fd880 msgr2=0x7fe2580ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:48:58.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.108+0000 7fe25f935700 1 -- 172.21.15.76:0/3408985250 shutdown_connections 2024-03-18T14:48:58.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.108+0000 7fe25f935700 1 -- 172.21.15.76:0/3408985250 wait complete. 2024-03-18T14:48:58.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.108+0000 7fe25f935700 1 Processor -- start 2024-03-18T14:48:58.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.108+0000 7fe25f935700 1 -- start start 2024-03-18T14:48:58.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.108+0000 7fe25f935700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe258102300 0x7fe258198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:48:58.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.108+0000 7fe25f935700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe258103500 0x7fe258198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:48:58.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.108+0000 7fe25f935700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe258103ea0 0x7fe25819fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:48:58.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.108+0000 7fe25f935700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe25810df40 con 0x7fe258103500 2024-03-18T14:48:58.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.108+0000 7fe25f935700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe25810ddc0 con 0x7fe258103ea0 2024-03-18T14:48:58.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.108+0000 7fe25f935700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe25810e0c0 con 0x7fe258102300 2024-03-18T14:48:58.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.108+0000 7fe25ced0700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe258103500 0x7fe258198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:48:58.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.109+0000 7fe25ced0700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe258103500 0x7fe258198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:42250/0 (socket says 172.21.15.76:42250) 2024-03-18T14:48:58.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.109+0000 7fe25ced0700 1 -- 172.21.15.76:0/3973459067 learned_addr learned my addr 172.21.15.76:0/3973459067 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:48:58.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.109+0000 7fe25d6d1700 1 --2- 172.21.15.76:0/3973459067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe258102300 0x7fe258198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:48:58.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.109+0000 7fe25ded2700 1 --2- 172.21.15.76:0/3973459067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe258103ea0 0x7fe25819fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:48:58.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.109+0000 7fe25ced0700 1 -- 172.21.15.76:0/3973459067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe258102300 msgr2=0x7fe258198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:48:58.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.109+0000 7fe25ced0700 1 --2- 172.21.15.76:0/3973459067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe258102300 0x7fe258198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:58.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.109+0000 7fe25ced0700 1 -- 172.21.15.76:0/3973459067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe258103ea0 msgr2=0x7fe25819fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:48:58.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.109+0000 7fe25ced0700 1 --2- 172.21.15.76:0/3973459067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe258103ea0 0x7fe25819fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:48:58.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.109+0000 7fe25ced0700 1 -- 172.21.15.76:0/3973459067 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2581a0180 con 0x7fe258103500 2024-03-18T14:48:58.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.109+0000 7fe25d6d1700 1 --2- 172.21.15.76:0/3973459067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe258102300 0x7fe258198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:48:58.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.109+0000 7fe25ced0700 1 --2- 172.21.15.76:0/3973459067 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe258103500 0x7fe258198a60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe24800b560 tx=0x7fe24800b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:48:58.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.109+0000 7fe24e7fc700 1 -- 172.21.15.76:0/3973459067 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe248015020 con 0x7fe258103500 2024-03-18T14:48:58.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.109+0000 7fe24e7fc700 1 -- 172.21.15.76:0/3973459067 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe248011d10 con 0x7fe258103500 2024-03-18T14:48:58.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.109+0000 7fe25f935700 1 -- 172.21.15.76:0/3973459067 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2581a0470 con 0x7fe258103500 2024-03-18T14:48:58.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.109+0000 7fe24e7fc700 1 -- 172.21.15.76:0/3973459067 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24800f2e0 con 0x7fe258103500 2024-03-18T14:48:58.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.109+0000 7fe25f935700 1 -- 172.21.15.76:0/3973459067 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2581a0a80 con 0x7fe258103500 2024-03-18T14:48:58.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.110+0000 7fe25f935700 1 -- 172.21.15.76:0/3973459067 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe258068f90 con 0x7fe258103500 2024-03-18T14:48:58.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.111+0000 7fe24e7fc700 1 -- 172.21.15.76:0/3973459067 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe248018020 con 0x7fe258103500 2024-03-18T14:48:58.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.112+0000 7fe24e7fc700 1 --2- 172.21.15.76:0/3973459067 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe2440a2910 0x7fe2440a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:48:58.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.112+0000 7fe25d6d1700 1 --2- 172.21.15.76:0/3973459067 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe2440a2910 0x7fe2440a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:48:58.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.112+0000 7fe24e7fc700 1 -- 172.21.15.76:0/3973459067 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5858+0+0 (secure 0 0 0) 0x7fe2480cdd30 con 0x7fe258103500 2024-03-18T14:48:58.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.112+0000 7fe25d6d1700 1 --2- 172.21.15.76:0/3973459067 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe2440a2910 0x7fe2440a4dd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe254003890 tx=0x7fe254005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:48:58.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.116+0000 7fe24e7fc700 1 -- 172.21.15.76:0/3973459067 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe24801f020 con 0x7fe258103500 2024-03-18T14:48:58.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:48:58.324+0000 7fe25f935700 1 -- 172.21.15.76:0/3973459067 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe258062710 con 0x7fe2440a2910 2024-03-18T14:48:58.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:58 smithi076 ceph-mon[32205]: Detected new or changed devices on smithi174 2024-03-18T14:48:58.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:58 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi174 to 5693M 2024-03-18T14:48:58.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:58 smithi076 ceph-mon[32205]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T14:48:58.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:58 smithi174 ceph-mon[36787]: Detected new or changed devices on smithi174 2024-03-18T14:48:58.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:58 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi174 to 5693M 2024-03-18T14:48:58.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:58 smithi174 ceph-mon[36787]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T14:48:58.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:58 smithi077 ceph-mon[35865]: Detected new or changed devices on smithi174 2024-03-18T14:48:58.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:58 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi174 to 5693M 2024-03-18T14:48:58.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:58 smithi077 ceph-mon[35865]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T14:48:59.207 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:59 smithi076 ceph-mon[32205]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:48:59.208 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:59 smithi076 ceph-mon[32205]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:48:59.208 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:59 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:48:59.208 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:59 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:48:59.208 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:48:59 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:59.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:59 smithi174 ceph-mon[36787]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:48:59.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:59 smithi174 ceph-mon[36787]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:48:59.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:59 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:48:59.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:59 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:48:59.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:48:59 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:48:59.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:59 smithi077 ceph-mon[35865]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:48:59.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:59 smithi077 ceph-mon[35865]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:48:59.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:48:59.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:48:59.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:48:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:00.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:00 smithi076 ceph-mon[32205]: osdmap e64: 9 total, 9 up, 9 in 2024-03-18T14:49:00.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:00 smithi174 ceph-mon[36787]: osdmap e64: 9 total, 9 up, 9 in 2024-03-18T14:49:00.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:00 smithi077 ceph-mon[35865]: osdmap e64: 9 total, 9 up, 9 in 2024-03-18T14:49:01.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:01 smithi076 ceph-mon[32205]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:01.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:01 smithi174 ceph-mon[36787]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:01.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:01 smithi077 ceph-mon[35865]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:02.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:02 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/779268657' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a818122-b7a3-4f62-adaf-0ec35156cf1c"}]: dispatch 2024-03-18T14:49:02.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:02 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/779268657' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a818122-b7a3-4f62-adaf-0ec35156cf1c"}]': finished 2024-03-18T14:49:02.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:02 smithi076 ceph-mon[32205]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T14:49:02.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:02 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:02.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:02 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1802518158' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:49:02.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:02 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/779268657' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a818122-b7a3-4f62-adaf-0ec35156cf1c"}]: dispatch 2024-03-18T14:49:02.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:02 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/779268657' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a818122-b7a3-4f62-adaf-0ec35156cf1c"}]': finished 2024-03-18T14:49:02.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:02 smithi174 ceph-mon[36787]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T14:49:02.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:02 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:02.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:02 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1802518158' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:49:02.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:02 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/779268657' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a818122-b7a3-4f62-adaf-0ec35156cf1c"}]: dispatch 2024-03-18T14:49:02.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:02 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/779268657' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a818122-b7a3-4f62-adaf-0ec35156cf1c"}]': finished 2024-03-18T14:49:02.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:02 smithi077 ceph-mon[35865]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T14:49:02.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:02 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:02.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:02 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1802518158' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:49:03.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:03 smithi174 ceph-mon[36787]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-18T14:49:03.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:03 smithi077 ceph-mon[35865]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-18T14:49:03.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:03 smithi076 ceph-mon[32205]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-18T14:49:05.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:05 smithi076 ceph-mon[32205]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-18T14:49:05.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:05 smithi174 ceph-mon[36787]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-18T14:49:05.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:05 smithi077 ceph-mon[35865]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-18T14:49:07.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:07 smithi076 ceph-mon[32205]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T14:49:07.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:07 smithi174 ceph-mon[36787]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T14:49:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:07 smithi077 ceph-mon[35865]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T14:49:09.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:09 smithi076 ceph-mon[32205]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-18T14:49:09.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:09 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T14:49:09.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:09 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:09.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:09 smithi174 ceph-mon[36787]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-18T14:49:09.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:09 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T14:49:09.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:09 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:09.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:09 smithi077 ceph-mon[35865]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-18T14:49:09.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:09 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T14:49:09.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:09 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:10.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:10 smithi076 ceph-mon[32205]: Deploying daemon osd.9 on smithi076 2024-03-18T14:49:10.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:10 smithi174 ceph-mon[36787]: Deploying daemon osd.9 on smithi076 2024-03-18T14:49:10.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:10 smithi077 ceph-mon[35865]: Deploying daemon osd.9 on smithi076 2024-03-18T14:49:11.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:11 smithi076 ceph-mon[32205]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T14:49:11.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:11 smithi174 ceph-mon[36787]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T14:49:11.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:11 smithi077 ceph-mon[35865]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T14:49:13.493 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:13 smithi076 ceph-mon[32205]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:13.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:13 smithi174 ceph-mon[36787]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:13.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:13 smithi077 ceph-mon[35865]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:15.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:15 smithi076 ceph-mon[32205]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:15.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:15 smithi174 ceph-mon[36787]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:15.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:15 smithi077 ceph-mon[35865]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:16.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:16 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:16.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:16 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:16.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:16 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:49:16.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:16 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:16.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:16 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:16.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:16 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:49:16.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:16 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:16.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:16 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:16.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:16 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:49:17.691 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:17 smithi076 ceph-mon[32205]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:17.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:17 smithi174 ceph-mon[36787]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:17.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:17 smithi077 ceph-mon[35865]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:19.583 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 9 on host 'smithi076' 2024-03-18T14:49:19.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:49:19.582+0000 7fe24e7fc700 1 -- 172.21.15.76:0/3973459067 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe258062710 con 0x7fe2440a2910 2024-03-18T14:49:19.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:49:19.584+0000 7fe25f935700 1 -- 172.21.15.76:0/3973459067 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe2440a2910 msgr2=0x7fe2440a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:49:19.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:49:19.584+0000 7fe25f935700 1 --2- 172.21.15.76:0/3973459067 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe2440a2910 0x7fe2440a4dd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe254003890 tx=0x7fe254005fd0 comp rx=0 tx=0).stop 2024-03-18T14:49:19.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:49:19.585+0000 7fe25f935700 1 -- 172.21.15.76:0/3973459067 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe258103500 msgr2=0x7fe258198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:49:19.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:49:19.585+0000 7fe25f935700 1 --2- 172.21.15.76:0/3973459067 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe258103500 0x7fe258198a60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe24800b560 tx=0x7fe24800b920 comp rx=0 tx=0).stop 2024-03-18T14:49:19.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:49:19.585+0000 7fe25f935700 1 -- 172.21.15.76:0/3973459067 shutdown_connections 2024-03-18T14:49:19.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:49:19.585+0000 7fe25f935700 1 --2- 172.21.15.76:0/3973459067 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe2440a2910 0x7fe2440a4dd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:19.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:49:19.585+0000 7fe25f935700 1 --2- 172.21.15.76:0/3973459067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe258103ea0 0x7fe25819fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:19.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:49:19.585+0000 7fe25f935700 1 --2- 172.21.15.76:0/3973459067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe258102300 0x7fe258198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:19.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:49:19.585+0000 7fe25f935700 1 --2- 172.21.15.76:0/3973459067 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe258103500 0x7fe258198a60 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:19.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:49:19.585+0000 7fe25f935700 1 -- 172.21.15.76:0/3973459067 >> 172.21.15.76:0/3973459067 conn(0x7fe2580fd880 msgr2=0x7fe258106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:49:19.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:49:19.585+0000 7fe25f935700 1 -- 172.21.15.76:0/3973459067 shutdown_connections 2024-03-18T14:49:19.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:49:19.585+0000 7fe25f935700 1 -- 172.21.15.76:0/3973459067 wait complete. 2024-03-18T14:49:19.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:19 smithi174 ceph-mon[36787]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:19.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:19.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:19.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:19.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:49:19.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:19 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:19.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:19 smithi174 ceph-mon[36787]: from='osd.9 [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T14:49:19.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:19 smithi077 ceph-mon[35865]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:19.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:19.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:19.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:19.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:49:19.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:19 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:19.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:19 smithi077 ceph-mon[35865]: from='osd.9 [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T14:49:19.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:19 smithi076 ceph-mon[32205]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:19.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:19.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:19.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:19.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:49:19.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:19 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:19.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:19 smithi076 ceph-mon[32205]: from='osd.9 [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T14:49:20.420 DEBUG:teuthology.orchestra.run.smithi076:osd.9> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.9.service 2024-03-18T14:49:20.422 INFO:tasks.cephadm:Deploying osd.10 on smithi077 with /dev/vg_nvme/lv_1... 2024-03-18T14:49:20.422 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T14:49:20.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:20 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:20.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:20 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:20.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:20 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:49:20.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:20 smithi076 ceph-mon[32205]: from='osd.9 [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T14:49:20.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:20 smithi076 ceph-mon[32205]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T14:49:20.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:20 smithi076 ceph-mon[32205]: from='osd.9 [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-18T14:49:20.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:20 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:20.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:20 smithi076 ceph-mon[32205]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:20.668 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T14:49:20.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:20 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:20 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:20 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:49:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:20 smithi077 ceph-mon[35865]: from='osd.9 [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T14:49:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:20 smithi077 ceph-mon[35865]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T14:49:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:20 smithi077 ceph-mon[35865]: from='osd.9 [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-18T14:49:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:20 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:20.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:20 smithi077 ceph-mon[35865]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:20.919 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 14:49:20 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T14:49:20.699+0000 7f6c93006700 -1 osd.9 0 waiting for initial osdmap 2024-03-18T14:49:20.920 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 14:49:20 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T14:49:20.710+0000 7f6c8edf9700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:49:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:20 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:20 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:20 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:49:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:20 smithi174 ceph-mon[36787]: from='osd.9 [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T14:49:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:20 smithi174 ceph-mon[36787]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T14:49:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:20 smithi174 ceph-mon[36787]: from='osd.9 [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-18T14:49:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:20 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:20 smithi174 ceph-mon[36787]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:21.717 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:21 smithi077 ceph-mon[35865]: from='osd.9 [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-18T14:49:21.717 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:21 smithi077 ceph-mon[35865]: osdmap e67: 10 total, 9 up, 10 in 2024-03-18T14:49:21.717 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:21 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:21.717 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:21 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:21.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:21 smithi076 ceph-mon[32205]: from='osd.9 [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-18T14:49:21.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:21 smithi076 ceph-mon[32205]: osdmap e67: 10 total, 9 up, 10 in 2024-03-18T14:49:21.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:21 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:21.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:21 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:21 smithi174 ceph-mon[36787]: from='osd.9 [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-18T14:49:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:21 smithi174 ceph-mon[36787]: osdmap e67: 10 total, 9 up, 10 in 2024-03-18T14:49:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:21 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:21 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:22.601 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T14:49:22.601 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T14:49:22.601 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-03-18T14:49:22.601 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-03-18T14:49:22.601 INFO:teuthology.orchestra.run.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113478 s, 924 MB/s 2024-03-18T14:49:22.601 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-03-18T14:49:22.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:22 smithi076 ceph-mon[32205]: purged_snaps scrub starts 2024-03-18T14:49:22.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:22 smithi076 ceph-mon[32205]: purged_snaps scrub ok 2024-03-18T14:49:22.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:22.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:22 smithi076 ceph-mon[32205]: osd.9 [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663] boot 2024-03-18T14:49:22.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:22 smithi076 ceph-mon[32205]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T14:49:22.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:22.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:22 smithi076 ceph-mon[32205]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:22.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:22 smithi174 ceph-mon[36787]: purged_snaps scrub starts 2024-03-18T14:49:22.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:22 smithi174 ceph-mon[36787]: purged_snaps scrub ok 2024-03-18T14:49:22.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:22 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:22.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:22 smithi174 ceph-mon[36787]: osd.9 [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663] boot 2024-03-18T14:49:22.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:22 smithi174 ceph-mon[36787]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T14:49:22.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:22 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:22.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:22 smithi174 ceph-mon[36787]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:22.975 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:22 smithi077 ceph-mon[35865]: purged_snaps scrub starts 2024-03-18T14:49:22.975 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:22 smithi077 ceph-mon[35865]: purged_snaps scrub ok 2024-03-18T14:49:22.975 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:22 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:22.975 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:22 smithi077 ceph-mon[35865]: osd.9 [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663] boot 2024-03-18T14:49:22.975 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:22 smithi077 ceph-mon[35865]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T14:49:22.975 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:22 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:49:22.975 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:22 smithi077 ceph-mon[35865]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-18T14:49:23.306 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi077:vg_nvme/lv_1 2024-03-18T14:49:23.549 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T14:49:23.888 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:23 smithi077 ceph-mon[35865]: osdmap e69: 10 total, 10 up, 10 in 2024-03-18T14:49:23.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:23 smithi174 ceph-mon[36787]: osdmap e69: 10 total, 10 up, 10 in 2024-03-18T14:49:24.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:23 smithi076 ceph-mon[32205]: osdmap e69: 10 total, 10 up, 10 in 2024-03-18T14:49:25.046 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:24 smithi077 ceph-mon[35865]: Detected new or changed devices on smithi076 2024-03-18T14:49:25.047 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:25.047 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:25.047 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:25.047 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:25.047 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:25.047 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:25.047 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:24 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi076 to 4272M 2024-03-18T14:49:25.047 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:25.047 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:25.047 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:49:25.047 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:25.047 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:24 smithi077 ceph-mon[35865]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:25.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.146+0000 7fdbead67700 1 -- 172.21.15.77:0/3646987353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbe4103ea0 msgr2=0x7fdbe410a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:49:25.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.146+0000 7fdbead67700 1 --2- 172.21.15.77:0/3646987353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbe4103ea0 0x7fdbe410a750 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00b0d0 tx=0x7fdbdc00b3e0 comp rx=0 tx=0).stop 2024-03-18T14:49:25.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.146+0000 7fdbead67700 1 -- 172.21.15.77:0/3646987353 shutdown_connections 2024-03-18T14:49:25.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.146+0000 7fdbead67700 1 --2- 172.21.15.77:0/3646987353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbe4103ea0 0x7fdbe410a750 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.146+0000 7fdbead67700 1 --2- 172.21.15.77:0/3646987353 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbe4102300 0x7fdbe4102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.146+0000 7fdbead67700 1 --2- 172.21.15.77:0/3646987353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbe4103500 0x7fdbe4103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.146+0000 7fdbead67700 1 -- 172.21.15.77:0/3646987353 >> 172.21.15.77:0/3646987353 conn(0x7fdbe40fd8e0 msgr2=0x7fdbe40ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:49:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.146+0000 7fdbead67700 1 -- 172.21.15.77:0/3646987353 shutdown_connections 2024-03-18T14:49:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.146+0000 7fdbead67700 1 -- 172.21.15.77:0/3646987353 wait complete. 2024-03-18T14:49:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.147+0000 7fdbead67700 1 Processor -- start 2024-03-18T14:49:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.147+0000 7fdbead67700 1 -- start start 2024-03-18T14:49:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.147+0000 7fdbead67700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbe4102300 0x7fdbe4076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:49:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.147+0000 7fdbead67700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbe4103500 0x7fdbe40746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:49:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.147+0000 7fdbead67700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbe4103ea0 0x7fdbe4074c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:49:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.147+0000 7fdbead67700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe410dfe0 con 0x7fdbe4102300 2024-03-18T14:49:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.147+0000 7fdbead67700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe410de60 con 0x7fdbe4103500 2024-03-18T14:49:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.147+0000 7fdbead67700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe410e160 con 0x7fdbe4103ea0 2024-03-18T14:49:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.147+0000 7fdbe3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbe4103500 0x7fdbe40746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:49:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.147+0000 7fdbe3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbe4103500 0x7fdbe40746c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47780/0 (socket says 172.21.15.77:47780) 2024-03-18T14:49:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.147+0000 7fdbe3fff700 1 -- 172.21.15.77:0/2588918099 learned_addr learned my addr 172.21.15.77:0/2588918099 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-18T14:49:25.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.148+0000 7fdbe9304700 1 --2- 172.21.15.77:0/2588918099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbe4103ea0 0x7fdbe4074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:49:25.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.148+0000 7fdbe8b03700 1 --2- 172.21.15.77:0/2588918099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbe4102300 0x7fdbe4076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:49:25.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.148+0000 7fdbe3fff700 1 -- 172.21.15.77:0/2588918099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbe4103ea0 msgr2=0x7fdbe4074c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:49:25.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.148+0000 7fdbe3fff700 1 --2- 172.21.15.77:0/2588918099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbe4103ea0 0x7fdbe4074c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:25.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.148+0000 7fdbe3fff700 1 -- 172.21.15.77:0/2588918099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbe4102300 msgr2=0x7fdbe4076070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:49:25.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.148+0000 7fdbe3fff700 1 --2- 172.21.15.77:0/2588918099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbe4102300 0x7fdbe4076070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:25.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.148+0000 7fdbe3fff700 1 -- 172.21.15.77:0/2588918099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe40754b0 con 0x7fdbe4103500 2024-03-18T14:49:25.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.148+0000 7fdbe9304700 1 --2- 172.21.15.77:0/2588918099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbe4103ea0 0x7fdbe4074c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:49:25.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.148+0000 7fdbe3fff700 1 --2- 172.21.15.77:0/2588918099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbe4103500 0x7fdbe40746c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdbd000b530 tx=0x7fdbd000b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:49:25.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.148+0000 7fdbe1ffb700 1 -- 172.21.15.77:0/2588918099 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0015020 con 0x7fdbe4103500 2024-03-18T14:49:25.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.148+0000 7fdbead67700 1 -- 172.21.15.77:0/2588918099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbe41a27b0 con 0x7fdbe4103500 2024-03-18T14:49:25.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.148+0000 7fdbe1ffb700 1 -- 172.21.15.77:0/2588918099 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbd0011d10 con 0x7fdbe4103500 2024-03-18T14:49:25.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.148+0000 7fdbe1ffb700 1 -- 172.21.15.77:0/2588918099 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0010e50 con 0x7fdbe4103500 2024-03-18T14:49:25.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.148+0000 7fdbead67700 1 -- 172.21.15.77:0/2588918099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbe41a2cf0 con 0x7fdbe4103500 2024-03-18T14:49:25.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.149+0000 7fdbead67700 1 -- 172.21.15.77:0/2588918099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbe4068f90 con 0x7fdbe4103500 2024-03-18T14:49:25.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.150+0000 7fdbe1ffb700 1 -- 172.21.15.77:0/2588918099 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fdbd0018020 con 0x7fdbe4103500 2024-03-18T14:49:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.151+0000 7fdbe1ffb700 1 --2- 172.21.15.77:0/2588918099 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fdbd40a2a30 0x7fdbd40a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:49:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.151+0000 7fdbe1ffb700 1 -- 172.21.15.77:0/2588918099 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7fdbd00cdb20 con 0x7fdbe4103500 2024-03-18T14:49:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.151+0000 7fdbe8b03700 1 --2- 172.21.15.77:0/2588918099 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fdbd40a2a30 0x7fdbd40a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:49:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.152+0000 7fdbe8b03700 1 --2- 172.21.15.77:0/2588918099 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fdbd40a2a30 0x7fdbd40a4ef0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8006fd0 tx=0x7fdbd8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:49:25.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.156+0000 7fdbe1ffb700 1 -- 172.21.15.77:0/2588918099 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdbd001f020 con 0x7fdbe4103500 2024-03-18T14:49:25.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:24 smithi076 ceph-mon[32205]: Detected new or changed devices on smithi076 2024-03-18T14:49:25.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:25.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:25.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:25.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:25.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:25.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:25.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:24 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi076 to 4272M 2024-03-18T14:49:25.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:25.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:25.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:49:25.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:25.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:24 smithi076 ceph-mon[32205]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:25.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:24 smithi174 ceph-mon[36787]: Detected new or changed devices on smithi076 2024-03-18T14:49:25.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:25.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:25.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:25.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:25.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:25.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:25.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:24 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi076 to 4272M 2024-03-18T14:49:25.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:25.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:25.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:49:25.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:25.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:24 smithi174 ceph-mon[36787]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:25.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:25.358+0000 7fdbead67700 1 -- 172.21.15.77:0/2588918099 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fdbe4062710 con 0x7fdbd40a2a30 2024-03-18T14:49:26.055 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:25 smithi077 ceph-mon[35865]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:49:26.056 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:25 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:49:26.056 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:25 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:49:26.056 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:25 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:26.123 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:25 smithi076 ceph-mon[32205]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:49:26.123 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:25 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:49:26.123 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:25 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:49:26.123 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:25 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:26.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:25 smithi174 ceph-mon[36787]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:49:26.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:25 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:49:26.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:25 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:49:26.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:25 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:26.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:26 smithi077 ceph-mon[35865]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:27.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:26 smithi076 ceph-mon[32205]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:27.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:26 smithi174 ceph-mon[36787]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:28.323 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:28 smithi174 ceph-mon[36787]: from='client.? 172.21.15.77:0/3764667151' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee1f73e8-94b3-4969-9944-da77e7f46bec"}]: dispatch 2024-03-18T14:49:28.323 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:28 smithi174 ceph-mon[36787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee1f73e8-94b3-4969-9944-da77e7f46bec"}]: dispatch 2024-03-18T14:49:28.323 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:28 smithi174 ceph-mon[36787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee1f73e8-94b3-4969-9944-da77e7f46bec"}]': finished 2024-03-18T14:49:28.323 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:28 smithi174 ceph-mon[36787]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T14:49:28.323 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:28 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:49:28.327 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:28 smithi077 ceph-mon[35865]: from='client.? 172.21.15.77:0/3764667151' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee1f73e8-94b3-4969-9944-da77e7f46bec"}]: dispatch 2024-03-18T14:49:28.327 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:28 smithi077 ceph-mon[35865]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee1f73e8-94b3-4969-9944-da77e7f46bec"}]: dispatch 2024-03-18T14:49:28.327 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:28 smithi077 ceph-mon[35865]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee1f73e8-94b3-4969-9944-da77e7f46bec"}]': finished 2024-03-18T14:49:28.327 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:28 smithi077 ceph-mon[35865]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T14:49:28.327 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:28 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:49:28.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:28 smithi076 ceph-mon[32205]: from='client.? 172.21.15.77:0/3764667151' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee1f73e8-94b3-4969-9944-da77e7f46bec"}]: dispatch 2024-03-18T14:49:28.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:28 smithi076 ceph-mon[32205]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee1f73e8-94b3-4969-9944-da77e7f46bec"}]: dispatch 2024-03-18T14:49:28.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:28 smithi076 ceph-mon[32205]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee1f73e8-94b3-4969-9944-da77e7f46bec"}]': finished 2024-03-18T14:49:28.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:28 smithi076 ceph-mon[32205]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T14:49:28.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:28 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:49:29.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:29 smithi174 ceph-mon[36787]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:29.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:29 smithi174 ceph-mon[36787]: from='client.? 172.21.15.77:0/3761936106' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:49:29.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:29 smithi076 ceph-mon[32205]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:29.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:29 smithi076 ceph-mon[32205]: from='client.? 172.21.15.77:0/3761936106' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:49:29.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:29 smithi077 ceph-mon[35865]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:29.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:29 smithi077 ceph-mon[35865]: from='client.? 172.21.15.77:0/3761936106' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:49:31.621 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:31 smithi077 ceph-mon[35865]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:31.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:31 smithi076 ceph-mon[32205]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:31.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:31 smithi174 ceph-mon[36787]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:33.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:33 smithi076 ceph-mon[32205]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:33.681 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:33 smithi077 ceph-mon[35865]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:33.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:33 smithi174 ceph-mon[36787]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:35.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:35 smithi076 ceph-mon[32205]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:35.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:35 smithi174 ceph-mon[36787]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:35.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:35 smithi077 ceph-mon[35865]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:36.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:36 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T14:49:36.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:36 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:36.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:36 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T14:49:36.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:36 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:36.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:36 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T14:49:36.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:36 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:37.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:37 smithi076 ceph-mon[32205]: Deploying daemon osd.10 on smithi077 2024-03-18T14:49:37.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:37 smithi076 ceph-mon[32205]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:37.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:37 smithi174 ceph-mon[36787]: Deploying daemon osd.10 on smithi077 2024-03-18T14:49:37.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:37 smithi174 ceph-mon[36787]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:37 smithi077 ceph-mon[35865]: Deploying daemon osd.10 on smithi077 2024-03-18T14:49:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:37 smithi077 ceph-mon[35865]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:39.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:39 smithi077 ceph-mon[35865]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:39.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:39 smithi076 ceph-mon[32205]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:39.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:39 smithi174 ceph-mon[36787]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:41.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:41 smithi076 ceph-mon[32205]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:41.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:41 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:41.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:41 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:41.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:41 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:49:41.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:41 smithi174 ceph-mon[36787]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:41.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:41 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:41.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:41 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:41.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:41 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:49:41.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:41 smithi077 ceph-mon[35865]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:41.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:41 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:41.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:41 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:41.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:41 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:49:43.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:43 smithi076 ceph-mon[32205]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:43.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:43 smithi174 ceph-mon[36787]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:43.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:43 smithi077 ceph-mon[35865]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:44.276 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 10 on host 'smithi077' 2024-03-18T14:49:44.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:44.275+0000 7fdbe1ffb700 1 -- 172.21.15.77:0/2588918099 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fdbe4062710 con 0x7fdbd40a2a30 2024-03-18T14:49:44.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:44.277+0000 7fdbead67700 1 -- 172.21.15.77:0/2588918099 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fdbd40a2a30 msgr2=0x7fdbd40a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:49:44.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:44.277+0000 7fdbead67700 1 --2- 172.21.15.77:0/2588918099 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fdbd40a2a30 0x7fdbd40a4ef0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8006fd0 tx=0x7fdbd8008040 comp rx=0 tx=0).stop 2024-03-18T14:49:44.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:44.277+0000 7fdbead67700 1 -- 172.21.15.77:0/2588918099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbe4103500 msgr2=0x7fdbe40746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:49:44.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:44.277+0000 7fdbead67700 1 --2- 172.21.15.77:0/2588918099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbe4103500 0x7fdbe40746c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdbd000b530 tx=0x7fdbd000b8f0 comp rx=0 tx=0).stop 2024-03-18T14:49:44.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:44.277+0000 7fdbead67700 1 -- 172.21.15.77:0/2588918099 shutdown_connections 2024-03-18T14:49:44.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:44.277+0000 7fdbead67700 1 --2- 172.21.15.77:0/2588918099 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fdbd40a2a30 0x7fdbd40a4ef0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:44.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:44.277+0000 7fdbead67700 1 --2- 172.21.15.77:0/2588918099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbe4103ea0 0x7fdbe4074c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:44.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:44.277+0000 7fdbead67700 1 --2- 172.21.15.77:0/2588918099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbe4102300 0x7fdbe4076070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:44.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:44.277+0000 7fdbead67700 1 --2- 172.21.15.77:0/2588918099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbe4103500 0x7fdbe40746c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:44.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:44.277+0000 7fdbead67700 1 -- 172.21.15.77:0/2588918099 >> 172.21.15.77:0/2588918099 conn(0x7fdbe40fd8e0 msgr2=0x7fdbe4106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:49:44.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:44.278+0000 7fdbead67700 1 -- 172.21.15.77:0/2588918099 shutdown_connections 2024-03-18T14:49:44.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-18T14:49:44.278+0000 7fdbead67700 1 -- 172.21.15.77:0/2588918099 wait complete. 2024-03-18T14:49:45.164 DEBUG:teuthology.orchestra.run.smithi077:osd.10> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.10.service 2024-03-18T14:49:45.166 INFO:tasks.cephadm:Deploying osd.11 on smithi174 with /dev/vg_nvme/lv_1... 2024-03-18T14:49:45.167 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T14:49:45.416 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T14:49:45.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:45 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:45.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:45 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:45.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:45 smithi174 ceph-mon[36787]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:45.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:45 smithi174 ceph-mon[36787]: from='osd.10 [v2:172.21.15.77:6824/4105457096,v1:172.21.15.77:6825/4105457096]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T14:49:45.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:45 smithi174 ceph-mon[36787]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T14:49:45.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:45 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:45.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:45 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:45.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:45 smithi077 ceph-mon[35865]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:45.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:45 smithi077 ceph-mon[35865]: from='osd.10 [v2:172.21.15.77:6824/4105457096,v1:172.21.15.77:6825/4105457096]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T14:49:45.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:45 smithi077 ceph-mon[35865]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T14:49:45.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:45 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:45.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:45 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:45.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:45 smithi076 ceph-mon[32205]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:45.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:45 smithi076 ceph-mon[32205]: from='osd.10 [v2:172.21.15.77:6824/4105457096,v1:172.21.15.77:6825/4105457096]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T14:49:45.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:45 smithi076 ceph-mon[32205]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T14:49:46.326 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:46 smithi077 ceph-mon[35865]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T14:49:46.326 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:46 smithi077 ceph-mon[35865]: from='osd.10 [v2:172.21.15.77:6824/4105457096,v1:172.21.15.77:6825/4105457096]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:49:46.326 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:46 smithi077 ceph-mon[35865]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T14:49:46.326 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:46 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:49:46.326 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:46 smithi077 ceph-mon[35865]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:49:46.326 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:46 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:46.326 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:46 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:46.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:46 smithi076 ceph-mon[32205]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T14:49:46.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:46 smithi076 ceph-mon[32205]: from='osd.10 [v2:172.21.15.77:6824/4105457096,v1:172.21.15.77:6825/4105457096]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:49:46.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:46 smithi076 ceph-mon[32205]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T14:49:46.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:46 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:49:46.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:46 smithi076 ceph-mon[32205]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:49:46.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:46 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:46.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:46 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:46.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:46 smithi174 ceph-mon[36787]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T14:49:46.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:46 smithi174 ceph-mon[36787]: from='osd.10 [v2:172.21.15.77:6824/4105457096,v1:172.21.15.77:6825/4105457096]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:49:46.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:46 smithi174 ceph-mon[36787]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T14:49:46.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:46 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:49:46.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:46 smithi174 ceph-mon[36787]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-18T14:49:46.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:46 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:46.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:46 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:46.973 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 14:49:46 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T14:49:46.657+0000 7fe64fa1e700 -1 osd.10 0 waiting for initial osdmap 2024-03-18T14:49:46.974 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 14:49:46 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T14:49:46.670+0000 7fe647809700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:49:47.430 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:47 smithi174 ceph-mon[36787]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-18T14:49:47.430 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:47 smithi174 ceph-mon[36787]: osdmap e72: 11 total, 10 up, 11 in 2024-03-18T14:49:47.430 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:47 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:49:47.430 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:47 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:49:47.430 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:47 smithi174 ceph-mon[36787]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:47.430 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:47 smithi174 ceph-mon[36787]: from='osd.10 ' entity='osd.10' 2024-03-18T14:49:47.430 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:47 smithi174 ceph-mon[36787]: osd.10 [v2:172.21.15.77:6824/4105457096,v1:172.21.15.77:6825/4105457096] boot 2024-03-18T14:49:47.431 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:47 smithi174 ceph-mon[36787]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T14:49:47.431 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:47 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:49:47.511 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:47 smithi077 ceph-mon[35865]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-18T14:49:47.511 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:47 smithi077 ceph-mon[35865]: osdmap e72: 11 total, 10 up, 11 in 2024-03-18T14:49:47.511 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:47 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:49:47.511 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:47 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:49:47.511 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:47 smithi077 ceph-mon[35865]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:47.512 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:47 smithi077 ceph-mon[35865]: from='osd.10 ' entity='osd.10' 2024-03-18T14:49:47.512 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:47 smithi077 ceph-mon[35865]: osd.10 [v2:172.21.15.77:6824/4105457096,v1:172.21.15.77:6825/4105457096] boot 2024-03-18T14:49:47.512 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:47 smithi077 ceph-mon[35865]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T14:49:47.512 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:47 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:49:47.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:47 smithi076 ceph-mon[32205]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-18T14:49:47.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:47 smithi076 ceph-mon[32205]: osdmap e72: 11 total, 10 up, 11 in 2024-03-18T14:49:47.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:47 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:49:47.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:47 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:49:47.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:47 smithi076 ceph-mon[32205]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-18T14:49:47.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:47 smithi076 ceph-mon[32205]: from='osd.10 ' entity='osd.10' 2024-03-18T14:49:47.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:47 smithi076 ceph-mon[32205]: osd.10 [v2:172.21.15.77:6824/4105457096,v1:172.21.15.77:6825/4105457096] boot 2024-03-18T14:49:47.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:47 smithi076 ceph-mon[32205]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T14:49:47.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:47 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:49:47.729 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T14:49:47.729 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T14:49:47.729 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-03-18T14:49:47.729 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-03-18T14:49:47.729 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112236 s, 934 MB/s 2024-03-18T14:49:47.729 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-03-18T14:49:48.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:48 smithi174 ceph-mon[36787]: purged_snaps scrub starts 2024-03-18T14:49:48.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:48 smithi174 ceph-mon[36787]: purged_snaps scrub ok 2024-03-18T14:49:48.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:48 smithi174 ceph-mon[36787]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T14:49:48.401 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:48 smithi077 ceph-mon[35865]: purged_snaps scrub starts 2024-03-18T14:49:48.401 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:48 smithi077 ceph-mon[35865]: purged_snaps scrub ok 2024-03-18T14:49:48.401 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:48 smithi077 ceph-mon[35865]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T14:49:48.472 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi174:vg_nvme/lv_1 2024-03-18T14:49:48.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:48 smithi076 ceph-mon[32205]: purged_snaps scrub starts 2024-03-18T14:49:48.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:48 smithi076 ceph-mon[32205]: purged_snaps scrub ok 2024-03-18T14:49:48.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:48 smithi076 ceph-mon[32205]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T14:49:48.709 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T14:49:49.428 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:49 smithi174 ceph-mon[36787]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:49:49.428 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:49.428 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:49.428 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:49.428 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:49.428 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:49.428 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:49.428 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:49.428 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:49.428 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:49:49.429 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:49.429 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:49:49.429 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:49.429 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:49:49.429 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:49.429 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:49 smithi174 ceph-mon[36787]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T14:49:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:49 smithi076 ceph-mon[32205]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:49:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:49.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:49.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:49.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:49.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:49.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:49.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:49:49.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:49.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:49:49.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:49.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:49:49.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:49.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:49 smithi076 ceph-mon[32205]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T14:49:49.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:49 smithi077 ceph-mon[35865]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:49:49.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:49.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:49.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:49.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:49.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:49:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:49:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:49:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:49:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:49:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:49 smithi077 ceph-mon[35865]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T14:49:50.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.303+0000 7fdebeb34700 1 -- 172.21.15.174:0/3784517391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdeb81034f0 msgr2=0x7fdeb8103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:49:50.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.303+0000 7fdebeb34700 1 --2- 172.21.15.174:0/3784517391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdeb81034f0 0x7fdeb8103950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdea4009990 tx=0x7fdea4009ca0 comp rx=0 tx=0).stop 2024-03-18T14:49:50.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebeb34700 1 -- 172.21.15.174:0/3784517391 shutdown_connections 2024-03-18T14:49:50.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebeb34700 1 --2- 172.21.15.174:0/3784517391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeb8103e90 0x7fdeb810a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:50.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebeb34700 1 --2- 172.21.15.174:0/3784517391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdeb81022f0 0x7fdeb8102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:50.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebeb34700 1 --2- 172.21.15.174:0/3784517391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdeb81034f0 0x7fdeb8103950 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:50.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebeb34700 1 -- 172.21.15.174:0/3784517391 >> 172.21.15.174:0/3784517391 conn(0x7fdeb80fd8b0 msgr2=0x7fdeb80ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:49:50.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebeb34700 1 -- 172.21.15.174:0/3784517391 shutdown_connections 2024-03-18T14:49:50.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebeb34700 1 -- 172.21.15.174:0/3784517391 wait complete. 2024-03-18T14:49:50.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebeb34700 1 Processor -- start 2024-03-18T14:49:50.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebeb34700 1 -- start start 2024-03-18T14:49:50.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebeb34700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdeb81022f0 0x7fdeb8076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:49:50.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebeb34700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdeb81034f0 0x7fdeb80746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:49:50.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebeb34700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeb8103e90 0x7fdeb8074c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:49:50.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebeb34700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb810df50 con 0x7fdeb81034f0 2024-03-18T14:49:50.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebeb34700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb810ddd0 con 0x7fdeb8103e90 2024-03-18T14:49:50.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebeb34700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb810e0d0 con 0x7fdeb81022f0 2024-03-18T14:49:50.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebc8d0700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdeb81022f0 0x7fdeb8076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:49:50.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebc8d0700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdeb81022f0 0x7fdeb8076070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:59558/0 (socket says 172.21.15.174:59558) 2024-03-18T14:49:50.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebc8d0700 1 -- 172.21.15.174:0/622822426 learned_addr learned my addr 172.21.15.174:0/622822426 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-18T14:49:50.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.304+0000 7fdebd0d1700 1 --2- 172.21.15.174:0/622822426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeb8103e90 0x7fdeb8074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:49:50.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.305+0000 7fdeb7fff700 1 --2- 172.21.15.174:0/622822426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdeb81034f0 0x7fdeb80746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:49:50.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.305+0000 7fdebc8d0700 1 -- 172.21.15.174:0/622822426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeb8103e90 msgr2=0x7fdeb8074c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:49:50.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.305+0000 7fdebc8d0700 1 --2- 172.21.15.174:0/622822426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeb8103e90 0x7fdeb8074c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:50.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.305+0000 7fdebc8d0700 1 -- 172.21.15.174:0/622822426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdeb81034f0 msgr2=0x7fdeb80746c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:49:50.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.305+0000 7fdebc8d0700 1 --2- 172.21.15.174:0/622822426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdeb81034f0 0x7fdeb80746c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:49:50.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.305+0000 7fdebc8d0700 1 -- 172.21.15.174:0/622822426 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb80754b0 con 0x7fdeb81022f0 2024-03-18T14:49:50.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.305+0000 7fdebc8d0700 1 --2- 172.21.15.174:0/622822426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdeb81022f0 0x7fdeb8076070 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00b500 tx=0x7fdeac00b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:49:50.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.305+0000 7fdeb5ffb700 1 -- 172.21.15.174:0/622822426 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac015020 con 0x7fdeb81022f0 2024-03-18T14:49:50.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.305+0000 7fdebeb34700 1 -- 172.21.15.174:0/622822426 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeb81a26a0 con 0x7fdeb81022f0 2024-03-18T14:49:50.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.305+0000 7fdeb5ffb700 1 -- 172.21.15.174:0/622822426 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdeac011d10 con 0x7fdeb81022f0 2024-03-18T14:49:50.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.305+0000 7fdeb5ffb700 1 -- 172.21.15.174:0/622822426 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac010e50 con 0x7fdeb81022f0 2024-03-18T14:49:50.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.305+0000 7fdebeb34700 1 -- 172.21.15.174:0/622822426 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeb81a2c60 con 0x7fdeb81022f0 2024-03-18T14:49:50.309 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:50 smithi174 ceph-mon[36787]: Detected new or changed devices on smithi077 2024-03-18T14:49:50.309 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:50 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi077 to 4270M 2024-03-18T14:49:50.309 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:50 smithi174 ceph-mon[36787]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T14:49:50.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.306+0000 7fdebeb34700 1 -- 172.21.15.174:0/622822426 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeb8068f90 con 0x7fdeb81022f0 2024-03-18T14:49:50.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.307+0000 7fdeb5ffb700 1 -- 172.21.15.174:0/622822426 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fdeac018020 con 0x7fdeb81022f0 2024-03-18T14:49:50.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.308+0000 7fdeb5ffb700 1 --2- 172.21.15.174:0/622822426 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fdea80a2a30 0x7fdea80a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:49:50.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.308+0000 7fdeb5ffb700 1 -- 172.21.15.174:0/622822426 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6694+0+0 (secure 0 0 0) 0x7fdeac0cdba0 con 0x7fdeb81022f0 2024-03-18T14:49:50.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.308+0000 7fdeb7fff700 1 --2- 172.21.15.174:0/622822426 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fdea80a2a30 0x7fdea80a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:49:50.311 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.308+0000 7fdeb7fff700 1 --2- 172.21.15.174:0/622822426 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fdea80a2a30 0x7fdea80a4ef0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdea40039c0 tx=0x7fdea4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:49:50.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.312+0000 7fdeb5ffb700 1 -- 172.21.15.174:0/622822426 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdeac01f020 con 0x7fdeb81022f0 2024-03-18T14:49:50.517 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:49:50.516+0000 7fdebeb34700 1 -- 172.21.15.174:0/622822426 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fdeb8062710 con 0x7fdea80a2a30 2024-03-18T14:49:50.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:50 smithi076 ceph-mon[32205]: Detected new or changed devices on smithi077 2024-03-18T14:49:50.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:50 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi077 to 4270M 2024-03-18T14:49:50.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:50 smithi076 ceph-mon[32205]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T14:49:50.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:50 smithi077 ceph-mon[35865]: Detected new or changed devices on smithi077 2024-03-18T14:49:50.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:50 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi077 to 4270M 2024-03-18T14:49:50.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:50 smithi077 ceph-mon[35865]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T14:49:51.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:51 smithi174 ceph-mon[36787]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:49:51.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:51 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:49:51.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:51 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:49:51.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:51 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:51.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:51 smithi174 ceph-mon[36787]: osdmap e77: 11 total, 11 up, 11 in 2024-03-18T14:49:51.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:51 smithi076 ceph-mon[32205]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:49:51.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:51 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:49:51.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:51 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:49:51.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:51 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:51.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:51 smithi076 ceph-mon[32205]: osdmap e77: 11 total, 11 up, 11 in 2024-03-18T14:49:51.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:51 smithi077 ceph-mon[35865]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:49:51.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:51 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T14:49:51.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:51 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T14:49:51.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:51 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:49:51.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:51 smithi077 ceph-mon[35865]: osdmap e77: 11 total, 11 up, 11 in 2024-03-18T14:49:52.416 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:52 smithi174 ceph-mon[36787]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:49:52.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:52 smithi076 ceph-mon[32205]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:49:52.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:52 smithi077 ceph-mon[35865]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:49:53.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:53 smithi076 ceph-mon[32205]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-18T14:49:53.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:53 smithi174 ceph-mon[36787]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-18T14:49:53.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:53 smithi077 ceph-mon[35865]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-18T14:49:54.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:54 smithi076 ceph-mon[32205]: from='client.? 172.21.15.174:0/1359123579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35839388-3d56-497b-a49c-0db8686e3a97"}]: dispatch 2024-03-18T14:49:54.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:54 smithi076 ceph-mon[32205]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35839388-3d56-497b-a49c-0db8686e3a97"}]: dispatch 2024-03-18T14:49:54.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:54 smithi076 ceph-mon[32205]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35839388-3d56-497b-a49c-0db8686e3a97"}]': finished 2024-03-18T14:49:54.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:54 smithi076 ceph-mon[32205]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T14:49:54.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:54 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:49:54.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:54 smithi076 ceph-mon[32205]: from='client.? 172.21.15.174:0/3646858597' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:49:54.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:54 smithi174 ceph-mon[36787]: from='client.? 172.21.15.174:0/1359123579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35839388-3d56-497b-a49c-0db8686e3a97"}]: dispatch 2024-03-18T14:49:54.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:54 smithi174 ceph-mon[36787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35839388-3d56-497b-a49c-0db8686e3a97"}]: dispatch 2024-03-18T14:49:54.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:54 smithi174 ceph-mon[36787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35839388-3d56-497b-a49c-0db8686e3a97"}]': finished 2024-03-18T14:49:54.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:54 smithi174 ceph-mon[36787]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T14:49:54.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:54 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:49:54.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:54 smithi174 ceph-mon[36787]: from='client.? 172.21.15.174:0/3646858597' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:49:54.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:54 smithi077 ceph-mon[35865]: from='client.? 172.21.15.174:0/1359123579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35839388-3d56-497b-a49c-0db8686e3a97"}]: dispatch 2024-03-18T14:49:54.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:54 smithi077 ceph-mon[35865]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35839388-3d56-497b-a49c-0db8686e3a97"}]: dispatch 2024-03-18T14:49:54.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:54 smithi077 ceph-mon[35865]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35839388-3d56-497b-a49c-0db8686e3a97"}]': finished 2024-03-18T14:49:54.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:54 smithi077 ceph-mon[35865]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T14:49:54.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:54 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:49:54.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:54 smithi077 ceph-mon[35865]: from='client.? 172.21.15.174:0/3646858597' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T14:49:55.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:55 smithi076 ceph-mon[32205]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-18T14:49:55.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:55 smithi174 ceph-mon[36787]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-18T14:49:55.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:55 smithi077 ceph-mon[35865]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-18T14:49:57.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:57 smithi076 ceph-mon[32205]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-18T14:49:57.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:57 smithi174 ceph-mon[36787]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-18T14:49:57.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:57 smithi077 ceph-mon[35865]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-18T14:49:59.590 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:49:59 smithi174 ceph-mon[36787]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T14:49:59.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:49:59 smithi076 ceph-mon[32205]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T14:49:59.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:49:59 smithi077 ceph-mon[35865]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T14:50:00.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:00 smithi076 ceph-mon[32205]: overall HEALTH_OK 2024-03-18T14:50:00.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:00 smithi174 ceph-mon[36787]: overall HEALTH_OK 2024-03-18T14:50:00.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:00 smithi077 ceph-mon[35865]: overall HEALTH_OK 2024-03-18T14:50:01.657 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:01 smithi174 ceph-mon[36787]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:01.657 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:01 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T14:50:01.658 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:01 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:01.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:01 smithi076 ceph-mon[32205]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:01.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:01 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T14:50:01.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:01 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:01.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:01 smithi077 ceph-mon[35865]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:01.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:01 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T14:50:01.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:01 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:02.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:02 smithi076 ceph-mon[32205]: Deploying daemon osd.11 on smithi174 2024-03-18T14:50:02.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:02 smithi174 ceph-mon[36787]: Deploying daemon osd.11 on smithi174 2024-03-18T14:50:02.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:02 smithi077 ceph-mon[35865]: Deploying daemon osd.11 on smithi174 2024-03-18T14:50:03.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:03 smithi076 ceph-mon[32205]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:03.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:03 smithi174 ceph-mon[36787]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:03.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:03 smithi077 ceph-mon[35865]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:05.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:05 smithi076 ceph-mon[32205]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:05.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:05 smithi174 ceph-mon[36787]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:05.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:05 smithi077 ceph-mon[35865]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:07.602 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:07 smithi174 ceph-mon[36787]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:07.602 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:07 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:07.602 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:07 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:07.602 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:07 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:50:07.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:07 smithi076 ceph-mon[32205]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:07.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:07 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:07.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:07 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:07.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:07 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:50:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:07 smithi077 ceph-mon[35865]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:07 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:07 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:07 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:50:09.646 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:09 smithi174 ceph-mon[36787]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:09.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:09 smithi076 ceph-mon[32205]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:09.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:09 smithi077 ceph-mon[35865]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:10.991 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:10.990+0000 7fdeb5ffb700 1 -- 172.21.15.174:0/622822426 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fdeb8062710 con 0x7fdea80a2a30 2024-03-18T14:50:10.991 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 11 on host 'smithi174' 2024-03-18T14:50:10.993 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:10.992+0000 7fdebeb34700 1 -- 172.21.15.174:0/622822426 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fdea80a2a30 msgr2=0x7fdea80a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:10.993 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:10.992+0000 7fdebeb34700 1 --2- 172.21.15.174:0/622822426 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fdea80a2a30 0x7fdea80a4ef0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdea40039c0 tx=0x7fdea4005fb0 comp rx=0 tx=0).stop 2024-03-18T14:50:10.993 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:10.992+0000 7fdebeb34700 1 -- 172.21.15.174:0/622822426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdeb81022f0 msgr2=0x7fdeb8076070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:10.994 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:10.992+0000 7fdebeb34700 1 --2- 172.21.15.174:0/622822426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdeb81022f0 0x7fdeb8076070 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00b500 tx=0x7fdeac00b810 comp rx=0 tx=0).stop 2024-03-18T14:50:10.994 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:10.992+0000 7fdebeb34700 1 -- 172.21.15.174:0/622822426 shutdown_connections 2024-03-18T14:50:10.994 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:10.992+0000 7fdebeb34700 1 --2- 172.21.15.174:0/622822426 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fdea80a2a30 0x7fdea80a4ef0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:10.994 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:10.992+0000 7fdebeb34700 1 --2- 172.21.15.174:0/622822426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeb8103e90 0x7fdeb8074c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:10.994 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:10.993+0000 7fdebeb34700 1 --2- 172.21.15.174:0/622822426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdeb81022f0 0x7fdeb8076070 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:10.994 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:10.993+0000 7fdebeb34700 1 --2- 172.21.15.174:0/622822426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdeb81034f0 0x7fdeb80746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:10.994 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:10.993+0000 7fdebeb34700 1 -- 172.21.15.174:0/622822426 >> 172.21.15.174:0/622822426 conn(0x7fdeb80fd8b0 msgr2=0x7fdeb8106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:10.994 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:10.993+0000 7fdebeb34700 1 -- 172.21.15.174:0/622822426 shutdown_connections 2024-03-18T14:50:10.994 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:10.993+0000 7fdebeb34700 1 -- 172.21.15.174:0/622822426 wait complete. 2024-03-18T14:50:11.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:11 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:11.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:11 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:11.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:11 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:11.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:11 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:50:11.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:11 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:11.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:11 smithi174 ceph-mon[36787]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:11.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:11 smithi174 ceph-mon[36787]: from='osd.11 [v2:172.21.15.174:6824/1489039553,v1:172.21.15.174:6825/1489039553]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T14:50:11.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:11 smithi174 ceph-mon[36787]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T14:50:11.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:11 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:11.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:11 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:50:11.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:11 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:11.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:11 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:11.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:11 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:11.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:11 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:11.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:11 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:50:11.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:11 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:11.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:11 smithi076 ceph-mon[32205]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:11.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:11 smithi076 ceph-mon[32205]: from='osd.11 [v2:172.21.15.174:6824/1489039553,v1:172.21.15.174:6825/1489039553]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T14:50:11.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:11 smithi076 ceph-mon[32205]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T14:50:11.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:11 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:11.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:11 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:50:11.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:11 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:11.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:11 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:11.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:11 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:11.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:11 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:11.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:11 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:50:11.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:11 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:11.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:11 smithi077 ceph-mon[35865]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:11.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:11 smithi077 ceph-mon[35865]: from='osd.11 [v2:172.21.15.174:6824/1489039553,v1:172.21.15.174:6825/1489039553]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T14:50:11.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:11 smithi077 ceph-mon[35865]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T14:50:11.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:11 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:11.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:11 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:50:11.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:11 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:11.660 DEBUG:teuthology.orchestra.run.smithi174:osd.11> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.11.service 2024-03-18T14:50:11.662 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-18T14:50:11.662 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-18T14:50:11.916 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:50:12.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:12 smithi174 ceph-mon[36787]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T14:50:12.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:12 smithi174 ceph-mon[36787]: from='osd.11 [v2:172.21.15.174:6824/1489039553,v1:172.21.15.174:6825/1489039553]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:50:12.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:12 smithi174 ceph-mon[36787]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T14:50:12.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:12 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:50:12.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:12 smithi174 ceph-mon[36787]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:50:12.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:12 smithi076 ceph-mon[32205]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T14:50:12.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:12 smithi076 ceph-mon[32205]: from='osd.11 [v2:172.21.15.174:6824/1489039553,v1:172.21.15.174:6825/1489039553]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:50:12.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:12 smithi076 ceph-mon[32205]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T14:50:12.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:12 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:50:12.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:12 smithi076 ceph-mon[32205]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:50:12.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:12 smithi077 ceph-mon[35865]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T14:50:12.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:12 smithi077 ceph-mon[35865]: from='osd.11 [v2:172.21.15.174:6824/1489039553,v1:172.21.15.174:6825/1489039553]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:50:12.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:12 smithi077 ceph-mon[35865]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T14:50:12.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:12 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:50:12.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:12 smithi077 ceph-mon[35865]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-18T14:50:12.517 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 14:50:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T14:50:12.501+0000 7f8984eca700 -1 osd.11 0 waiting for initial osdmap 2024-03-18T14:50:12.518 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 14:50:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T14:50:12.516+0000 7f897ccb5700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:50:13.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:13 smithi174 ceph-mon[36787]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-18T14:50:13.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:13 smithi174 ceph-mon[36787]: osdmap e80: 12 total, 11 up, 12 in 2024-03-18T14:50:13.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:13 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:50:13.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:13 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:50:13.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:13 smithi174 ceph-mon[36787]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:13.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:13 smithi174 ceph-mon[36787]: from='osd.11 ' entity='osd.11' 2024-03-18T14:50:13.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:13 smithi076 ceph-mon[32205]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-18T14:50:13.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:13 smithi076 ceph-mon[32205]: osdmap e80: 12 total, 11 up, 12 in 2024-03-18T14:50:13.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:13 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:50:13.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:13 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:50:13.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:13 smithi076 ceph-mon[32205]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:13.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:13 smithi076 ceph-mon[32205]: from='osd.11 ' entity='osd.11' 2024-03-18T14:50:13.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:13 smithi077 ceph-mon[35865]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-18T14:50:13.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:13 smithi077 ceph-mon[35865]: osdmap e80: 12 total, 11 up, 12 in 2024-03-18T14:50:13.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:13 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:50:13.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:13 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:50:13.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:13 smithi077 ceph-mon[35865]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-18T14:50:13.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:13 smithi077 ceph-mon[35865]: from='osd.11 ' entity='osd.11' 2024-03-18T14:50:13.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.930+0000 7f97aaf57700 1 -- 172.21.15.76:0/4074333602 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a4105760 msgr2=0x7f97a4107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.930+0000 7f97aaf57700 1 --2- 172.21.15.76:0/4074333602 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a4105760 0x7f97a4107b50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9794009990 tx=0x7f9794009ca0 comp rx=0 tx=0).stop 2024-03-18T14:50:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.930+0000 7f97aaf57700 1 -- 172.21.15.76:0/4074333602 shutdown_connections 2024-03-18T14:50:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.930+0000 7f97aaf57700 1 --2- 172.21.15.76:0/4074333602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97a4108090 0x7f97a410a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.930+0000 7f97aaf57700 1 --2- 172.21.15.76:0/4074333602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97a4102e30 0x7f97a4105220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.930+0000 7f97aaf57700 1 --2- 172.21.15.76:0/4074333602 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a4105760 0x7f97a4107b50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.930+0000 7f97aaf57700 1 -- 172.21.15.76:0/4074333602 >> 172.21.15.76:0/4074333602 conn(0x7f97a40fc7c0 msgr2=0x7f97a40febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.930+0000 7f97aaf57700 1 -- 172.21.15.76:0/4074333602 shutdown_connections 2024-03-18T14:50:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.930+0000 7f97aaf57700 1 -- 172.21.15.76:0/4074333602 wait complete. 2024-03-18T14:50:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.930+0000 7f97aaf57700 1 Processor -- start 2024-03-18T14:50:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.930+0000 7f97aaf57700 1 -- start start 2024-03-18T14:50:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f97aaf57700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97a4102e30 0x7f97a4198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f97aaf57700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97a4105760 0x7f97a4198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f97aaf57700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a4108090 0x7f97a419fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f97aaf57700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f97a410df10 con 0x7f97a4108090 2024-03-18T14:50:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f97aaf57700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f97a410dd90 con 0x7f97a4105760 2024-03-18T14:50:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f97aaf57700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f97a410e090 con 0x7f97a4102e30 2024-03-18T14:50:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f97a94f4700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a4108090 0x7f97a419fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f97a94f4700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a4108090 0x7f97a419fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:42272/0 (socket says 172.21.15.76:42272) 2024-03-18T14:50:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f97a8cf3700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97a4102e30 0x7f97a4198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f97a94f4700 1 -- 172.21.15.76:0/3095821064 learned_addr learned my addr 172.21.15.76:0/3095821064 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f979bfff700 1 --2- 172.21.15.76:0/3095821064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97a4105760 0x7f97a4198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f97a94f4700 1 -- 172.21.15.76:0/3095821064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97a4102e30 msgr2=0x7f97a4198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f97a94f4700 1 --2- 172.21.15.76:0/3095821064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97a4102e30 0x7f97a4198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f97a94f4700 1 -- 172.21.15.76:0/3095821064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97a4105760 msgr2=0x7f97a4198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f97a94f4700 1 --2- 172.21.15.76:0/3095821064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97a4105760 0x7f97a4198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f97a94f4700 1 -- 172.21.15.76:0/3095821064 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a41a0170 con 0x7f97a4108090 2024-03-18T14:50:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f97a94f4700 1 --2- 172.21.15.76:0/3095821064 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a4108090 0x7f97a419fad0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f979c007ee0 tx=0x7f979c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f9799ffb700 1 -- 172.21.15.76:0/3095821064 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979c004db0 con 0x7f97a4108090 2024-03-18T14:50:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f97aaf57700 1 -- 172.21.15.76:0/3095821064 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a41a0460 con 0x7f97a4108090 2024-03-18T14:50:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f9799ffb700 1 -- 172.21.15.76:0/3095821064 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f979c005020 con 0x7f97a4108090 2024-03-18T14:50:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.931+0000 7f9799ffb700 1 -- 172.21.15.76:0/3095821064 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979c015660 con 0x7f97a4108090 2024-03-18T14:50:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.932+0000 7f97aaf57700 1 -- 172.21.15.76:0/3095821064 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a41a0950 con 0x7f97a4108090 2024-03-18T14:50:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.933+0000 7f97aaf57700 1 -- 172.21.15.76:0/3095821064 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a4068f90 con 0x7f97a4108090 2024-03-18T14:50:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.934+0000 7f9799ffb700 1 -- 172.21.15.76:0/3095821064 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f979c005560 con 0x7f97a4108090 2024-03-18T14:50:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.935+0000 7f9799ffb700 1 --2- 172.21.15.76:0/3095821064 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f978c0a2890 0x7f978c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.935+0000 7f97a8cf3700 1 --2- 172.21.15.76:0/3095821064 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f978c0a2890 0x7f978c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.935+0000 7f9799ffb700 1 -- 172.21.15.76:0/3095821064 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f979c0cdfd0 con 0x7f97a4108090 2024-03-18T14:50:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.935+0000 7f97a8cf3700 1 --2- 172.21.15.76:0/3095821064 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f978c0a2890 0x7f978c0a4d50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009c60 tx=0x7f97a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:13.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:13.941+0000 7f9799ffb700 1 -- 172.21.15.76:0/3095821064 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f979c010070 con 0x7f97a4108090 2024-03-18T14:50:14.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:14.139+0000 7f97aaf57700 1 -- 172.21.15.76:0/3095821064 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f97a402db70 con 0x7f97a4108090 2024-03-18T14:50:14.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:14.139+0000 7f9799ffb700 1 -- 172.21.15.76:0/3095821064 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f979c09b510 con 0x7f97a4108090 2024-03-18T14:50:14.140 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:14.141+0000 7f97aaf57700 1 -- 172.21.15.76:0/3095821064 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f978c0a2890 msgr2=0x7f978c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:14.141+0000 7f97aaf57700 1 --2- 172.21.15.76:0/3095821064 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f978c0a2890 0x7f978c0a4d50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009c60 tx=0x7f97a0009260 comp rx=0 tx=0).stop 2024-03-18T14:50:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:14.141+0000 7f97aaf57700 1 -- 172.21.15.76:0/3095821064 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a4108090 msgr2=0x7f97a419fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:14.141+0000 7f97aaf57700 1 --2- 172.21.15.76:0/3095821064 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a4108090 0x7f97a419fad0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f979c007ee0 tx=0x7f979c00d3b0 comp rx=0 tx=0).stop 2024-03-18T14:50:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:14.141+0000 7f97aaf57700 1 -- 172.21.15.76:0/3095821064 shutdown_connections 2024-03-18T14:50:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:14.141+0000 7f97aaf57700 1 --2- 172.21.15.76:0/3095821064 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f978c0a2890 0x7f978c0a4d50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:14.141+0000 7f97aaf57700 1 --2- 172.21.15.76:0/3095821064 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a4108090 0x7f97a419fad0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:14.141+0000 7f97aaf57700 1 --2- 172.21.15.76:0/3095821064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97a4102e30 0x7f97a4198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:14.141+0000 7f97aaf57700 1 --2- 172.21.15.76:0/3095821064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97a4105760 0x7f97a4198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:14.141+0000 7f97aaf57700 1 -- 172.21.15.76:0/3095821064 >> 172.21.15.76:0/3095821064 conn(0x7f97a40fc7c0 msgr2=0x7f97a41038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:14.141+0000 7f97aaf57700 1 -- 172.21.15.76:0/3095821064 shutdown_connections 2024-03-18T14:50:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:14.141+0000 7f97aaf57700 1 -- 172.21.15.76:0/3095821064 wait complete. 2024-03-18T14:50:14.186 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:14 smithi076 ceph-mon[32205]: purged_snaps scrub starts 2024-03-18T14:50:14.186 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:14 smithi076 ceph-mon[32205]: purged_snaps scrub ok 2024-03-18T14:50:14.186 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:14 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:50:14.186 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:14 smithi076 ceph-mon[32205]: osd.11 [v2:172.21.15.174:6824/1489039553,v1:172.21.15.174:6825/1489039553] boot 2024-03-18T14:50:14.186 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:14 smithi076 ceph-mon[32205]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T14:50:14.186 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:14 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:50:14.423 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:14 smithi174 ceph-mon[36787]: purged_snaps scrub starts 2024-03-18T14:50:14.424 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:14 smithi174 ceph-mon[36787]: purged_snaps scrub ok 2024-03-18T14:50:14.424 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:14 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:50:14.424 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:14 smithi174 ceph-mon[36787]: osd.11 [v2:172.21.15.174:6824/1489039553,v1:172.21.15.174:6825/1489039553] boot 2024-03-18T14:50:14.424 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:14 smithi174 ceph-mon[36787]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T14:50:14.424 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:14 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:50:14.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:14 smithi077 ceph-mon[35865]: purged_snaps scrub starts 2024-03-18T14:50:14.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:14 smithi077 ceph-mon[35865]: purged_snaps scrub ok 2024-03-18T14:50:14.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:14 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:50:14.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:14 smithi077 ceph-mon[35865]: osd.11 [v2:172.21.15.174:6824/1489039553,v1:172.21.15.174:6825/1489039553] boot 2024-03-18T14:50:14.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:14 smithi077 ceph-mon[35865]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T14:50:14.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:14 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:50:14.951 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1710773413,"num_in_osds":12,"osd_in_since":1710773393,"num_remapped_pgs":0} 2024-03-18T14:50:14.952 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T14:50:15.200 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:50:15.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:15 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3095821064' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T14:50:15.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:15 smithi076 ceph-mon[32205]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:15.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:15 smithi076 ceph-mon[32205]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T14:50:15.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:15.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:15.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:50:15.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:50:15.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:50:15.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:50:15.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:15.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:15.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:50:15.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:15 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:15.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:15 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3095821064' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T14:50:15.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:15 smithi174 ceph-mon[36787]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:15.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:15 smithi174 ceph-mon[36787]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T14:50:15.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:15.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:15.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:50:15.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:50:15.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:50:15.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:50:15.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:15.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:15.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:50:15.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:15 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:15.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:15 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3095821064' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T14:50:15.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:15 smithi077 ceph-mon[35865]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:15.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:15 smithi077 ceph-mon[35865]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T14:50:15.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:15.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:15.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:50:15.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:50:15.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:50:15.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:50:15.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:15.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:15.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:50:15.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:15 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:16.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:16 smithi076 ceph-mon[32205]: Detected new or changed devices on smithi174 2024-03-18T14:50:16.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:16 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi174 to 4270M 2024-03-18T14:50:16.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:16 smithi174 ceph-mon[36787]: Detected new or changed devices on smithi174 2024-03-18T14:50:16.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:16 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi174 to 4270M 2024-03-18T14:50:16.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:16 smithi077 ceph-mon[35865]: Detected new or changed devices on smithi174 2024-03-18T14:50:16.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:16 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi174 to 4270M 2024-03-18T14:50:17.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.044+0000 7fc2887d0700 1 -- 172.21.15.76:0/2880991403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc280074e20 msgr2=0x7fc28010e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:17.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.045+0000 7fc2887d0700 1 --2- 172.21.15.76:0/2880991403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc280074e20 0x7fc28010e790 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc270009990 tx=0x7fc270009ca0 comp rx=0 tx=0).stop 2024-03-18T14:50:17.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.045+0000 7fc2887d0700 1 -- 172.21.15.76:0/2880991403 shutdown_connections 2024-03-18T14:50:17.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.045+0000 7fc2887d0700 1 --2- 172.21.15.76:0/2880991403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc28010ecd0 0x7fc2801110c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:17.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.045+0000 7fc2887d0700 1 --2- 172.21.15.76:0/2880991403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc280074500 0x7fc2800748e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:17.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.045+0000 7fc2887d0700 1 --2- 172.21.15.76:0/2880991403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc280074e20 0x7fc28010e790 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:17.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.045+0000 7fc2887d0700 1 -- 172.21.15.76:0/2880991403 >> 172.21.15.76:0/2880991403 conn(0x7fc280079400 msgr2=0x7fc280079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:17.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.045+0000 7fc2887d0700 1 -- 172.21.15.76:0/2880991403 shutdown_connections 2024-03-18T14:50:17.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.045+0000 7fc2887d0700 1 -- 172.21.15.76:0/2880991403 wait complete. 2024-03-18T14:50:17.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.045+0000 7fc2887d0700 1 Processor -- start 2024-03-18T14:50:17.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.045+0000 7fc2887d0700 1 -- start start 2024-03-18T14:50:17.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc2887d0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc280074500 0x7fc2801a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:17.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc2887d0700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc280074e20 0x7fc2801a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:17.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc2887d0700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc28010ecd0 0x7fc28019aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:17.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc2887d0700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc2801147a0 con 0x7fc28010ecd0 2024-03-18T14:50:17.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc2887d0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc280114620 con 0x7fc280074500 2024-03-18T14:50:17.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc2887d0700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc280114920 con 0x7fc280074e20 2024-03-18T14:50:17.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc286d6d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc28010ecd0 0x7fc28019aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:17.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc286d6d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc28010ecd0 0x7fc28019aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:42290/0 (socket says 172.21.15.76:42290) 2024-03-18T14:50:17.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc286d6d700 1 -- 172.21.15.76:0/4232381006 learned_addr learned my addr 172.21.15.76:0/4232381006 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:17.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc28656c700 1 --2- 172.21.15.76:0/4232381006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc280074500 0x7fc2801a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:17.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc285d6b700 1 --2- 172.21.15.76:0/4232381006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc280074e20 0x7fc2801a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:17.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc286d6d700 1 -- 172.21.15.76:0/4232381006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc280074e20 msgr2=0x7fc2801a0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:17.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc286d6d700 1 --2- 172.21.15.76:0/4232381006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc280074e20 0x7fc2801a0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:17.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc286d6d700 1 -- 172.21.15.76:0/4232381006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc280074500 msgr2=0x7fc2801a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:17.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc286d6d700 1 --2- 172.21.15.76:0/4232381006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc280074500 0x7fc2801a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:17.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc286d6d700 1 -- 172.21.15.76:0/4232381006 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc28019b1e0 con 0x7fc28010ecd0 2024-03-18T14:50:17.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc28656c700 1 --2- 172.21.15.76:0/4232381006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc280074500 0x7fc2801a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:50:17.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc286d6d700 1 --2- 172.21.15.76:0/4232381006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc28010ecd0 0x7fc28019aa80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc278007ee0 tx=0x7fc27800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:17.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.046+0000 7fc2777fe700 1 -- 172.21.15.76:0/4232381006 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc278004bb0 con 0x7fc28010ecd0 2024-03-18T14:50:17.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.047+0000 7fc2887d0700 1 -- 172.21.15.76:0/4232381006 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc28019b4d0 con 0x7fc28010ecd0 2024-03-18T14:50:17.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.047+0000 7fc2777fe700 1 -- 172.21.15.76:0/4232381006 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc278004d50 con 0x7fc28010ecd0 2024-03-18T14:50:17.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.047+0000 7fc2777fe700 1 -- 172.21.15.76:0/4232381006 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc278005960 con 0x7fc28010ecd0 2024-03-18T14:50:17.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.047+0000 7fc2887d0700 1 -- 172.21.15.76:0/4232381006 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2801a7790 con 0x7fc28010ecd0 2024-03-18T14:50:17.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.047+0000 7fc2887d0700 1 -- 172.21.15.76:0/4232381006 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc280068f90 con 0x7fc28010ecd0 2024-03-18T14:50:17.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.049+0000 7fc2777fe700 1 -- 172.21.15.76:0/4232381006 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc27801a020 con 0x7fc28010ecd0 2024-03-18T14:50:17.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.049+0000 7fc2777fe700 1 --2- 172.21.15.76:0/4232381006 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc26c0a2890 0x7fc26c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:17.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.049+0000 7fc28656c700 1 --2- 172.21.15.76:0/4232381006 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc26c0a2890 0x7fc26c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:17.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.049+0000 7fc2777fe700 1 -- 172.21.15.76:0/4232381006 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc2780ce340 con 0x7fc28010ecd0 2024-03-18T14:50:17.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.050+0000 7fc28656c700 1 --2- 172.21.15.76:0/4232381006 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc26c0a2890 0x7fc26c0a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc27c005950 tx=0x7fc27c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:17.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.053+0000 7fc2777fe700 1 -- 172.21.15.76:0/4232381006 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc278010070 con 0x7fc28010ecd0 2024-03-18T14:50:17.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.249+0000 7fc2887d0700 1 -- 172.21.15.76:0/4232381006 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc28002db70 con 0x7fc28010ecd0 2024-03-18T14:50:17.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.250+0000 7fc2777fe700 1 -- 172.21.15.76:0/4232381006 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18429 (secure 0 0 0) 0x7fc27809b880 con 0x7fc28010ecd0 2024-03-18T14:50:17.252 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:17.252 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":82,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:50:14.506203+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:50:17.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.252+0000 7fc2887d0700 1 -- 172.21.15.76:0/4232381006 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc26c0a2890 msgr2=0x7fc26c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:17.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.252+0000 7fc2887d0700 1 --2- 172.21.15.76:0/4232381006 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc26c0a2890 0x7fc26c0a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc27c005950 tx=0x7fc27c00a300 comp rx=0 tx=0).stop 2024-03-18T14:50:17.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.252+0000 7fc2887d0700 1 -- 172.21.15.76:0/4232381006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc28010ecd0 msgr2=0x7fc28019aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:17.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.252+0000 7fc2887d0700 1 --2- 172.21.15.76:0/4232381006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc28010ecd0 0x7fc28019aa80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc278007ee0 tx=0x7fc27800d3b0 comp rx=0 tx=0).stop 2024-03-18T14:50:17.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.252+0000 7fc2887d0700 1 -- 172.21.15.76:0/4232381006 shutdown_connections 2024-03-18T14:50:17.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.252+0000 7fc2887d0700 1 --2- 172.21.15.76:0/4232381006 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc26c0a2890 0x7fc26c0a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:17.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.252+0000 7fc2887d0700 1 --2- 172.21.15.76:0/4232381006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc28010ecd0 0x7fc28019aa80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:17.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.252+0000 7fc2887d0700 1 --2- 172.21.15.76:0/4232381006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc280074500 0x7fc2801a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:17.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.252+0000 7fc2887d0700 1 --2- 172.21.15.76:0/4232381006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc280074e20 0x7fc2801a0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:17.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.252+0000 7fc2887d0700 1 -- 172.21.15.76:0/4232381006 >> 172.21.15.76:0/4232381006 conn(0x7fc280079400 msgr2=0x7fc2800ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:17.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.252+0000 7fc2887d0700 1 -- 172.21.15.76:0/4232381006 shutdown_connections 2024-03-18T14:50:17.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:17.253+0000 7fc2887d0700 1 -- 172.21.15.76:0/4232381006 wait complete. 2024-03-18T14:50:17.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:17 smithi174 ceph-mon[36787]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:17.378 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:17 smithi077 ceph-mon[35865]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:17.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:17 smithi076 ceph-mon[32205]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:18.040 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:46:26.588384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:50:18.041 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-18T14:50:18.284 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:50:18.405 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:18 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/4232381006' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:50:18.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:18 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/4232381006' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:50:18.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:18 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/4232381006' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:50:19.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:19 smithi076 ceph-mon[32205]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:19.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:19 smithi174 ceph-mon[36787]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:19.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:19 smithi077 ceph-mon[35865]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:20.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.001+0000 7faf0250d700 1 -- 172.21.15.76:0/1594601264 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faefc06a920 msgr2=0x7faefc10f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:20.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.001+0000 7faf0250d700 1 --2- 172.21.15.76:0/1594601264 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faefc06a920 0x7faefc10f160 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7faef400b0d0 tx=0x7faef400b3e0 comp rx=0 tx=0).stop 2024-03-18T14:50:20.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.001+0000 7faf0250d700 1 -- 172.21.15.76:0/1594601264 shutdown_connections 2024-03-18T14:50:20.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.001+0000 7faf0250d700 1 --2- 172.21.15.76:0/1594601264 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faefc06a920 0x7faefc10f160 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:20.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.001+0000 7faf0250d700 1 --2- 172.21.15.76:0/1594601264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faefc108320 0x7faefc108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:20.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.001+0000 7faf0250d700 1 --2- 172.21.15.76:0/1594601264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faefc069f80 0x7faefc06a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:20.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.001+0000 7faf0250d700 1 -- 172.21.15.76:0/1594601264 >> 172.21.15.76:0/1594601264 conn(0x7faefc078100 msgr2=0x7faefc078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:20.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.001+0000 7faf0250d700 1 -- 172.21.15.76:0/1594601264 shutdown_connections 2024-03-18T14:50:20.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.001+0000 7faf0250d700 1 -- 172.21.15.76:0/1594601264 wait complete. 2024-03-18T14:50:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.002+0000 7faf0250d700 1 Processor -- start 2024-03-18T14:50:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.002+0000 7faf0250d700 1 -- start start 2024-03-18T14:50:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.002+0000 7faf0250d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faefc069f80 0x7faefc19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.002+0000 7faf0250d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faefc06a920 0x7faefc19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.002+0000 7faf0250d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faefc108320 0x7faefc198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.002+0000 7faf0250d700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7faefc112570 con 0x7faefc069f80 2024-03-18T14:50:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.002+0000 7faf0250d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faefc1123f0 con 0x7faefc06a920 2024-03-18T14:50:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.002+0000 7faf0250d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faefc1126f0 con 0x7faefc108320 2024-03-18T14:50:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.002+0000 7faefbfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faefc069f80 0x7faefc19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.002+0000 7faefbfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faefc069f80 0x7faefc19e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54668/0 (socket says 172.21.15.76:54668) 2024-03-18T14:50:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.003+0000 7faefbfff700 1 -- 172.21.15.76:0/2890763376 learned_addr learned my addr 172.21.15.76:0/2890763376 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.003+0000 7faefb7fe700 1 --2- 172.21.15.76:0/2890763376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faefc06a920 0x7faefc19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.003+0000 7faf00aaa700 1 --2- 172.21.15.76:0/2890763376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faefc108320 0x7faefc198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.003+0000 7faefbfff700 1 -- 172.21.15.76:0/2890763376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faefc108320 msgr2=0x7faefc198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.003+0000 7faefbfff700 1 --2- 172.21.15.76:0/2890763376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faefc108320 0x7faefc198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.003+0000 7faefbfff700 1 -- 172.21.15.76:0/2890763376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faefc06a920 msgr2=0x7faefc19ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.003+0000 7faefbfff700 1 --2- 172.21.15.76:0/2890763376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faefc06a920 0x7faefc19ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.003+0000 7faefbfff700 1 -- 172.21.15.76:0/2890763376 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faefc198ff0 con 0x7faefc069f80 2024-03-18T14:50:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.003+0000 7faf00aaa700 1 --2- 172.21.15.76:0/2890763376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faefc108320 0x7faefc198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:50:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.003+0000 7faefbfff700 1 --2- 172.21.15.76:0/2890763376 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faefc069f80 0x7faefc19e690 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faeec00d980 tx=0x7faeec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.003+0000 7faef97fa700 1 -- 172.21.15.76:0/2890763376 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeec016070 con 0x7faefc069f80 2024-03-18T14:50:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.003+0000 7faf0250d700 1 -- 172.21.15.76:0/2890763376 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faefc1992e0 con 0x7faefc069f80 2024-03-18T14:50:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.003+0000 7faef97fa700 1 -- 172.21.15.76:0/2890763376 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faeec004d10 con 0x7faefc069f80 2024-03-18T14:50:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.003+0000 7faef97fa700 1 -- 172.21.15.76:0/2890763376 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeec0055f0 con 0x7faefc069f80 2024-03-18T14:50:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.003+0000 7faf0250d700 1 -- 172.21.15.76:0/2890763376 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faefc1a54f0 con 0x7faefc069f80 2024-03-18T14:50:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.004+0000 7faf0250d700 1 -- 172.21.15.76:0/2890763376 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faefc1039d0 con 0x7faefc069f80 2024-03-18T14:50:20.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.005+0000 7faef97fa700 1 -- 172.21.15.76:0/2890763376 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7faeec004750 con 0x7faefc069f80 2024-03-18T14:50:20.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.006+0000 7faef97fa700 1 --2- 172.21.15.76:0/2890763376 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7faee40a2890 0x7faee40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:20.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.006+0000 7faefb7fe700 1 --2- 172.21.15.76:0/2890763376 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7faee40a2890 0x7faee40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:20.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.006+0000 7faef97fa700 1 -- 172.21.15.76:0/2890763376 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7faeec0ce8a0 con 0x7faefc069f80 2024-03-18T14:50:20.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.006+0000 7faefb7fe700 1 --2- 172.21.15.76:0/2890763376 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7faee40a2890 0x7faee40a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7faefc078690 tx=0x7faef000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:20.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.009+0000 7faef97fa700 1 -- 172.21.15.76:0/2890763376 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faeec01f020 con 0x7faefc069f80 2024-03-18T14:50:20.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.204+0000 7faf0250d700 1 -- 172.21.15.76:0/2890763376 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faefc199d40 con 0x7faefc069f80 2024-03-18T14:50:20.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.205+0000 7faef97fa700 1 -- 172.21.15.76:0/2890763376 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7faeec09bde0 con 0x7faefc069f80 2024-03-18T14:50:20.206 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-18T14:50:20.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.207+0000 7faf0250d700 1 -- 172.21.15.76:0/2890763376 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7faee40a2890 msgr2=0x7faee40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:20.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.207+0000 7faf0250d700 1 --2- 172.21.15.76:0/2890763376 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7faee40a2890 0x7faee40a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7faefc078690 tx=0x7faef000b410 comp rx=0 tx=0).stop 2024-03-18T14:50:20.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.207+0000 7faf0250d700 1 -- 172.21.15.76:0/2890763376 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faefc069f80 msgr2=0x7faefc19e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:20.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.207+0000 7faf0250d700 1 --2- 172.21.15.76:0/2890763376 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faefc069f80 0x7faefc19e690 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faeec00d980 tx=0x7faeec00dc90 comp rx=0 tx=0).stop 2024-03-18T14:50:20.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.207+0000 7faf0250d700 1 -- 172.21.15.76:0/2890763376 shutdown_connections 2024-03-18T14:50:20.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.207+0000 7faf0250d700 1 --2- 172.21.15.76:0/2890763376 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7faee40a2890 0x7faee40a4d50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:20.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.207+0000 7faf0250d700 1 --2- 172.21.15.76:0/2890763376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faefc108320 0x7faefc198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:20.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.207+0000 7faf0250d700 1 --2- 172.21.15.76:0/2890763376 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faefc069f80 0x7faefc19e690 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:20.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.207+0000 7faf0250d700 1 --2- 172.21.15.76:0/2890763376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faefc06a920 0x7faefc19ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:20.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.207+0000 7faf0250d700 1 -- 172.21.15.76:0/2890763376 >> 172.21.15.76:0/2890763376 conn(0x7faefc078100 msgr2=0x7faefc0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:20.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.207+0000 7faf0250d700 1 -- 172.21.15.76:0/2890763376 shutdown_connections 2024-03-18T14:50:20.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:20.207+0000 7faf0250d700 1 -- 172.21.15.76:0/2890763376 wait complete. 2024-03-18T14:50:20.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:20 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2890763376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:50:20.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:20 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2890763376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:50:20.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:20 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2890763376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:50:20.906 INFO:tasks.cephadm:Adding mds.a on smithi076 2024-03-18T14:50:20.906 INFO:tasks.cephadm:Adding mds.d on smithi076 2024-03-18T14:50:20.906 INFO:tasks.cephadm:Adding mds.g on smithi076 2024-03-18T14:50:20.906 INFO:tasks.cephadm:Adding mds.j on smithi076 2024-03-18T14:50:20.906 INFO:tasks.cephadm:Adding mds.b on smithi077 2024-03-18T14:50:20.906 INFO:tasks.cephadm:Adding mds.e on smithi077 2024-03-18T14:50:20.906 INFO:tasks.cephadm:Adding mds.h on smithi077 2024-03-18T14:50:20.906 INFO:tasks.cephadm:Adding mds.k on smithi077 2024-03-18T14:50:20.906 INFO:tasks.cephadm:Adding mds.c on smithi174 2024-03-18T14:50:20.906 INFO:tasks.cephadm:Adding mds.f on smithi174 2024-03-18T14:50:20.906 INFO:tasks.cephadm:Adding mds.i on smithi174 2024-03-18T14:50:20.907 INFO:tasks.cephadm:Adding mds.l on smithi174 2024-03-18T14:50:20.907 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l' 2024-03-18T14:50:21.156 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T14:50:21.545 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:21 smithi174 ceph-mon[36787]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:21.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:21 smithi076 ceph-mon[32205]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:21.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:21 smithi077 ceph-mon[35865]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:22.784 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.781+0000 7f5c639f5700 1 -- 172.21.15.174:0/2416353135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c5c102dd0 msgr2=0x7f5c5c109680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:22.784 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.781+0000 7f5c639f5700 1 --2- 172.21.15.174:0/2416353135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c5c102dd0 0x7f5c5c109680 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5c54009990 tx=0x7f5c54009ca0 comp rx=0 tx=0).stop 2024-03-18T14:50:22.784 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.781+0000 7f5c639f5700 1 -- 172.21.15.174:0/2416353135 shutdown_connections 2024-03-18T14:50:22.784 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.781+0000 7f5c639f5700 1 --2- 172.21.15.174:0/2416353135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c5c102dd0 0x7f5c5c109680 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:22.784 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.781+0000 7f5c639f5700 1 --2- 172.21.15.174:0/2416353135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c5c101230 0x7f5c5c101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:22.784 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.781+0000 7f5c639f5700 1 --2- 172.21.15.174:0/2416353135 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c5c102430 0x7f5c5c102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:22.784 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.781+0000 7f5c639f5700 1 -- 172.21.15.174:0/2416353135 >> 172.21.15.174:0/2416353135 conn(0x7f5c5c0fc7d0 msgr2=0x7f5c5c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:22.784 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.781+0000 7f5c639f5700 1 -- 172.21.15.174:0/2416353135 shutdown_connections 2024-03-18T14:50:22.784 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.781+0000 7f5c639f5700 1 -- 172.21.15.174:0/2416353135 wait complete. 2024-03-18T14:50:22.784 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.781+0000 7f5c639f5700 1 Processor -- start 2024-03-18T14:50:22.784 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c639f5700 1 -- start start 2024-03-18T14:50:22.784 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c639f5700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c5c101230 0x7f5c5c19a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:22.785 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c639f5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c5c102430 0x7f5c5c19aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:22.785 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c639f5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c5c102dd0 0x7f5c5c1a1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:22.785 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c639f5700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5c10ce10 con 0x7f5c5c101230 2024-03-18T14:50:22.785 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c639f5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5c10cc90 con 0x7f5c5c102dd0 2024-03-18T14:50:22.785 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c639f5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5c10cf90 con 0x7f5c5c102430 2024-03-18T14:50:22.785 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c60f90700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c5c102430 0x7f5c5c19aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:22.785 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c60f90700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c5c102430 0x7f5c5c19aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:34738/0 (socket says 172.21.15.174:34738) 2024-03-18T14:50:22.785 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c60f90700 1 -- 172.21.15.174:0/3955316248 learned_addr learned my addr 172.21.15.174:0/3955316248 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-18T14:50:22.785 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c61f92700 1 --2- 172.21.15.174:0/3955316248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c5c102dd0 0x7f5c5c1a1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:22.785 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c61791700 1 --2- 172.21.15.174:0/3955316248 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c5c101230 0x7f5c5c19a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:22.785 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c60f90700 1 -- 172.21.15.174:0/3955316248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c5c102dd0 msgr2=0x7f5c5c1a1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:22.785 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c60f90700 1 --2- 172.21.15.174:0/3955316248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c5c102dd0 0x7f5c5c1a1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:22.785 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c60f90700 1 -- 172.21.15.174:0/3955316248 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c5c101230 msgr2=0x7f5c5c19a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:22.785 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c60f90700 1 --2- 172.21.15.174:0/3955316248 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c5c101230 0x7f5c5c19a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:22.785 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c60f90700 1 -- 172.21.15.174:0/3955316248 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c5c1a22c0 con 0x7f5c5c102430 2024-03-18T14:50:22.786 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.782+0000 7f5c60f90700 1 --2- 172.21.15.174:0/3955316248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c5c102430 0x7f5c5c19aba0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c00d980 tx=0x7f5c4c00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:22.786 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.783+0000 7f5c527fc700 1 -- 172.21.15.174:0/3955316248 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c016070 con 0x7f5c5c102430 2024-03-18T14:50:22.786 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.783+0000 7f5c527fc700 1 -- 172.21.15.174:0/3955316248 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c4c009db0 con 0x7f5c5c102430 2024-03-18T14:50:22.786 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.783+0000 7f5c639f5700 1 -- 172.21.15.174:0/3955316248 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c5c1a25b0 con 0x7f5c5c102430 2024-03-18T14:50:22.786 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.783+0000 7f5c527fc700 1 -- 172.21.15.174:0/3955316248 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c00f3c0 con 0x7f5c5c102430 2024-03-18T14:50:22.786 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.783+0000 7f5c639f5700 1 -- 172.21.15.174:0/3955316248 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c5c1a2af0 con 0x7f5c5c102430 2024-03-18T14:50:22.787 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.784+0000 7f5c639f5700 1 -- 172.21.15.174:0/3955316248 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c5c068f90 con 0x7f5c5c102430 2024-03-18T14:50:22.787 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.784+0000 7f5c527fc700 1 -- 172.21.15.174:0/3955316248 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5c4c019070 con 0x7f5c5c102430 2024-03-18T14:50:22.787 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.785+0000 7f5c527fc700 1 --2- 172.21.15.174:0/3955316248 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5c480a2890 0x7f5c480a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:22.787 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.785+0000 7f5c527fc700 1 -- 172.21.15.174:0/3955316248 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f5c4c0ce090 con 0x7f5c5c102430 2024-03-18T14:50:22.787 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.785+0000 7f5c61791700 1 --2- 172.21.15.174:0/3955316248 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5c480a2890 0x7f5c480a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:22.787 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.786+0000 7f5c61791700 1 --2- 172.21.15.174:0/3955316248 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5c480a2890 0x7f5c480a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c102290 tx=0x7f5c5800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:22.790 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.790+0000 7f5c527fc700 1 -- 172.21.15.174:0/3955316248 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5c4c09b5d0 con 0x7f5c5c102430 2024-03-18T14:50:22.998 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:22.998+0000 7f5c639f5700 1 -- 172.21.15.174:0/3955316248 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l", "target": ["mon-mgr", ""]}) -- 0x7f5c5c06b180 con 0x7f5c480a2890 2024-03-18T14:50:23.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:23.005+0000 7f5c527fc700 1 -- 172.21.15.174:0/3955316248 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f5c5c06b180 con 0x7f5c480a2890 2024-03-18T14:50:23.006 INFO:teuthology.orchestra.run.smithi174.stdout:Scheduled mds.all update... 2024-03-18T14:50:23.008 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:23.008+0000 7f5c639f5700 1 -- 172.21.15.174:0/3955316248 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5c480a2890 msgr2=0x7f5c480a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:23.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:23.008+0000 7f5c639f5700 1 --2- 172.21.15.174:0/3955316248 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5c480a2890 0x7f5c480a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c102290 tx=0x7f5c5800b040 comp rx=0 tx=0).stop 2024-03-18T14:50:23.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:23.008+0000 7f5c639f5700 1 -- 172.21.15.174:0/3955316248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c5c102430 msgr2=0x7f5c5c19aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:23.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:23.008+0000 7f5c639f5700 1 --2- 172.21.15.174:0/3955316248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c5c102430 0x7f5c5c19aba0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c00d980 tx=0x7f5c4c00dc90 comp rx=0 tx=0).stop 2024-03-18T14:50:23.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:23.008+0000 7f5c639f5700 1 -- 172.21.15.174:0/3955316248 shutdown_connections 2024-03-18T14:50:23.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:23.008+0000 7f5c639f5700 1 --2- 172.21.15.174:0/3955316248 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5c480a2890 0x7f5c480a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:23.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:23.008+0000 7f5c639f5700 1 --2- 172.21.15.174:0/3955316248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c5c102dd0 0x7f5c5c1a1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:23.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:23.008+0000 7f5c639f5700 1 --2- 172.21.15.174:0/3955316248 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c5c101230 0x7f5c5c19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:23.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:23.008+0000 7f5c639f5700 1 --2- 172.21.15.174:0/3955316248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c5c102430 0x7f5c5c19aba0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:23.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:23.008+0000 7f5c639f5700 1 -- 172.21.15.174:0/3955316248 >> 172.21.15.174:0/3955316248 conn(0x7f5c5c0fc7d0 msgr2=0x7f5c5c06a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:23.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:23.008+0000 7f5c639f5700 1 -- 172.21.15.174:0/3955316248 shutdown_connections 2024-03-18T14:50:23.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-18T14:50:23.008+0000 7f5c639f5700 1 -- 172.21.15.174:0/3955316248 wait complete. 2024-03-18T14:50:23.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:23 smithi076 ceph-mon[32205]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:23.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:23 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:23.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:23 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:50:23.715 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:23 smithi174 ceph-mon[36787]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:23.715 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:23 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:23.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:23 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:50:23.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:23 smithi077 ceph-mon[35865]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:23.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:23 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:23.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:23 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:50:23.752 DEBUG:teuthology.orchestra.run.smithi076:mds.a> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.a.service 2024-03-18T14:50:23.755 DEBUG:teuthology.orchestra.run.smithi076:mds.d> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.d.service 2024-03-18T14:50:23.799 DEBUG:teuthology.orchestra.run.smithi076:mds.g> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.g.service 2024-03-18T14:50:23.842 DEBUG:teuthology.orchestra.run.smithi076:mds.j> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.j.service 2024-03-18T14:50:23.885 DEBUG:teuthology.orchestra.run.smithi077:mds.b> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.b.service 2024-03-18T14:50:23.888 DEBUG:teuthology.orchestra.run.smithi077:mds.e> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.e.service 2024-03-18T14:50:23.932 DEBUG:teuthology.orchestra.run.smithi077:mds.h> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.h.service 2024-03-18T14:50:23.974 DEBUG:teuthology.orchestra.run.smithi077:mds.k> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.k.service 2024-03-18T14:50:24.017 DEBUG:teuthology.orchestra.run.smithi174:mds.c> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.c.service 2024-03-18T14:50:24.020 DEBUG:teuthology.orchestra.run.smithi174:mds.f> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.f.service 2024-03-18T14:50:24.062 DEBUG:teuthology.orchestra.run.smithi174:mds.i> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.i.service 2024-03-18T14:50:24.106 DEBUG:teuthology.orchestra.run.smithi174:mds.l> sudo journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.l.service 2024-03-18T14:50:24.149 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-18T14:50:24.150 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:50:24.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.263+0000 7fd90c9cd640 1 -- 172.21.15.76:0/111595006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908101800 msgr2=0x7fd908101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:24.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.263+0000 7fd90c9cd640 1 --2- 172.21.15.76:0/111595006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908101800 0x7fd908101c20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8009900 tx=0x7fd8f802bc30 comp rx=0 tx=0).stop 2024-03-18T14:50:24.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.263+0000 7fd90c9cd640 1 -- 172.21.15.76:0/111595006 shutdown_connections 2024-03-18T14:50:24.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.263+0000 7fd90c9cd640 1 --2- 172.21.15.76:0/111595006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9081033b0 0x7fd908109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.263+0000 7fd90c9cd640 1 --2- 172.21.15.76:0/111595006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9081029f0 0x7fd908102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.263+0000 7fd90c9cd640 1 --2- 172.21.15.76:0/111595006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908101800 0x7fd908101c20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.263+0000 7fd90c9cd640 1 -- 172.21.15.76:0/111595006 >> 172.21.15.76:0/111595006 conn(0x7fd9080fcd90 msgr2=0x7fd9080ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:24.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.263+0000 7fd90c9cd640 1 -- 172.21.15.76:0/111595006 shutdown_connections 2024-03-18T14:50:24.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.263+0000 7fd90c9cd640 1 -- 172.21.15.76:0/111595006 wait complete. 2024-03-18T14:50:24.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd90c9cd640 1 Processor -- start 2024-03-18T14:50:24.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd90c9cd640 1 -- start start 2024-03-18T14:50:24.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd90c9cd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908101800 0x7fd908197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:24.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd90c9cd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9081029f0 0x7fd908198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:24.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd90c9cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9081033b0 0x7fd90819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:24.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd90c9cd640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd90810d480 con 0x7fd908101800 2024-03-18T14:50:24.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd90c9cd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd90810d300 con 0x7fd9081033b0 2024-03-18T14:50:24.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd90c9cd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd90810d600 con 0x7fd9081029f0 2024-03-18T14:50:24.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd90659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908101800 0x7fd908197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:24.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd90659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908101800 0x7fd908197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54696/0 (socket says 172.21.15.76:54696) 2024-03-18T14:50:24.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd90659c640 1 -- 172.21.15.76:0/4049106527 learned_addr learned my addr 172.21.15.76:0/4049106527 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:24.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd906d9d640 1 --2- 172.21.15.76:0/4049106527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9081033b0 0x7fd90819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:24.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd8fffff640 1 --2- 172.21.15.76:0/4049106527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9081029f0 0x7fd908198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:24.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd90659c640 1 -- 172.21.15.76:0/4049106527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9081029f0 msgr2=0x7fd908198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:24.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd90659c640 1 --2- 172.21.15.76:0/4049106527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9081029f0 0x7fd908198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd90659c640 1 -- 172.21.15.76:0/4049106527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9081033b0 msgr2=0x7fd90819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:24.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd90659c640 1 --2- 172.21.15.76:0/4049106527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9081033b0 0x7fd90819f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.264+0000 7fd90659c640 1 -- 172.21.15.76:0/4049106527 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd90819f740 con 0x7fd908101800 2024-03-18T14:50:24.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.265+0000 7fd90659c640 1 --2- 172.21.15.76:0/4049106527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908101800 0x7fd908197b20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8005ab0 tx=0x7fd8f8005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:24.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.265+0000 7fd8ff7fe640 1 -- 172.21.15.76:0/4049106527 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8041070 con 0x7fd908101800 2024-03-18T14:50:24.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.265+0000 7fd90c9cd640 1 -- 172.21.15.76:0/4049106527 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd90819f9d0 con 0x7fd908101800 2024-03-18T14:50:24.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.265+0000 7fd8ff7fe640 1 -- 172.21.15.76:0/4049106527 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8f80054e0 con 0x7fd908101800 2024-03-18T14:50:24.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.265+0000 7fd8ff7fe640 1 -- 172.21.15.76:0/4049106527 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8034440 con 0x7fd908101800 2024-03-18T14:50:24.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.265+0000 7fd90c9cd640 1 -- 172.21.15.76:0/4049106527 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd90819ff10 con 0x7fd908101800 2024-03-18T14:50:24.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.266+0000 7fd90c9cd640 1 -- 172.21.15.76:0/4049106527 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd908108b20 con 0x7fd908101800 2024-03-18T14:50:24.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.267+0000 7fd8ff7fe640 1 -- 172.21.15.76:0/4049106527 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd8f8044070 con 0x7fd908101800 2024-03-18T14:50:24.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.267+0000 7fd8ff7fe640 1 --2- 172.21.15.76:0/4049106527 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd8f40a2800 0x7fd8f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:24.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.268+0000 7fd8fffff640 1 --2- 172.21.15.76:0/4049106527 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd8f40a2800 0x7fd8f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:24.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.268+0000 7fd8ff7fe640 1 -- 172.21.15.76:0/4049106527 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd8f80f0200 con 0x7fd908101800 2024-03-18T14:50:24.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.268+0000 7fd8fffff640 1 --2- 172.21.15.76:0/4049106527 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd8f40a2800 0x7fd8f40a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0005fd0 tx=0x7fd8f0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:24.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.272+0000 7fd8ff7fe640 1 -- 172.21.15.76:0/4049106527 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd8f80bd740 con 0x7fd908101800 2024-03-18T14:50:24.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.450+0000 7fd90c9cd640 1 -- 172.21.15.76:0/4049106527 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd90802d3f0 con 0x7fd908101800 2024-03-18T14:50:24.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.451+0000 7fd8ff7fe640 1 -- 172.21.15.76:0/4049106527 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18429 (secure 0 0 0) 0x7fd8f80bd040 con 0x7fd908101800 2024-03-18T14:50:24.452 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:24.453 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":82,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:50:14.506203+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:50:24.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.453+0000 7fd90c9cd640 1 -- 172.21.15.76:0/4049106527 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd8f40a2800 msgr2=0x7fd8f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:24.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.453+0000 7fd90c9cd640 1 --2- 172.21.15.76:0/4049106527 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd8f40a2800 0x7fd8f40a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0005fd0 tx=0x7fd8f0005850 comp rx=0 tx=0).stop 2024-03-18T14:50:24.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.453+0000 7fd90c9cd640 1 -- 172.21.15.76:0/4049106527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908101800 msgr2=0x7fd908197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:24.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.453+0000 7fd90c9cd640 1 --2- 172.21.15.76:0/4049106527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908101800 0x7fd908197b20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8005ab0 tx=0x7fd8f8005dc0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.453+0000 7fd90c9cd640 1 -- 172.21.15.76:0/4049106527 shutdown_connections 2024-03-18T14:50:24.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.453+0000 7fd90c9cd640 1 --2- 172.21.15.76:0/4049106527 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd8f40a2800 0x7fd8f40a4cc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.453+0000 7fd90c9cd640 1 --2- 172.21.15.76:0/4049106527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9081029f0 0x7fd908198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.453+0000 7fd90c9cd640 1 --2- 172.21.15.76:0/4049106527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9081033b0 0x7fd90819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.453+0000 7fd90c9cd640 1 --2- 172.21.15.76:0/4049106527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908101800 0x7fd908197b20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.453+0000 7fd90c9cd640 1 -- 172.21.15.76:0/4049106527 >> 172.21.15.76:0/4049106527 conn(0x7fd9080fcd90 msgr2=0x7fd908105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:24.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.453+0000 7fd90c9cd640 1 -- 172.21.15.76:0/4049106527 shutdown_connections 2024-03-18T14:50:24.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.453+0000 7fd90c9cd640 1 -- 172.21.15.76:0/4049106527 wait complete. 2024-03-18T14:50:24.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:24 smithi076 ceph-mon[32205]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:50:24.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:24 smithi076 ceph-mon[32205]: Saving service mds.all spec with placement smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l;count:12 2024-03-18T14:50:24.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:24.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:50:24.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:24.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:24.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:24.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:24 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:24.466 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:46:26.588384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:50:24.467 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:50:24.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.571+0000 7f152b8e8640 1 -- 172.21.15.76:0/3382844923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15241006f0 msgr2=0x7f1524100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:24.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.571+0000 7f152b8e8640 1 --2- 172.21.15.76:0/3382844923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15241006f0 0x7f1524100b10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1520009900 tx=0x7f152002bc30 comp rx=0 tx=0).stop 2024-03-18T14:50:24.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.571+0000 7f152b8e8640 1 -- 172.21.15.76:0/3382844923 shutdown_connections 2024-03-18T14:50:24.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.571+0000 7f152b8e8640 1 --2- 172.21.15.76:0/3382844923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15241022a0 0x7f1524108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.571+0000 7f152b8e8640 1 --2- 172.21.15.76:0/3382844923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15241018e0 0x7f1524101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.571+0000 7f152b8e8640 1 --2- 172.21.15.76:0/3382844923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15241006f0 0x7f1524100b10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.571+0000 7f152b8e8640 1 -- 172.21.15.76:0/3382844923 >> 172.21.15.76:0/3382844923 conn(0x7f15240fbc80 msgr2=0x7f15240fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:24.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.571+0000 7f152b8e8640 1 -- 172.21.15.76:0/3382844923 shutdown_connections 2024-03-18T14:50:24.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.571+0000 7f152b8e8640 1 -- 172.21.15.76:0/3382844923 wait complete. 2024-03-18T14:50:24.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f152b8e8640 1 Processor -- start 2024-03-18T14:50:24.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f152b8e8640 1 -- start start 2024-03-18T14:50:24.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f152b8e8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15241006f0 0x7f152406a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:24.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f152b8e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15241018e0 0x7f152419ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:24.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f152b8e8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15241022a0 0x7f15241a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:24.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f152b8e8640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f152410c3c0 con 0x7f15241006f0 2024-03-18T14:50:24.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f152b8e8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f152410c240 con 0x7f15241018e0 2024-03-18T14:50:24.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f152b8e8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f152410c540 con 0x7f15241022a0 2024-03-18T14:50:24.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f1529684640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15241006f0 0x7f152406a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:24.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f1529684640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15241006f0 0x7f152406a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54712/0 (socket says 172.21.15.76:54712) 2024-03-18T14:50:24.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f1529684640 1 -- 172.21.15.76:0/4154102659 learned_addr learned my addr 172.21.15.76:0/4154102659 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:24.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f1529e85640 1 --2- 172.21.15.76:0/4154102659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15241022a0 0x7f15241a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:24.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f1528e83640 1 --2- 172.21.15.76:0/4154102659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15241018e0 0x7f152419ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:24.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f1529684640 1 -- 172.21.15.76:0/4154102659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15241022a0 msgr2=0x7f15241a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:24.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f1529684640 1 --2- 172.21.15.76:0/4154102659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15241022a0 0x7f15241a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f1529684640 1 -- 172.21.15.76:0/4154102659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15241018e0 msgr2=0x7f152419ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:24.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f1529684640 1 --2- 172.21.15.76:0/4154102659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15241018e0 0x7f152419ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f1529684640 1 -- 172.21.15.76:0/4154102659 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15241a1aa0 con 0x7f15241006f0 2024-03-18T14:50:24.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f1529e85640 1 --2- 172.21.15.76:0/4154102659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15241022a0 0x7f15241a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:50:24.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f1529684640 1 --2- 172.21.15.76:0/4154102659 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15241006f0 0x7f152406a920 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1520005ab0 tx=0x7f1520005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:24.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f151a7fc640 1 -- 172.21.15.76:0/4154102659 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1520041070 con 0x7f15241006f0 2024-03-18T14:50:24.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f151a7fc640 1 -- 172.21.15.76:0/4154102659 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15200054e0 con 0x7f15241006f0 2024-03-18T14:50:24.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f152b8e8640 1 -- 172.21.15.76:0/4154102659 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15241a1cd0 con 0x7f15241006f0 2024-03-18T14:50:24.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f151a7fc640 1 -- 172.21.15.76:0/4154102659 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1520034440 con 0x7f15241006f0 2024-03-18T14:50:24.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.572+0000 7f152b8e8640 1 -- 172.21.15.76:0/4154102659 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15241a21c0 con 0x7f15241006f0 2024-03-18T14:50:24.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.573+0000 7f152b8e8640 1 -- 172.21.15.76:0/4154102659 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1524107ab0 con 0x7f15241006f0 2024-03-18T14:50:24.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.574+0000 7f151a7fc640 1 -- 172.21.15.76:0/4154102659 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1520044070 con 0x7f15241006f0 2024-03-18T14:50:24.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.575+0000 7f151a7fc640 1 --2- 172.21.15.76:0/4154102659 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f15100a28b0 0x7f15100a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:24.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.575+0000 7f1528e83640 1 --2- 172.21.15.76:0/4154102659 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f15100a28b0 0x7f15100a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:24.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.575+0000 7f151a7fc640 1 -- 172.21.15.76:0/4154102659 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f15200f0280 con 0x7f15241006f0 2024-03-18T14:50:24.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.576+0000 7f1528e83640 1 --2- 172.21.15.76:0/4154102659 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f15100a28b0 0x7f15100a4d70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f152406a690 tx=0x7f1514008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:24.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.580+0000 7f151a7fc640 1 -- 172.21.15.76:0/4154102659 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f15200bd720 con 0x7f15241006f0 2024-03-18T14:50:24.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:24 smithi174 ceph-mon[36787]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:50:24.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:24 smithi174 ceph-mon[36787]: Saving service mds.all spec with placement smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l;count:12 2024-03-18T14:50:24.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:24.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:50:24.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:24.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:24.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:24.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:24 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:24.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:24 smithi077 ceph-mon[35865]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:50:24.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:24 smithi077 ceph-mon[35865]: Saving service mds.all spec with placement smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l;count:12 2024-03-18T14:50:24.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:24.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:50:24.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:24.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:24.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:24.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:24 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:24.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.755+0000 7f152b8e8640 1 -- 172.21.15.76:0/4154102659 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1524062480 con 0x7f15241006f0 2024-03-18T14:50:24.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.755+0000 7f151a7fc640 1 -- 172.21.15.76:0/4154102659 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f15200bd020 con 0x7f15241006f0 2024-03-18T14:50:24.756 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-18T14:50:24.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.757+0000 7f152b8e8640 1 -- 172.21.15.76:0/4154102659 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f15100a28b0 msgr2=0x7f15100a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:24.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.757+0000 7f152b8e8640 1 --2- 172.21.15.76:0/4154102659 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f15100a28b0 0x7f15100a4d70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f152406a690 tx=0x7f1514008040 comp rx=0 tx=0).stop 2024-03-18T14:50:24.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.757+0000 7f152b8e8640 1 -- 172.21.15.76:0/4154102659 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15241006f0 msgr2=0x7f152406a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:24.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.757+0000 7f152b8e8640 1 --2- 172.21.15.76:0/4154102659 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15241006f0 0x7f152406a920 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1520005ab0 tx=0x7f1520005dc0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.757+0000 7f152b8e8640 1 -- 172.21.15.76:0/4154102659 shutdown_connections 2024-03-18T14:50:24.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.757+0000 7f152b8e8640 1 --2- 172.21.15.76:0/4154102659 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f15100a28b0 0x7f15100a4d70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.757+0000 7f152b8e8640 1 --2- 172.21.15.76:0/4154102659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15241022a0 0x7f15241a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.757+0000 7f152b8e8640 1 --2- 172.21.15.76:0/4154102659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15241018e0 0x7f152419ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.757+0000 7f152b8e8640 1 --2- 172.21.15.76:0/4154102659 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15241006f0 0x7f152406a920 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.757+0000 7f152b8e8640 1 -- 172.21.15.76:0/4154102659 >> 172.21.15.76:0/4154102659 conn(0x7f15240fbc80 msgr2=0x7f1524104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:24.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.757+0000 7f152b8e8640 1 -- 172.21.15.76:0/4154102659 shutdown_connections 2024-03-18T14:50:24.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.757+0000 7f152b8e8640 1 -- 172.21.15.76:0/4154102659 wait complete. 2024-03-18T14:50:24.769 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:50:24.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.869+0000 7f7f95b22640 1 -- 172.21.15.76:0/1411157514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f90104c30 msgr2=0x7f7f90107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.869+0000 7f7f95b22640 1 --2- 172.21.15.76:0/1411157514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f90104c30 0x7f7f90107020 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7f84009900 tx=0x7f7f8402bc30 comp rx=0 tx=0).stop 2024-03-18T14:50:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.869+0000 7f7f95b22640 1 -- 172.21.15.76:0/1411157514 shutdown_connections 2024-03-18T14:50:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.869+0000 7f7f95b22640 1 --2- 172.21.15.76:0/1411157514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f90107560 0x7f7f901099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.869+0000 7f7f95b22640 1 --2- 172.21.15.76:0/1411157514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f90104c30 0x7f7f90107020 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.869+0000 7f7f95b22640 1 --2- 172.21.15.76:0/1411157514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f90102300 0x7f7f901046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.869+0000 7f7f95b22640 1 -- 172.21.15.76:0/1411157514 >> 172.21.15.76:0/1411157514 conn(0x7f7f900fbd10 msgr2=0x7f7f900fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.869+0000 7f7f95b22640 1 -- 172.21.15.76:0/1411157514 shutdown_connections 2024-03-18T14:50:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.869+0000 7f7f95b22640 1 -- 172.21.15.76:0/1411157514 wait complete. 2024-03-18T14:50:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.870+0000 7f7f95b22640 1 Processor -- start 2024-03-18T14:50:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.870+0000 7f7f95b22640 1 -- start start 2024-03-18T14:50:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.870+0000 7f7f95b22640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f90102300 0x7f7f90197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.870+0000 7f7f95b22640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f90104c30 0x7f7f90198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.870+0000 7f7f95b22640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f90107560 0x7f7f9019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.870+0000 7f7f95b22640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9010d4a0 con 0x7f7f90107560 2024-03-18T14:50:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.870+0000 7f7f95b22640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9010d320 con 0x7f7f90104c30 2024-03-18T14:50:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.870+0000 7f7f95b22640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9010d620 con 0x7f7f90102300 2024-03-18T14:50:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.870+0000 7f7f8ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f90107560 0x7f7f9019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.870+0000 7f7f8ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f90107560 0x7f7f9019f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54736/0 (socket says 172.21.15.76:54736) 2024-03-18T14:50:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.870+0000 7f7f8ffff640 1 -- 172.21.15.76:0/3063954303 learned_addr learned my addr 172.21.15.76:0/3063954303 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.870+0000 7f7f8effd640 1 --2- 172.21.15.76:0/3063954303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f90104c30 0x7f7f90198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.870+0000 7f7f8f7fe640 1 --2- 172.21.15.76:0/3063954303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f90102300 0x7f7f90197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.870+0000 7f7f8ffff640 1 -- 172.21.15.76:0/3063954303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f90102300 msgr2=0x7f7f90197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.871+0000 7f7f8ffff640 1 --2- 172.21.15.76:0/3063954303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f90102300 0x7f7f90197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.871+0000 7f7f8ffff640 1 -- 172.21.15.76:0/3063954303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f90104c30 msgr2=0x7f7f90198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.871+0000 7f7f8ffff640 1 --2- 172.21.15.76:0/3063954303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f90104c30 0x7f7f90198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.871+0000 7f7f8ffff640 1 -- 172.21.15.76:0/3063954303 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f9019f6e0 con 0x7f7f90107560 2024-03-18T14:50:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.871+0000 7f7f8ffff640 1 --2- 172.21.15.76:0/3063954303 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f90107560 0x7f7f9019f070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7f8800cce0 tx=0x7f7f8800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.871+0000 7f7f8cff9640 1 -- 172.21.15.76:0/3063954303 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8800f450 con 0x7f7f90107560 2024-03-18T14:50:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.871+0000 7f7f8cff9640 1 -- 172.21.15.76:0/3063954303 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f8800fad0 con 0x7f7f90107560 2024-03-18T14:50:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.871+0000 7f7f95b22640 1 -- 172.21.15.76:0/3063954303 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f9019f9d0 con 0x7f7f90107560 2024-03-18T14:50:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.871+0000 7f7f8cff9640 1 -- 172.21.15.76:0/3063954303 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f88018c10 con 0x7f7f90107560 2024-03-18T14:50:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.871+0000 7f7f95b22640 1 -- 172.21.15.76:0/3063954303 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f9019fef0 con 0x7f7f90107560 2024-03-18T14:50:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.872+0000 7f7f95b22640 1 -- 172.21.15.76:0/3063954303 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f900fd800 con 0x7f7f90107560 2024-03-18T14:50:24.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.873+0000 7f7f8cff9640 1 -- 172.21.15.76:0/3063954303 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7f8800f5f0 con 0x7f7f90107560 2024-03-18T14:50:24.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.873+0000 7f7f8cff9640 1 --2- 172.21.15.76:0/3063954303 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7f780a2880 0x7f7f780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:24.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.873+0000 7f7f8f7fe640 1 --2- 172.21.15.76:0/3063954303 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7f780a2880 0x7f7f780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:24.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.874+0000 7f7f8cff9640 1 -- 172.21.15.76:0/3063954303 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f7f880cec40 con 0x7f7f90107560 2024-03-18T14:50:24.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.874+0000 7f7f8f7fe640 1 --2- 172.21.15.76:0/3063954303 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7f780a2880 0x7f7f780a4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7f80005fd0 tx=0x7f7f80005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:24.878+0000 7f7f8cff9640 1 -- 172.21.15.76:0/3063954303 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7f88014030 con 0x7f7f90107560 2024-03-18T14:50:25.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:25.089+0000 7f7f95b22640 1 -- 172.21.15.76:0/3063954303 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f900623e0 con 0x7f7f90107560 2024-03-18T14:50:25.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:25.089+0000 7f7f8cff9640 1 -- 172.21.15.76:0/3063954303 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f7f8809c180 con 0x7f7f90107560 2024-03-18T14:50:25.091 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:25.091 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-18T14:50:25.091 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 1 2024-03-18T14:50:25.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:25.091+0000 7f7f95b22640 1 -- 172.21.15.76:0/3063954303 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7f780a2880 msgr2=0x7f7f780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:25.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:25.091+0000 7f7f95b22640 1 --2- 172.21.15.76:0/3063954303 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7f780a2880 0x7f7f780a4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7f80005fd0 tx=0x7f7f80005850 comp rx=0 tx=0).stop 2024-03-18T14:50:25.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:25.091+0000 7f7f95b22640 1 -- 172.21.15.76:0/3063954303 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f90107560 msgr2=0x7f7f9019f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:25.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:25.091+0000 7f7f95b22640 1 --2- 172.21.15.76:0/3063954303 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f90107560 0x7f7f9019f070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7f8800cce0 tx=0x7f7f8800a620 comp rx=0 tx=0).stop 2024-03-18T14:50:25.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:25.091+0000 7f7f95b22640 1 -- 172.21.15.76:0/3063954303 shutdown_connections 2024-03-18T14:50:25.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:25.091+0000 7f7f95b22640 1 --2- 172.21.15.76:0/3063954303 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7f780a2880 0x7f7f780a4d40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:25.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:25.091+0000 7f7f95b22640 1 --2- 172.21.15.76:0/3063954303 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f90107560 0x7f7f9019f070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:25.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:25.091+0000 7f7f95b22640 1 --2- 172.21.15.76:0/3063954303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f90104c30 0x7f7f90198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:25.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:25.091+0000 7f7f95b22640 1 --2- 172.21.15.76:0/3063954303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f90102300 0x7f7f90197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:25.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:25.091+0000 7f7f95b22640 1 -- 172.21.15.76:0/3063954303 >> 172.21.15.76:0/3063954303 conn(0x7f7f900fbd10 msgr2=0x7f7f90102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:25.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:25.092+0000 7f7f95b22640 1 -- 172.21.15.76:0/3063954303 shutdown_connections 2024-03-18T14:50:25.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:25.092+0000 7f7f95b22640 1 -- 172.21.15.76:0/3063954303 wait complete. 2024-03-18T14:50:25.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:25 smithi174 ceph-mon[36787]: Deploying daemon mds.l on smithi174 2024-03-18T14:50:25.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:25 smithi174 ceph-mon[36787]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:25.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:25 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/4049106527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:50:25.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:25 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/4154102659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:50:25.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:25 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3063954303' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:25.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:25 smithi076 ceph-mon[32205]: Deploying daemon mds.l on smithi174 2024-03-18T14:50:25.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:25 smithi076 ceph-mon[32205]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:25.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:25 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/4049106527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:50:25.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:25 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/4154102659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:50:25.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:25 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3063954303' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:25.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:25 smithi077 ceph-mon[35865]: Deploying daemon mds.l on smithi174 2024-03-18T14:50:25.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:25 smithi077 ceph-mon[35865]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:25.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:25 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/4049106527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:50:25.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:25 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/4154102659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:50:25.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:25 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3063954303' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:25.896 INFO:journalctl@ceph.mds.l.smithi174.stdout:Mar 18 14:50:25 smithi174 systemd[1]: Starting Ceph mds.l for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T14:50:26.178 INFO:journalctl@ceph.mds.l.smithi174.stdout:Mar 18 14:50:25 smithi174 podman[66677]: 2024-03-18 14:50:25.799223755 +0000 UTC m=+0.017574512 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T14:50:26.178 INFO:journalctl@ceph.mds.l.smithi174.stdout:Mar 18 14:50:25 smithi174 podman[66677]: 2024-03-18 14:50:25.91769076 +0000 UTC m=+0.136041525 container create a4e76b74a47bc0fde90f445f9b9f72cb76d32f65a878d7fc20089fc687963303 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-18T14:50:26.179 INFO:journalctl@ceph.mds.l.smithi174.stdout:Mar 18 14:50:26 smithi174 podman[66677]: 2024-03-18 14:50:26.178460365 +0000 UTC m=+0.396811125 container init a4e76b74a47bc0fde90f445f9b9f72cb76d32f65a878d7fc20089fc687963303 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-l, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T14:50:26.467 INFO:journalctl@ceph.mds.l.smithi174.stdout:Mar 18 14:50:26 smithi174 podman[66677]: 2024-03-18 14:50:26.181002605 +0000 UTC m=+0.399353354 container start a4e76b74a47bc0fde90f445f9b9f72cb76d32f65a878d7fc20089fc687963303 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-l, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-18T14:50:26.467 INFO:journalctl@ceph.mds.l.smithi174.stdout:Mar 18 14:50:26 smithi174 ceph-mds[66695]: -- 172.21.15.174:0/245418674 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564f8ea91a40 con 0x564f8ea64c00 2024-03-18T14:50:26.467 INFO:journalctl@ceph.mds.l.smithi174.stdout:Mar 18 14:50:26 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-l[66691]: starting mds.l at 2024-03-18T14:50:26.467 INFO:journalctl@ceph.mds.l.smithi174.stdout:Mar 18 14:50:26 smithi174 bash[66677]: a4e76b74a47bc0fde90f445f9b9f72cb76d32f65a878d7fc20089fc687963303 2024-03-18T14:50:26.468 INFO:journalctl@ceph.mds.l.smithi174.stdout:Mar 18 14:50:26 smithi174 systemd[1]: Started Ceph mds.l for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:50:27.105 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:50:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.220+0000 7f0f9b5a9640 1 -- 172.21.15.76:0/4160677606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f94102a60 msgr2=0x7f0f94104ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.220+0000 7f0f9b5a9640 1 --2- 172.21.15.76:0/4160677606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f94102a60 0x7f0f94104ed0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0f90009900 tx=0x7f0f9002bc30 comp rx=0 tx=0).stop 2024-03-18T14:50:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.220+0000 7f0f9b5a9640 1 -- 172.21.15.76:0/4160677606 shutdown_connections 2024-03-18T14:50:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.220+0000 7f0f9b5a9640 1 --2- 172.21.15.76:0/4160677606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f94105410 0x7f0f94105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.220+0000 7f0f9b5a9640 1 --2- 172.21.15.76:0/4160677606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f94102a60 0x7f0f94104ed0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.220+0000 7f0f9b5a9640 1 --2- 172.21.15.76:0/4160677606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f941000f0 0x7f0f94102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.220+0000 7f0f9b5a9640 1 -- 172.21.15.76:0/4160677606 >> 172.21.15.76:0/4160677606 conn(0x7f0f940fbd10 msgr2=0x7f0f940fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.220+0000 7f0f9b5a9640 1 -- 172.21.15.76:0/4160677606 shutdown_connections 2024-03-18T14:50:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.220+0000 7f0f9b5a9640 1 -- 172.21.15.76:0/4160677606 wait complete. 2024-03-18T14:50:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f9b5a9640 1 Processor -- start 2024-03-18T14:50:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f9b5a9640 1 -- start start 2024-03-18T14:50:27.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f9b5a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f941000f0 0x7f0f9419e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:27.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f9b5a9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f94102a60 0x7f0f9419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:27.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f9b5a9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f94105410 0x7f0f941a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:27.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f9b5a9640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0f94113a80 con 0x7f0f94102a60 2024-03-18T14:50:27.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f9b5a9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0f94113900 con 0x7f0f941000f0 2024-03-18T14:50:27.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f9b5a9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0f94113c00 con 0x7f0f94105410 2024-03-18T14:50:27.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f98b44640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f94102a60 0x7f0f9419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:27.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f98b44640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f94102a60 0x7f0f9419e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54760/0 (socket says 172.21.15.76:54760) 2024-03-18T14:50:27.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f98b44640 1 -- 172.21.15.76:0/2022079970 learned_addr learned my addr 172.21.15.76:0/2022079970 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:27.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f99345640 1 --2- 172.21.15.76:0/2022079970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f941000f0 0x7f0f9419e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:27.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f99b46640 1 --2- 172.21.15.76:0/2022079970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f94105410 0x7f0f941a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:27.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f98b44640 1 -- 172.21.15.76:0/2022079970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f94105410 msgr2=0x7f0f941a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:27.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f98b44640 1 --2- 172.21.15.76:0/2022079970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f94105410 0x7f0f941a5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:27.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f98b44640 1 -- 172.21.15.76:0/2022079970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f941000f0 msgr2=0x7f0f9419e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:27.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f98b44640 1 --2- 172.21.15.76:0/2022079970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f941000f0 0x7f0f9419e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:27.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.221+0000 7f0f98b44640 1 -- 172.21.15.76:0/2022079970 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f941a5ce0 con 0x7f0f94102a60 2024-03-18T14:50:27.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.222+0000 7f0f98b44640 1 --2- 172.21.15.76:0/2022079970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f94102a60 0x7f0f9419e600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0f90009230 tx=0x7f0f90005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:27.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.225+0000 7f0f8a7fc640 1 -- 172.21.15.76:0/2022079970 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f90041070 con 0x7f0f94102a60 2024-03-18T14:50:27.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.225+0000 7f0f8a7fc640 1 -- 172.21.15.76:0/2022079970 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f900044d0 con 0x7f0f94102a60 2024-03-18T14:50:27.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.225+0000 7f0f9b5a9640 1 -- 172.21.15.76:0/2022079970 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f941a5f70 con 0x7f0f94102a60 2024-03-18T14:50:27.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.225+0000 7f0f8a7fc640 1 -- 172.21.15.76:0/2022079970 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f90034440 con 0x7f0f94102a60 2024-03-18T14:50:27.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.225+0000 7f0f9b5a9640 1 -- 172.21.15.76:0/2022079970 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f941a63e0 con 0x7f0f94102a60 2024-03-18T14:50:27.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.226+0000 7f0f9b5a9640 1 -- 172.21.15.76:0/2022079970 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f940fd800 con 0x7f0f94102a60 2024-03-18T14:50:27.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.227+0000 7f0f8a7fc640 1 -- 172.21.15.76:0/2022079970 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0f90044070 con 0x7f0f94102a60 2024-03-18T14:50:27.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.228+0000 7f0f8a7fc640 1 --2- 172.21.15.76:0/2022079970 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0f800a2880 0x7f0f800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:27.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.228+0000 7f0f99345640 1 --2- 172.21.15.76:0/2022079970 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0f800a2880 0x7f0f800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:27.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.228+0000 7f0f8a7fc640 1 -- 172.21.15.76:0/2022079970 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f0f900f00d0 con 0x7f0f94102a60 2024-03-18T14:50:27.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.228+0000 7f0f99345640 1 --2- 172.21.15.76:0/2022079970 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0f800a2880 0x7f0f800a4d40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0f84005fd0 tx=0x7f0f8400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:27.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.232+0000 7f0f8a7fc640 1 -- 172.21.15.76:0/2022079970 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0f900bcfc0 con 0x7f0f94102a60 2024-03-18T14:50:27.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:27 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.174:6832/3569060524,v1:172.21.15.174:6833/3569060524] up:boot 2024-03-18T14:50:27.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:27 smithi076 ceph-mon[32205]: fsmap 1 up:standby 2024-03-18T14:50:27.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:27 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T14:50:27.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:27 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:27.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:27 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:27.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:27 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:27.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:27 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:27.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:27 smithi076 ceph-mon[32205]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:27.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:27 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:27.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:27 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:27.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:27 smithi076 ceph-mon[32205]: Deploying daemon mds.k on smithi077 2024-03-18T14:50:27.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.449+0000 7f0f9b5a9640 1 -- 172.21.15.76:0/2022079970 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f9402d3d0 con 0x7f0f94102a60 2024-03-18T14:50:27.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.449+0000 7f0f8a7fc640 1 -- 172.21.15.76:0/2022079970 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f0f900c1060 con 0x7f0f94102a60 2024-03-18T14:50:27.451 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:27.451 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T14:50:27.451 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 2 2024-03-18T14:50:27.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.451+0000 7f0f9b5a9640 1 -- 172.21.15.76:0/2022079970 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0f800a2880 msgr2=0x7f0f800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:27.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.451+0000 7f0f9b5a9640 1 --2- 172.21.15.76:0/2022079970 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0f800a2880 0x7f0f800a4d40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0f84005fd0 tx=0x7f0f8400b390 comp rx=0 tx=0).stop 2024-03-18T14:50:27.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.451+0000 7f0f9b5a9640 1 -- 172.21.15.76:0/2022079970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f94102a60 msgr2=0x7f0f9419e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:27.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.451+0000 7f0f9b5a9640 1 --2- 172.21.15.76:0/2022079970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f94102a60 0x7f0f9419e600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0f90009230 tx=0x7f0f90005d50 comp rx=0 tx=0).stop 2024-03-18T14:50:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.451+0000 7f0f9b5a9640 1 -- 172.21.15.76:0/2022079970 shutdown_connections 2024-03-18T14:50:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.451+0000 7f0f9b5a9640 1 --2- 172.21.15.76:0/2022079970 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0f800a2880 0x7f0f800a4d40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.451+0000 7f0f9b5a9640 1 --2- 172.21.15.76:0/2022079970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f94105410 0x7f0f941a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.451+0000 7f0f9b5a9640 1 --2- 172.21.15.76:0/2022079970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f94102a60 0x7f0f9419e600 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.451+0000 7f0f9b5a9640 1 --2- 172.21.15.76:0/2022079970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f941000f0 0x7f0f9419e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.451+0000 7f0f9b5a9640 1 -- 172.21.15.76:0/2022079970 >> 172.21.15.76:0/2022079970 conn(0x7f0f940fbd10 msgr2=0x7f0f94100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.451+0000 7f0f9b5a9640 1 -- 172.21.15.76:0/2022079970 shutdown_connections 2024-03-18T14:50:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:27.451+0000 7f0f9b5a9640 1 -- 172.21.15.76:0/2022079970 wait complete. 2024-03-18T14:50:27.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:27 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.174:6832/3569060524,v1:172.21.15.174:6833/3569060524] up:boot 2024-03-18T14:50:27.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:27 smithi077 ceph-mon[35865]: fsmap 1 up:standby 2024-03-18T14:50:27.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:27 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T14:50:27.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:27 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:27.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:27 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:27.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:27 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:27.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:27 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:27.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:27 smithi077 ceph-mon[35865]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:27.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:27 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:27.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:27 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:27.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:27 smithi077 ceph-mon[35865]: Deploying daemon mds.k on smithi077 2024-03-18T14:50:27.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:27 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.174:6832/3569060524,v1:172.21.15.174:6833/3569060524] up:boot 2024-03-18T14:50:27.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:27 smithi174 ceph-mon[36787]: fsmap 1 up:standby 2024-03-18T14:50:27.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:27 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T14:50:27.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:27 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:27.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:27 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:27.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:27 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:27.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:27 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:27.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:27 smithi174 ceph-mon[36787]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:27.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:27 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:27.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:27 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:27.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:27 smithi174 ceph-mon[36787]: Deploying daemon mds.k on smithi077 2024-03-18T14:50:28.644 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:28 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2022079970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:28.644 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 14:50:28 smithi077 systemd[1]: Starting Ceph mds.k for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T14:50:28.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:28 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2022079970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:28.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:28 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2022079970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:28.925 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 14:50:28 smithi077 podman[65064]: 2024-03-18 14:50:28.684267452 +0000 UTC m=+0.017697505 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T14:50:28.925 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 14:50:28 smithi077 podman[65064]: 2024-03-18 14:50:28.810304571 +0000 UTC m=+0.143734624 container create 1f0f58e394561c34f01bfa03edff8b6015613ca77108b30b68fc52f6937bc463 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-k, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-18T14:50:29.228 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 14:50:29 smithi077 podman[65064]: 2024-03-18 14:50:29.06322403 +0000 UTC m=+0.396654084 container init 1f0f58e394561c34f01bfa03edff8b6015613ca77108b30b68fc52f6937bc463 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-18T14:50:29.228 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 14:50:29 smithi077 podman[65064]: 2024-03-18 14:50:29.065793763 +0000 UTC m=+0.399223813 container start 1f0f58e394561c34f01bfa03edff8b6015613ca77108b30b68fc52f6937bc463 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-k, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-03-18T14:50:29.228 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 14:50:29 smithi077 ceph-mds[65083]: -- 172.21.15.77:0/2681138403 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562eb1103a40 con 0x562eb10d6800 2024-03-18T14:50:29.228 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 14:50:29 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-k[65079]: starting mds.k at 2024-03-18T14:50:29.228 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 14:50:29 smithi077 bash[65064]: 1f0f58e394561c34f01bfa03edff8b6015613ca77108b30b68fc52f6937bc463 2024-03-18T14:50:29.228 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 14:50:29 smithi077 systemd[1]: Started Ceph mds.k for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:50:29.465 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:50:29.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.569+0000 7fd80355b640 1 -- 172.21.15.76:0/1095678017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc1022e0 msgr2=0x7fd7fc1046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:29.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.569+0000 7fd80355b640 1 --2- 172.21.15.76:0/1095678017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc1022e0 0x7fd7fc1046d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec009900 tx=0x7fd7ec02bc30 comp rx=0 tx=0).stop 2024-03-18T14:50:29.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.569+0000 7fd80355b640 1 -- 172.21.15.76:0/1095678017 shutdown_connections 2024-03-18T14:50:29.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.569+0000 7fd80355b640 1 --2- 172.21.15.76:0/1095678017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7fc104c10 0x7fd7fc107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:29.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.569+0000 7fd80355b640 1 --2- 172.21.15.76:0/1095678017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7fc107540 0x7fd7fc109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:29.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.569+0000 7fd80355b640 1 --2- 172.21.15.76:0/1095678017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc1022e0 0x7fd7fc1046d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:29.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.569+0000 7fd80355b640 1 -- 172.21.15.76:0/1095678017 >> 172.21.15.76:0/1095678017 conn(0x7fd7fc0fbc70 msgr2=0x7fd7fc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:29.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.570+0000 7fd80355b640 1 -- 172.21.15.76:0/1095678017 shutdown_connections 2024-03-18T14:50:29.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.570+0000 7fd80355b640 1 -- 172.21.15.76:0/1095678017 wait complete. 2024-03-18T14:50:29.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.570+0000 7fd80355b640 1 Processor -- start 2024-03-18T14:50:29.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.570+0000 7fd80355b640 1 -- start start 2024-03-18T14:50:29.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.570+0000 7fd80355b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7fc1022e0 0x7fd7fc197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:29.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.570+0000 7fd80355b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7fc104c10 0x7fd7fc1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:29.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.570+0000 7fd80355b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc107540 0x7fd7fc19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:29.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.570+0000 7fd80355b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd7fc10d4a0 con 0x7fd7fc107540 2024-03-18T14:50:29.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.570+0000 7fd80355b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd7fc10d320 con 0x7fd7fc104c10 2024-03-18T14:50:29.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.570+0000 7fd80355b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd7fc10d620 con 0x7fd7fc1022e0 2024-03-18T14:50:29.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.570+0000 7fd801af8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc107540 0x7fd7fc19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:29.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.570+0000 7fd801af8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc107540 0x7fd7fc19f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57466/0 (socket says 172.21.15.76:57466) 2024-03-18T14:50:29.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.570+0000 7fd801af8640 1 -- 172.21.15.76:0/599735243 learned_addr learned my addr 172.21.15.76:0/599735243 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:29.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.571+0000 7fd8012f7640 1 --2- 172.21.15.76:0/599735243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7fc1022e0 0x7fd7fc197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:29.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.571+0000 7fd800af6640 1 --2- 172.21.15.76:0/599735243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7fc104c10 0x7fd7fc1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:29.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.571+0000 7fd801af8640 1 -- 172.21.15.76:0/599735243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7fc1022e0 msgr2=0x7fd7fc197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:29.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.571+0000 7fd801af8640 1 --2- 172.21.15.76:0/599735243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7fc1022e0 0x7fd7fc197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:29.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.571+0000 7fd801af8640 1 -- 172.21.15.76:0/599735243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7fc104c10 msgr2=0x7fd7fc1980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:29.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.571+0000 7fd801af8640 1 --2- 172.21.15.76:0/599735243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7fc104c10 0x7fd7fc1980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:29.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.571+0000 7fd801af8640 1 -- 172.21.15.76:0/599735243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7fc19f790 con 0x7fd7fc107540 2024-03-18T14:50:29.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.571+0000 7fd801af8640 1 --2- 172.21.15.76:0/599735243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc107540 0x7fd7fc19f120 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd7f400eb10 tx=0x7fd7f400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:29.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.571+0000 7fd7f27fc640 1 -- 172.21.15.76:0/599735243 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f4004b20 con 0x7fd7fc107540 2024-03-18T14:50:29.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.571+0000 7fd80355b640 1 -- 172.21.15.76:0/599735243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7fc19fa80 con 0x7fd7fc107540 2024-03-18T14:50:29.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.571+0000 7fd7f27fc640 1 -- 172.21.15.76:0/599735243 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7f4004cc0 con 0x7fd7fc107540 2024-03-18T14:50:29.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.571+0000 7fd7f27fc640 1 -- 172.21.15.76:0/599735243 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f4010760 con 0x7fd7fc107540 2024-03-18T14:50:29.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.571+0000 7fd80355b640 1 -- 172.21.15.76:0/599735243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7fc19ff80 con 0x7fd7fc107540 2024-03-18T14:50:29.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.572+0000 7fd80355b640 1 -- 172.21.15.76:0/599735243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7fc0fd760 con 0x7fd7fc107540 2024-03-18T14:50:29.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.574+0000 7fd7f27fc640 1 -- 172.21.15.76:0/599735243 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd7f4010900 con 0x7fd7fc107540 2024-03-18T14:50:29.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.574+0000 7fd7f27fc640 1 --2- 172.21.15.76:0/599735243 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd7e80a2800 0x7fd7e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:29.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.575+0000 7fd8012f7640 1 --2- 172.21.15.76:0/599735243 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd7e80a2800 0x7fd7e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:29.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.575+0000 7fd7f27fc640 1 -- 172.21.15.76:0/599735243 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd7f40ced60 con 0x7fd7fc107540 2024-03-18T14:50:29.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.575+0000 7fd8012f7640 1 --2- 172.21.15.76:0/599735243 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd7e80a2800 0x7fd7e80a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec02c7b0 tx=0x7fd7ec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:29.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.577+0000 7fd7f27fc640 1 -- 172.21.15.76:0/599735243 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd7f4053740 con 0x7fd7fc107540 2024-03-18T14:50:29.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:29 smithi076 ceph-mon[32205]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:29.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:29 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:29.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:29 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:29.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:29 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:29.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:29 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:29.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:29 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:29.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:29 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:29.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:29 smithi174 ceph-mon[36787]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:29.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:29 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:29.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:29 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:29.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:29 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:29.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:29 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:29.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:29 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:29.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:29 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:29.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:29 smithi077 ceph-mon[35865]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:29.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:29 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:29.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:29 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:29.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:29 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:29.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:29 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:29.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:29 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:29.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:29 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:29.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.801+0000 7fd80355b640 1 -- 172.21.15.76:0/599735243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7fc02d1b0 con 0x7fd7fc107540 2024-03-18T14:50:29.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.801+0000 7fd7f27fc640 1 -- 172.21.15.76:0/599735243 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7fd7f409c2a0 con 0x7fd7fc107540 2024-03-18T14:50:29.803 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:29.803 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-18T14:50:29.803 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 3 2024-03-18T14:50:29.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.803+0000 7fd80355b640 1 -- 172.21.15.76:0/599735243 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd7e80a2800 msgr2=0x7fd7e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:29.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.804+0000 7fd80355b640 1 --2- 172.21.15.76:0/599735243 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd7e80a2800 0x7fd7e80a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec02c7b0 tx=0x7fd7ec005fd0 comp rx=0 tx=0).stop 2024-03-18T14:50:29.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.804+0000 7fd80355b640 1 -- 172.21.15.76:0/599735243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc107540 msgr2=0x7fd7fc19f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:29.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.804+0000 7fd80355b640 1 --2- 172.21.15.76:0/599735243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc107540 0x7fd7fc19f120 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd7f400eb10 tx=0x7fd7f400c2d0 comp rx=0 tx=0).stop 2024-03-18T14:50:29.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.804+0000 7fd80355b640 1 -- 172.21.15.76:0/599735243 shutdown_connections 2024-03-18T14:50:29.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.804+0000 7fd80355b640 1 --2- 172.21.15.76:0/599735243 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd7e80a2800 0x7fd7e80a4cc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:29.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.804+0000 7fd80355b640 1 --2- 172.21.15.76:0/599735243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc107540 0x7fd7fc19f120 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:29.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.804+0000 7fd80355b640 1 --2- 172.21.15.76:0/599735243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7fc104c10 0x7fd7fc1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:29.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.804+0000 7fd80355b640 1 --2- 172.21.15.76:0/599735243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7fc1022e0 0x7fd7fc197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:29.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.804+0000 7fd80355b640 1 -- 172.21.15.76:0/599735243 >> 172.21.15.76:0/599735243 conn(0x7fd7fc0fbc70 msgr2=0x7fd7fc102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:29.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.804+0000 7fd80355b640 1 -- 172.21.15.76:0/599735243 shutdown_connections 2024-03-18T14:50:29.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:29.804+0000 7fd80355b640 1 -- 172.21.15.76:0/599735243 wait complete. 2024-03-18T14:50:30.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:30 smithi076 ceph-mon[32205]: Deploying daemon mds.j on smithi076 2024-03-18T14:50:30.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:30 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.77:6832/2726911960,v1:172.21.15.77:6833/2726911960] up:boot 2024-03-18T14:50:30.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:30 smithi076 ceph-mon[32205]: fsmap 2 up:standby 2024-03-18T14:50:30.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:30 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T14:50:30.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:30 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/599735243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:30.474 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:30 smithi077 ceph-mon[35865]: Deploying daemon mds.j on smithi076 2024-03-18T14:50:30.474 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:30 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.77:6832/2726911960,v1:172.21.15.77:6833/2726911960] up:boot 2024-03-18T14:50:30.474 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:30 smithi077 ceph-mon[35865]: fsmap 2 up:standby 2024-03-18T14:50:30.474 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:30 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T14:50:30.474 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:30 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/599735243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:30.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:30 smithi174 ceph-mon[36787]: Deploying daemon mds.j on smithi076 2024-03-18T14:50:30.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:30 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.77:6832/2726911960,v1:172.21.15.77:6833/2726911960] up:boot 2024-03-18T14:50:30.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:30 smithi174 ceph-mon[36787]: fsmap 2 up:standby 2024-03-18T14:50:30.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:30 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T14:50:30.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:30 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/599735243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:31.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:31 smithi076 ceph-mon[32205]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:31.661 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 18 14:50:31 smithi076 systemd[1]: Starting Ceph mds.j for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T14:50:31.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:31 smithi174 ceph-mon[36787]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:31.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:31 smithi077 ceph-mon[35865]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:31.818 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:50:31.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.926+0000 7fa0ba66c640 1 -- 172.21.15.76:0/2933973081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0b4100280 msgr2=0x7fa0b4109e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:31.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.926+0000 7fa0ba66c640 1 --2- 172.21.15.76:0/2933973081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0b4100280 0x7fa0b4109e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac009ca0 tx=0x7fa0ac02c0f0 comp rx=0 tx=0).stop 2024-03-18T14:50:31.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.926+0000 7fa0ba66c640 1 -- 172.21.15.76:0/2933973081 shutdown_connections 2024-03-18T14:50:31.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.926+0000 7fa0ba66c640 1 --2- 172.21.15.76:0/2933973081 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b40ff8c0 0x7fa0b40ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:31.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.926+0000 7fa0ba66c640 1 --2- 172.21.15.76:0/2933973081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0b4100280 0x7fa0b4109e30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:31.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.926+0000 7fa0ba66c640 1 --2- 172.21.15.76:0/2933973081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b4102e50 0x7fa0b4103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:31.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.926+0000 7fa0ba66c640 1 -- 172.21.15.76:0/2933973081 >> 172.21.15.76:0/2933973081 conn(0x7fa0b40fb420 msgr2=0x7fa0b40fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:31.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.926+0000 7fa0ba66c640 1 -- 172.21.15.76:0/2933973081 shutdown_connections 2024-03-18T14:50:31.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.926+0000 7fa0ba66c640 1 -- 172.21.15.76:0/2933973081 wait complete. 2024-03-18T14:50:31.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0ba66c640 1 Processor -- start 2024-03-18T14:50:31.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0ba66c640 1 -- start start 2024-03-18T14:50:31.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0ba66c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b40ff8c0 0x7fa0b4197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:31.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0ba66c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0b4100280 0x7fa0b4197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:31.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0ba66c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b4102e50 0x7fa0b419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:31.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0ba66c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b410d4b0 con 0x7fa0b4100280 2024-03-18T14:50:31.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0ba66c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b410d330 con 0x7fa0b40ff8c0 2024-03-18T14:50:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0ba66c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b410d630 con 0x7fa0b4102e50 2024-03-18T14:50:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0b37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0b4100280 0x7fa0b4197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0b37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0b4100280 0x7fa0b4197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57482/0 (socket says 172.21.15.76:57482) 2024-03-18T14:50:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0b37fe640 1 -- 172.21.15.76:0/1817800860 learned_addr learned my addr 172.21.15.76:0/1817800860 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0b37fe640 1 -- 172.21.15.76:0/1817800860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b4102e50 msgr2=0x7fa0b419f040 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T14:50:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0b37fe640 1 --2- 172.21.15.76:0/1817800860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b4102e50 0x7fa0b419f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0b37fe640 1 -- 172.21.15.76:0/1817800860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b40ff8c0 msgr2=0x7fa0b4197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0b3fff640 1 --2- 172.21.15.76:0/1817800860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b40ff8c0 0x7fa0b4197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0b37fe640 1 --2- 172.21.15.76:0/1817800860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b40ff8c0 0x7fa0b4197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0b37fe640 1 -- 172.21.15.76:0/1817800860 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b419f6b0 con 0x7fa0b4100280 2024-03-18T14:50:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.927+0000 7fa0b37fe640 1 --2- 172.21.15.76:0/1817800860 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0b4100280 0x7fa0b4197fd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa0a800e9f0 tx=0x7fa0a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.928+0000 7fa0b17fa640 1 -- 172.21.15.76:0/1817800860 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a800cd80 con 0x7fa0b4100280 2024-03-18T14:50:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.928+0000 7fa0ba66c640 1 -- 172.21.15.76:0/1817800860 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0b419f9a0 con 0x7fa0b4100280 2024-03-18T14:50:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.928+0000 7fa0b17fa640 1 -- 172.21.15.76:0/1817800860 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0a8005500 con 0x7fa0b4100280 2024-03-18T14:50:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.928+0000 7fa0b17fa640 1 -- 172.21.15.76:0/1817800860 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8010690 con 0x7fa0b4100280 2024-03-18T14:50:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.928+0000 7fa0ba66c640 1 -- 172.21.15.76:0/1817800860 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0b419fef0 con 0x7fa0b4100280 2024-03-18T14:50:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.929+0000 7fa0ba66c640 1 -- 172.21.15.76:0/1817800860 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0b4108cb0 con 0x7fa0b4100280 2024-03-18T14:50:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.930+0000 7fa0b17fa640 1 -- 172.21.15.76:0/1817800860 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa0a80046c0 con 0x7fa0b4100280 2024-03-18T14:50:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.931+0000 7fa0b17fa640 1 --2- 172.21.15.76:0/1817800860 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa09c0a2800 0x7fa09c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:31.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.931+0000 7fa0b3fff640 1 --2- 172.21.15.76:0/1817800860 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa09c0a2800 0x7fa09c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:31.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.931+0000 7fa0b17fa640 1 -- 172.21.15.76:0/1817800860 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa0a80ced50 con 0x7fa0b4100280 2024-03-18T14:50:31.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.932+0000 7fa0b3fff640 1 --2- 172.21.15.76:0/1817800860 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa09c0a2800 0x7fa09c0a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4005fd0 tx=0x7fa0a4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:31.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:31.934+0000 7fa0b17fa640 1 -- 172.21.15.76:0/1817800860 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa0a8053740 con 0x7fa0b4100280 2024-03-18T14:50:32.003 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 18 14:50:31 smithi076 podman[72013]: 2024-03-18 14:50:31.628305121 +0000 UTC m=+0.018531513 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T14:50:32.003 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 18 14:50:31 smithi076 podman[72013]: 2024-03-18 14:50:31.749985979 +0000 UTC m=+0.140212370 container create 4e1a88725374a383045efd1033c1096d715e10581c32173c30a957201ebb30cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-j, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T14:50:32.003 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 18 14:50:32 smithi076 podman[72013]: 2024-03-18 14:50:32.002811461 +0000 UTC m=+0.393037868 container init 4e1a88725374a383045efd1033c1096d715e10581c32173c30a957201ebb30cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-j, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T14:50:32.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:32.162+0000 7fa0ba66c640 1 -- 172.21.15.76:0/1817800860 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0b402d1b0 con 0x7fa0b4100280 2024-03-18T14:50:32.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:32.164+0000 7fa0b17fa640 1 -- 172.21.15.76:0/1817800860 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7fa0a801f340 con 0x7fa0b4100280 2024-03-18T14:50:32.165 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:32.166 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-18T14:50:32.166 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 3 2024-03-18T14:50:32.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:32.167+0000 7fa0ba66c640 1 -- 172.21.15.76:0/1817800860 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa09c0a2800 msgr2=0x7fa09c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:32.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:32.167+0000 7fa0ba66c640 1 --2- 172.21.15.76:0/1817800860 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa09c0a2800 0x7fa09c0a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4005fd0 tx=0x7fa0a4005850 comp rx=0 tx=0).stop 2024-03-18T14:50:32.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:32.167+0000 7fa0ba66c640 1 -- 172.21.15.76:0/1817800860 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0b4100280 msgr2=0x7fa0b4197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:32.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:32.167+0000 7fa0ba66c640 1 --2- 172.21.15.76:0/1817800860 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0b4100280 0x7fa0b4197fd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa0a800e9f0 tx=0x7fa0a800edf0 comp rx=0 tx=0).stop 2024-03-18T14:50:32.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:32.167+0000 7fa0ba66c640 1 -- 172.21.15.76:0/1817800860 shutdown_connections 2024-03-18T14:50:32.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:32.167+0000 7fa0ba66c640 1 --2- 172.21.15.76:0/1817800860 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa09c0a2800 0x7fa09c0a4cc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:32.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:32.167+0000 7fa0ba66c640 1 --2- 172.21.15.76:0/1817800860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b4102e50 0x7fa0b419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:32.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:32.167+0000 7fa0ba66c640 1 --2- 172.21.15.76:0/1817800860 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0b4100280 0x7fa0b4197fd0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:32.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:32.167+0000 7fa0ba66c640 1 --2- 172.21.15.76:0/1817800860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b40ff8c0 0x7fa0b4197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:32.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:32.167+0000 7fa0ba66c640 1 -- 172.21.15.76:0/1817800860 >> 172.21.15.76:0/1817800860 conn(0x7fa0b40fb420 msgr2=0x7fa0b4105e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:32.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:32.167+0000 7fa0ba66c640 1 -- 172.21.15.76:0/1817800860 shutdown_connections 2024-03-18T14:50:32.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:32.167+0000 7fa0ba66c640 1 -- 172.21.15.76:0/1817800860 wait complete. 2024-03-18T14:50:32.411 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 18 14:50:32 smithi076 podman[72013]: 2024-03-18 14:50:32.005570688 +0000 UTC m=+0.395797079 container start 4e1a88725374a383045efd1033c1096d715e10581c32173c30a957201ebb30cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-j, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T14:50:32.411 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 18 14:50:32 smithi076 ceph-mds[72071]: -- 172.21.15.76:0/818381855 <== mon.0 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dcfe375a40 con 0x55dcfe349000 2024-03-18T14:50:32.411 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 18 14:50:32 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-j[72048]: starting mds.j at 2024-03-18T14:50:32.411 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 18 14:50:32 smithi076 bash[72013]: 4e1a88725374a383045efd1033c1096d715e10581c32173c30a957201ebb30cb 2024-03-18T14:50:32.411 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 18 14:50:32 smithi076 systemd[1]: Started Ceph mds.j for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:50:33.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:33 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:33.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:33 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1817800860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:33.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:33 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:33.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:33 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:33.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:33 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:33.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:33 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:33.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:33 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:33.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:33 smithi076 ceph-mon[32205]: Deploying daemon mds.d on smithi076 2024-03-18T14:50:33.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:33 smithi076 ceph-mon[32205]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:33.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:33 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.76:6834/3585848317,v1:172.21.15.76:6835/3585848317] up:boot 2024-03-18T14:50:33.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:33 smithi076 ceph-mon[32205]: fsmap 3 up:standby 2024-03-18T14:50:33.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:33 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T14:50:33.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:33 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:33.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:33 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1817800860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:33.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:33 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:33.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:33 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:33.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:33 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:33.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:33 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:33.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:33 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:33.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:33 smithi174 ceph-mon[36787]: Deploying daemon mds.d on smithi076 2024-03-18T14:50:33.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:33 smithi174 ceph-mon[36787]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:33.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:33 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.76:6834/3585848317,v1:172.21.15.76:6835/3585848317] up:boot 2024-03-18T14:50:33.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:33 smithi174 ceph-mon[36787]: fsmap 3 up:standby 2024-03-18T14:50:33.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:33 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T14:50:33.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:33 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:33.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:33 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1817800860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:33.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:33 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:33.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:33 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:33.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:33 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:33.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:33 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:33.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:33 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:33.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:33 smithi077 ceph-mon[35865]: Deploying daemon mds.d on smithi076 2024-03-18T14:50:33.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:33 smithi077 ceph-mon[35865]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:33.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:33 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.76:6834/3585848317,v1:172.21.15.76:6835/3585848317] up:boot 2024-03-18T14:50:33.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:33 smithi077 ceph-mon[35865]: fsmap 3 up:standby 2024-03-18T14:50:33.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:33 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T14:50:34.180 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:50:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.303+0000 7fcc3e4ea640 1 -- 172.21.15.76:0/323732294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc381018f0 msgr2=0x7fcc38101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.303+0000 7fcc3e4ea640 1 --2- 172.21.15.76:0/323732294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc381018f0 0x7fcc38101d70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c009900 tx=0x7fcc2c02bc30 comp rx=0 tx=0).stop 2024-03-18T14:50:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.303+0000 7fcc3e4ea640 1 -- 172.21.15.76:0/323732294 shutdown_connections 2024-03-18T14:50:34.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.303+0000 7fcc3e4ea640 1 --2- 172.21.15.76:0/323732294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc381022b0 0x7fcc38108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:34.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.303+0000 7fcc3e4ea640 1 --2- 172.21.15.76:0/323732294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc381018f0 0x7fcc38101d70 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:34.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.303+0000 7fcc3e4ea640 1 --2- 172.21.15.76:0/323732294 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc38100700 0x7fcc38100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:34.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.303+0000 7fcc3e4ea640 1 -- 172.21.15.76:0/323732294 >> 172.21.15.76:0/323732294 conn(0x7fcc380fbc70 msgr2=0x7fcc380fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:34.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.303+0000 7fcc3e4ea640 1 -- 172.21.15.76:0/323732294 shutdown_connections 2024-03-18T14:50:34.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.303+0000 7fcc3e4ea640 1 -- 172.21.15.76:0/323732294 wait complete. 2024-03-18T14:50:34.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.304+0000 7fcc3e4ea640 1 Processor -- start 2024-03-18T14:50:34.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.304+0000 7fcc3e4ea640 1 -- start start 2024-03-18T14:50:34.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc3e4ea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc38100700 0x7fcc38196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:34.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc3e4ea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc381018f0 0x7fcc38196fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:34.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc3e4ea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc381022b0 0x7fcc3819e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:34.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc3e4ea640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3810c3f0 con 0x7fcc381022b0 2024-03-18T14:50:34.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc3e4ea640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3810c270 con 0x7fcc381018f0 2024-03-18T14:50:34.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc3e4ea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3810c570 con 0x7fcc38100700 2024-03-18T14:50:34.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc3ca87640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc381022b0 0x7fcc3819e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:34.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc3ca87640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc381022b0 0x7fcc3819e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57514/0 (socket says 172.21.15.76:57514) 2024-03-18T14:50:34.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc3ca87640 1 -- 172.21.15.76:0/1148432626 learned_addr learned my addr 172.21.15.76:0/1148432626 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:34.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc377fe640 1 --2- 172.21.15.76:0/1148432626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc381018f0 0x7fcc38196fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:34.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc37fff640 1 --2- 172.21.15.76:0/1148432626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc38100700 0x7fcc38196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:34.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc3ca87640 1 -- 172.21.15.76:0/1148432626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc38100700 msgr2=0x7fcc38196a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:34.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc3ca87640 1 --2- 172.21.15.76:0/1148432626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc38100700 0x7fcc38196a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:34.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc3ca87640 1 -- 172.21.15.76:0/1148432626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc381018f0 msgr2=0x7fcc38196fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:34.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc3ca87640 1 --2- 172.21.15.76:0/1148432626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc381018f0 0x7fcc38196fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:34.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc3ca87640 1 -- 172.21.15.76:0/1148432626 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc3819e6b0 con 0x7fcc381022b0 2024-03-18T14:50:34.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc3ca87640 1 --2- 172.21.15.76:0/1148432626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc381022b0 0x7fcc3819e040 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000eaa0 tx=0x7fcc3000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:34.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc357fa640 1 -- 172.21.15.76:0/1148432626 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3000cdb0 con 0x7fcc381022b0 2024-03-18T14:50:34.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc357fa640 1 -- 172.21.15.76:0/1148432626 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc30005020 con 0x7fcc381022b0 2024-03-18T14:50:34.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.305+0000 7fcc3e4ea640 1 -- 172.21.15.76:0/1148432626 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc3819e9a0 con 0x7fcc381022b0 2024-03-18T14:50:34.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.306+0000 7fcc357fa640 1 -- 172.21.15.76:0/1148432626 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc300116b0 con 0x7fcc381022b0 2024-03-18T14:50:34.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.306+0000 7fcc3e4ea640 1 -- 172.21.15.76:0/1148432626 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc3819ee70 con 0x7fcc381022b0 2024-03-18T14:50:34.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.307+0000 7fcc3e4ea640 1 -- 172.21.15.76:0/1148432626 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc38107a20 con 0x7fcc381022b0 2024-03-18T14:50:34.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.308+0000 7fcc357fa640 1 -- 172.21.15.76:0/1148432626 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fcc30005560 con 0x7fcc381022b0 2024-03-18T14:50:34.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.308+0000 7fcc357fa640 1 --2- 172.21.15.76:0/1148432626 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fcc200a2800 0x7fcc200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:34.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.308+0000 7fcc37fff640 1 --2- 172.21.15.76:0/1148432626 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fcc200a2800 0x7fcc200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:34.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.308+0000 7fcc357fa640 1 -- 172.21.15.76:0/1148432626 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fcc300ce830 con 0x7fcc381022b0 2024-03-18T14:50:34.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.309+0000 7fcc37fff640 1 --2- 172.21.15.76:0/1148432626 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fcc200a2800 0x7fcc200a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcc38101750 tx=0x7fcc28008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:34.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.313+0000 7fcc357fa640 1 -- 172.21.15.76:0/1148432626 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcc30016030 con 0x7fcc381022b0 2024-03-18T14:50:34.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.554+0000 7fcc3e4ea640 1 -- 172.21.15.76:0/1148432626 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc380623e0 con 0x7fcc381022b0 2024-03-18T14:50:34.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.556+0000 7fcc357fa640 1 -- 172.21.15.76:0/1148432626 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7fcc3009bd70 con 0x7fcc381022b0 2024-03-18T14:50:34.557 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:34.557 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-18T14:50:34.557 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 4 2024-03-18T14:50:34.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.558+0000 7fcc3e4ea640 1 -- 172.21.15.76:0/1148432626 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fcc200a2800 msgr2=0x7fcc200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:34.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.558+0000 7fcc3e4ea640 1 --2- 172.21.15.76:0/1148432626 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fcc200a2800 0x7fcc200a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcc38101750 tx=0x7fcc28008040 comp rx=0 tx=0).stop 2024-03-18T14:50:34.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.558+0000 7fcc3e4ea640 1 -- 172.21.15.76:0/1148432626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc381022b0 msgr2=0x7fcc3819e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:34.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.558+0000 7fcc3e4ea640 1 --2- 172.21.15.76:0/1148432626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc381022b0 0x7fcc3819e040 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000eaa0 tx=0x7fcc3000ef60 comp rx=0 tx=0).stop 2024-03-18T14:50:34.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.558+0000 7fcc3e4ea640 1 -- 172.21.15.76:0/1148432626 shutdown_connections 2024-03-18T14:50:34.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.558+0000 7fcc3e4ea640 1 --2- 172.21.15.76:0/1148432626 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fcc200a2800 0x7fcc200a4cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:34.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.558+0000 7fcc3e4ea640 1 --2- 172.21.15.76:0/1148432626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc381022b0 0x7fcc3819e040 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:34.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.558+0000 7fcc3e4ea640 1 --2- 172.21.15.76:0/1148432626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc381018f0 0x7fcc38196fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:34.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.558+0000 7fcc3e4ea640 1 --2- 172.21.15.76:0/1148432626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc38100700 0x7fcc38196a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:34.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.558+0000 7fcc3e4ea640 1 -- 172.21.15.76:0/1148432626 >> 172.21.15.76:0/1148432626 conn(0x7fcc380fbc70 msgr2=0x7fcc38104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:34.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.558+0000 7fcc3e4ea640 1 -- 172.21.15.76:0/1148432626 shutdown_connections 2024-03-18T14:50:34.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:34.558+0000 7fcc3e4ea640 1 -- 172.21.15.76:0/1148432626 wait complete. 2024-03-18T14:50:34.569 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 18 14:50:34 smithi076 systemd[1]: Starting Ceph mds.d for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T14:50:34.909 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 18 14:50:34 smithi076 podman[72401]: 2024-03-18 14:50:34.552816028 +0000 UTC m=+0.017709445 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T14:50:34.909 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 18 14:50:34 smithi076 podman[72401]: 2024-03-18 14:50:34.673741547 +0000 UTC m=+0.138634987 container create 3e7989df5e0c51441016dae1cfc6ae987e5a9ff1fd5ad3d0a92d9f80bc8c4cd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T14:50:34.910 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 18 14:50:34 smithi076 podman[72401]: 2024-03-18 14:50:34.90955809 +0000 UTC m=+0.374451494 container init 3e7989df5e0c51441016dae1cfc6ae987e5a9ff1fd5ad3d0a92d9f80bc8c4cd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T14:50:35.161 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 18 14:50:34 smithi076 podman[72401]: 2024-03-18 14:50:34.91222302 +0000 UTC m=+0.377116421 container start 3e7989df5e0c51441016dae1cfc6ae987e5a9ff1fd5ad3d0a92d9f80bc8c4cd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-d, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-18T14:50:35.161 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 18 14:50:34 smithi076 ceph-mds[72422]: -- 172.21.15.76:0/2270705165 <== mon.0 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556b78dd1a40 con 0x556b78da4800 2024-03-18T14:50:35.161 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 18 14:50:34 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-d[72418]: starting mds.d at 2024-03-18T14:50:35.161 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 18 14:50:35 smithi076 bash[72401]: 3e7989df5e0c51441016dae1cfc6ae987e5a9ff1fd5ad3d0a92d9f80bc8c4cd1 2024-03-18T14:50:35.161 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 18 14:50:35 smithi076 systemd[1]: Started Ceph mds.d for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:50:35.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:35 smithi076 ceph-mon[32205]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:35.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:35 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1148432626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:35.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:35 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:35.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:35 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:35.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:35 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:35.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:35 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:35.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:35 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:35.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:35 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:35.704 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:35 smithi077 ceph-mon[35865]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:35.704 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:35 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1148432626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:35.704 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:35 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:35.705 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:35 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:35.705 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:35 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:35.705 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:35 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:35.705 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:35 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:35.705 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:35 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:35.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:35 smithi174 ceph-mon[36787]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:35.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:35 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1148432626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:35.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:35 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:35.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:35 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:35.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:35 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:35.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:35 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:35.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:35 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:35.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:35 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:36.571 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:50:36.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:36 smithi076 ceph-mon[32205]: Deploying daemon mds.e on smithi077 2024-03-18T14:50:36.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:36 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.76:6836/3302809366,v1:172.21.15.76:6837/3302809366] up:boot 2024-03-18T14:50:36.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:36 smithi076 ceph-mon[32205]: fsmap 4 up:standby 2024-03-18T14:50:36.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:36 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T14:50:36.680 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:36 smithi077 ceph-mon[35865]: Deploying daemon mds.e on smithi077 2024-03-18T14:50:36.680 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:36 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.76:6836/3302809366,v1:172.21.15.76:6837/3302809366] up:boot 2024-03-18T14:50:36.680 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:36 smithi077 ceph-mon[35865]: fsmap 4 up:standby 2024-03-18T14:50:36.680 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:36 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T14:50:36.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.692+0000 7fd4132ea640 1 -- 172.21.15.76:0/1340966518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd40c101970 msgr2=0x7fd40c101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:36.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.692+0000 7fd4132ea640 1 --2- 172.21.15.76:0/1340966518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd40c101970 0x7fd40c101df0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd408009900 tx=0x7fd40802bc30 comp rx=0 tx=0).stop 2024-03-18T14:50:36.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.692+0000 7fd4132ea640 1 -- 172.21.15.76:0/1340966518 shutdown_connections 2024-03-18T14:50:36.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.692+0000 7fd4132ea640 1 --2- 172.21.15.76:0/1340966518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd40c102330 0x7fd40c108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:36.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.692+0000 7fd4132ea640 1 --2- 172.21.15.76:0/1340966518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd40c101970 0x7fd40c101df0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:36.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.692+0000 7fd4132ea640 1 --2- 172.21.15.76:0/1340966518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd40c100780 0x7fd40c100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:36.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.692+0000 7fd4132ea640 1 -- 172.21.15.76:0/1340966518 >> 172.21.15.76:0/1340966518 conn(0x7fd40c0fbd10 msgr2=0x7fd40c0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:36.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.692+0000 7fd4132ea640 1 -- 172.21.15.76:0/1340966518 shutdown_connections 2024-03-18T14:50:36.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.692+0000 7fd4132ea640 1 -- 172.21.15.76:0/1340966518 wait complete. 2024-03-18T14:50:36.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd4132ea640 1 Processor -- start 2024-03-18T14:50:36.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd4132ea640 1 -- start start 2024-03-18T14:50:36.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd4132ea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd40c100780 0x7fd40c197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:36.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd4132ea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd40c101970 0x7fd40c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:36.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd4132ea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd40c102330 0x7fd40c19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:36.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd4132ea640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd40c10c370 con 0x7fd40c102330 2024-03-18T14:50:36.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd4132ea640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd40c10c1f0 con 0x7fd40c101970 2024-03-18T14:50:36.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd4132ea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd40c10c4f0 con 0x7fd40c100780 2024-03-18T14:50:36.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd411887640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd40c102330 0x7fd40c19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:36.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd411887640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd40c102330 0x7fd40c19f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57538/0 (socket says 172.21.15.76:57538) 2024-03-18T14:50:36.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd411887640 1 -- 172.21.15.76:0/3333027751 learned_addr learned my addr 172.21.15.76:0/3333027751 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:36.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd410885640 1 --2- 172.21.15.76:0/3333027751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd40c101970 0x7fd40c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:36.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd411086640 1 --2- 172.21.15.76:0/3333027751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd40c100780 0x7fd40c197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:36.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd411887640 1 -- 172.21.15.76:0/3333027751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd40c100780 msgr2=0x7fd40c197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:36.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd411887640 1 --2- 172.21.15.76:0/3333027751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd40c100780 0x7fd40c197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:36.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd411887640 1 -- 172.21.15.76:0/3333027751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd40c101970 msgr2=0x7fd40c198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:36.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd411887640 1 --2- 172.21.15.76:0/3333027751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd40c101970 0x7fd40c198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:36.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.693+0000 7fd411887640 1 -- 172.21.15.76:0/3333027751 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40c19f760 con 0x7fd40c102330 2024-03-18T14:50:36.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.694+0000 7fd411887640 1 --2- 172.21.15.76:0/3333027751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd40c102330 0x7fd40c19f0f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd40400eaa0 tx=0x7fd40400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:36.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.694+0000 7fd3fe7fc640 1 -- 172.21.15.76:0/3333027751 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40400cdb0 con 0x7fd40c102330 2024-03-18T14:50:36.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.694+0000 7fd3fe7fc640 1 -- 172.21.15.76:0/3333027751 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd404005020 con 0x7fd40c102330 2024-03-18T14:50:36.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.694+0000 7fd4132ea640 1 -- 172.21.15.76:0/3333027751 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd40c19fa50 con 0x7fd40c102330 2024-03-18T14:50:36.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.694+0000 7fd3fe7fc640 1 -- 172.21.15.76:0/3333027751 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4040116b0 con 0x7fd40c102330 2024-03-18T14:50:36.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.694+0000 7fd4132ea640 1 -- 172.21.15.76:0/3333027751 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd40c19ff20 con 0x7fd40c102330 2024-03-18T14:50:36.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.695+0000 7fd4132ea640 1 -- 172.21.15.76:0/3333027751 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd40c1a0210 con 0x7fd40c102330 2024-03-18T14:50:36.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.696+0000 7fd3fe7fc640 1 -- 172.21.15.76:0/3333027751 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd404005560 con 0x7fd40c102330 2024-03-18T14:50:36.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.697+0000 7fd3fe7fc640 1 --2- 172.21.15.76:0/3333027751 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd3f80a2800 0x7fd3f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:36.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.697+0000 7fd411086640 1 --2- 172.21.15.76:0/3333027751 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd3f80a2800 0x7fd3f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:36.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.697+0000 7fd3fe7fc640 1 -- 172.21.15.76:0/3333027751 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd4040ce770 con 0x7fd40c102330 2024-03-18T14:50:36.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.697+0000 7fd411086640 1 --2- 172.21.15.76:0/3333027751 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd3f80a2800 0x7fd3f80a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd40c1017d0 tx=0x7fd40000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:36.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.701+0000 7fd3fe7fc640 1 -- 172.21.15.76:0/3333027751 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd404016030 con 0x7fd40c102330 2024-03-18T14:50:36.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:36 smithi174 ceph-mon[36787]: Deploying daemon mds.e on smithi077 2024-03-18T14:50:36.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:36 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.76:6836/3302809366,v1:172.21.15.76:6837/3302809366] up:boot 2024-03-18T14:50:36.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:36 smithi174 ceph-mon[36787]: fsmap 4 up:standby 2024-03-18T14:50:36.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:36 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T14:50:36.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.914+0000 7fd4132ea640 1 -- 172.21.15.76:0/3333027751 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd40c1a0480 con 0x7fd40c102330 2024-03-18T14:50:36.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.914+0000 7fd3fe7fc640 1 -- 172.21.15.76:0/3333027751 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7fd40409b660 con 0x7fd40c102330 2024-03-18T14:50:36.915 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:36.915 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-18T14:50:36.916 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 5 2024-03-18T14:50:36.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.916+0000 7fd4132ea640 1 -- 172.21.15.76:0/3333027751 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd3f80a2800 msgr2=0x7fd3f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:36.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.916+0000 7fd4132ea640 1 --2- 172.21.15.76:0/3333027751 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd3f80a2800 0x7fd3f80a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd40c1017d0 tx=0x7fd40000b390 comp rx=0 tx=0).stop 2024-03-18T14:50:36.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.916+0000 7fd4132ea640 1 -- 172.21.15.76:0/3333027751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd40c102330 msgr2=0x7fd40c19f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:36.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.916+0000 7fd4132ea640 1 --2- 172.21.15.76:0/3333027751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd40c102330 0x7fd40c19f0f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd40400eaa0 tx=0x7fd40400ef60 comp rx=0 tx=0).stop 2024-03-18T14:50:36.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.916+0000 7fd4132ea640 1 -- 172.21.15.76:0/3333027751 shutdown_connections 2024-03-18T14:50:36.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.916+0000 7fd4132ea640 1 --2- 172.21.15.76:0/3333027751 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd3f80a2800 0x7fd3f80a4cc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:36.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.916+0000 7fd4132ea640 1 --2- 172.21.15.76:0/3333027751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd40c102330 0x7fd40c19f0f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:36.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.916+0000 7fd4132ea640 1 --2- 172.21.15.76:0/3333027751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd40c101970 0x7fd40c198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:36.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.916+0000 7fd4132ea640 1 --2- 172.21.15.76:0/3333027751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd40c100780 0x7fd40c197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:36.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.916+0000 7fd4132ea640 1 -- 172.21.15.76:0/3333027751 >> 172.21.15.76:0/3333027751 conn(0x7fd40c0fbd10 msgr2=0x7fd40c0fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:36.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.916+0000 7fd4132ea640 1 -- 172.21.15.76:0/3333027751 shutdown_connections 2024-03-18T14:50:36.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:36.916+0000 7fd4132ea640 1 -- 172.21.15.76:0/3333027751 wait complete. 2024-03-18T14:50:37.377 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:50:37 smithi077 systemd[1]: Starting Ceph mds.e for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T14:50:37.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:37 smithi076 ceph-mon[32205]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:37.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:37 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3333027751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:37.685 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:37 smithi077 ceph-mon[35865]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:37.686 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:37 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3333027751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:37.686 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:50:37 smithi077 podman[65385]: 2024-03-18 14:50:37.279595905 +0000 UTC m=+0.017604998 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T14:50:37.686 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:50:37 smithi077 podman[65385]: 2024-03-18 14:50:37.383123939 +0000 UTC m=+0.121133073 container create d4bb2c6604eda4b0d04ae9368fe4baf9070cbc52621f73f2dfee45f33dda921d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-18T14:50:37.686 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:50:37 smithi077 podman[65385]: 2024-03-18 14:50:37.594237376 +0000 UTC m=+0.332246478 container init d4bb2c6604eda4b0d04ae9368fe4baf9070cbc52621f73f2dfee45f33dda921d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T14:50:37.686 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:50:37 smithi077 podman[65385]: 2024-03-18 14:50:37.597108157 +0000 UTC m=+0.335117250 container start d4bb2c6604eda4b0d04ae9368fe4baf9070cbc52621f73f2dfee45f33dda921d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-18T14:50:37.686 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:50:37 smithi077 ceph-mds[65404]: -- 172.21.15.77:0/2570030619 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5576b70c7a40 con 0x5576b709ac00 2024-03-18T14:50:37.686 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:50:37 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: starting mds.e at 2024-03-18T14:50:37.686 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:50:37 smithi077 bash[65385]: d4bb2c6604eda4b0d04ae9368fe4baf9070cbc52621f73f2dfee45f33dda921d 2024-03-18T14:50:37.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:37 smithi174 ceph-mon[36787]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:37.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:37 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3333027751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:37.977 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:50:37 smithi077 systemd[1]: Started Ceph mds.e for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:50:38.929 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.041+0000 7fec866c6640 1 -- 172.21.15.76:0/255725720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec80102300 msgr2=0x7fec801046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.041+0000 7fec866c6640 1 --2- 172.21.15.76:0/255725720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec80102300 0x7fec801046f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fec70009900 tx=0x7fec7002bc30 comp rx=0 tx=0).stop 2024-03-18T14:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.041+0000 7fec866c6640 1 -- 172.21.15.76:0/255725720 shutdown_connections 2024-03-18T14:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.041+0000 7fec866c6640 1 --2- 172.21.15.76:0/255725720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec80104c30 0x7fec80107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.041+0000 7fec866c6640 1 --2- 172.21.15.76:0/255725720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec80107560 0x7fec801099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.041+0000 7fec866c6640 1 --2- 172.21.15.76:0/255725720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec80102300 0x7fec801046f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:39.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.041+0000 7fec866c6640 1 -- 172.21.15.76:0/255725720 >> 172.21.15.76:0/255725720 conn(0x7fec800fbd10 msgr2=0x7fec800fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:39.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.041+0000 7fec866c6640 1 -- 172.21.15.76:0/255725720 shutdown_connections 2024-03-18T14:50:39.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.041+0000 7fec866c6640 1 -- 172.21.15.76:0/255725720 wait complete. 2024-03-18T14:50:39.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.042+0000 7fec866c6640 1 Processor -- start 2024-03-18T14:50:39.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.042+0000 7fec866c6640 1 -- start start 2024-03-18T14:50:39.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.042+0000 7fec866c6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec80102300 0x7fec80197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:39.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.042+0000 7fec866c6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec80104c30 0x7fec80198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:39.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.042+0000 7fec866c6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec80107560 0x7fec8019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:39.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.042+0000 7fec866c6640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fec8010d490 con 0x7fec80102300 2024-03-18T14:50:39.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.042+0000 7fec866c6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fec8010d310 con 0x7fec80104c30 2024-03-18T14:50:39.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.042+0000 7fec866c6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fec8010d610 con 0x7fec80107560 2024-03-18T14:50:39.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.042+0000 7fec7ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec80102300 0x7fec80197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:39.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.042+0000 7fec7ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec80102300 0x7fec80197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46984/0 (socket says 172.21.15.76:46984) 2024-03-18T14:50:39.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.042+0000 7fec7ffff640 1 -- 172.21.15.76:0/3169003699 learned_addr learned my addr 172.21.15.76:0/3169003699 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:39.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.042+0000 7fec7f7fe640 1 --2- 172.21.15.76:0/3169003699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec80104c30 0x7fec80198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:39.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.042+0000 7fec84c63640 1 --2- 172.21.15.76:0/3169003699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec80107560 0x7fec8019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:39.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.043+0000 7fec7ffff640 1 -- 172.21.15.76:0/3169003699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec80107560 msgr2=0x7fec8019f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:39.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.043+0000 7fec7ffff640 1 --2- 172.21.15.76:0/3169003699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec80107560 0x7fec8019f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:39.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.043+0000 7fec7ffff640 1 -- 172.21.15.76:0/3169003699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec80104c30 msgr2=0x7fec80198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:39.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.043+0000 7fec7ffff640 1 --2- 172.21.15.76:0/3169003699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec80104c30 0x7fec80198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:39.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.043+0000 7fec7ffff640 1 -- 172.21.15.76:0/3169003699 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8019f760 con 0x7fec80102300 2024-03-18T14:50:39.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.043+0000 7fec7ffff640 1 --2- 172.21.15.76:0/3169003699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec80102300 0x7fec80197b40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fec70009230 tx=0x7fec70005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:39.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.043+0000 7fec7d7fa640 1 -- 172.21.15.76:0/3169003699 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70041070 con 0x7fec80102300 2024-03-18T14:50:39.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.043+0000 7fec7d7fa640 1 -- 172.21.15.76:0/3169003699 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec700054e0 con 0x7fec80102300 2024-03-18T14:50:39.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.043+0000 7fec866c6640 1 -- 172.21.15.76:0/3169003699 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec8019f9f0 con 0x7fec80102300 2024-03-18T14:50:39.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.043+0000 7fec7d7fa640 1 -- 172.21.15.76:0/3169003699 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70034440 con 0x7fec80102300 2024-03-18T14:50:39.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.043+0000 7fec866c6640 1 -- 172.21.15.76:0/3169003699 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec8019fee0 con 0x7fec80102300 2024-03-18T14:50:39.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.044+0000 7fec866c6640 1 -- 172.21.15.76:0/3169003699 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec800fd800 con 0x7fec80102300 2024-03-18T14:50:39.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.045+0000 7fec7d7fa640 1 -- 172.21.15.76:0/3169003699 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fec70044070 con 0x7fec80102300 2024-03-18T14:50:39.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.045+0000 7fec7d7fa640 1 --2- 172.21.15.76:0/3169003699 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fec680a2800 0x7fec680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:39.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.046+0000 7fec7f7fe640 1 --2- 172.21.15.76:0/3169003699 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fec680a2800 0x7fec680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:39.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.046+0000 7fec7d7fa640 1 -- 172.21.15.76:0/3169003699 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fec700f0220 con 0x7fec80102300 2024-03-18T14:50:39.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.046+0000 7fec7f7fe640 1 --2- 172.21.15.76:0/3169003699 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fec680a2800 0x7fec680a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fec80199050 tx=0x7fec7400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:39.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.050+0000 7fec7d7fa640 1 -- 172.21.15.76:0/3169003699 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fec700bd110 con 0x7fec80102300 2024-03-18T14:50:39.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:38 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:39.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:38 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:39.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:38 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:39.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:38 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:39.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:38 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:39.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:38 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:39.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:38 smithi076 ceph-mon[32205]: Deploying daemon mds.f on smithi174 2024-03-18T14:50:39.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:38 smithi076 ceph-mon[32205]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:39.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:38 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.77:6834/3551464902,v1:172.21.15.77:6835/3551464902] up:boot 2024-03-18T14:50:39.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:38 smithi076 ceph-mon[32205]: fsmap 5 up:standby 2024-03-18T14:50:39.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:38 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:50:39.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:38 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:39.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:38 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:39.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:38 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:39.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:38 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:39.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:38 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:39.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:38 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:39.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:38 smithi174 ceph-mon[36787]: Deploying daemon mds.f on smithi174 2024-03-18T14:50:39.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:38 smithi174 ceph-mon[36787]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:39.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:38 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.77:6834/3551464902,v1:172.21.15.77:6835/3551464902] up:boot 2024-03-18T14:50:39.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:38 smithi174 ceph-mon[36787]: fsmap 5 up:standby 2024-03-18T14:50:39.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:38 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:50:39.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:38 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:39.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:38 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:39.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:38 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:39.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:38 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:39.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:38 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:39.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:38 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:39.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:38 smithi077 ceph-mon[35865]: Deploying daemon mds.f on smithi174 2024-03-18T14:50:39.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:38 smithi077 ceph-mon[35865]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:39.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:38 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.77:6834/3551464902,v1:172.21.15.77:6835/3551464902] up:boot 2024-03-18T14:50:39.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:38 smithi077 ceph-mon[35865]: fsmap 5 up:standby 2024-03-18T14:50:39.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:38 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:50:39.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.264+0000 7fec866c6640 1 -- 172.21.15.76:0/3169003699 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec8002d1b0 con 0x7fec80102300 2024-03-18T14:50:39.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.264+0000 7fec7d7fa640 1 -- 172.21.15.76:0/3169003699 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7fec700c11b0 con 0x7fec80102300 2024-03-18T14:50:39.265 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:39.265 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-18T14:50:39.265 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 6 2024-03-18T14:50:39.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.266+0000 7fec866c6640 1 -- 172.21.15.76:0/3169003699 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fec680a2800 msgr2=0x7fec680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:39.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.266+0000 7fec866c6640 1 --2- 172.21.15.76:0/3169003699 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fec680a2800 0x7fec680a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fec80199050 tx=0x7fec7400b410 comp rx=0 tx=0).stop 2024-03-18T14:50:39.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.266+0000 7fec866c6640 1 -- 172.21.15.76:0/3169003699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec80102300 msgr2=0x7fec80197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:39.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.266+0000 7fec866c6640 1 --2- 172.21.15.76:0/3169003699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec80102300 0x7fec80197b40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fec70009230 tx=0x7fec70005dd0 comp rx=0 tx=0).stop 2024-03-18T14:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.266+0000 7fec866c6640 1 -- 172.21.15.76:0/3169003699 shutdown_connections 2024-03-18T14:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.266+0000 7fec866c6640 1 --2- 172.21.15.76:0/3169003699 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fec680a2800 0x7fec680a4cc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.266+0000 7fec866c6640 1 --2- 172.21.15.76:0/3169003699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec80107560 0x7fec8019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.266+0000 7fec866c6640 1 --2- 172.21.15.76:0/3169003699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec80104c30 0x7fec80198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.266+0000 7fec866c6640 1 --2- 172.21.15.76:0/3169003699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec80102300 0x7fec80197b40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.266+0000 7fec866c6640 1 -- 172.21.15.76:0/3169003699 >> 172.21.15.76:0/3169003699 conn(0x7fec800fbd10 msgr2=0x7fec80102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.266+0000 7fec866c6640 1 -- 172.21.15.76:0/3169003699 shutdown_connections 2024-03-18T14:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:39.266+0000 7fec866c6640 1 -- 172.21.15.76:0/3169003699 wait complete. 2024-03-18T14:50:39.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:39 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3169003699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:39.967 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 14:50:39 smithi174 systemd[1]: Starting Ceph mds.f for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T14:50:40.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:39 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3169003699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:40.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:39 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3169003699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:40.341 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 14:50:40 smithi174 podman[66985]: 2024-03-18 14:50:39.989170755 +0000 UTC m=+0.017125925 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T14:50:40.341 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 14:50:40 smithi174 podman[66985]: 2024-03-18 14:50:40.104926964 +0000 UTC m=+0.132882163 container create 45a90165cc166940f0e6036ebca8d6222c991402c47daa71dd11e950af99ad3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-f, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-18T14:50:40.341 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 14:50:40 smithi174 podman[66985]: 2024-03-18 14:50:40.34103974 +0000 UTC m=+0.368994909 container init 45a90165cc166940f0e6036ebca8d6222c991402c47daa71dd11e950af99ad3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-f, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T14:50:40.717 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 14:50:40 smithi174 podman[66985]: 2024-03-18 14:50:40.343904684 +0000 UTC m=+0.371859833 container start 45a90165cc166940f0e6036ebca8d6222c991402c47daa71dd11e950af99ad3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-f, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-18T14:50:40.717 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 14:50:40 smithi174 ceph-mds[67005]: -- 172.21.15.174:0/3942075234 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562759a11a40 con 0x5627599e4800 2024-03-18T14:50:40.717 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 14:50:40 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-f[67001]: starting mds.f at 2024-03-18T14:50:40.717 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 14:50:40 smithi174 bash[66985]: 45a90165cc166940f0e6036ebca8d6222c991402c47daa71dd11e950af99ad3c 2024-03-18T14:50:40.717 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 14:50:40 smithi174 systemd[1]: Started Ceph mds.f for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:50:41.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:40 smithi076 ceph-mon[32205]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:41.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:41.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:41.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:41.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:41.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:41.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:40 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:41.163 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:40 smithi174 ceph-mon[36787]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:41.163 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:41.163 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:41.163 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:41.163 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:41.163 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:41.163 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:40 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:41.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:40 smithi077 ceph-mon[35865]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:41.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:41.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:41.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:41.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:41.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:41.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:40 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:41.279 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:50:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.390+0000 7fda8b0f8640 1 -- 172.21.15.76:0/2480788690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda841018f0 msgr2=0x7fda84101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.390+0000 7fda8b0f8640 1 --2- 172.21.15.76:0/2480788690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda841018f0 0x7fda84101d70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fda70009900 tx=0x7fda7002bc30 comp rx=0 tx=0).stop 2024-03-18T14:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.390+0000 7fda8b0f8640 1 -- 172.21.15.76:0/2480788690 shutdown_connections 2024-03-18T14:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.390+0000 7fda8b0f8640 1 --2- 172.21.15.76:0/2480788690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda841022b0 0x7fda84108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.390+0000 7fda8b0f8640 1 --2- 172.21.15.76:0/2480788690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda841018f0 0x7fda84101d70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.390+0000 7fda8b0f8640 1 --2- 172.21.15.76:0/2480788690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda84100700 0x7fda84100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.390+0000 7fda8b0f8640 1 -- 172.21.15.76:0/2480788690 >> 172.21.15.76:0/2480788690 conn(0x7fda840fbc70 msgr2=0x7fda840fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.390+0000 7fda8b0f8640 1 -- 172.21.15.76:0/2480788690 shutdown_connections 2024-03-18T14:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.390+0000 7fda8b0f8640 1 -- 172.21.15.76:0/2480788690 wait complete. 2024-03-18T14:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.390+0000 7fda8b0f8640 1 Processor -- start 2024-03-18T14:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.390+0000 7fda8b0f8640 1 -- start start 2024-03-18T14:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda8b0f8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda84100700 0x7fda84197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda8b0f8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda841018f0 0x7fda84198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda8b0f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda841022b0 0x7fda8419f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda8b0f8640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fda8410c380 con 0x7fda841018f0 2024-03-18T14:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda8b0f8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fda8410c200 con 0x7fda841022b0 2024-03-18T14:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda8b0f8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fda8410c500 con 0x7fda84100700 2024-03-18T14:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda83fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda841018f0 0x7fda84198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda83fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda841018f0 0x7fda84198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46996/0 (socket says 172.21.15.76:46996) 2024-03-18T14:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda83fff640 1 -- 172.21.15.76:0/3971107836 learned_addr learned my addr 172.21.15.76:0/3971107836 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda88e94640 1 --2- 172.21.15.76:0/3971107836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda84100700 0x7fda84197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda89695640 1 --2- 172.21.15.76:0/3971107836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda841022b0 0x7fda8419f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda83fff640 1 -- 172.21.15.76:0/3971107836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda84100700 msgr2=0x7fda84197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda83fff640 1 --2- 172.21.15.76:0/3971107836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda84100700 0x7fda84197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda83fff640 1 -- 172.21.15.76:0/3971107836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda841022b0 msgr2=0x7fda8419f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda83fff640 1 --2- 172.21.15.76:0/3971107836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda841022b0 0x7fda8419f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda83fff640 1 -- 172.21.15.76:0/3971107836 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8419f760 con 0x7fda841018f0 2024-03-18T14:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda83fff640 1 --2- 172.21.15.76:0/3971107836 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda841018f0 0x7fda84198080 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fda70009230 tx=0x7fda70005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda81ffb640 1 -- 172.21.15.76:0/3971107836 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda70041070 con 0x7fda841018f0 2024-03-18T14:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda81ffb640 1 -- 172.21.15.76:0/3971107836 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda700044d0 con 0x7fda841018f0 2024-03-18T14:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda81ffb640 1 -- 172.21.15.76:0/3971107836 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda70034440 con 0x7fda841018f0 2024-03-18T14:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.391+0000 7fda8b0f8640 1 -- 172.21.15.76:0/3971107836 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda8419f9f0 con 0x7fda841018f0 2024-03-18T14:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.392+0000 7fda8b0f8640 1 -- 172.21.15.76:0/3971107836 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda8419ff30 con 0x7fda841018f0 2024-03-18T14:50:41.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.393+0000 7fda8b0f8640 1 -- 172.21.15.76:0/3971107836 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda841a0160 con 0x7fda841018f0 2024-03-18T14:50:41.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.394+0000 7fda81ffb640 1 -- 172.21.15.76:0/3971107836 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fda70044070 con 0x7fda841018f0 2024-03-18T14:50:41.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.394+0000 7fda81ffb640 1 --2- 172.21.15.76:0/3971107836 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fda740a2880 0x7fda740a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:41.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.394+0000 7fda88e94640 1 --2- 172.21.15.76:0/3971107836 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fda740a2880 0x7fda740a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:41.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.394+0000 7fda81ffb640 1 -- 172.21.15.76:0/3971107836 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fda700f0140 con 0x7fda841018f0 2024-03-18T14:50:41.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.395+0000 7fda88e94640 1 --2- 172.21.15.76:0/3971107836 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fda740a2880 0x7fda740a4d40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fda78005fd0 tx=0x7fda7800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:41.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.399+0000 7fda81ffb640 1 -- 172.21.15.76:0/3971107836 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fda700bd030 con 0x7fda841018f0 2024-03-18T14:50:41.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.612+0000 7fda8b0f8640 1 -- 172.21.15.76:0/3971107836 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda841a0330 con 0x7fda841018f0 2024-03-18T14:50:41.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.613+0000 7fda81ffb640 1 -- 172.21.15.76:0/3971107836 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7fda700c10d0 con 0x7fda841018f0 2024-03-18T14:50:41.614 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:41.614 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-18T14:50:41.614 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 7 2024-03-18T14:50:41.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.615+0000 7fda8b0f8640 1 -- 172.21.15.76:0/3971107836 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fda740a2880 msgr2=0x7fda740a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:41.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.615+0000 7fda8b0f8640 1 --2- 172.21.15.76:0/3971107836 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fda740a2880 0x7fda740a4d40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fda78005fd0 tx=0x7fda7800b410 comp rx=0 tx=0).stop 2024-03-18T14:50:41.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.615+0000 7fda8b0f8640 1 -- 172.21.15.76:0/3971107836 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda841018f0 msgr2=0x7fda84198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:41.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.615+0000 7fda8b0f8640 1 --2- 172.21.15.76:0/3971107836 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda841018f0 0x7fda84198080 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fda70009230 tx=0x7fda70005d50 comp rx=0 tx=0).stop 2024-03-18T14:50:41.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.615+0000 7fda8b0f8640 1 -- 172.21.15.76:0/3971107836 shutdown_connections 2024-03-18T14:50:41.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.615+0000 7fda8b0f8640 1 --2- 172.21.15.76:0/3971107836 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fda740a2880 0x7fda740a4d40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:41.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.615+0000 7fda8b0f8640 1 --2- 172.21.15.76:0/3971107836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda841022b0 0x7fda8419f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:41.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.615+0000 7fda8b0f8640 1 --2- 172.21.15.76:0/3971107836 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda841018f0 0x7fda84198080 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:41.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.615+0000 7fda8b0f8640 1 --2- 172.21.15.76:0/3971107836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda84100700 0x7fda84197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:41.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.615+0000 7fda8b0f8640 1 -- 172.21.15.76:0/3971107836 >> 172.21.15.76:0/3971107836 conn(0x7fda840fbc70 msgr2=0x7fda84068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:41.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.615+0000 7fda8b0f8640 1 -- 172.21.15.76:0/3971107836 shutdown_connections 2024-03-18T14:50:41.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:41.615+0000 7fda8b0f8640 1 -- 172.21.15.76:0/3971107836 wait complete. 2024-03-18T14:50:41.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:41 smithi076 ceph-mon[32205]: Deploying daemon mds.c on smithi174 2024-03-18T14:50:41.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:41 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.174:6834/603893267,v1:172.21.15.174:6835/603893267] up:boot 2024-03-18T14:50:41.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:41 smithi076 ceph-mon[32205]: fsmap 6 up:standby 2024-03-18T14:50:41.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:41 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T14:50:41.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:41 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3971107836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:42.099 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:41 smithi174 ceph-mon[36787]: Deploying daemon mds.c on smithi174 2024-03-18T14:50:42.099 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:41 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.174:6834/603893267,v1:172.21.15.174:6835/603893267] up:boot 2024-03-18T14:50:42.099 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:41 smithi174 ceph-mon[36787]: fsmap 6 up:standby 2024-03-18T14:50:42.099 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:41 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T14:50:42.100 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:41 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3971107836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:42.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:41 smithi077 ceph-mon[35865]: Deploying daemon mds.c on smithi174 2024-03-18T14:50:42.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:41 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.174:6834/603893267,v1:172.21.15.174:6835/603893267] up:boot 2024-03-18T14:50:42.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:41 smithi077 ceph-mon[35865]: fsmap 6 up:standby 2024-03-18T14:50:42.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:41 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T14:50:42.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:41 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3971107836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:42.717 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:50:42 smithi174 systemd[1]: Starting Ceph mds.c for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T14:50:43.079 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:42 smithi174 ceph-mon[36787]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:43.079 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:50:42 smithi174 podman[67295]: 2024-03-18 14:50:42.708269846 +0000 UTC m=+0.017727178 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T14:50:43.079 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:50:42 smithi174 podman[67295]: 2024-03-18 14:50:42.809334938 +0000 UTC m=+0.118792284 container create 717e2533657d739e5f0eac09689104a830e2e4b0046635a377e0021f6f2b8783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-03-18T14:50:43.079 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:50:43 smithi174 podman[67295]: 2024-03-18 14:50:43.079142617 +0000 UTC m=+0.388599961 container init 717e2533657d739e5f0eac09689104a830e2e4b0046635a377e0021f6f2b8783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-18T14:50:43.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:42 smithi076 ceph-mon[32205]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:43.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:42 smithi077 ceph-mon[35865]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:43.467 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:50:43 smithi174 podman[67295]: 2024-03-18 14:50:43.08176348 +0000 UTC m=+0.391220798 container start 717e2533657d739e5f0eac09689104a830e2e4b0046635a377e0021f6f2b8783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-18T14:50:43.467 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:50:43 smithi174 ceph-mds[67314]: -- 172.21.15.174:0/3663899769 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5587db29da40 con 0x5587db270c00 2024-03-18T14:50:43.467 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:50:43 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: starting mds.c at 2024-03-18T14:50:43.467 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:50:43 smithi174 bash[67295]: 717e2533657d739e5f0eac09689104a830e2e4b0046635a377e0021f6f2b8783 2024-03-18T14:50:43.467 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:50:43 smithi174 systemd[1]: Started Ceph mds.c for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:50:43.628 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:50:43.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.742+0000 7f89ce297640 1 -- 172.21.15.76:0/1757937136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89c8101970 msgr2=0x7f89c8101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:43.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.742+0000 7f89ce297640 1 --2- 172.21.15.76:0/1757937136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89c8101970 0x7f89c8101df0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f89bc009900 tx=0x7f89bc02bc30 comp rx=0 tx=0).stop 2024-03-18T14:50:43.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.742+0000 7f89ce297640 1 -- 172.21.15.76:0/1757937136 shutdown_connections 2024-03-18T14:50:43.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.742+0000 7f89ce297640 1 --2- 172.21.15.76:0/1757937136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89c8102330 0x7f89c8108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:43.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.742+0000 7f89ce297640 1 --2- 172.21.15.76:0/1757937136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89c8101970 0x7f89c8101df0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:43.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.742+0000 7f89ce297640 1 --2- 172.21.15.76:0/1757937136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89c8100780 0x7f89c8100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:43.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.742+0000 7f89ce297640 1 -- 172.21.15.76:0/1757937136 >> 172.21.15.76:0/1757937136 conn(0x7f89c80fbd10 msgr2=0x7f89c80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:43.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.742+0000 7f89ce297640 1 -- 172.21.15.76:0/1757937136 shutdown_connections 2024-03-18T14:50:43.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.742+0000 7f89ce297640 1 -- 172.21.15.76:0/1757937136 wait complete. 2024-03-18T14:50:43.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89ce297640 1 Processor -- start 2024-03-18T14:50:43.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89ce297640 1 -- start start 2024-03-18T14:50:43.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89ce297640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89c8100780 0x7f89c8197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:43.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89ce297640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89c8101970 0x7f89c8197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:43.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89ce297640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89c8102330 0x7f89c819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:43.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89ce297640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f89c810c370 con 0x7f89c8101970 2024-03-18T14:50:43.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89ce297640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f89c810c1f0 con 0x7f89c8102330 2024-03-18T14:50:43.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89ce297640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f89c810c4f0 con 0x7f89c8100780 2024-03-18T14:50:43.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89c77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89c8101970 0x7f89c8197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:43.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89c77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89c8101970 0x7f89c8197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47008/0 (socket says 172.21.15.76:47008) 2024-03-18T14:50:43.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89c77fe640 1 -- 172.21.15.76:0/3820540868 learned_addr learned my addr 172.21.15.76:0/3820540868 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:43.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89cc834640 1 --2- 172.21.15.76:0/3820540868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89c8102330 0x7f89c819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:43.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89c7fff640 1 --2- 172.21.15.76:0/3820540868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89c8100780 0x7f89c8197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:43.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89c77fe640 1 -- 172.21.15.76:0/3820540868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89c8100780 msgr2=0x7f89c8197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:43.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89c77fe640 1 --2- 172.21.15.76:0/3820540868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89c8100780 0x7f89c8197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:43.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89c77fe640 1 -- 172.21.15.76:0/3820540868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89c8102330 msgr2=0x7f89c819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:43.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89c77fe640 1 --2- 172.21.15.76:0/3820540868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89c8102330 0x7f89c819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:43.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.743+0000 7f89c77fe640 1 -- 172.21.15.76:0/3820540868 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c819f690 con 0x7f89c8101970 2024-03-18T14:50:43.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.744+0000 7f89c77fe640 1 --2- 172.21.15.76:0/3820540868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89c8101970 0x7f89c8197fb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f89bc009230 tx=0x7f89bc005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:43.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.744+0000 7f89c57fa640 1 -- 172.21.15.76:0/3820540868 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc041070 con 0x7f89c8101970 2024-03-18T14:50:43.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.744+0000 7f89ce297640 1 -- 172.21.15.76:0/3820540868 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c819f920 con 0x7f89c8101970 2024-03-18T14:50:43.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.744+0000 7f89c57fa640 1 -- 172.21.15.76:0/3820540868 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89bc0044d0 con 0x7f89c8101970 2024-03-18T14:50:43.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.744+0000 7f89c57fa640 1 -- 172.21.15.76:0/3820540868 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc034440 con 0x7f89c8101970 2024-03-18T14:50:43.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.744+0000 7f89ce297640 1 -- 172.21.15.76:0/3820540868 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c819fe60 con 0x7f89c8101970 2024-03-18T14:50:43.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.745+0000 7f89ce297640 1 -- 172.21.15.76:0/3820540868 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c81a0090 con 0x7f89c8101970 2024-03-18T14:50:43.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.746+0000 7f89c57fa640 1 -- 172.21.15.76:0/3820540868 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f89bc044070 con 0x7f89c8101970 2024-03-18T14:50:43.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.746+0000 7f89c57fa640 1 --2- 172.21.15.76:0/3820540868 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89b00a2800 0x7f89b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:43.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.747+0000 7f89c7fff640 1 --2- 172.21.15.76:0/3820540868 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89b00a2800 0x7f89b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:43.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.747+0000 7f89c57fa640 1 -- 172.21.15.76:0/3820540868 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f89bc0f0170 con 0x7f89c8101970 2024-03-18T14:50:43.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.747+0000 7f89c7fff640 1 --2- 172.21.15.76:0/3820540868 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89b00a2800 0x7f89b00a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f89c81017d0 tx=0x7f89b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:43.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.751+0000 7f89c57fa640 1 -- 172.21.15.76:0/3820540868 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f89bc0bd060 con 0x7f89c8101970 2024-03-18T14:50:43.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.963+0000 7f89ce297640 1 -- 172.21.15.76:0/3820540868 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89c81a0260 con 0x7f89c8101970 2024-03-18T14:50:43.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.964+0000 7f89c57fa640 1 -- 172.21.15.76:0/3820540868 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6346 (secure 0 0 0) 0x7f89bc0c1100 con 0x7f89c8101970 2024-03-18T14:50:43.965 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:43.965 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-18T14:50:43.965 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 8 2024-03-18T14:50:43.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.966+0000 7f89ce297640 1 -- 172.21.15.76:0/3820540868 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89b00a2800 msgr2=0x7f89b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:43.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.966+0000 7f89ce297640 1 --2- 172.21.15.76:0/3820540868 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89b00a2800 0x7f89b00a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f89c81017d0 tx=0x7f89b8005f90 comp rx=0 tx=0).stop 2024-03-18T14:50:43.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.966+0000 7f89ce297640 1 -- 172.21.15.76:0/3820540868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89c8101970 msgr2=0x7f89c8197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:43.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.966+0000 7f89ce297640 1 --2- 172.21.15.76:0/3820540868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89c8101970 0x7f89c8197fb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f89bc009230 tx=0x7f89bc005d50 comp rx=0 tx=0).stop 2024-03-18T14:50:43.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.966+0000 7f89ce297640 1 -- 172.21.15.76:0/3820540868 shutdown_connections 2024-03-18T14:50:43.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.966+0000 7f89ce297640 1 --2- 172.21.15.76:0/3820540868 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89b00a2800 0x7f89b00a4cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:43.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.966+0000 7f89ce297640 1 --2- 172.21.15.76:0/3820540868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89c8102330 0x7f89c819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:43.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.966+0000 7f89ce297640 1 --2- 172.21.15.76:0/3820540868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89c8101970 0x7f89c8197fb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:43.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.966+0000 7f89ce297640 1 --2- 172.21.15.76:0/3820540868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89c8100780 0x7f89c8197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:43.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.966+0000 7f89ce297640 1 -- 172.21.15.76:0/3820540868 >> 172.21.15.76:0/3820540868 conn(0x7f89c80fbd10 msgr2=0x7f89c80fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:43.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.966+0000 7f89ce297640 1 -- 172.21.15.76:0/3820540868 shutdown_connections 2024-03-18T14:50:43.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:43.966+0000 7f89ce297640 1 -- 172.21.15.76:0/3820540868 wait complete. 2024-03-18T14:50:44.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:44 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:44.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:44 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:44.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:44 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:44.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:44 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:44.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:44 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:44.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:44 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:44.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:44 smithi076 ceph-mon[32205]: Deploying daemon mds.h on smithi077 2024-03-18T14:50:44.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:44 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] up:boot 2024-03-18T14:50:44.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:44 smithi076 ceph-mon[32205]: fsmap 7 up:standby 2024-03-18T14:50:44.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:44 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T14:50:44.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:44 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3820540868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:44.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:44 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:44.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:44 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:44.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:44 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:44.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:44 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:44.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:44 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:44.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:44 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:44.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:44 smithi174 ceph-mon[36787]: Deploying daemon mds.h on smithi077 2024-03-18T14:50:44.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:44 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] up:boot 2024-03-18T14:50:44.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:44 smithi174 ceph-mon[36787]: fsmap 7 up:standby 2024-03-18T14:50:44.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:44 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T14:50:44.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:44 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3820540868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:44.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:44 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:44.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:44 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:44.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:44 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:44.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:44 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:44.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:44 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:44.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:44 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:44.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:44 smithi077 ceph-mon[35865]: Deploying daemon mds.h on smithi077 2024-03-18T14:50:44.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:44 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] up:boot 2024-03-18T14:50:44.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:44 smithi077 ceph-mon[35865]: fsmap 7 up:standby 2024-03-18T14:50:44.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:44 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T14:50:44.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:44 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3820540868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:45.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:45 smithi077 ceph-mon[35865]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:45.589 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:45 smithi076 ceph-mon[32205]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:45.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:45 smithi174 ceph-mon[36787]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:45.978 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:50:46.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.080+0000 7f9fbe189640 1 -- 172.21.15.76:0/3313985492 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fb81022b0 msgr2=0x7f9fb8108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:46.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.080+0000 7f9fbe189640 1 --2- 172.21.15.76:0/3313985492 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fb81022b0 0x7f9fb8108b40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9fb0009900 tx=0x7f9fb002bc30 comp rx=0 tx=0).stop 2024-03-18T14:50:46.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.080+0000 7f9fbe189640 1 -- 172.21.15.76:0/3313985492 shutdown_connections 2024-03-18T14:50:46.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.080+0000 7f9fbe189640 1 --2- 172.21.15.76:0/3313985492 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fb81022b0 0x7f9fb8108b40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:46.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.080+0000 7f9fbe189640 1 --2- 172.21.15.76:0/3313985492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fb81018f0 0x7f9fb8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:46.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.080+0000 7f9fbe189640 1 --2- 172.21.15.76:0/3313985492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fb8100700 0x7f9fb8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:46.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.080+0000 7f9fbe189640 1 -- 172.21.15.76:0/3313985492 >> 172.21.15.76:0/3313985492 conn(0x7f9fb80fbc70 msgr2=0x7f9fb80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:46.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.081+0000 7f9fbe189640 1 -- 172.21.15.76:0/3313985492 shutdown_connections 2024-03-18T14:50:46.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.081+0000 7f9fbe189640 1 -- 172.21.15.76:0/3313985492 wait complete. 2024-03-18T14:50:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.081+0000 7f9fbe189640 1 Processor -- start 2024-03-18T14:50:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.081+0000 7f9fbe189640 1 -- start start 2024-03-18T14:50:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.081+0000 7f9fbe189640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fb8100700 0x7f9fb8199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.081+0000 7f9fbe189640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fb81018f0 0x7f9fb819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.081+0000 7f9fbe189640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fb81022b0 0x7f9fb81a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.081+0000 7f9fbe189640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb810c380 con 0x7f9fb81022b0 2024-03-18T14:50:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.081+0000 7f9fbe189640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb810c200 con 0x7f9fb8100700 2024-03-18T14:50:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.081+0000 7f9fbe189640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb810c500 con 0x7f9fb81018f0 2024-03-18T14:50:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fb7fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fb81022b0 0x7f9fb81a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fb77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fb8100700 0x7f9fb8199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fb77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fb8100700 0x7f9fb8199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:51168/0 (socket says 172.21.15.76:51168) 2024-03-18T14:50:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fb7fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fb81022b0 0x7f9fb81a1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47026/0 (socket says 172.21.15.76:47026) 2024-03-18T14:50:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fb77fe640 1 -- 172.21.15.76:0/2454763768 learned_addr learned my addr 172.21.15.76:0/2454763768 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fb6ffd640 1 --2- 172.21.15.76:0/2454763768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fb81018f0 0x7f9fb819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fb77fe640 1 -- 172.21.15.76:0/2454763768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fb81018f0 msgr2=0x7f9fb819a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fb77fe640 1 --2- 172.21.15.76:0/2454763768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fb81018f0 0x7f9fb819a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fb77fe640 1 -- 172.21.15.76:0/2454763768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fb81022b0 msgr2=0x7f9fb81a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fb77fe640 1 --2- 172.21.15.76:0/2454763768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fb81022b0 0x7f9fb81a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fb77fe640 1 -- 172.21.15.76:0/2454763768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb81a1920 con 0x7f9fb8100700 2024-03-18T14:50:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fb7fff640 1 --2- 172.21.15.76:0/2454763768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fb81022b0 0x7f9fb81a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:50:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fb77fe640 1 --2- 172.21.15.76:0/2454763768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fb8100700 0x7f9fb8199d10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800eaa0 tx=0x7f9fa800ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fb4ff9640 1 -- 172.21.15.76:0/2454763768 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8016070 con 0x7f9fb8100700 2024-03-18T14:50:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fbe189640 1 -- 172.21.15.76:0/2454763768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fb81a1c10 con 0x7f9fb8100700 2024-03-18T14:50:46.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fbe189640 1 -- 172.21.15.76:0/2454763768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fb81a2160 con 0x7f9fb8100700 2024-03-18T14:50:46.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fb4ff9640 1 -- 172.21.15.76:0/2454763768 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fa8003e50 con 0x7f9fb8100700 2024-03-18T14:50:46.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.082+0000 7f9fb4ff9640 1 -- 172.21.15.76:0/2454763768 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa80116d0 con 0x7f9fb8100700 2024-03-18T14:50:46.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.083+0000 7f9fbe189640 1 -- 172.21.15.76:0/2454763768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fb81a2390 con 0x7f9fb8100700 2024-03-18T14:50:46.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.084+0000 7f9fb4ff9640 1 -- 172.21.15.76:0/2454763768 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9fa8019070 con 0x7f9fb8100700 2024-03-18T14:50:46.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.085+0000 7f9fb4ff9640 1 --2- 172.21.15.76:0/2454763768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9fa00a2800 0x7f9fa00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:46.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.085+0000 7f9fb6ffd640 1 --2- 172.21.15.76:0/2454763768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9fa00a2800 0x7f9fa00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:46.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.085+0000 7f9fb4ff9640 1 -- 172.21.15.76:0/2454763768 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f9fa80ceba0 con 0x7f9fb8100700 2024-03-18T14:50:46.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.086+0000 7f9fb6ffd640 1 --2- 172.21.15.76:0/2454763768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9fa00a2800 0x7f9fa00a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9fac0095f0 tx=0x7f9fac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:46.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.089+0000 7f9fb4ff9640 1 -- 172.21.15.76:0/2454763768 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9fa80cee20 con 0x7f9fb8100700 2024-03-18T14:50:46.227 INFO:journalctl@ceph.mds.h.smithi077.stdout:Mar 18 14:50:46 smithi077 systemd[1]: Started Ceph mds.h for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:50:46.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.311+0000 7f9fbe189640 1 -- 172.21.15.76:0/2454763768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9fb81a2560 con 0x7f9fb8100700 2024-03-18T14:50:46.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.312+0000 7f9fb4ff9640 1 -- 172.21.15.76:0/2454763768 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7171 (secure 0 0 0) 0x7f9fa809b9f0 con 0x7f9fb8100700 2024-03-18T14:50:46.313 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:46.313 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-18T14:50:46.313 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 9 2024-03-18T14:50:46.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.314+0000 7f9fbe189640 1 -- 172.21.15.76:0/2454763768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9fa00a2800 msgr2=0x7f9fa00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:46.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.314+0000 7f9fbe189640 1 --2- 172.21.15.76:0/2454763768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9fa00a2800 0x7f9fa00a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9fac0095f0 tx=0x7f9fac005f90 comp rx=0 tx=0).stop 2024-03-18T14:50:46.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.314+0000 7f9fbe189640 1 -- 172.21.15.76:0/2454763768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fb8100700 msgr2=0x7f9fb8199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:46.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.314+0000 7f9fbe189640 1 --2- 172.21.15.76:0/2454763768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fb8100700 0x7f9fb8199d10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800eaa0 tx=0x7f9fa800ef60 comp rx=0 tx=0).stop 2024-03-18T14:50:46.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.314+0000 7f9fbe189640 1 -- 172.21.15.76:0/2454763768 shutdown_connections 2024-03-18T14:50:46.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.314+0000 7f9fbe189640 1 --2- 172.21.15.76:0/2454763768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9fa00a2800 0x7f9fa00a4cc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:46.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.314+0000 7f9fbe189640 1 --2- 172.21.15.76:0/2454763768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fb81018f0 0x7f9fb819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:46.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.314+0000 7f9fbe189640 1 --2- 172.21.15.76:0/2454763768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fb81022b0 0x7f9fb81a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:46.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.314+0000 7f9fbe189640 1 --2- 172.21.15.76:0/2454763768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fb8100700 0x7f9fb8199d10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:46.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.314+0000 7f9fbe189640 1 -- 172.21.15.76:0/2454763768 >> 172.21.15.76:0/2454763768 conn(0x7f9fb80fbc70 msgr2=0x7f9fb80fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:46.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.314+0000 7f9fbe189640 1 -- 172.21.15.76:0/2454763768 shutdown_connections 2024-03-18T14:50:46.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:46.314+0000 7f9fbe189640 1 -- 172.21.15.76:0/2454763768 wait complete. 2024-03-18T14:50:47.398 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:47 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:47.399 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:47 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:47.399 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:47 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:47.399 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:47 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:47.399 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:47 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:47.399 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:47 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:47.399 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:47 smithi077 ceph-mon[35865]: Deploying daemon mds.i on smithi174 2024-03-18T14:50:47.399 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:47 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.77:6836/2568866075,v1:172.21.15.77:6837/2568866075] up:boot 2024-03-18T14:50:47.399 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:47 smithi077 ceph-mon[35865]: fsmap 8 up:standby 2024-03-18T14:50:47.399 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:47 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T14:50:47.399 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:47 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2454763768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:47.399 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:47 smithi077 ceph-mon[35865]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:47.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:47 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:47.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:47 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:47.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:47 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:47.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:47 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:47.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:47 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:47.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:47 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:47.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:47 smithi076 ceph-mon[32205]: Deploying daemon mds.i on smithi174 2024-03-18T14:50:47.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:47 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.77:6836/2568866075,v1:172.21.15.77:6837/2568866075] up:boot 2024-03-18T14:50:47.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:47 smithi076 ceph-mon[32205]: fsmap 8 up:standby 2024-03-18T14:50:47.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:47 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T14:50:47.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:47 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2454763768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:47.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:47 smithi076 ceph-mon[32205]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:47.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:47 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:47.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:47 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:47.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:47 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:47.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:47 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:47.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:47 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:47.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:47 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:47.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:47 smithi174 ceph-mon[36787]: Deploying daemon mds.i on smithi174 2024-03-18T14:50:47.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:47 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.77:6836/2568866075,v1:172.21.15.77:6837/2568866075] up:boot 2024-03-18T14:50:47.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:47 smithi174 ceph-mon[36787]: fsmap 8 up:standby 2024-03-18T14:50:47.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:47 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T14:50:47.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:47 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2454763768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:47.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:47 smithi174 ceph-mon[36787]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:48.327 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:50:48.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.444+0000 7fb5d2d77640 1 -- 172.21.15.76:0/3924301667 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb5cc1022b0 msgr2=0x7fb5cc108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:48.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.444+0000 7fb5d2d77640 1 --2- 172.21.15.76:0/3924301667 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb5cc1022b0 0x7fb5cc108b40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4009ca0 tx=0x7fb5c402c0f0 comp rx=0 tx=0).stop 2024-03-18T14:50:48.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.444+0000 7fb5d2d77640 1 -- 172.21.15.76:0/3924301667 shutdown_connections 2024-03-18T14:50:48.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.444+0000 7fb5d2d77640 1 --2- 172.21.15.76:0/3924301667 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb5cc1022b0 0x7fb5cc108b40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:48.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.444+0000 7fb5d2d77640 1 --2- 172.21.15.76:0/3924301667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5cc1018f0 0x7fb5cc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:48.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.444+0000 7fb5d2d77640 1 --2- 172.21.15.76:0/3924301667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5cc100700 0x7fb5cc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:48.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.444+0000 7fb5d2d77640 1 -- 172.21.15.76:0/3924301667 >> 172.21.15.76:0/3924301667 conn(0x7fb5cc0fbc70 msgr2=0x7fb5cc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:48.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.444+0000 7fb5d2d77640 1 -- 172.21.15.76:0/3924301667 shutdown_connections 2024-03-18T14:50:48.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.444+0000 7fb5d2d77640 1 -- 172.21.15.76:0/3924301667 wait complete. 2024-03-18T14:50:48.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.444+0000 7fb5d2d77640 1 Processor -- start 2024-03-18T14:50:48.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.444+0000 7fb5d2d77640 1 -- start start 2024-03-18T14:50:48.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5d2d77640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb5cc100700 0x7fb5cc197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:48.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5d2d77640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5cc1018f0 0x7fb5cc198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:48.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5d2d77640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5cc1022b0 0x7fb5cc19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:48.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5d2d77640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc10c380 con 0x7fb5cc100700 2024-03-18T14:50:48.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5d2d77640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc10c200 con 0x7fb5cc1018f0 2024-03-18T14:50:48.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5d2d77640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc10c500 con 0x7fb5cc1022b0 2024-03-18T14:50:48.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5d0b13640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb5cc100700 0x7fb5cc197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:48.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5d0b13640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb5cc100700 0x7fb5cc197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:42556/0 (socket says 172.21.15.76:42556) 2024-03-18T14:50:48.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5d0b13640 1 -- 172.21.15.76:0/370188502 learned_addr learned my addr 172.21.15.76:0/370188502 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:48.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5cbfff640 1 --2- 172.21.15.76:0/370188502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5cc1018f0 0x7fb5cc198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:48.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5d1314640 1 --2- 172.21.15.76:0/370188502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5cc1022b0 0x7fb5cc19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:48.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5d0b13640 1 -- 172.21.15.76:0/370188502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5cc1022b0 msgr2=0x7fb5cc19f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:48.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5d0b13640 1 --2- 172.21.15.76:0/370188502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5cc1022b0 0x7fb5cc19f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:48.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5d0b13640 1 -- 172.21.15.76:0/370188502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5cc1018f0 msgr2=0x7fb5cc198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:48.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5d0b13640 1 --2- 172.21.15.76:0/370188502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5cc1018f0 0x7fb5cc198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:48.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5d0b13640 1 -- 172.21.15.76:0/370188502 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5cc19f700 con 0x7fb5cc100700 2024-03-18T14:50:48.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5cbfff640 1 --2- 172.21.15.76:0/370188502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5cc1018f0 0x7fb5cc198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:50:48.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5d0b13640 1 --2- 172.21.15.76:0/370188502 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb5cc100700 0x7fb5cc197ae0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00e9c0 tx=0x7fb5bc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:48.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5c9ffb640 1 -- 172.21.15.76:0/370188502 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc00cd80 con 0x7fb5cc100700 2024-03-18T14:50:48.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5c9ffb640 1 -- 172.21.15.76:0/370188502 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5bc0035f0 con 0x7fb5cc100700 2024-03-18T14:50:48.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5d2d77640 1 -- 172.21.15.76:0/370188502 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5cc19f9f0 con 0x7fb5cc100700 2024-03-18T14:50:48.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.445+0000 7fb5c9ffb640 1 -- 172.21.15.76:0/370188502 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc010690 con 0x7fb5cc100700 2024-03-18T14:50:48.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.446+0000 7fb5d2d77640 1 -- 172.21.15.76:0/370188502 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5cc19ff40 con 0x7fb5cc100700 2024-03-18T14:50:48.450 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:50:48 smithi174 systemd[1]: Starting Ceph mds.i for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T14:50:48.450 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:50:48 smithi174 podman[67602]: 2024-03-18 14:50:48.348303528 +0000 UTC m=+0.017668860 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T14:50:48.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.447+0000 7fb5d2d77640 1 -- 172.21.15.76:0/370188502 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5cc1a01d0 con 0x7fb5cc100700 2024-03-18T14:50:48.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.448+0000 7fb5c9ffb640 1 -- 172.21.15.76:0/370188502 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb5bc0037a0 con 0x7fb5cc100700 2024-03-18T14:50:48.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.448+0000 7fb5c9ffb640 1 --2- 172.21.15.76:0/370188502 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb5b40a2880 0x7fb5b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:48.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.448+0000 7fb5cbfff640 1 --2- 172.21.15.76:0/370188502 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb5b40a2880 0x7fb5b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:48.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.448+0000 7fb5c9ffb640 1 -- 172.21.15.76:0/370188502 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fb5bc0ce9d0 con 0x7fb5cc100700 2024-03-18T14:50:48.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.449+0000 7fb5cbfff640 1 --2- 172.21.15.76:0/370188502 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb5b40a2880 0x7fb5b40a4d40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc198ff0 tx=0x7fb5c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:48.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.453+0000 7fb5c9ffb640 1 -- 172.21.15.76:0/370188502 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb5bc09bf10 con 0x7fb5cc100700 2024-03-18T14:50:48.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.666+0000 7fb5d2d77640 1 -- 172.21.15.76:0/370188502 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5cc1a03a0 con 0x7fb5cc100700 2024-03-18T14:50:48.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.666+0000 7fb5c9ffb640 1 -- 172.21.15.76:0/370188502 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7171 (secure 0 0 0) 0x7fb5bc09b810 con 0x7fb5cc100700 2024-03-18T14:50:48.667 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:48.667 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-18T14:50:48.667 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 9 2024-03-18T14:50:48.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.668+0000 7fb5d2d77640 1 -- 172.21.15.76:0/370188502 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb5b40a2880 msgr2=0x7fb5b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:48.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.668+0000 7fb5d2d77640 1 --2- 172.21.15.76:0/370188502 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb5b40a2880 0x7fb5b40a4d40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc198ff0 tx=0x7fb5c0005f90 comp rx=0 tx=0).stop 2024-03-18T14:50:48.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.668+0000 7fb5d2d77640 1 -- 172.21.15.76:0/370188502 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb5cc100700 msgr2=0x7fb5cc197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:48.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.668+0000 7fb5d2d77640 1 --2- 172.21.15.76:0/370188502 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb5cc100700 0x7fb5cc197ae0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00e9c0 tx=0x7fb5bc00ee80 comp rx=0 tx=0).stop 2024-03-18T14:50:48.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.668+0000 7fb5d2d77640 1 -- 172.21.15.76:0/370188502 shutdown_connections 2024-03-18T14:50:48.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.668+0000 7fb5d2d77640 1 --2- 172.21.15.76:0/370188502 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb5b40a2880 0x7fb5b40a4d40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:48.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.668+0000 7fb5d2d77640 1 --2- 172.21.15.76:0/370188502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5cc1022b0 0x7fb5cc19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:48.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.668+0000 7fb5d2d77640 1 --2- 172.21.15.76:0/370188502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5cc1018f0 0x7fb5cc198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:48.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.668+0000 7fb5d2d77640 1 --2- 172.21.15.76:0/370188502 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb5cc100700 0x7fb5cc197ae0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:48.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.668+0000 7fb5d2d77640 1 -- 172.21.15.76:0/370188502 >> 172.21.15.76:0/370188502 conn(0x7fb5cc0fbc70 msgr2=0x7fb5cc0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:48.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.668+0000 7fb5d2d77640 1 -- 172.21.15.76:0/370188502 shutdown_connections 2024-03-18T14:50:48.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:48.668+0000 7fb5d2d77640 1 -- 172.21.15.76:0/370188502 wait complete. 2024-03-18T14:50:48.698 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:50:48 smithi174 podman[67602]: 2024-03-18 14:50:48.475450515 +0000 UTC m=+0.144815828 container create 6ed05e4295d7c9d28a4b0678f68ed915ba32baff30815650d5ae07b43c12854c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-18T14:50:48.967 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:50:48 smithi174 podman[67602]: 2024-03-18 14:50:48.698215018 +0000 UTC m=+0.367580350 container init 6ed05e4295d7c9d28a4b0678f68ed915ba32baff30815650d5ae07b43c12854c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-18T14:50:48.967 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:50:48 smithi174 podman[67602]: 2024-03-18 14:50:48.701045365 +0000 UTC m=+0.370410677 container start 6ed05e4295d7c9d28a4b0678f68ed915ba32baff30815650d5ae07b43c12854c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-03-18T14:50:48.967 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:50:48 smithi174 ceph-mds[67622]: -- 172.21.15.174:0/1382835048 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562d028ada40 con 0x562d02881000 2024-03-18T14:50:48.967 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:50:48 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: starting mds.i at 2024-03-18T14:50:48.967 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:50:48 smithi174 bash[67602]: 6ed05e4295d7c9d28a4b0678f68ed915ba32baff30815650d5ae07b43c12854c 2024-03-18T14:50:48.967 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:50:48 smithi174 systemd[1]: Started Ceph mds.i for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:50:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:49 smithi076 ceph-mon[32205]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:49 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/370188502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:49.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:49.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:49.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:49.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:49 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:49 smithi174 ceph-mon[36787]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:49 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/370188502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:49 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:49.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:49 smithi077 ceph-mon[35865]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:49.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:49 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/370188502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:49 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:50.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:50 smithi076 ceph-mon[32205]: Deploying daemon mds.g on smithi076 2024-03-18T14:50:50.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:50 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] up:boot 2024-03-18T14:50:50.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:50 smithi076 ceph-mon[32205]: fsmap 9 up:standby 2024-03-18T14:50:50.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:50 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T14:50:50.681 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:50:50.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:50 smithi174 ceph-mon[36787]: Deploying daemon mds.g on smithi076 2024-03-18T14:50:50.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:50 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] up:boot 2024-03-18T14:50:50.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:50 smithi174 ceph-mon[36787]: fsmap 9 up:standby 2024-03-18T14:50:50.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:50 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T14:50:50.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:50 smithi077 ceph-mon[35865]: Deploying daemon mds.g on smithi076 2024-03-18T14:50:50.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:50 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] up:boot 2024-03-18T14:50:50.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:50 smithi077 ceph-mon[35865]: fsmap 9 up:standby 2024-03-18T14:50:50.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:50 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T14:50:50.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.793+0000 7f80ee2e8640 1 -- 172.21.15.76:0/2947187846 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80e81018f0 msgr2=0x7f80e8101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:50.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.793+0000 7f80ee2e8640 1 --2- 172.21.15.76:0/2947187846 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80e81018f0 0x7f80e8101d70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f80dc009900 tx=0x7f80dc02bc30 comp rx=0 tx=0).stop 2024-03-18T14:50:50.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.793+0000 7f80ee2e8640 1 -- 172.21.15.76:0/2947187846 shutdown_connections 2024-03-18T14:50:50.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.793+0000 7f80ee2e8640 1 --2- 172.21.15.76:0/2947187846 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80e81022b0 0x7f80e8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:50.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.793+0000 7f80ee2e8640 1 --2- 172.21.15.76:0/2947187846 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80e81018f0 0x7f80e8101d70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.793+0000 7f80ee2e8640 1 --2- 172.21.15.76:0/2947187846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80e8100700 0x7f80e8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.793+0000 7f80ee2e8640 1 -- 172.21.15.76:0/2947187846 >> 172.21.15.76:0/2947187846 conn(0x7f80e80fbc70 msgr2=0x7f80e80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.793+0000 7f80ee2e8640 1 -- 172.21.15.76:0/2947187846 shutdown_connections 2024-03-18T14:50:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.793+0000 7f80ee2e8640 1 -- 172.21.15.76:0/2947187846 wait complete. 2024-03-18T14:50:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.794+0000 7f80ee2e8640 1 Processor -- start 2024-03-18T14:50:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.794+0000 7f80ee2e8640 1 -- start start 2024-03-18T14:50:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.794+0000 7f80ee2e8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80e8100700 0x7f80e8197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.794+0000 7f80ee2e8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80e81018f0 0x7f80e81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.794+0000 7f80ee2e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80e81022b0 0x7f80e819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.794+0000 7f80ee2e8640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f80e810c360 con 0x7f80e81018f0 2024-03-18T14:50:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.794+0000 7f80ee2e8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f80e810c1e0 con 0x7f80e81022b0 2024-03-18T14:50:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.794+0000 7f80ee2e8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f80e810c4e0 con 0x7f80e8100700 2024-03-18T14:50:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.794+0000 7f80e77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80e81018f0 0x7f80e81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.794+0000 7f80e7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80e8100700 0x7f80e8197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.794+0000 7f80e77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80e81018f0 0x7f80e81980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:42574/0 (socket says 172.21.15.76:42574) 2024-03-18T14:50:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.794+0000 7f80e77fe640 1 -- 172.21.15.76:0/3405917064 learned_addr learned my addr 172.21.15.76:0/3405917064 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.794+0000 7f80ec885640 1 --2- 172.21.15.76:0/3405917064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80e81022b0 0x7f80e819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.794+0000 7f80e77fe640 1 -- 172.21.15.76:0/3405917064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80e8100700 msgr2=0x7f80e8197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.794+0000 7f80e77fe640 1 --2- 172.21.15.76:0/3405917064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80e8100700 0x7f80e8197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.794+0000 7f80e77fe640 1 -- 172.21.15.76:0/3405917064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80e81022b0 msgr2=0x7f80e819f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.794+0000 7f80e77fe640 1 --2- 172.21.15.76:0/3405917064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80e81022b0 0x7f80e819f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.795+0000 7f80e77fe640 1 -- 172.21.15.76:0/3405917064 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e819f730 con 0x7f80e81018f0 2024-03-18T14:50:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.795+0000 7f80ec885640 1 --2- 172.21.15.76:0/3405917064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80e81022b0 0x7f80e819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:50:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.795+0000 7f80e7fff640 1 --2- 172.21.15.76:0/3405917064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80e8100700 0x7f80e8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:50:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.795+0000 7f80e77fe640 1 --2- 172.21.15.76:0/3405917064 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80e81018f0 0x7f80e81980a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f80dc009230 tx=0x7f80dc005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.795+0000 7f80e57fa640 1 -- 172.21.15.76:0/3405917064 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc041070 con 0x7f80e81018f0 2024-03-18T14:50:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.795+0000 7f80ee2e8640 1 -- 172.21.15.76:0/3405917064 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e819f9c0 con 0x7f80e81018f0 2024-03-18T14:50:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.795+0000 7f80e57fa640 1 -- 172.21.15.76:0/3405917064 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80dc0044d0 con 0x7f80e81018f0 2024-03-18T14:50:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.795+0000 7f80e57fa640 1 -- 172.21.15.76:0/3405917064 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc034440 con 0x7f80e81018f0 2024-03-18T14:50:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.795+0000 7f80ee2e8640 1 -- 172.21.15.76:0/3405917064 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e819feb0 con 0x7f80e81018f0 2024-03-18T14:50:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.796+0000 7f80ee2e8640 1 -- 172.21.15.76:0/3405917064 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80e81a00e0 con 0x7f80e81018f0 2024-03-18T14:50:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.797+0000 7f80e57fa640 1 -- 172.21.15.76:0/3405917064 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f80dc044070 con 0x7f80e81018f0 2024-03-18T14:50:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.798+0000 7f80e57fa640 1 --2- 172.21.15.76:0/3405917064 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f80d00a2800 0x7f80d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.798+0000 7f80e7fff640 1 --2- 172.21.15.76:0/3405917064 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f80d00a2800 0x7f80d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.798+0000 7f80e57fa640 1 -- 172.21.15.76:0/3405917064 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f80dc0f0540 con 0x7f80e81018f0 2024-03-18T14:50:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.798+0000 7f80e7fff640 1 --2- 172.21.15.76:0/3405917064 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f80d00a2800 0x7f80d00a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f80d8005fd0 tx=0x7f80d800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:50.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:50.802+0000 7f80e57fa640 1 -- 172.21.15.76:0/3405917064 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f80dc0bda80 con 0x7f80e81018f0 2024-03-18T14:50:51.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:51.036+0000 7f80ee2e8640 1 -- 172.21.15.76:0/3405917064 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80e81a02b0 con 0x7f80e81018f0 2024-03-18T14:50:51.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:51.037+0000 7f80e57fa640 1 -- 172.21.15.76:0/3405917064 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7998 (secure 0 0 0) 0x7f80dc0bd380 con 0x7f80e81018f0 2024-03-18T14:50:51.038 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:51.038 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24490,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/399810652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":399810652},{"type":"v1","addr":"172.21.15.174:6839","nonce":399810652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T14:50:51.038 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 10 2024-03-18T14:50:51.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:51.040+0000 7f80ee2e8640 1 -- 172.21.15.76:0/3405917064 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f80d00a2800 msgr2=0x7f80d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:51.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:51.040+0000 7f80ee2e8640 1 --2- 172.21.15.76:0/3405917064 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f80d00a2800 0x7f80d00a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f80d8005fd0 tx=0x7f80d800b410 comp rx=0 tx=0).stop 2024-03-18T14:50:51.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:51.040+0000 7f80ee2e8640 1 -- 172.21.15.76:0/3405917064 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80e81018f0 msgr2=0x7f80e81980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:51.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:51.040+0000 7f80ee2e8640 1 --2- 172.21.15.76:0/3405917064 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80e81018f0 0x7f80e81980a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f80dc009230 tx=0x7f80dc005d50 comp rx=0 tx=0).stop 2024-03-18T14:50:51.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:51.040+0000 7f80ee2e8640 1 -- 172.21.15.76:0/3405917064 shutdown_connections 2024-03-18T14:50:51.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:51.040+0000 7f80ee2e8640 1 --2- 172.21.15.76:0/3405917064 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f80d00a2800 0x7f80d00a4cc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:51.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:51.040+0000 7f80ee2e8640 1 --2- 172.21.15.76:0/3405917064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80e81022b0 0x7f80e819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:51.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:51.040+0000 7f80ee2e8640 1 --2- 172.21.15.76:0/3405917064 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80e81018f0 0x7f80e81980a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:51.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:51.040+0000 7f80ee2e8640 1 --2- 172.21.15.76:0/3405917064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80e8100700 0x7f80e8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:51.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:51.040+0000 7f80ee2e8640 1 -- 172.21.15.76:0/3405917064 >> 172.21.15.76:0/3405917064 conn(0x7f80e80fbc70 msgr2=0x7f80e80fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:51.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:51.040+0000 7f80ee2e8640 1 -- 172.21.15.76:0/3405917064 shutdown_connections 2024-03-18T14:50:51.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:51.040+0000 7f80ee2e8640 1 -- 172.21.15.76:0/3405917064 wait complete. 2024-03-18T14:50:51.385 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 18 14:50:51 smithi076 systemd[1]: Starting Ceph mds.g for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T14:50:51.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:51 smithi076 ceph-mon[32205]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:51.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:51 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3405917064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:51.661 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 18 14:50:51 smithi076 podman[73012]: 2024-03-18 14:50:51.287450656 +0000 UTC m=+0.017516668 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T14:50:51.661 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 18 14:50:51 smithi076 podman[73012]: 2024-03-18 14:50:51.389087542 +0000 UTC m=+0.119153555 container create d5e4e51ec334da5a584b8d05e551820678c799202a334b66b039fc76f89572bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-g, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T14:50:51.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:51 smithi174 ceph-mon[36787]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:51.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:51 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3405917064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:51.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:51 smithi077 ceph-mon[35865]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:51.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:51 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3405917064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:52.161 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 18 14:50:51 smithi076 podman[73012]: 2024-03-18 14:50:51.686463075 +0000 UTC m=+0.416529101 container init d5e4e51ec334da5a584b8d05e551820678c799202a334b66b039fc76f89572bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-g, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T14:50:52.161 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 18 14:50:51 smithi076 podman[73012]: 2024-03-18 14:50:51.689154424 +0000 UTC m=+0.419220440 container start d5e4e51ec334da5a584b8d05e551820678c799202a334b66b039fc76f89572bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-g, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-18T14:50:52.161 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 18 14:50:51 smithi076 ceph-mds[73031]: -- 172.21.15.76:0/345966119 <== mon.0 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56380c969a40 con 0x56380c93cc00 2024-03-18T14:50:52.161 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 18 14:50:51 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-g[73027]: starting mds.g at 2024-03-18T14:50:52.161 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 18 14:50:51 smithi076 bash[73012]: d5e4e51ec334da5a584b8d05e551820678c799202a334b66b039fc76f89572bb 2024-03-18T14:50:52.161 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 18 14:50:51 smithi076 systemd[1]: Started Ceph mds.g for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:50:53.056 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:50:53.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:52 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:53.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:52 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:53.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:52 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:53.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:52 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:53.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:52 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:53.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:52 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:53.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:52 smithi076 ceph-mon[32205]: Deploying daemon mds.a on smithi076 2024-03-18T14:50:53.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:52 smithi076 ceph-mon[32205]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:53.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:52 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.76:6838/2801069369,v1:172.21.15.76:6839/2801069369] up:boot 2024-03-18T14:50:53.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:52 smithi076 ceph-mon[32205]: fsmap 10 up:standby 2024-03-18T14:50:53.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:52 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T14:50:53.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.164+0000 7f101a9c4640 1 -- 172.21.15.76:0/3037770690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10141029f0 msgr2=0x7f1014102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:53.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.164+0000 7f101a9c4640 1 --2- 172.21.15.76:0/3037770690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10141029f0 0x7f1014102e70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1008009900 tx=0x7f100802bc30 comp rx=0 tx=0).stop 2024-03-18T14:50:53.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.164+0000 7f101a9c4640 1 -- 172.21.15.76:0/3037770690 shutdown_connections 2024-03-18T14:50:53.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.164+0000 7f101a9c4640 1 --2- 172.21.15.76:0/3037770690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10141033b0 0x7f1014109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:53.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.164+0000 7f101a9c4640 1 --2- 172.21.15.76:0/3037770690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10141029f0 0x7f1014102e70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:53.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.164+0000 7f101a9c4640 1 --2- 172.21.15.76:0/3037770690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1014101800 0x7f1014101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:53.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.164+0000 7f101a9c4640 1 -- 172.21.15.76:0/3037770690 >> 172.21.15.76:0/3037770690 conn(0x7f10140fcd90 msgr2=0x7f10140ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:53.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.164+0000 7f101a9c4640 1 -- 172.21.15.76:0/3037770690 shutdown_connections 2024-03-18T14:50:53.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.164+0000 7f101a9c4640 1 -- 172.21.15.76:0/3037770690 wait complete. 2024-03-18T14:50:53.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f101a9c4640 1 Processor -- start 2024-03-18T14:50:53.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f101a9c4640 1 -- start start 2024-03-18T14:50:53.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f101a9c4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1014101800 0x7f1014197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:53.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f101a9c4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10141029f0 0x7f1014198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:53.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f101a9c4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10141033b0 0x7f101419f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:53.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f101a9c4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f101410d4f0 con 0x7f10141029f0 2024-03-18T14:50:53.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f101a9c4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f101410d370 con 0x7f10141033b0 2024-03-18T14:50:53.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f101a9c4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f101410d670 con 0x7f1014101800 2024-03-18T14:50:53.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f10137fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10141029f0 0x7f1014198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:53.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f10137fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10141029f0 0x7f1014198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:42594/0 (socket says 172.21.15.76:42594) 2024-03-18T14:50:53.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f10137fe640 1 -- 172.21.15.76:0/4090558076 learned_addr learned my addr 172.21.15.76:0/4090558076 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:53.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f1018f61640 1 --2- 172.21.15.76:0/4090558076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10141033b0 0x7f101419f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:53.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f1013fff640 1 --2- 172.21.15.76:0/4090558076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1014101800 0x7f1014197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:53.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f10137fe640 1 -- 172.21.15.76:0/4090558076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1014101800 msgr2=0x7f1014197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:53.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f10137fe640 1 --2- 172.21.15.76:0/4090558076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1014101800 0x7f1014197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:53.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f10137fe640 1 -- 172.21.15.76:0/4090558076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10141033b0 msgr2=0x7f101419f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:53.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f10137fe640 1 --2- 172.21.15.76:0/4090558076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10141033b0 0x7f101419f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:53.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f10137fe640 1 -- 172.21.15.76:0/4090558076 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f101419f7e0 con 0x7f10141029f0 2024-03-18T14:50:53.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f1018f61640 1 --2- 172.21.15.76:0/4090558076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10141033b0 0x7f101419f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:50:53.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.165+0000 7f1013fff640 1 --2- 172.21.15.76:0/4090558076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1014101800 0x7f1014197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:50:53.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.166+0000 7f10137fe640 1 --2- 172.21.15.76:0/4090558076 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10141029f0 0x7f1014198100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1008009230 tx=0x7f1008004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:53.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.166+0000 7f10117fa640 1 -- 172.21.15.76:0/4090558076 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1008041070 con 0x7f10141029f0 2024-03-18T14:50:53.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.166+0000 7f10117fa640 1 -- 172.21.15.76:0/4090558076 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10080054e0 con 0x7f10141029f0 2024-03-18T14:50:53.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.166+0000 7f101a9c4640 1 -- 172.21.15.76:0/4090558076 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f101419fa70 con 0x7f10141029f0 2024-03-18T14:50:53.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.166+0000 7f10117fa640 1 -- 172.21.15.76:0/4090558076 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10080346a0 con 0x7f10141029f0 2024-03-18T14:50:53.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.166+0000 7f101a9c4640 1 -- 172.21.15.76:0/4090558076 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f101419ff60 con 0x7f10141029f0 2024-03-18T14:50:53.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.167+0000 7f101a9c4640 1 -- 172.21.15.76:0/4090558076 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1014108b70 con 0x7f10141029f0 2024-03-18T14:50:53.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.168+0000 7f10117fa640 1 -- 172.21.15.76:0/4090558076 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1008044070 con 0x7f10141029f0 2024-03-18T14:50:53.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.168+0000 7f10117fa640 1 --2- 172.21.15.76:0/4090558076 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0ffc0a2800 0x7f0ffc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:53.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.169+0000 7f1013fff640 1 --2- 172.21.15.76:0/4090558076 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0ffc0a2800 0x7f0ffc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:53.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.169+0000 7f10117fa640 1 -- 172.21.15.76:0/4090558076 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f10080f0410 con 0x7f10141029f0 2024-03-18T14:50:53.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.169+0000 7f1013fff640 1 --2- 172.21.15.76:0/4090558076 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0ffc0a2800 0x7f0ffc0a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1004006fd0 tx=0x7f1004008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:53.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.173+0000 7f10117fa640 1 -- 172.21.15.76:0/4090558076 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f10080bd950 con 0x7f10141029f0 2024-03-18T14:50:53.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:52 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:53.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:52 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:53.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:52 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:53.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:52 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:53.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:52 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:53.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:52 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:53.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:52 smithi174 ceph-mon[36787]: Deploying daemon mds.a on smithi076 2024-03-18T14:50:53.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:52 smithi174 ceph-mon[36787]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:53.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:52 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.76:6838/2801069369,v1:172.21.15.76:6839/2801069369] up:boot 2024-03-18T14:50:53.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:52 smithi174 ceph-mon[36787]: fsmap 10 up:standby 2024-03-18T14:50:53.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:52 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T14:50:53.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:52 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:53.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:52 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:53.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:52 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:53.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:52 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:53.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:52 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:53.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:52 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:53.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:52 smithi077 ceph-mon[35865]: Deploying daemon mds.a on smithi076 2024-03-18T14:50:53.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:52 smithi077 ceph-mon[35865]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:53.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:52 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.76:6838/2801069369,v1:172.21.15.76:6839/2801069369] up:boot 2024-03-18T14:50:53.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:52 smithi077 ceph-mon[35865]: fsmap 10 up:standby 2024-03-18T14:50:53.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:52 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T14:50:53.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.387+0000 7f101a9c4640 1 -- 172.21.15.76:0/4090558076 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f101402d380 con 0x7f10141029f0 2024-03-18T14:50:53.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.388+0000 7f10117fa640 1 -- 172.21.15.76:0/4090558076 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8824 (secure 0 0 0) 0x7f10080bd250 con 0x7f10141029f0 2024-03-18T14:50:53.389 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:53.389 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/2801069369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":2801069369},{"type":"v1","addr":"172.21.15.76:6839","nonce":2801069369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24490,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/399810652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":399810652},{"type":"v1","addr":"172.21.15.174:6839","nonce":399810652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T14:50:53.389 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 11 2024-03-18T14:50:53.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.390+0000 7f101a9c4640 1 -- 172.21.15.76:0/4090558076 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0ffc0a2800 msgr2=0x7f0ffc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:53.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.390+0000 7f101a9c4640 1 --2- 172.21.15.76:0/4090558076 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0ffc0a2800 0x7f0ffc0a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1004006fd0 tx=0x7f1004008040 comp rx=0 tx=0).stop 2024-03-18T14:50:53.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.390+0000 7f101a9c4640 1 -- 172.21.15.76:0/4090558076 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10141029f0 msgr2=0x7f1014198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:53.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.390+0000 7f101a9c4640 1 --2- 172.21.15.76:0/4090558076 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10141029f0 0x7f1014198100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1008009230 tx=0x7f1008004040 comp rx=0 tx=0).stop 2024-03-18T14:50:53.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.390+0000 7f101a9c4640 1 -- 172.21.15.76:0/4090558076 shutdown_connections 2024-03-18T14:50:53.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.390+0000 7f101a9c4640 1 --2- 172.21.15.76:0/4090558076 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0ffc0a2800 0x7f0ffc0a4cc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:53.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.390+0000 7f101a9c4640 1 --2- 172.21.15.76:0/4090558076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10141033b0 0x7f101419f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:53.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.390+0000 7f101a9c4640 1 --2- 172.21.15.76:0/4090558076 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10141029f0 0x7f1014198100 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:53.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.390+0000 7f101a9c4640 1 --2- 172.21.15.76:0/4090558076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1014101800 0x7f1014197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:53.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.390+0000 7f101a9c4640 1 -- 172.21.15.76:0/4090558076 >> 172.21.15.76:0/4090558076 conn(0x7f10140fcd90 msgr2=0x7f1014105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:53.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.390+0000 7f101a9c4640 1 -- 172.21.15.76:0/4090558076 shutdown_connections 2024-03-18T14:50:53.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:53.390+0000 7f101a9c4640 1 -- 172.21.15.76:0/4090558076 wait complete. 2024-03-18T14:50:54.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:53 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/4090558076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:54.161 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 18 14:50:54 smithi076 systemd[1]: Starting Ceph mds.a for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T14:50:54.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:53 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/4090558076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:54.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:53 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/4090558076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:54.649 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 18 14:50:54 smithi076 podman[73359]: 2024-03-18 14:50:54.126338838 +0000 UTC m=+0.017297745 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T14:50:54.649 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 18 14:50:54 smithi076 podman[73359]: 2024-03-18 14:50:54.246169395 +0000 UTC m=+0.137128343 container create bf67b06ccf43e9b6831399025156f8545bac483c3828056b92fc48c48f1d4a26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-18T14:50:54.649 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 18 14:50:54 smithi076 podman[73359]: 2024-03-18 14:50:54.649237336 +0000 UTC m=+0.540196251 container init bf67b06ccf43e9b6831399025156f8545bac483c3828056b92fc48c48f1d4a26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T14:50:54.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:54 smithi076 ceph-mon[32205]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:54.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:54 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:54.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:54 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:54.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:54 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:54.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:54 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:54.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:54 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:54.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:54 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:54.912 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 18 14:50:54 smithi076 podman[73359]: 2024-03-18 14:50:54.651916715 +0000 UTC m=+0.542875621 container start bf67b06ccf43e9b6831399025156f8545bac483c3828056b92fc48c48f1d4a26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-18T14:50:54.912 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 18 14:50:54 smithi076 ceph-mds[73378]: -- 172.21.15.76:0/1071503358 <== mon.0 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5598b681fa40 con 0x5598b67f2c00 2024-03-18T14:50:54.912 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 18 14:50:54 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-a[73374]: starting mds.a at 2024-03-18T14:50:54.912 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 18 14:50:54 smithi076 bash[73359]: bf67b06ccf43e9b6831399025156f8545bac483c3828056b92fc48c48f1d4a26 2024-03-18T14:50:54.912 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 18 14:50:54 smithi076 systemd[1]: Started Ceph mds.a for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:50:55.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:54 smithi174 ceph-mon[36787]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:55.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:54 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:55.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:54 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:55.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:54 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:55.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:54 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:55.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:54 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:55.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:54 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:55.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:54 smithi077 ceph-mon[35865]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:55.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:54 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:55.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:54 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:55.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:54 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:55.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:54 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T14:50:55.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:54 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T14:50:55.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:54 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:50:55.403 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:50:55.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.509+0000 7fa019137640 1 -- 172.21.15.76:0/2170135279 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa014102300 msgr2=0x7fa0141046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:55.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.509+0000 7fa019137640 1 --2- 172.21.15.76:0/2170135279 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa014102300 0x7fa0141046f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa004009ca0 tx=0x7fa00402c0f0 comp rx=0 tx=0).stop 2024-03-18T14:50:55.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.509+0000 7fa019137640 1 -- 172.21.15.76:0/2170135279 shutdown_connections 2024-03-18T14:50:55.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.509+0000 7fa019137640 1 --2- 172.21.15.76:0/2170135279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa014107560 0x7fa0141099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:55.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.509+0000 7fa019137640 1 --2- 172.21.15.76:0/2170135279 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa014104c30 0x7fa014107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:55.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.509+0000 7fa019137640 1 --2- 172.21.15.76:0/2170135279 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa014102300 0x7fa0141046f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:55.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.509+0000 7fa019137640 1 -- 172.21.15.76:0/2170135279 >> 172.21.15.76:0/2170135279 conn(0x7fa0140fbd10 msgr2=0x7fa0140fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:55.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.509+0000 7fa019137640 1 -- 172.21.15.76:0/2170135279 shutdown_connections 2024-03-18T14:50:55.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.509+0000 7fa019137640 1 -- 172.21.15.76:0/2170135279 wait complete. 2024-03-18T14:50:55.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.510+0000 7fa019137640 1 Processor -- start 2024-03-18T14:50:55.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.510+0000 7fa019137640 1 -- start start 2024-03-18T14:50:55.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.510+0000 7fa019137640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa014102300 0x7fa014101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:55.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.510+0000 7fa019137640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa014104c30 0x7fa0141002b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:55.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.510+0000 7fa019137640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa014107560 0x7fa0141007f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:55.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.510+0000 7fa019137640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa01410d4b0 con 0x7fa014102300 2024-03-18T14:50:55.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.510+0000 7fa019137640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa01410d330 con 0x7fa014107560 2024-03-18T14:50:55.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.510+0000 7fa019137640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa01410d630 con 0x7fa014104c30 2024-03-18T14:50:55.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.511+0000 7fa012d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa014102300 0x7fa014101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:55.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.511+0000 7fa012d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa014102300 0x7fa014101c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:42638/0 (socket says 172.21.15.76:42638) 2024-03-18T14:50:55.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.511+0000 7fa012d9d640 1 -- 172.21.15.76:0/3313733653 learned_addr learned my addr 172.21.15.76:0/3313733653 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:55.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.511+0000 7fa01359e640 1 --2- 172.21.15.76:0/3313733653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa014107560 0x7fa0141007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:55.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.511+0000 7fa01259c640 1 --2- 172.21.15.76:0/3313733653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa014104c30 0x7fa0141002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:55.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.511+0000 7fa012d9d640 1 -- 172.21.15.76:0/3313733653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa014104c30 msgr2=0x7fa0141002b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:55.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.511+0000 7fa012d9d640 1 --2- 172.21.15.76:0/3313733653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa014104c30 0x7fa0141002b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:55.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.511+0000 7fa012d9d640 1 -- 172.21.15.76:0/3313733653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa014107560 msgr2=0x7fa0141007f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:55.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.511+0000 7fa012d9d640 1 --2- 172.21.15.76:0/3313733653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa014107560 0x7fa0141007f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:55.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.511+0000 7fa012d9d640 1 -- 172.21.15.76:0/3313733653 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa014101080 con 0x7fa014102300 2024-03-18T14:50:55.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.511+0000 7fa012d9d640 1 --2- 172.21.15.76:0/3313733653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa014102300 0x7fa014101c70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa0040325f0 tx=0x7fa004032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:55.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.511+0000 7f9ffbfff640 1 -- 172.21.15.76:0/3313733653 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa004041020 con 0x7fa014102300 2024-03-18T14:50:55.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.511+0000 7f9ffbfff640 1 -- 172.21.15.76:0/3313733653 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa004034cd0 con 0x7fa014102300 2024-03-18T14:50:55.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.511+0000 7f9ffbfff640 1 -- 172.21.15.76:0/3313733653 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa004039040 con 0x7fa014102300 2024-03-18T14:50:55.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.511+0000 7fa019137640 1 -- 172.21.15.76:0/3313733653 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0141a1c20 con 0x7fa014102300 2024-03-18T14:50:55.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.511+0000 7fa019137640 1 -- 172.21.15.76:0/3313733653 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0141a2190 con 0x7fa014102300 2024-03-18T14:50:55.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.512+0000 7fa019137640 1 -- 172.21.15.76:0/3313733653 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0140fd800 con 0x7fa014102300 2024-03-18T14:50:55.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.513+0000 7f9ffbfff640 1 -- 172.21.15.76:0/3313733653 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa004044020 con 0x7fa014102300 2024-03-18T14:50:55.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.514+0000 7f9ffbfff640 1 --2- 172.21.15.76:0/3313733653 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9ffc0a2880 0x7f9ffc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:55.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.514+0000 7fa01259c640 1 --2- 172.21.15.76:0/3313733653 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9ffc0a2880 0x7f9ffc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:55.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.514+0000 7f9ffbfff640 1 -- 172.21.15.76:0/3313733653 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa0040f0480 con 0x7fa014102300 2024-03-18T14:50:55.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.514+0000 7fa01259c640 1 --2- 172.21.15.76:0/3313733653 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9ffc0a2880 0x7f9ffc0a4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa0141019e0 tx=0x7fa008005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:55.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.518+0000 7f9ffbfff640 1 -- 172.21.15.76:0/3313733653 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa00404b020 con 0x7fa014102300 2024-03-18T14:50:55.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.732+0000 7fa019137640 1 -- 172.21.15.76:0/3313733653 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa01402d1b0 con 0x7fa014102300 2024-03-18T14:50:55.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.733+0000 7f9ffbfff640 1 -- 172.21.15.76:0/3313733653 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9650 (secure 0 0 0) 0x7fa0040bd9c0 con 0x7fa014102300 2024-03-18T14:50:55.734 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:55.734 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/2801069369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":2801069369},{"type":"v1","addr":"172.21.15.76:6839","nonce":2801069369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2645229232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2645229232},{"type":"v1","addr":"172.21.15.76:6841","nonce":2645229232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24490,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/399810652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":399810652},{"type":"v1","addr":"172.21.15.174:6839","nonce":399810652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T14:50:55.734 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 12 2024-03-18T14:50:55.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.735+0000 7fa019137640 1 -- 172.21.15.76:0/3313733653 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9ffc0a2880 msgr2=0x7f9ffc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:55.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.735+0000 7fa019137640 1 --2- 172.21.15.76:0/3313733653 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9ffc0a2880 0x7f9ffc0a4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa0141019e0 tx=0x7fa008005f90 comp rx=0 tx=0).stop 2024-03-18T14:50:55.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.735+0000 7fa019137640 1 -- 172.21.15.76:0/3313733653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa014102300 msgr2=0x7fa014101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:55.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.735+0000 7fa019137640 1 --2- 172.21.15.76:0/3313733653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa014102300 0x7fa014101c70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa0040325f0 tx=0x7fa004032ec0 comp rx=0 tx=0).stop 2024-03-18T14:50:55.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.735+0000 7fa019137640 1 -- 172.21.15.76:0/3313733653 shutdown_connections 2024-03-18T14:50:55.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.735+0000 7fa019137640 1 --2- 172.21.15.76:0/3313733653 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9ffc0a2880 0x7f9ffc0a4d40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:55.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.735+0000 7fa019137640 1 --2- 172.21.15.76:0/3313733653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa014104c30 0x7fa0141002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:55.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.735+0000 7fa019137640 1 --2- 172.21.15.76:0/3313733653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa014107560 0x7fa0141007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:55.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.735+0000 7fa019137640 1 --2- 172.21.15.76:0/3313733653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa014102300 0x7fa014101c70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:55.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.735+0000 7fa019137640 1 -- 172.21.15.76:0/3313733653 >> 172.21.15.76:0/3313733653 conn(0x7fa0140fbd10 msgr2=0x7fa014102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:55.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.735+0000 7fa019137640 1 -- 172.21.15.76:0/3313733653 shutdown_connections 2024-03-18T14:50:55.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:55.735+0000 7fa019137640 1 -- 172.21.15.76:0/3313733653 wait complete. 2024-03-18T14:50:55.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:55 smithi077 ceph-mon[35865]: Deploying daemon mds.b on smithi077 2024-03-18T14:50:55.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:55 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.76:6840/2645229232,v1:172.21.15.76:6841/2645229232] up:boot 2024-03-18T14:50:55.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:55 smithi077 ceph-mon[35865]: fsmap 11 up:standby 2024-03-18T14:50:55.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:55 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T14:50:55.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:55 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3313733653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:55 smithi076 ceph-mon[32205]: Deploying daemon mds.b on smithi077 2024-03-18T14:50:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:55 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.76:6840/2645229232,v1:172.21.15.76:6841/2645229232] up:boot 2024-03-18T14:50:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:55 smithi076 ceph-mon[32205]: fsmap 11 up:standby 2024-03-18T14:50:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:55 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T14:50:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:55 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3313733653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:56.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:55 smithi174 ceph-mon[36787]: Deploying daemon mds.b on smithi077 2024-03-18T14:50:56.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:55 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.76:6840/2645229232,v1:172.21.15.76:6841/2645229232] up:boot 2024-03-18T14:50:56.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:55 smithi174 ceph-mon[36787]: fsmap 11 up:standby 2024-03-18T14:50:56.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:55 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T14:50:56.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:55 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3313733653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:57.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:56 smithi076 ceph-mon[32205]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:57.187 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:56 smithi077 ceph-mon[35865]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:57.187 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 14:50:57 smithi077 systemd[1]: Starting Ceph mds.b for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T14:50:57.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:56 smithi174 ceph-mon[36787]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:57.477 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 14:50:57 smithi077 podman[66013]: 2024-03-18 14:50:57.303556657 +0000 UTC m=+0.017468248 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T14:50:57.477 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 14:50:57 smithi077 podman[66013]: 2024-03-18 14:50:57.412339095 +0000 UTC m=+0.126250714 container create 7709a3c15615b82ae888d42e45f2729bbbe2ca826e886c28a89d75f9a865e30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-18T14:50:57.748 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:50:57.816 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 14:50:57 smithi077 podman[66013]: 2024-03-18 14:50:57.64878702 +0000 UTC m=+0.362698606 container init 7709a3c15615b82ae888d42e45f2729bbbe2ca826e886c28a89d75f9a865e30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-b, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-18T14:50:57.816 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 14:50:57 smithi077 podman[66013]: 2024-03-18 14:50:57.65140994 +0000 UTC m=+0.365321517 container start 7709a3c15615b82ae888d42e45f2729bbbe2ca826e886c28a89d75f9a865e30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-18T14:50:57.816 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 14:50:57 smithi077 ceph-mds[66032]: -- 172.21.15.77:0/511087622 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a74307ba40 con 0x55a74304ec00 2024-03-18T14:50:57.816 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 14:50:57 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-b[66028]: starting mds.b at 2024-03-18T14:50:57.816 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 14:50:57 smithi077 bash[66013]: 7709a3c15615b82ae888d42e45f2729bbbe2ca826e886c28a89d75f9a865e30d 2024-03-18T14:50:57.816 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 14:50:57 smithi077 systemd[1]: Started Ceph mds.b for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T14:50:57.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.863+0000 7fa7efd52640 1 -- 172.21.15.76:0/2360898811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7e81033d0 msgr2=0x7fa7e8109c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:57.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.863+0000 7fa7efd52640 1 --2- 172.21.15.76:0/2360898811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7e81033d0 0x7fa7e8109c60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8009ca0 tx=0x7fa7d802c0f0 comp rx=0 tx=0).stop 2024-03-18T14:50:57.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.863+0000 7fa7efd52640 1 -- 172.21.15.76:0/2360898811 shutdown_connections 2024-03-18T14:50:57.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.863+0000 7fa7efd52640 1 --2- 172.21.15.76:0/2360898811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7e8102a10 0x7fa7e8102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:57.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.863+0000 7fa7efd52640 1 --2- 172.21.15.76:0/2360898811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7e81033d0 0x7fa7e8109c60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:57.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.863+0000 7fa7efd52640 1 --2- 172.21.15.76:0/2360898811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e8101820 0x7fa7e8101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:57.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.863+0000 7fa7efd52640 1 -- 172.21.15.76:0/2360898811 >> 172.21.15.76:0/2360898811 conn(0x7fa7e80fcdb0 msgr2=0x7fa7e80ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:57.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.863+0000 7fa7efd52640 1 -- 172.21.15.76:0/2360898811 shutdown_connections 2024-03-18T14:50:57.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.863+0000 7fa7efd52640 1 -- 172.21.15.76:0/2360898811 wait complete. 2024-03-18T14:50:57.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7efd52640 1 Processor -- start 2024-03-18T14:50:57.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7efd52640 1 -- start start 2024-03-18T14:50:57.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7efd52640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7e8101820 0x7fa7e8197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:57.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7efd52640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7e8102a10 0x7fa7e8197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:57.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7efd52640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e81033d0 0x7fa7e819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:57.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7efd52640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e810d4a0 con 0x7fa7e8102a10 2024-03-18T14:50:57.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7efd52640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e810d320 con 0x7fa7e81033d0 2024-03-18T14:50:57.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7efd52640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e810d620 con 0x7fa7e8101820 2024-03-18T14:50:57.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7ed2ed640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7e8102a10 0x7fa7e8197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:57.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7ed2ed640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7e8102a10 0x7fa7e8197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:42646/0 (socket says 172.21.15.76:42646) 2024-03-18T14:50:57.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7ed2ed640 1 -- 172.21.15.76:0/3471996531 learned_addr learned my addr 172.21.15.76:0/3471996531 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:57.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7edaee640 1 --2- 172.21.15.76:0/3471996531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7e8101820 0x7fa7e8197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:57.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7ee2ef640 1 --2- 172.21.15.76:0/3471996531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e81033d0 0x7fa7e819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:57.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7ed2ed640 1 -- 172.21.15.76:0/3471996531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7e8101820 msgr2=0x7fa7e8197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:57.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7ed2ed640 1 --2- 172.21.15.76:0/3471996531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7e8101820 0x7fa7e8197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:57.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7ed2ed640 1 -- 172.21.15.76:0/3471996531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e81033d0 msgr2=0x7fa7e819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:57.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7ed2ed640 1 --2- 172.21.15.76:0/3471996531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e81033d0 0x7fa7e819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:57.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7ed2ed640 1 -- 172.21.15.76:0/3471996531 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e819f6d0 con 0x7fa7e8102a10 2024-03-18T14:50:57.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.864+0000 7fa7ed2ed640 1 --2- 172.21.15.76:0/3471996531 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7e8102a10 0x7fa7e8197ff0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc00d770 tx=0x7fa7dc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:57.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.865+0000 7fa7e6ffd640 1 -- 172.21.15.76:0/3471996531 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc017070 con 0x7fa7e8102a10 2024-03-18T14:50:57.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.865+0000 7fa7e6ffd640 1 -- 172.21.15.76:0/3471996531 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7dc005500 con 0x7fa7e8102a10 2024-03-18T14:50:57.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.865+0000 7fa7efd52640 1 -- 172.21.15.76:0/3471996531 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e819f9c0 con 0x7fa7e8102a10 2024-03-18T14:50:57.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.865+0000 7fa7e6ffd640 1 -- 172.21.15.76:0/3471996531 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc004e00 con 0x7fa7e8102a10 2024-03-18T14:50:57.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.865+0000 7fa7efd52640 1 -- 172.21.15.76:0/3471996531 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e819fec0 con 0x7fa7e8102a10 2024-03-18T14:50:57.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.866+0000 7fa7efd52640 1 -- 172.21.15.76:0/3471996531 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e8108b40 con 0x7fa7e8102a10 2024-03-18T14:50:57.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.867+0000 7fa7e6ffd640 1 -- 172.21.15.76:0/3471996531 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa7dc004fa0 con 0x7fa7e8102a10 2024-03-18T14:50:57.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.868+0000 7fa7e6ffd640 1 --2- 172.21.15.76:0/3471996531 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa7d00a2800 0x7fa7d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:57.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.868+0000 7fa7edaee640 1 --2- 172.21.15.76:0/3471996531 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa7d00a2800 0x7fa7d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:57.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.868+0000 7fa7e6ffd640 1 -- 172.21.15.76:0/3471996531 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa7dc0ce350 con 0x7fa7e8102a10 2024-03-18T14:50:57.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.869+0000 7fa7edaee640 1 --2- 172.21.15.76:0/3471996531 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa7d00a2800 0x7fa7d00a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8102870 tx=0x7fa7e000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:57.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:57.872+0000 7fa7e6ffd640 1 -- 172.21.15.76:0/3471996531 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa7dc09b240 con 0x7fa7e8102a10 2024-03-18T14:50:58.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.093+0000 7fa7efd52640 1 -- 172.21.15.76:0/3471996531 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7e802d370 con 0x7fa7e8102a10 2024-03-18T14:50:58.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.094+0000 7fa7e6ffd640 1 -- 172.21.15.76:0/3471996531 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10476 (secure 0 0 0) 0x7fa7dc09f2e0 con 0x7fa7e8102a10 2024-03-18T14:50:58.095 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:58.096 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/2801069369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":2801069369},{"type":"v1","addr":"172.21.15.76:6839","nonce":2801069369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2645229232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2645229232},{"type":"v1","addr":"172.21.15.76:6841","nonce":2645229232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24490,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/399810652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":399810652},{"type":"v1","addr":"172.21.15.174:6839","nonce":399810652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24524,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1685041985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1685041985},{"type":"v1","addr":"172.21.15.77:6839","nonce":1685041985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-18T14:50:58.096 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 13 2024-03-18T14:50:58.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.096+0000 7fa7efd52640 1 -- 172.21.15.76:0/3471996531 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa7d00a2800 msgr2=0x7fa7d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:58.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.096+0000 7fa7efd52640 1 --2- 172.21.15.76:0/3471996531 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa7d00a2800 0x7fa7d00a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8102870 tx=0x7fa7e000b390 comp rx=0 tx=0).stop 2024-03-18T14:50:58.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.096+0000 7fa7efd52640 1 -- 172.21.15.76:0/3471996531 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7e8102a10 msgr2=0x7fa7e8197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:58.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.096+0000 7fa7efd52640 1 --2- 172.21.15.76:0/3471996531 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7e8102a10 0x7fa7e8197ff0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc00d770 tx=0x7fa7dc00dc30 comp rx=0 tx=0).stop 2024-03-18T14:50:58.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.096+0000 7fa7efd52640 1 -- 172.21.15.76:0/3471996531 shutdown_connections 2024-03-18T14:50:58.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.096+0000 7fa7efd52640 1 --2- 172.21.15.76:0/3471996531 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa7d00a2800 0x7fa7d00a4cc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.096+0000 7fa7efd52640 1 --2- 172.21.15.76:0/3471996531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e81033d0 0x7fa7e819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.096+0000 7fa7efd52640 1 --2- 172.21.15.76:0/3471996531 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7e8102a10 0x7fa7e8197ff0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.096+0000 7fa7efd52640 1 --2- 172.21.15.76:0/3471996531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7e8101820 0x7fa7e8197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.096+0000 7fa7efd52640 1 -- 172.21.15.76:0/3471996531 >> 172.21.15.76:0/3471996531 conn(0x7fa7e80fcdb0 msgr2=0x7fa7e8105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:58.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.096+0000 7fa7efd52640 1 -- 172.21.15.76:0/3471996531 shutdown_connections 2024-03-18T14:50:58.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.096+0000 7fa7efd52640 1 -- 172.21.15.76:0/3471996531 wait complete. 2024-03-18T14:50:58.109 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:50:58.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.212+0000 7ff20c4dc640 1 -- 172.21.15.76:0/3369942267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2041022b0 msgr2=0x7ff204108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:58.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.212+0000 7ff20c4dc640 1 --2- 172.21.15.76:0/3369942267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2041022b0 0x7ff204108b40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc009900 tx=0x7ff1fc02bc30 comp rx=0 tx=0).stop 2024-03-18T14:50:58.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.212+0000 7ff20c4dc640 1 -- 172.21.15.76:0/3369942267 shutdown_connections 2024-03-18T14:50:58.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.212+0000 7ff20c4dc640 1 --2- 172.21.15.76:0/3369942267 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2041018f0 0x7ff204101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.212+0000 7ff20c4dc640 1 --2- 172.21.15.76:0/3369942267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2041022b0 0x7ff204108b40 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.212+0000 7ff20c4dc640 1 --2- 172.21.15.76:0/3369942267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff204100700 0x7ff204100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.212+0000 7ff20c4dc640 1 -- 172.21.15.76:0/3369942267 >> 172.21.15.76:0/3369942267 conn(0x7ff2040fbc70 msgr2=0x7ff2040fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:58.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.212+0000 7ff20c4dc640 1 -- 172.21.15.76:0/3369942267 shutdown_connections 2024-03-18T14:50:58.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.212+0000 7ff20c4dc640 1 -- 172.21.15.76:0/3369942267 wait complete. 2024-03-18T14:50:58.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20c4dc640 1 Processor -- start 2024-03-18T14:50:58.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20c4dc640 1 -- start start 2024-03-18T14:50:58.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20c4dc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff204100700 0x7ff204197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:58.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20c4dc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2041018f0 0x7ff204198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:58.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20c4dc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2041022b0 0x7ff20419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:58.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20c4dc640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff20410c360 con 0x7ff2041022b0 2024-03-18T14:50:58.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20c4dc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff20410c1e0 con 0x7ff2041018f0 2024-03-18T14:50:58.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20c4dc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff20410c4e0 con 0x7ff204100700 2024-03-18T14:50:58.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20aa79640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2041022b0 0x7ff20419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:58.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20aa79640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2041022b0 0x7ff20419f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:42676/0 (socket says 172.21.15.76:42676) 2024-03-18T14:50:58.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20aa79640 1 -- 172.21.15.76:0/217144655 learned_addr learned my addr 172.21.15.76:0/217144655 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:58.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20a278640 1 --2- 172.21.15.76:0/217144655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff204100700 0x7ff204197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:58.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff209a77640 1 --2- 172.21.15.76:0/217144655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2041018f0 0x7ff204198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:58.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20aa79640 1 -- 172.21.15.76:0/217144655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff204100700 msgr2=0x7ff204197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:58.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20aa79640 1 --2- 172.21.15.76:0/217144655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff204100700 0x7ff204197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20aa79640 1 -- 172.21.15.76:0/217144655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2041018f0 msgr2=0x7ff204198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:58.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20aa79640 1 --2- 172.21.15.76:0/217144655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2041018f0 0x7ff204198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20aa79640 1 -- 172.21.15.76:0/217144655 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff20419f6e0 con 0x7ff2041022b0 2024-03-18T14:50:58.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.213+0000 7ff20aa79640 1 --2- 172.21.15.76:0/217144655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2041022b0 0x7ff20419f070 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc02ce70 tx=0x7ff1fc005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:58.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.214+0000 7ff1fb7fe640 1 -- 172.21.15.76:0/217144655 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc041070 con 0x7ff2041022b0 2024-03-18T14:50:58.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.214+0000 7ff20c4dc640 1 -- 172.21.15.76:0/217144655 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff20419f970 con 0x7ff2041022b0 2024-03-18T14:50:58.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.214+0000 7ff1fb7fe640 1 -- 172.21.15.76:0/217144655 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1fc0054e0 con 0x7ff2041022b0 2024-03-18T14:50:58.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.214+0000 7ff1fb7fe640 1 -- 172.21.15.76:0/217144655 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc0344a0 con 0x7ff2041022b0 2024-03-18T14:50:58.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.214+0000 7ff20c4dc640 1 -- 172.21.15.76:0/217144655 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff20419fe60 con 0x7ff2041022b0 2024-03-18T14:50:58.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.215+0000 7ff20c4dc640 1 -- 172.21.15.76:0/217144655 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2041a00f0 con 0x7ff2041022b0 2024-03-18T14:50:58.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.216+0000 7ff1fb7fe640 1 -- 172.21.15.76:0/217144655 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff1fc044070 con 0x7ff2041022b0 2024-03-18T14:50:58.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.216+0000 7ff1fb7fe640 1 --2- 172.21.15.76:0/217144655 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff1f00a2800 0x7ff1f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:58.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.216+0000 7ff20a278640 1 --2- 172.21.15.76:0/217144655 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff1f00a2800 0x7ff1f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:58.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.216+0000 7ff1fb7fe640 1 -- 172.21.15.76:0/217144655 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ff1fc0f0620 con 0x7ff2041022b0 2024-03-18T14:50:58.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.217+0000 7ff20a278640 1 --2- 172.21.15.76:0/217144655 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff1f00a2800 0x7ff1f00a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff204101750 tx=0x7ff200009720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:58.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.224+0000 7ff1fb7fe640 1 -- 172.21.15.76:0/217144655 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff1fc0bd510 con 0x7ff2041022b0 2024-03-18T14:50:58.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.409+0000 7ff20c4dc640 1 -- 172.21.15.76:0/217144655 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff2041a0480 con 0x7ff2041022b0 2024-03-18T14:50:58.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.411+0000 7ff1fb7fe640 1 -- 172.21.15.76:0/217144655 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18429 (secure 0 0 0) 0x7ff2041a0480 con 0x7ff2041022b0 2024-03-18T14:50:58.413 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:58.413 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":82,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:50:14.506203+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:50:58.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.413+0000 7ff20c4dc640 1 -- 172.21.15.76:0/217144655 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff1f00a2800 msgr2=0x7ff1f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:58.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.413+0000 7ff20c4dc640 1 --2- 172.21.15.76:0/217144655 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff1f00a2800 0x7ff1f00a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff204101750 tx=0x7ff200009720 comp rx=0 tx=0).stop 2024-03-18T14:50:58.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.413+0000 7ff20c4dc640 1 -- 172.21.15.76:0/217144655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2041022b0 msgr2=0x7ff20419f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:58.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.413+0000 7ff20c4dc640 1 --2- 172.21.15.76:0/217144655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2041022b0 0x7ff20419f070 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc02ce70 tx=0x7ff1fc005e40 comp rx=0 tx=0).stop 2024-03-18T14:50:58.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.413+0000 7ff20c4dc640 1 -- 172.21.15.76:0/217144655 shutdown_connections 2024-03-18T14:50:58.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.413+0000 7ff20c4dc640 1 --2- 172.21.15.76:0/217144655 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff1f00a2800 0x7ff1f00a4cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.413+0000 7ff20c4dc640 1 --2- 172.21.15.76:0/217144655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2041022b0 0x7ff20419f070 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.413+0000 7ff20c4dc640 1 --2- 172.21.15.76:0/217144655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2041018f0 0x7ff204198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.413+0000 7ff20c4dc640 1 --2- 172.21.15.76:0/217144655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff204100700 0x7ff204197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.413+0000 7ff20c4dc640 1 -- 172.21.15.76:0/217144655 >> 172.21.15.76:0/217144655 conn(0x7ff2040fbc70 msgr2=0x7ff2040fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:58.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.413+0000 7ff20c4dc640 1 -- 172.21.15.76:0/217144655 shutdown_connections 2024-03-18T14:50:58.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.413+0000 7ff20c4dc640 1 -- 172.21.15.76:0/217144655 wait complete. 2024-03-18T14:50:58.425 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:46:26.588384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:50:58.426 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:50:58.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.544+0000 7fc439e0b640 1 -- 172.21.15.76:0/1850747242 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4341022b0 msgr2=0x7fc434108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:58.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.544+0000 7fc439e0b640 1 --2- 172.21.15.76:0/1850747242 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4341022b0 0x7fc434108b40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009ca0 tx=0x7fc42c02c0f0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.544+0000 7fc439e0b640 1 -- 172.21.15.76:0/1850747242 shutdown_connections 2024-03-18T14:50:58.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.544+0000 7fc439e0b640 1 --2- 172.21.15.76:0/1850747242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4341018f0 0x7fc434101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.544+0000 7fc439e0b640 1 --2- 172.21.15.76:0/1850747242 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4341022b0 0x7fc434108b40 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.544+0000 7fc439e0b640 1 --2- 172.21.15.76:0/1850747242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc434100700 0x7fc434100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.544+0000 7fc439e0b640 1 -- 172.21.15.76:0/1850747242 >> 172.21.15.76:0/1850747242 conn(0x7fc4340fbc90 msgr2=0x7fc4340fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:58.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.544+0000 7fc439e0b640 1 -- 172.21.15.76:0/1850747242 shutdown_connections 2024-03-18T14:50:58.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.544+0000 7fc439e0b640 1 -- 172.21.15.76:0/1850747242 wait complete. 2024-03-18T14:50:58.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.544+0000 7fc439e0b640 1 Processor -- start 2024-03-18T14:50:58.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc439e0b640 1 -- start start 2024-03-18T14:50:58.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc439e0b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc434100700 0x7fc43406a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:58.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc439e0b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4341018f0 0x7fc43419c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:58.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc439e0b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4341022b0 0x7fc43419ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:58.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc439e0b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc43410c360 con 0x7fc4341022b0 2024-03-18T14:50:58.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc439e0b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc43410c1e0 con 0x7fc4341018f0 2024-03-18T14:50:58.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc439e0b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc43410c4e0 con 0x7fc434100700 2024-03-18T14:50:58.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc433fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4341022b0 0x7fc43419ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:58.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc433fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4341022b0 0x7fc43419ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57946/0 (socket says 172.21.15.76:57946) 2024-03-18T14:50:58.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc433fff640 1 -- 172.21.15.76:0/3396821970 learned_addr learned my addr 172.21.15.76:0/3396821970 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:58.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc432ffd640 1 --2- 172.21.15.76:0/3396821970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4341018f0 0x7fc43419c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:58.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc4337fe640 1 --2- 172.21.15.76:0/3396821970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc434100700 0x7fc43406a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:58.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc433fff640 1 -- 172.21.15.76:0/3396821970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc434100700 msgr2=0x7fc43406a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:58.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc433fff640 1 --2- 172.21.15.76:0/3396821970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc434100700 0x7fc43406a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc433fff640 1 -- 172.21.15.76:0/3396821970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4341018f0 msgr2=0x7fc43419c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:58.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc433fff640 1 --2- 172.21.15.76:0/3396821970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4341018f0 0x7fc43419c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc433fff640 1 -- 172.21.15.76:0/3396821970 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc43419f6c0 con 0x7fc4341022b0 2024-03-18T14:50:58.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.545+0000 7fc432ffd640 1 --2- 172.21.15.76:0/3396821970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4341018f0 0x7fc43419c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:50:58.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.546+0000 7fc433fff640 1 --2- 172.21.15.76:0/3396821970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4341022b0 0x7fc43419ee00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc42c037040 tx=0x7fc42c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:58.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.546+0000 7fc430ff9640 1 -- 172.21.15.76:0/3396821970 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c041020 con 0x7fc4341022b0 2024-03-18T14:50:58.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.546+0000 7fc439e0b640 1 -- 172.21.15.76:0/3396821970 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc43419f950 con 0x7fc4341022b0 2024-03-18T14:50:58.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.546+0000 7fc430ff9640 1 -- 172.21.15.76:0/3396821970 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc42c034d90 con 0x7fc4341022b0 2024-03-18T14:50:58.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.546+0000 7fc430ff9640 1 -- 172.21.15.76:0/3396821970 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c039040 con 0x7fc4341022b0 2024-03-18T14:50:58.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.546+0000 7fc439e0b640 1 -- 172.21.15.76:0/3396821970 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc43419fe90 con 0x7fc4341022b0 2024-03-18T14:50:58.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.547+0000 7fc439e0b640 1 -- 172.21.15.76:0/3396821970 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4341079c0 con 0x7fc4341022b0 2024-03-18T14:50:58.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.549+0000 7fc430ff9640 1 -- 172.21.15.76:0/3396821970 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc42c044020 con 0x7fc4341022b0 2024-03-18T14:50:58.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.549+0000 7fc430ff9640 1 --2- 172.21.15.76:0/3396821970 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc41c0a2800 0x7fc41c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:58.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.549+0000 7fc4337fe640 1 --2- 172.21.15.76:0/3396821970 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc41c0a2800 0x7fc41c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:58.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.549+0000 7fc430ff9640 1 -- 172.21.15.76:0/3396821970 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc42c0f0870 con 0x7fc4341022b0 2024-03-18T14:50:58.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.550+0000 7fc4337fe640 1 --2- 172.21.15.76:0/3396821970 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc41c0a2800 0x7fc41c0a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc434101750 tx=0x7fc424005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:58.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.552+0000 7fc430ff9640 1 -- 172.21.15.76:0/3396821970 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc42c04b020 con 0x7fc4341022b0 2024-03-18T14:50:58.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.734+0000 7fc439e0b640 1 -- 172.21.15.76:0/3396821970 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc43402d1b0 con 0x7fc4341022b0 2024-03-18T14:50:58.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.735+0000 7fc430ff9640 1 -- 172.21.15.76:0/3396821970 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fc42c0dd070 con 0x7fc4341022b0 2024-03-18T14:50:58.736 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-18T14:50:58.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.736+0000 7fc439e0b640 1 -- 172.21.15.76:0/3396821970 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc41c0a2800 msgr2=0x7fc41c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:58.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.736+0000 7fc439e0b640 1 --2- 172.21.15.76:0/3396821970 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc41c0a2800 0x7fc41c0a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc434101750 tx=0x7fc424005f90 comp rx=0 tx=0).stop 2024-03-18T14:50:58.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.736+0000 7fc439e0b640 1 -- 172.21.15.76:0/3396821970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4341022b0 msgr2=0x7fc43419ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:58.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.736+0000 7fc439e0b640 1 --2- 172.21.15.76:0/3396821970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4341022b0 0x7fc43419ee00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc42c037040 tx=0x7fc42c032f20 comp rx=0 tx=0).stop 2024-03-18T14:50:58.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.736+0000 7fc439e0b640 1 -- 172.21.15.76:0/3396821970 shutdown_connections 2024-03-18T14:50:58.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.736+0000 7fc439e0b640 1 --2- 172.21.15.76:0/3396821970 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc41c0a2800 0x7fc41c0a4cc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.736+0000 7fc439e0b640 1 --2- 172.21.15.76:0/3396821970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4341022b0 0x7fc43419ee00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.736+0000 7fc439e0b640 1 --2- 172.21.15.76:0/3396821970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4341018f0 0x7fc43419c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.736+0000 7fc439e0b640 1 --2- 172.21.15.76:0/3396821970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc434100700 0x7fc43406a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.736+0000 7fc439e0b640 1 -- 172.21.15.76:0/3396821970 >> 172.21.15.76:0/3396821970 conn(0x7fc4340fbc90 msgr2=0x7fc434104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:58.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.737+0000 7fc439e0b640 1 -- 172.21.15.76:0/3396821970 shutdown_connections 2024-03-18T14:50:58.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.737+0000 7fc439e0b640 1 -- 172.21.15.76:0/3396821970 wait complete. 2024-03-18T14:50:58.749 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T14:50:58.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.850+0000 7fe76cf10640 1 -- 172.21.15.76:0/3932824452 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe768104c30 msgr2=0x7fe768107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:58.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.850+0000 7fe76cf10640 1 --2- 172.21.15.76:0/3932824452 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe768104c30 0x7fe768107020 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe750009900 tx=0x7fe75002bc30 comp rx=0 tx=0).stop 2024-03-18T14:50:58.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.850+0000 7fe76cf10640 1 -- 172.21.15.76:0/3932824452 shutdown_connections 2024-03-18T14:50:58.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.850+0000 7fe76cf10640 1 --2- 172.21.15.76:0/3932824452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe768107560 0x7fe7681099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.850+0000 7fe76cf10640 1 --2- 172.21.15.76:0/3932824452 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe768104c30 0x7fe768107020 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.850+0000 7fe76cf10640 1 --2- 172.21.15.76:0/3932824452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe768102300 0x7fe7681046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.850+0000 7fe76cf10640 1 -- 172.21.15.76:0/3932824452 >> 172.21.15.76:0/3932824452 conn(0x7fe7680fbd10 msgr2=0x7fe7680fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:58.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.850+0000 7fe76cf10640 1 -- 172.21.15.76:0/3932824452 shutdown_connections 2024-03-18T14:50:58.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.850+0000 7fe76cf10640 1 -- 172.21.15.76:0/3932824452 wait complete. 2024-03-18T14:50:58.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.851+0000 7fe76cf10640 1 Processor -- start 2024-03-18T14:50:58.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.851+0000 7fe76cf10640 1 -- start start 2024-03-18T14:50:58.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.851+0000 7fe76cf10640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe768102300 0x7fe768197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:58.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.851+0000 7fe76cf10640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe768104c30 0x7fe768198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:58.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.851+0000 7fe76cf10640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe768107560 0x7fe76819f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:58.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.851+0000 7fe76cf10640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe76810d480 con 0x7fe768102300 2024-03-18T14:50:58.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.851+0000 7fe76cf10640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe76810d300 con 0x7fe768107560 2024-03-18T14:50:58.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.851+0000 7fe76cf10640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe76810d600 con 0x7fe768104c30 2024-03-18T14:50:58.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.851+0000 7fe76659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe768102300 0x7fe768197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:58.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.851+0000 7fe76659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe768102300 0x7fe768197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57960/0 (socket says 172.21.15.76:57960) 2024-03-18T14:50:58.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.851+0000 7fe76659c640 1 -- 172.21.15.76:0/2507281010 learned_addr learned my addr 172.21.15.76:0/2507281010 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:58.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.851+0000 7fe766d9d640 1 --2- 172.21.15.76:0/2507281010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe768107560 0x7fe76819f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:58.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.851+0000 7fe75dd9b640 1 --2- 172.21.15.76:0/2507281010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe768104c30 0x7fe768198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:58.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.852+0000 7fe76659c640 1 -- 172.21.15.76:0/2507281010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe768104c30 msgr2=0x7fe768198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:58.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.852+0000 7fe76659c640 1 --2- 172.21.15.76:0/2507281010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe768104c30 0x7fe768198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.852+0000 7fe76659c640 1 -- 172.21.15.76:0/2507281010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe768107560 msgr2=0x7fe76819f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:58.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.852+0000 7fe76659c640 1 --2- 172.21.15.76:0/2507281010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe768107560 0x7fe76819f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:58.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.852+0000 7fe76659c640 1 -- 172.21.15.76:0/2507281010 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe76819f750 con 0x7fe768102300 2024-03-18T14:50:58.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.852+0000 7fe76659c640 1 --2- 172.21.15.76:0/2507281010 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe768102300 0x7fe768197b30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe75800d950 tx=0x7fe75800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:58.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.852+0000 7fe75ffff640 1 -- 172.21.15.76:0/2507281010 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe758017030 con 0x7fe768102300 2024-03-18T14:50:58.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.852+0000 7fe76cf10640 1 -- 172.21.15.76:0/2507281010 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe76819fa40 con 0x7fe768102300 2024-03-18T14:50:58.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.852+0000 7fe75ffff640 1 -- 172.21.15.76:0/2507281010 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe758004020 con 0x7fe768102300 2024-03-18T14:50:58.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.852+0000 7fe75ffff640 1 -- 172.21.15.76:0/2507281010 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7580047a0 con 0x7fe768102300 2024-03-18T14:50:58.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.852+0000 7fe76cf10640 1 -- 172.21.15.76:0/2507281010 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe76819ffb0 con 0x7fe768102300 2024-03-18T14:50:58.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.853+0000 7fe76cf10640 1 -- 172.21.15.76:0/2507281010 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7680fd800 con 0x7fe768102300 2024-03-18T14:50:58.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.854+0000 7fe75ffff640 1 -- 172.21.15.76:0/2507281010 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe7580041d0 con 0x7fe768102300 2024-03-18T14:50:58.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.854+0000 7fe75ffff640 1 --2- 172.21.15.76:0/2507281010 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe7540a2800 0x7fe7540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:58.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.854+0000 7fe75dd9b640 1 --2- 172.21.15.76:0/2507281010 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe7540a2800 0x7fe7540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:58.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.855+0000 7fe75ffff640 1 -- 172.21.15.76:0/2507281010 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe7580ce040 con 0x7fe768102300 2024-03-18T14:50:58.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.855+0000 7fe75dd9b640 1 --2- 172.21.15.76:0/2507281010 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe7540a2800 0x7fe7540a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe75002c7b0 tx=0x7fe750005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:58.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:58.860+0000 7fe75ffff640 1 -- 172.21.15.76:0/2507281010 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe75809b580 con 0x7fe768102300 2024-03-18T14:50:59.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:58 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:59.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:58 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:59.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:58 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:59.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:58 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:59.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:58 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:59.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:58 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] up:boot 2024-03-18T14:50:59.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:58 smithi076 ceph-mon[32205]: fsmap 12 up:standby 2024-03-18T14:50:59.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:58 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T14:50:59.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:58 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:50:59.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:58 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3471996531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:59.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:58 smithi076 ceph-mon[32205]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:59.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:58 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/217144655' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:50:59.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:58 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3396821970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:50:59.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.046+0000 7fe76cf10640 1 -- 172.21.15.76:0/2507281010 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fe76802d460 con 0x7fe768102300 2024-03-18T14:50:59.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.047+0000 7fe75ffff640 1 -- 172.21.15.76:0/2507281010 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7fe75809ae80 con 0x7fe768102300 2024-03-18T14:50:59.049 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:50:59.049 INFO:teuthology.orchestra.run.smithi076.stdout:[ 2024-03-18T14:50:59.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:50:59.049 INFO:teuthology.orchestra.run.smithi076.stdout: "poolnum": 1, 2024-03-18T14:50:59.049 INFO:teuthology.orchestra.run.smithi076.stdout: "poolname": ".mgr" 2024-03-18T14:50:59.049 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T14:50:59.049 INFO:teuthology.orchestra.run.smithi076.stdout:] 2024-03-18T14:50:59.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.049+0000 7fe76cf10640 1 -- 172.21.15.76:0/2507281010 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe7540a2800 msgr2=0x7fe7540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:59.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.049+0000 7fe76cf10640 1 --2- 172.21.15.76:0/2507281010 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe7540a2800 0x7fe7540a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe75002c7b0 tx=0x7fe750005fd0 comp rx=0 tx=0).stop 2024-03-18T14:50:59.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.049+0000 7fe76cf10640 1 -- 172.21.15.76:0/2507281010 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe768102300 msgr2=0x7fe768197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:59.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.049+0000 7fe76cf10640 1 --2- 172.21.15.76:0/2507281010 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe768102300 0x7fe768197b30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe75800d950 tx=0x7fe75800de10 comp rx=0 tx=0).stop 2024-03-18T14:50:59.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.049+0000 7fe76cf10640 1 -- 172.21.15.76:0/2507281010 shutdown_connections 2024-03-18T14:50:59.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.049+0000 7fe76cf10640 1 --2- 172.21.15.76:0/2507281010 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe7540a2800 0x7fe7540a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:59.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.049+0000 7fe76cf10640 1 --2- 172.21.15.76:0/2507281010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe768104c30 0x7fe768198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:59.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.049+0000 7fe76cf10640 1 --2- 172.21.15.76:0/2507281010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe768107560 0x7fe76819f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:59.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.049+0000 7fe76cf10640 1 --2- 172.21.15.76:0/2507281010 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe768102300 0x7fe768197b30 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:59.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.049+0000 7fe76cf10640 1 -- 172.21.15.76:0/2507281010 >> 172.21.15.76:0/2507281010 conn(0x7fe7680fbd10 msgr2=0x7fe768102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:59.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.049+0000 7fe76cf10640 1 -- 172.21.15.76:0/2507281010 shutdown_connections 2024-03-18T14:50:59.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.049+0000 7fe76cf10640 1 -- 172.21.15.76:0/2507281010 wait complete. 2024-03-18T14:50:59.052 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:58 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:59.053 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:58 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:59.053 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:58 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:59.053 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:58 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:59.053 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:58 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:59.053 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:58 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] up:boot 2024-03-18T14:50:59.053 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:58 smithi077 ceph-mon[35865]: fsmap 12 up:standby 2024-03-18T14:50:59.053 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:58 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T14:50:59.053 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:58 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:50:59.053 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:58 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3471996531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:59.053 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:58 smithi077 ceph-mon[35865]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:59.053 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:58 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/217144655' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:50:59.054 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:58 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3396821970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:50:59.062 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T14:50:59.062 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T14:50:59.116 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:58 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:59.116 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:58 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:59.116 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:58 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:59.116 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:58 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:59.116 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:58 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:50:59.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:58 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] up:boot 2024-03-18T14:50:59.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:58 smithi174 ceph-mon[36787]: fsmap 12 up:standby 2024-03-18T14:50:59.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:58 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T14:50:59.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:58 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:50:59.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:58 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3471996531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:50:59.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:58 smithi174 ceph-mon[36787]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:50:59.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:58 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/217144655' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:50:59.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:58 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3396821970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:50:59.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.168+0000 7f84ff475640 1 -- 172.21.15.76:0/4114201640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84f81022a0 msgr2=0x7f84f8108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:59.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.168+0000 7f84ff475640 1 --2- 172.21.15.76:0/4114201640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84f81022a0 0x7f84f8108b30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f84f4009900 tx=0x7f84f402bc30 comp rx=0 tx=0).stop 2024-03-18T14:50:59.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.168+0000 7f84ff475640 1 -- 172.21.15.76:0/4114201640 shutdown_connections 2024-03-18T14:50:59.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.168+0000 7f84ff475640 1 --2- 172.21.15.76:0/4114201640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84f81018e0 0x7f84f8101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:59.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.168+0000 7f84ff475640 1 --2- 172.21.15.76:0/4114201640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84f81022a0 0x7f84f8108b30 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:59.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.168+0000 7f84ff475640 1 --2- 172.21.15.76:0/4114201640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84f81006f0 0x7f84f8100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:59.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.168+0000 7f84ff475640 1 -- 172.21.15.76:0/4114201640 >> 172.21.15.76:0/4114201640 conn(0x7f84f80fbc80 msgr2=0x7f84f80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:59.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.168+0000 7f84ff475640 1 -- 172.21.15.76:0/4114201640 shutdown_connections 2024-03-18T14:50:59.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.168+0000 7f84ff475640 1 -- 172.21.15.76:0/4114201640 wait complete. 2024-03-18T14:50:59.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.168+0000 7f84ff475640 1 Processor -- start 2024-03-18T14:50:59.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84ff475640 1 -- start start 2024-03-18T14:50:59.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84ff475640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84f81006f0 0x7f84f806a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:59.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84ff475640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84f81018e0 0x7f84f819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:59.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84ff475640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84f81022a0 0x7f84f819ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:59.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84ff475640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f84f810c370 con 0x7f84f81006f0 2024-03-18T14:50:59.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84ff475640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f84f810c1f0 con 0x7f84f81022a0 2024-03-18T14:50:59.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84ff475640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f84f810c4f0 con 0x7f84f81018e0 2024-03-18T14:50:59.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84fd211640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84f81006f0 0x7f84f806a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:59.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84fd211640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84f81006f0 0x7f84f806a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57992/0 (socket says 172.21.15.76:57992) 2024-03-18T14:50:59.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84fd211640 1 -- 172.21.15.76:0/1455669544 learned_addr learned my addr 172.21.15.76:0/1455669544 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:50:59.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84fca10640 1 --2- 172.21.15.76:0/1455669544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84f81018e0 0x7f84f819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:59.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84fda12640 1 --2- 172.21.15.76:0/1455669544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84f81022a0 0x7f84f819ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:59.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84fd211640 1 -- 172.21.15.76:0/1455669544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84f81018e0 msgr2=0x7f84f819c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:59.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84fd211640 1 --2- 172.21.15.76:0/1455669544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84f81018e0 0x7f84f819c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:59.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84fd211640 1 -- 172.21.15.76:0/1455669544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84f81022a0 msgr2=0x7f84f819ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:59.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84fd211640 1 --2- 172.21.15.76:0/1455669544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84f81022a0 0x7f84f819ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:59.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84fd211640 1 -- 172.21.15.76:0/1455669544 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f819f740 con 0x7f84f81006f0 2024-03-18T14:50:59.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84fd211640 1 --2- 172.21.15.76:0/1455669544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84f81006f0 0x7f84f806a940 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f84f000ebc0 tx=0x7f84f000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:59.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84ee7fc640 1 -- 172.21.15.76:0/1455669544 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0003b70 con 0x7f84f81006f0 2024-03-18T14:50:59.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84ff475640 1 -- 172.21.15.76:0/1455669544 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84f819f9d0 con 0x7f84f81006f0 2024-03-18T14:50:59.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84ee7fc640 1 -- 172.21.15.76:0/1455669544 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84f0003d10 con 0x7f84f81006f0 2024-03-18T14:50:59.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.169+0000 7f84ee7fc640 1 -- 172.21.15.76:0/1455669544 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0019980 con 0x7f84f81006f0 2024-03-18T14:50:59.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.170+0000 7f84ff475640 1 -- 172.21.15.76:0/1455669544 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84f819ff70 con 0x7f84f81006f0 2024-03-18T14:50:59.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.170+0000 7f84ff475640 1 -- 172.21.15.76:0/1455669544 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84f8107a10 con 0x7f84f81006f0 2024-03-18T14:50:59.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.171+0000 7f84ee7fc640 1 -- 172.21.15.76:0/1455669544 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f84f0019b20 con 0x7f84f81006f0 2024-03-18T14:50:59.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.172+0000 7f84ee7fc640 1 --2- 172.21.15.76:0/1455669544 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f84e40a2800 0x7f84e40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:50:59.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.172+0000 7f84fca10640 1 --2- 172.21.15.76:0/1455669544 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f84e40a2800 0x7f84e40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:50:59.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.172+0000 7f84ee7fc640 1 -- 172.21.15.76:0/1455669544 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f84f00ceda0 con 0x7f84f81006f0 2024-03-18T14:50:59.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.172+0000 7f84fca10640 1 --2- 172.21.15.76:0/1455669544 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f84e40a2800 0x7f84e40a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f84f806a6b0 tx=0x7f84e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:50:59.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.176+0000 7f84ee7fc640 1 -- 172.21.15.76:0/1455669544 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f84f009c2e0 con 0x7f84f81006f0 2024-03-18T14:50:59.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.356+0000 7f84ff475640 1 -- 172.21.15.76:0/1455669544 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f84f802d3c0 con 0x7f84f81006f0 2024-03-18T14:50:59.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.913+0000 7f84ee7fc640 1 -- 172.21.15.76:0/1455669544 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7f84f009bbe0 con 0x7f84f81006f0 2024-03-18T14:50:59.914 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'cephfs_metadata' created 2024-03-18T14:50:59.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.916+0000 7f84ff475640 1 -- 172.21.15.76:0/1455669544 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f84e40a2800 msgr2=0x7f84e40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:59.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.916+0000 7f84ff475640 1 --2- 172.21.15.76:0/1455669544 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f84e40a2800 0x7f84e40a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f84f806a6b0 tx=0x7f84e8005f90 comp rx=0 tx=0).stop 2024-03-18T14:50:59.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.916+0000 7f84ff475640 1 -- 172.21.15.76:0/1455669544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84f81006f0 msgr2=0x7f84f806a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:50:59.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.916+0000 7f84ff475640 1 --2- 172.21.15.76:0/1455669544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84f81006f0 0x7f84f806a940 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f84f000ebc0 tx=0x7f84f000c6a0 comp rx=0 tx=0).stop 2024-03-18T14:50:59.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.916+0000 7f84ff475640 1 -- 172.21.15.76:0/1455669544 shutdown_connections 2024-03-18T14:50:59.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.916+0000 7f84ff475640 1 --2- 172.21.15.76:0/1455669544 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f84e40a2800 0x7f84e40a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:59.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.916+0000 7f84ff475640 1 --2- 172.21.15.76:0/1455669544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84f81018e0 0x7f84f819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:59.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.916+0000 7f84ff475640 1 --2- 172.21.15.76:0/1455669544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84f81022a0 0x7f84f819ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:59.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.916+0000 7f84ff475640 1 --2- 172.21.15.76:0/1455669544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84f81006f0 0x7f84f806a940 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:50:59.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.916+0000 7f84ff475640 1 -- 172.21.15.76:0/1455669544 >> 172.21.15.76:0/1455669544 conn(0x7f84f80fbc80 msgr2=0x7f84f8104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:50:59.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.916+0000 7f84ff475640 1 -- 172.21.15.76:0/1455669544 shutdown_connections 2024-03-18T14:50:59.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:50:59.916+0000 7f84ff475640 1 -- 172.21.15.76:0/1455669544 wait complete. 2024-03-18T14:50:59.930 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T14:51:00.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.042+0000 7efd480b2640 1 -- 172.21.15.76:0/1930241549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd40102290 msgr2=0x7efd40108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:00.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.042+0000 7efd480b2640 1 --2- 172.21.15.76:0/1930241549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd40102290 0x7efd40108b20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efd38009ca0 tx=0x7efd3802c0f0 comp rx=0 tx=0).stop 2024-03-18T14:51:00.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.042+0000 7efd480b2640 1 -- 172.21.15.76:0/1930241549 shutdown_connections 2024-03-18T14:51:00.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.042+0000 7efd480b2640 1 --2- 172.21.15.76:0/1930241549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd401018d0 0x7efd40101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:00.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.042+0000 7efd480b2640 1 --2- 172.21.15.76:0/1930241549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd40102290 0x7efd40108b20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:00.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.042+0000 7efd480b2640 1 --2- 172.21.15.76:0/1930241549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd401006e0 0x7efd40100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:00.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.042+0000 7efd480b2640 1 -- 172.21.15.76:0/1930241549 >> 172.21.15.76:0/1930241549 conn(0x7efd400fbc70 msgr2=0x7efd400fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:00.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.042+0000 7efd480b2640 1 -- 172.21.15.76:0/1930241549 shutdown_connections 2024-03-18T14:51:00.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.042+0000 7efd480b2640 1 -- 172.21.15.76:0/1930241549 wait complete. 2024-03-18T14:51:00.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd480b2640 1 Processor -- start 2024-03-18T14:51:00.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd480b2640 1 -- start start 2024-03-18T14:51:00.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd480b2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd401006e0 0x7efd40197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:00.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd480b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd401018d0 0x7efd40198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:00.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd480b2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd40102290 0x7efd4019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:00.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd480b2640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efd4010c360 con 0x7efd40102290 2024-03-18T14:51:00.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd480b2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efd4010c1e0 con 0x7efd401018d0 2024-03-18T14:51:00.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd480b2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efd4010c4e0 con 0x7efd401006e0 2024-03-18T14:51:00.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd4664f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd40102290 0x7efd4019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:00.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd4664f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd40102290 0x7efd4019f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58010/0 (socket says 172.21.15.76:58010) 2024-03-18T14:51:00.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd4664f640 1 -- 172.21.15.76:0/1304398132 learned_addr learned my addr 172.21.15.76:0/1304398132 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:00.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd45e4e640 1 --2- 172.21.15.76:0/1304398132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd401006e0 0x7efd40197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:00.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd4664f640 1 -- 172.21.15.76:0/1304398132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd401006e0 msgr2=0x7efd40197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:00.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd4664f640 1 --2- 172.21.15.76:0/1304398132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd401006e0 0x7efd40197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:00.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd4664f640 1 -- 172.21.15.76:0/1304398132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd401018d0 msgr2=0x7efd40198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:00.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd4564d640 1 --2- 172.21.15.76:0/1304398132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd401018d0 0x7efd40198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:00.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd4664f640 1 --2- 172.21.15.76:0/1304398132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd401018d0 0x7efd40198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:00.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.043+0000 7efd4664f640 1 -- 172.21.15.76:0/1304398132 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd4019f720 con 0x7efd40102290 2024-03-18T14:51:00.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.044+0000 7efd4664f640 1 --2- 172.21.15.76:0/1304398132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd40102290 0x7efd4019f0b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efd38037040 tx=0x7efd38032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:00.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.044+0000 7efd36ffd640 1 -- 172.21.15.76:0/1304398132 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd38041020 con 0x7efd40102290 2024-03-18T14:51:00.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.044+0000 7efd480b2640 1 -- 172.21.15.76:0/1304398132 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd4019f9b0 con 0x7efd40102290 2024-03-18T14:51:00.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.044+0000 7efd36ffd640 1 -- 172.21.15.76:0/1304398132 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd38034cd0 con 0x7efd40102290 2024-03-18T14:51:00.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.044+0000 7efd36ffd640 1 -- 172.21.15.76:0/1304398132 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd38039040 con 0x7efd40102290 2024-03-18T14:51:00.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.044+0000 7efd480b2640 1 -- 172.21.15.76:0/1304398132 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd4019fea0 con 0x7efd40102290 2024-03-18T14:51:00.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.045+0000 7efd480b2640 1 -- 172.21.15.76:0/1304398132 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd401a00d0 con 0x7efd40102290 2024-03-18T14:51:00.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.046+0000 7efd36ffd640 1 -- 172.21.15.76:0/1304398132 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7efd38044020 con 0x7efd40102290 2024-03-18T14:51:00.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.046+0000 7efd36ffd640 1 --2- 172.21.15.76:0/1304398132 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efd2c0a2800 0x7efd2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:00.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.047+0000 7efd45e4e640 1 --2- 172.21.15.76:0/1304398132 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efd2c0a2800 0x7efd2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:00.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.047+0000 7efd36ffd640 1 -- 172.21.15.76:0/1304398132 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7efd380f09f0 con 0x7efd40102290 2024-03-18T14:51:00.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.047+0000 7efd45e4e640 1 --2- 172.21.15.76:0/1304398132 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efd2c0a2800 0x7efd2c0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efd40101730 tx=0x7efd30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:00.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.050+0000 7efd36ffd640 1 -- 172.21.15.76:0/1304398132 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efd3804b020 con 0x7efd40102290 2024-03-18T14:51:00.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:59 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2507281010' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T14:51:00.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:59 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1455669544' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T14:51:00.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:59 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:00.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:59 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:00.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:59 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:00.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:50:59 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:00.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:59 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2507281010' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T14:51:00.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:59 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1455669544' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T14:51:00.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:59 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:00.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:59 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:00.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:59 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:00.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:50:59 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:00.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:59 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2507281010' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T14:51:00.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:59 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1455669544' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T14:51:00.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:00.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:00.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:00.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:50:59 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:00.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.228+0000 7efd480b2640 1 -- 172.21.15.76:0/1304398132 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7efd401a02a0 con 0x7efd40102290 2024-03-18T14:51:00.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.915+0000 7efd36ffd640 1 -- 172.21.15.76:0/1304398132 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) ==== 170+0+0 (secure 0 0 0) 0x7efd380bde50 con 0x7efd40102290 2024-03-18T14:51:00.916 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'cephfs_data' created 2024-03-18T14:51:00.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.918+0000 7efd480b2640 1 -- 172.21.15.76:0/1304398132 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efd2c0a2800 msgr2=0x7efd2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:00.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.918+0000 7efd480b2640 1 --2- 172.21.15.76:0/1304398132 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efd2c0a2800 0x7efd2c0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efd40101730 tx=0x7efd30005f90 comp rx=0 tx=0).stop 2024-03-18T14:51:00.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.918+0000 7efd480b2640 1 -- 172.21.15.76:0/1304398132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd40102290 msgr2=0x7efd4019f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:00.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.918+0000 7efd480b2640 1 --2- 172.21.15.76:0/1304398132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd40102290 0x7efd4019f0b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efd38037040 tx=0x7efd38032f40 comp rx=0 tx=0).stop 2024-03-18T14:51:00.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.918+0000 7efd480b2640 1 -- 172.21.15.76:0/1304398132 shutdown_connections 2024-03-18T14:51:00.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.918+0000 7efd480b2640 1 --2- 172.21.15.76:0/1304398132 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efd2c0a2800 0x7efd2c0a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:00.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.918+0000 7efd480b2640 1 --2- 172.21.15.76:0/1304398132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd40102290 0x7efd4019f0b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:00.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.918+0000 7efd480b2640 1 --2- 172.21.15.76:0/1304398132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd401018d0 0x7efd40198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:00.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.918+0000 7efd480b2640 1 --2- 172.21.15.76:0/1304398132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd401006e0 0x7efd40197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:00.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.918+0000 7efd480b2640 1 -- 172.21.15.76:0/1304398132 >> 172.21.15.76:0/1304398132 conn(0x7efd400fbc70 msgr2=0x7efd40068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:00.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.918+0000 7efd480b2640 1 -- 172.21.15.76:0/1304398132 shutdown_connections 2024-03-18T14:51:00.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:00.918+0000 7efd480b2640 1 -- 172.21.15.76:0/1304398132 wait complete. 2024-03-18T14:51:00.941 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T14:51:01.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.050+0000 7f46e813b640 1 -- 172.21.15.76:0/908179186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e01018d0 msgr2=0x7f46e0101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:01.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.050+0000 7f46e813b640 1 --2- 172.21.15.76:0/908179186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e01018d0 0x7f46e0101d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f46dc009900 tx=0x7f46dc02bc30 comp rx=0 tx=0).stop 2024-03-18T14:51:01.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.050+0000 7f46e813b640 1 -- 172.21.15.76:0/908179186 shutdown_connections 2024-03-18T14:51:01.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.050+0000 7f46e813b640 1 --2- 172.21.15.76:0/908179186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46e0102290 0x7f46e0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:01.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.050+0000 7f46e813b640 1 --2- 172.21.15.76:0/908179186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e01018d0 0x7f46e0101d50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:01.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.050+0000 7f46e813b640 1 --2- 172.21.15.76:0/908179186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46e01006e0 0x7f46e0100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:01.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.050+0000 7f46e813b640 1 -- 172.21.15.76:0/908179186 >> 172.21.15.76:0/908179186 conn(0x7f46e00fbc70 msgr2=0x7f46e00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:01.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.050+0000 7f46e813b640 1 -- 172.21.15.76:0/908179186 shutdown_connections 2024-03-18T14:51:01.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.050+0000 7f46e813b640 1 -- 172.21.15.76:0/908179186 wait complete. 2024-03-18T14:51:01.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.051+0000 7f46e813b640 1 Processor -- start 2024-03-18T14:51:01.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.051+0000 7f46e813b640 1 -- start start 2024-03-18T14:51:01.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.051+0000 7f46e813b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46e01006e0 0x7f46e0196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:01.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.051+0000 7f46e813b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46e01018d0 0x7f46e0196fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:01.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.051+0000 7f46e813b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e0102290 0x7f46e019e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:01.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.051+0000 7f46e813b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f46e010c340 con 0x7f46e0102290 2024-03-18T14:51:01.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.051+0000 7f46e813b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f46e010c1c0 con 0x7f46e01018d0 2024-03-18T14:51:01.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.051+0000 7f46e813b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f46e010c4c0 con 0x7f46e01006e0 2024-03-18T14:51:01.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.051+0000 7f46e66d8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e0102290 0x7f46e019e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:01.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.051+0000 7f46e66d8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e0102290 0x7f46e019e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58030/0 (socket says 172.21.15.76:58030) 2024-03-18T14:51:01.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.051+0000 7f46e66d8640 1 -- 172.21.15.76:0/2489182130 learned_addr learned my addr 172.21.15.76:0/2489182130 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:01.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.051+0000 7f46e56d6640 1 --2- 172.21.15.76:0/2489182130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46e01018d0 0x7f46e0196fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:01.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.051+0000 7f46e5ed7640 1 --2- 172.21.15.76:0/2489182130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46e01006e0 0x7f46e0196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:01.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.052+0000 7f46e66d8640 1 -- 172.21.15.76:0/2489182130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46e01006e0 msgr2=0x7f46e0196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:01.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.052+0000 7f46e66d8640 1 --2- 172.21.15.76:0/2489182130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46e01006e0 0x7f46e0196a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:01.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.052+0000 7f46e66d8640 1 -- 172.21.15.76:0/2489182130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46e01018d0 msgr2=0x7f46e0196fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:01.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.052+0000 7f46e66d8640 1 --2- 172.21.15.76:0/2489182130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46e01018d0 0x7f46e0196fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:01.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.052+0000 7f46e66d8640 1 -- 172.21.15.76:0/2489182130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e019e680 con 0x7f46e0102290 2024-03-18T14:51:01.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.052+0000 7f46e5ed7640 1 --2- 172.21.15.76:0/2489182130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46e01006e0 0x7f46e0196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:51:01.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.052+0000 7f46e66d8640 1 --2- 172.21.15.76:0/2489182130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e0102290 0x7f46e019e010 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f46d8009fd0 tx=0x7f46d800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:01.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.052+0000 7f46d6ffd640 1 -- 172.21.15.76:0/2489182130 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8017070 con 0x7f46e0102290 2024-03-18T14:51:01.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.052+0000 7f46d6ffd640 1 -- 172.21.15.76:0/2489182130 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46d8003e50 con 0x7f46e0102290 2024-03-18T14:51:01.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.052+0000 7f46d6ffd640 1 -- 172.21.15.76:0/2489182130 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d80106c0 con 0x7f46e0102290 2024-03-18T14:51:01.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.052+0000 7f46e813b640 1 -- 172.21.15.76:0/2489182130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46e019e970 con 0x7f46e0102290 2024-03-18T14:51:01.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.052+0000 7f46e813b640 1 -- 172.21.15.76:0/2489182130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46e019ee90 con 0x7f46e0102290 2024-03-18T14:51:01.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.053+0000 7f46e813b640 1 -- 172.21.15.76:0/2489182130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46e0107b20 con 0x7f46e0102290 2024-03-18T14:51:01.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.054+0000 7f46d6ffd640 1 -- 172.21.15.76:0/2489182130 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f46d8003970 con 0x7f46e0102290 2024-03-18T14:51:01.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.055+0000 7f46d6ffd640 1 --2- 172.21.15.76:0/2489182130 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f46cc0a2880 0x7f46cc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:01.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.055+0000 7f46e5ed7640 1 --2- 172.21.15.76:0/2489182130 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f46cc0a2880 0x7f46cc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:01.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.055+0000 7f46d6ffd640 1 -- 172.21.15.76:0/2489182130 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7f46d80ce590 con 0x7f46e0102290 2024-03-18T14:51:01.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.056+0000 7f46e5ed7640 1 --2- 172.21.15.76:0/2489182130 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f46cc0a2880 0x7f46cc0a4d40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f46e0101730 tx=0x7f46d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:01.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.059+0000 7f46d6ffd640 1 -- 172.21.15.76:0/2489182130 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f46d809b840 con 0x7f46e0102290 2024-03-18T14:51:01.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:00 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1455669544' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T14:51:01.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:00 smithi174 ceph-mon[36787]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T14:51:01.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:00 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:01.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:00 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:01.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:00 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1304398132' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T14:51:01.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:00 smithi174 ceph-mon[36787]: pgmap v280: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:51:01.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:00 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:51:01.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:00 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:51:01.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:00 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:01.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:00 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:51:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:00 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1455669544' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T14:51:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:00 smithi077 ceph-mon[35865]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T14:51:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:00 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:00 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:00 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1304398132' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T14:51:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:00 smithi077 ceph-mon[35865]: pgmap v280: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:51:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:00 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:51:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:00 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:51:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:00 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:01.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:00 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:51:01.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.290+0000 7f46e813b640 1 -- 172.21.15.76:0/2489182130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f46e002d1b0 con 0x7f46e0102290 2024-03-18T14:51:01.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:00 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1455669544' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T14:51:01.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:00 smithi076 ceph-mon[32205]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T14:51:01.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:00 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:01.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:00 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:01.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:00 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1304398132' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T14:51:01.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:00 smithi076 ceph-mon[32205]: pgmap v280: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:51:01.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:00 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:51:01.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:00 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:51:01.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:00 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:01.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:00 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:51:01.929 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:01 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1304398132' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T14:51:01.930 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:01 smithi174 ceph-mon[36787]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T14:51:01.930 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:01 smithi174 ceph-mon[36787]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T14:51:01.930 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:01 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1304398132' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T14:51:01.931 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:01 smithi077 ceph-mon[35865]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T14:51:01.931 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:01 smithi077 ceph-mon[35865]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T14:51:01.931 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:01 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2489182130' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T14:51:01.931 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:01 smithi077 ceph-mon[35865]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T14:51:01.931 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:01 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:01.931 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:01 smithi077 ceph-mon[35865]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T14:51:01.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.993+0000 7f46d6ffd640 1 -- 172.21.15.76:0/2489182130 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f46d809b140 con 0x7f46e0102290 2024-03-18T14:51:01.994 INFO:teuthology.orchestra.run.smithi076.stderr:new fs with metadata pool 2 and data pool 3 2024-03-18T14:51:01.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.996+0000 7f46e813b640 1 -- 172.21.15.76:0/2489182130 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f46cc0a2880 msgr2=0x7f46cc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:01.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.996+0000 7f46e813b640 1 --2- 172.21.15.76:0/2489182130 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f46cc0a2880 0x7f46cc0a4d40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f46e0101730 tx=0x7f46d0005f90 comp rx=0 tx=0).stop 2024-03-18T14:51:01.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.996+0000 7f46e813b640 1 -- 172.21.15.76:0/2489182130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e0102290 msgr2=0x7f46e019e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:01.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.996+0000 7f46e813b640 1 --2- 172.21.15.76:0/2489182130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e0102290 0x7f46e019e010 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f46d8009fd0 tx=0x7f46d800b260 comp rx=0 tx=0).stop 2024-03-18T14:51:01.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.996+0000 7f46e813b640 1 -- 172.21.15.76:0/2489182130 shutdown_connections 2024-03-18T14:51:01.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.996+0000 7f46e813b640 1 --2- 172.21.15.76:0/2489182130 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f46cc0a2880 0x7f46cc0a4d40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:01.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.996+0000 7f46e813b640 1 --2- 172.21.15.76:0/2489182130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e0102290 0x7f46e019e010 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:01.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.996+0000 7f46e813b640 1 --2- 172.21.15.76:0/2489182130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46e01018d0 0x7f46e0196fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:01.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.996+0000 7f46e813b640 1 --2- 172.21.15.76:0/2489182130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46e01006e0 0x7f46e0196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:01.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.996+0000 7f46e813b640 1 -- 172.21.15.76:0/2489182130 >> 172.21.15.76:0/2489182130 conn(0x7f46e00fbc70 msgr2=0x7f46e0104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:01.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.997+0000 7f46e813b640 1 -- 172.21.15.76:0/2489182130 shutdown_connections 2024-03-18T14:51:01.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:01.997+0000 7f46e813b640 1 -- 172.21.15.76:0/2489182130 wait complete. 2024-03-18T14:51:02.011 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:51:02.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.125+0000 7f7ee1937640 1 -- 172.21.15.76:0/1209040126 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7edc1018f0 msgr2=0x7f7edc101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:02.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.125+0000 7f7ee1937640 1 --2- 172.21.15.76:0/1209040126 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7edc1018f0 0x7f7edc101d70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0009900 tx=0x7f7ed002bc30 comp rx=0 tx=0).stop 2024-03-18T14:51:02.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.126+0000 7f7ee1937640 1 -- 172.21.15.76:0/1209040126 shutdown_connections 2024-03-18T14:51:02.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.126+0000 7f7ee1937640 1 --2- 172.21.15.76:0/1209040126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7edc1022b0 0x7f7edc108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.126+0000 7f7ee1937640 1 --2- 172.21.15.76:0/1209040126 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7edc1018f0 0x7f7edc101d70 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.126+0000 7f7ee1937640 1 --2- 172.21.15.76:0/1209040126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc100700 0x7f7edc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.126+0000 7f7ee1937640 1 -- 172.21.15.76:0/1209040126 >> 172.21.15.76:0/1209040126 conn(0x7f7edc0fbc70 msgr2=0x7f7edc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:02.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.126+0000 7f7ee1937640 1 -- 172.21.15.76:0/1209040126 shutdown_connections 2024-03-18T14:51:02.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.126+0000 7f7ee1937640 1 -- 172.21.15.76:0/1209040126 wait complete. 2024-03-18T14:51:02.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.126+0000 7f7ee1937640 1 Processor -- start 2024-03-18T14:51:02.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.126+0000 7f7ee1937640 1 -- start start 2024-03-18T14:51:02.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.126+0000 7f7ee1937640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7edc100700 0x7f7edc196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:02.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.126+0000 7f7ee1937640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc1018f0 0x7f7edc196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:02.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.126+0000 7f7ee1937640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7edc1022b0 0x7f7edc19dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:02.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7ee1937640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7edc10c380 con 0x7f7edc100700 2024-03-18T14:51:02.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7ee1937640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7edc10c200 con 0x7f7edc1022b0 2024-03-18T14:51:02.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7ee1937640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7edc10c500 con 0x7f7edc1018f0 2024-03-18T14:51:02.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7edaffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7edc100700 0x7f7edc196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:02.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7edaffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7edc100700 0x7f7edc196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58044/0 (socket says 172.21.15.76:58044) 2024-03-18T14:51:02.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7edaffd640 1 -- 172.21.15.76:0/3047970011 learned_addr learned my addr 172.21.15.76:0/3047970011 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:02.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7edb7fe640 1 --2- 172.21.15.76:0/3047970011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7edc1022b0 0x7f7edc19dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:02.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7eda7fc640 1 --2- 172.21.15.76:0/3047970011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc1018f0 0x7f7edc196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:02.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7edaffd640 1 -- 172.21.15.76:0/3047970011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc1018f0 msgr2=0x7f7edc196f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:02.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7edaffd640 1 --2- 172.21.15.76:0/3047970011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc1018f0 0x7f7edc196f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7edaffd640 1 -- 172.21.15.76:0/3047970011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7edc1022b0 msgr2=0x7f7edc19dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:02.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7edaffd640 1 --2- 172.21.15.76:0/3047970011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7edc1022b0 0x7f7edc19dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7edaffd640 1 -- 172.21.15.76:0/3047970011 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7edc19e640 con 0x7f7edc100700 2024-03-18T14:51:02.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7eda7fc640 1 --2- 172.21.15.76:0/3047970011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc1018f0 0x7f7edc196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:51:02.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7edaffd640 1 --2- 172.21.15.76:0/3047970011 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7edc100700 0x7f7edc196a20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400e9c0 tx=0x7f7ed400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:02.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7ee0935640 1 -- 172.21.15.76:0/3047970011 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed400cd80 con 0x7f7edc100700 2024-03-18T14:51:02.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7ee1937640 1 -- 172.21.15.76:0/3047970011 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7edc19e930 con 0x7f7edc100700 2024-03-18T14:51:02.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7ee0935640 1 -- 172.21.15.76:0/3047970011 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ed40035f0 con 0x7f7edc100700 2024-03-18T14:51:02.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7ee0935640 1 -- 172.21.15.76:0/3047970011 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed4010690 con 0x7f7edc100700 2024-03-18T14:51:02.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.127+0000 7f7ee1937640 1 -- 172.21.15.76:0/3047970011 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7edc19ee80 con 0x7f7edc100700 2024-03-18T14:51:02.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.129+0000 7f7ee1937640 1 -- 172.21.15.76:0/3047970011 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7edc107aa0 con 0x7f7edc100700 2024-03-18T14:51:02.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.129+0000 7f7ee0935640 1 -- 172.21.15.76:0/3047970011 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7ed40037a0 con 0x7f7edc100700 2024-03-18T14:51:02.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.130+0000 7f7ee0935640 1 --2- 172.21.15.76:0/3047970011 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7ec40a2800 0x7f7ec40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:02.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.130+0000 7f7eda7fc640 1 --2- 172.21.15.76:0/3047970011 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7ec40a2800 0x7f7ec40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:02.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.130+0000 7f7ee0935640 1 -- 172.21.15.76:0/3047970011 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f7ed40ced20 con 0x7f7edc100700 2024-03-18T14:51:02.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.130+0000 7f7eda7fc640 1 --2- 172.21.15.76:0/3047970011 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7ec40a2800 0x7f7ec40a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0036040 tx=0x7f7ed0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:02.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.136+0000 7f7ee0935640 1 -- 172.21.15.76:0/3047970011 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7ed409bf10 con 0x7f7edc100700 2024-03-18T14:51:02.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:01 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2489182130' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T14:51:02.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:01 smithi174 ceph-mon[36787]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T14:51:02.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:01 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:02.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:01 smithi174 ceph-mon[36787]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T14:51:02.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.376+0000 7f7ee1937640 1 -- 172.21.15.76:0/3047970011 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7edc02d350 con 0x7f7edc100700 2024-03-18T14:51:02.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.377+0000 7f7ee0935640 1 -- 172.21.15.76:0/3047970011 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22192 (secure 0 0 0) 0x7f7ed409b810 con 0x7f7edc100700 2024-03-18T14:51:02.379 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:51:02.379 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":86,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:51:02.338985+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:51:02.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.380+0000 7f7ee1937640 1 -- 172.21.15.76:0/3047970011 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7ec40a2800 msgr2=0x7f7ec40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:02.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.380+0000 7f7ee1937640 1 --2- 172.21.15.76:0/3047970011 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7ec40a2800 0x7f7ec40a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0036040 tx=0x7f7ed0000bc0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.380+0000 7f7ee1937640 1 -- 172.21.15.76:0/3047970011 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7edc100700 msgr2=0x7f7edc196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:02.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.380+0000 7f7ee1937640 1 --2- 172.21.15.76:0/3047970011 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7edc100700 0x7f7edc196a20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400e9c0 tx=0x7f7ed400ee80 comp rx=0 tx=0).stop 2024-03-18T14:51:02.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.380+0000 7f7ee1937640 1 -- 172.21.15.76:0/3047970011 shutdown_connections 2024-03-18T14:51:02.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.380+0000 7f7ee1937640 1 --2- 172.21.15.76:0/3047970011 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f7ec40a2800 0x7f7ec40a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.380+0000 7f7ee1937640 1 --2- 172.21.15.76:0/3047970011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc1018f0 0x7f7edc196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.380+0000 7f7ee1937640 1 --2- 172.21.15.76:0/3047970011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7edc1022b0 0x7f7edc19dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.380+0000 7f7ee1937640 1 --2- 172.21.15.76:0/3047970011 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7edc100700 0x7f7edc196a20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.380+0000 7f7ee1937640 1 -- 172.21.15.76:0/3047970011 >> 172.21.15.76:0/3047970011 conn(0x7f7edc0fbc70 msgr2=0x7f7edc104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:02.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.380+0000 7f7ee1937640 1 -- 172.21.15.76:0/3047970011 shutdown_connections 2024-03-18T14:51:02.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.380+0000 7f7ee1937640 1 -- 172.21.15.76:0/3047970011 wait complete. 2024-03-18T14:51:02.391 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:01 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1304398132' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T14:51:02.391 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:01 smithi076 ceph-mon[32205]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T14:51:02.391 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:01 smithi076 ceph-mon[32205]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T14:51:02.391 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:01 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2489182130' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T14:51:02.391 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:01 smithi076 ceph-mon[32205]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T14:51:02.392 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:01 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:02.392 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:01 smithi076 ceph-mon[32205]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T14:51:02.393 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:51:02.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.507+0000 7ff15fc6a640 1 -- 172.21.15.76:0/2327796716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff158100700 msgr2=0x7ff158100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:02.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.507+0000 7ff15fc6a640 1 --2- 172.21.15.76:0/2327796716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff158100700 0x7ff158100b20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff154009900 tx=0x7ff15402bc30 comp rx=0 tx=0).stop 2024-03-18T14:51:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.507+0000 7ff15fc6a640 1 -- 172.21.15.76:0/2327796716 shutdown_connections 2024-03-18T14:51:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.507+0000 7ff15fc6a640 1 --2- 172.21.15.76:0/2327796716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1581018f0 0x7ff158101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.507+0000 7ff15fc6a640 1 --2- 172.21.15.76:0/2327796716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1581022b0 0x7ff158108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.507+0000 7ff15fc6a640 1 --2- 172.21.15.76:0/2327796716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff158100700 0x7ff158100b20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.507+0000 7ff15fc6a640 1 -- 172.21.15.76:0/2327796716 >> 172.21.15.76:0/2327796716 conn(0x7ff1580fbc70 msgr2=0x7ff1580fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.507+0000 7ff15fc6a640 1 -- 172.21.15.76:0/2327796716 shutdown_connections 2024-03-18T14:51:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.507+0000 7ff15fc6a640 1 -- 172.21.15.76:0/2327796716 wait complete. 2024-03-18T14:51:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.507+0000 7ff15fc6a640 1 Processor -- start 2024-03-18T14:51:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.507+0000 7ff15fc6a640 1 -- start start 2024-03-18T14:51:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15fc6a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff158100700 0x7ff158197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15fc6a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1581018f0 0x7ff158197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15fc6a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1581022b0 0x7ff15819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15fc6a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff15810c350 con 0x7ff158100700 2024-03-18T14:51:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15fc6a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff15810c1d0 con 0x7ff1581022b0 2024-03-18T14:51:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15fc6a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff15810c4d0 con 0x7ff1581018f0 2024-03-18T14:51:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15da06640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff158100700 0x7ff158197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15da06640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff158100700 0x7ff158197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58060/0 (socket says 172.21.15.76:58060) 2024-03-18T14:51:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15da06640 1 -- 172.21.15.76:0/2537678269 learned_addr learned my addr 172.21.15.76:0/2537678269 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15d205640 1 --2- 172.21.15.76:0/2537678269 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1581018f0 0x7ff158197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15e207640 1 --2- 172.21.15.76:0/2537678269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1581022b0 0x7ff15819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15da06640 1 -- 172.21.15.76:0/2537678269 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1581018f0 msgr2=0x7ff158197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15da06640 1 --2- 172.21.15.76:0/2537678269 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1581018f0 0x7ff158197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15da06640 1 -- 172.21.15.76:0/2537678269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1581022b0 msgr2=0x7ff15819f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15da06640 1 --2- 172.21.15.76:0/2537678269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1581022b0 0x7ff15819f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15da06640 1 -- 172.21.15.76:0/2537678269 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15819f670 con 0x7ff158100700 2024-03-18T14:51:02.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15da06640 1 --2- 172.21.15.76:0/2537678269 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff158100700 0x7ff158197a50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff154000c00 tx=0x7ff154004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:02.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff14effd640 1 -- 172.21.15.76:0/2537678269 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff154041070 con 0x7ff158100700 2024-03-18T14:51:02.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff14effd640 1 -- 172.21.15.76:0/2537678269 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1540054e0 con 0x7ff158100700 2024-03-18T14:51:02.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15fc6a640 1 -- 172.21.15.76:0/2537678269 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff15819f900 con 0x7ff158100700 2024-03-18T14:51:02.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff14effd640 1 -- 172.21.15.76:0/2537678269 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1540346a0 con 0x7ff158100700 2024-03-18T14:51:02.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.508+0000 7ff15fc6a640 1 -- 172.21.15.76:0/2537678269 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff15819fdf0 con 0x7ff158100700 2024-03-18T14:51:02.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.510+0000 7ff15fc6a640 1 -- 172.21.15.76:0/2537678269 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff15819fff0 con 0x7ff158100700 2024-03-18T14:51:02.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.511+0000 7ff14effd640 1 -- 172.21.15.76:0/2537678269 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff154044070 con 0x7ff158100700 2024-03-18T14:51:02.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.512+0000 7ff14effd640 1 --2- 172.21.15.76:0/2537678269 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff1440a2800 0x7ff1440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:02.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.512+0000 7ff15d205640 1 --2- 172.21.15.76:0/2537678269 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff1440a2800 0x7ff1440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:02.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.512+0000 7ff14effd640 1 -- 172.21.15.76:0/2537678269 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7ff1540f0810 con 0x7ff158100700 2024-03-18T14:51:02.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.512+0000 7ff15d205640 1 --2- 172.21.15.76:0/2537678269 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff1440a2800 0x7ff1440a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff158198f60 tx=0x7ff14800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:02.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.515+0000 7ff14effd640 1 -- 172.21.15.76:0/2537678269 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff1540bd9a0 con 0x7ff158100700 2024-03-18T14:51:02.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.695+0000 7ff15fc6a640 1 -- 172.21.15.76:0/2537678269 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff1581a01c0 con 0x7ff158100700 2024-03-18T14:51:02.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.696+0000 7ff14effd640 1 -- 172.21.15.76:0/2537678269 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22192 (secure 0 0 0) 0x7ff154003e20 con 0x7ff158100700 2024-03-18T14:51:02.697 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:51:02.697 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":86,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:51:02.338985+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:51:02.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.698+0000 7ff15fc6a640 1 -- 172.21.15.76:0/2537678269 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff1440a2800 msgr2=0x7ff1440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:02.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.698+0000 7ff15fc6a640 1 --2- 172.21.15.76:0/2537678269 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff1440a2800 0x7ff1440a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff158198f60 tx=0x7ff14800a2d0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.698+0000 7ff15fc6a640 1 -- 172.21.15.76:0/2537678269 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff158100700 msgr2=0x7ff158197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:02.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.698+0000 7ff15fc6a640 1 --2- 172.21.15.76:0/2537678269 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff158100700 0x7ff158197a50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff154000c00 tx=0x7ff154004040 comp rx=0 tx=0).stop 2024-03-18T14:51:02.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.698+0000 7ff15fc6a640 1 -- 172.21.15.76:0/2537678269 shutdown_connections 2024-03-18T14:51:02.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.698+0000 7ff15fc6a640 1 --2- 172.21.15.76:0/2537678269 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff1440a2800 0x7ff1440a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.698+0000 7ff15fc6a640 1 --2- 172.21.15.76:0/2537678269 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1581018f0 0x7ff158197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.698+0000 7ff15fc6a640 1 --2- 172.21.15.76:0/2537678269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1581022b0 0x7ff15819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.698+0000 7ff15fc6a640 1 --2- 172.21.15.76:0/2537678269 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff158100700 0x7ff158197a50 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.698+0000 7ff15fc6a640 1 -- 172.21.15.76:0/2537678269 >> 172.21.15.76:0/2537678269 conn(0x7ff1580fbc70 msgr2=0x7ff1580fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:02.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.699+0000 7ff15fc6a640 1 -- 172.21.15.76:0/2537678269 shutdown_connections 2024-03-18T14:51:02.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.699+0000 7ff15fc6a640 1 -- 172.21.15.76:0/2537678269 wait complete. 2024-03-18T14:51:02.710 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T14:51:02.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.815+0000 7f0d73189640 1 -- 172.21.15.76:0/217393573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c102290 msgr2=0x7f0d6c108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:02.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.815+0000 7f0d73189640 1 --2- 172.21.15.76:0/217393573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c102290 0x7f0d6c108b20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0d64009ca0 tx=0x7f0d6402c0f0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.815+0000 7f0d73189640 1 -- 172.21.15.76:0/217393573 shutdown_connections 2024-03-18T14:51:02.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.815+0000 7f0d73189640 1 --2- 172.21.15.76:0/217393573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d6c1018d0 0x7f0d6c101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.815+0000 7f0d73189640 1 --2- 172.21.15.76:0/217393573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c102290 0x7f0d6c108b20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.815+0000 7f0d73189640 1 --2- 172.21.15.76:0/217393573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d6c1006e0 0x7f0d6c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.815+0000 7f0d73189640 1 -- 172.21.15.76:0/217393573 >> 172.21.15.76:0/217393573 conn(0x7f0d6c0fbc70 msgr2=0x7f0d6c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:02.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.815+0000 7f0d73189640 1 -- 172.21.15.76:0/217393573 shutdown_connections 2024-03-18T14:51:02.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.815+0000 7f0d73189640 1 -- 172.21.15.76:0/217393573 wait complete. 2024-03-18T14:51:02.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.815+0000 7f0d73189640 1 Processor -- start 2024-03-18T14:51:02.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.815+0000 7f0d73189640 1 -- start start 2024-03-18T14:51:02.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d73189640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d6c1006e0 0x7f0d6c069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:02.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d73189640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d6c1018d0 0x7f0d6c19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:02.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d73189640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c102290 0x7f0d6c19ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:02.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d73189640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6c10c360 con 0x7f0d6c102290 2024-03-18T14:51:02.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d73189640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6c10c1e0 con 0x7f0d6c1018d0 2024-03-18T14:51:02.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d73189640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6c10c4e0 con 0x7f0d6c1006e0 2024-03-18T14:51:02.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d71726640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c102290 0x7f0d6c19ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:02.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d71726640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c102290 0x7f0d6c19ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58080/0 (socket says 172.21.15.76:58080) 2024-03-18T14:51:02.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d71726640 1 -- 172.21.15.76:0/2702788205 learned_addr learned my addr 172.21.15.76:0/2702788205 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:02.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d6bfff640 1 --2- 172.21.15.76:0/2702788205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d6c1018d0 0x7f0d6c19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:02.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d70f25640 1 --2- 172.21.15.76:0/2702788205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d6c1006e0 0x7f0d6c069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:02.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d71726640 1 -- 172.21.15.76:0/2702788205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d6c1006e0 msgr2=0x7f0d6c069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:02.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d71726640 1 --2- 172.21.15.76:0/2702788205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d6c1006e0 0x7f0d6c069510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d71726640 1 -- 172.21.15.76:0/2702788205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d6c1018d0 msgr2=0x7f0d6c19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:02.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d71726640 1 --2- 172.21.15.76:0/2702788205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d6c1018d0 0x7f0d6c19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:02.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d71726640 1 -- 172.21.15.76:0/2702788205 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d6c19f6f0 con 0x7f0d6c102290 2024-03-18T14:51:02.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d71726640 1 --2- 172.21.15.76:0/2702788205 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c102290 0x7f0d6c19ee30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0d6402c700 tx=0x7f0d64032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:02.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d69ffb640 1 -- 172.21.15.76:0/2702788205 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d64039040 con 0x7f0d6c102290 2024-03-18T14:51:02.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d73189640 1 -- 172.21.15.76:0/2702788205 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d6c19f980 con 0x7f0d6c102290 2024-03-18T14:51:02.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d69ffb640 1 -- 172.21.15.76:0/2702788205 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d64044070 con 0x7f0d6c102290 2024-03-18T14:51:02.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.816+0000 7f0d69ffb640 1 -- 172.21.15.76:0/2702788205 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d640042a0 con 0x7f0d6c102290 2024-03-18T14:51:02.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.817+0000 7f0d73189640 1 -- 172.21.15.76:0/2702788205 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d6c19fe70 con 0x7f0d6c102290 2024-03-18T14:51:02.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.817+0000 7f0d73189640 1 -- 172.21.15.76:0/2702788205 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d6c107a00 con 0x7f0d6c102290 2024-03-18T14:51:02.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.819+0000 7f0d69ffb640 1 -- 172.21.15.76:0/2702788205 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0d64003970 con 0x7f0d6c102290 2024-03-18T14:51:02.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.819+0000 7f0d69ffb640 1 --2- 172.21.15.76:0/2702788205 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0d5c0a2800 0x7f0d5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:02.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.819+0000 7f0d70f25640 1 --2- 172.21.15.76:0/2702788205 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0d5c0a2800 0x7f0d5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:02.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.819+0000 7f0d69ffb640 1 -- 172.21.15.76:0/2702788205 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f0d640f1520 con 0x7f0d6c102290 2024-03-18T14:51:02.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.820+0000 7f0d70f25640 1 --2- 172.21.15.76:0/2702788205 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0d5c0a2800 0x7f0d5c0a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0d60005db0 tx=0x7f0d60005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:02.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:02.823+0000 7f0d69ffb640 1 -- 172.21.15.76:0/2702788205 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0d640be8c0 con 0x7f0d6c102290 2024-03-18T14:51:03.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:03.038+0000 7f0d73189640 1 -- 172.21.15.76:0/2702788205 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f0d6c02d1b0 con 0x7f0d6c102290 2024-03-18T14:51:03.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:03 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2489182130' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T14:51:03.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:03 smithi076 ceph-mon[32205]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T14:51:03.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:03 smithi076 ceph-mon[32205]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T14:51:03.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:03 smithi076 ceph-mon[32205]: pgmap v283: 97 pgs: 21 active+clean, 24 creating+peering, 52 unknown; 578 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-03-18T14:51:03.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:03 smithi076 ceph-mon[32205]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T14:51:03.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:03 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3047970011' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:51:03.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:03 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:03 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:03 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:03 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:03 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:03 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:03 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:51:03.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:03 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:51:03.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:03 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:03 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2537678269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:51:03.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:03 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2489182130' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T14:51:03.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:03 smithi174 ceph-mon[36787]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T14:51:03.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:03 smithi174 ceph-mon[36787]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T14:51:03.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:03 smithi174 ceph-mon[36787]: pgmap v283: 97 pgs: 21 active+clean, 24 creating+peering, 52 unknown; 578 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-03-18T14:51:03.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:03 smithi174 ceph-mon[36787]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T14:51:03.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:03 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3047970011' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:51:03.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:03 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:03 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:03 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:03 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:03 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:03 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:03 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:51:03.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:03 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:51:03.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:03 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:03 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2537678269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:51:03.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:03 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2489182130' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T14:51:03.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:03 smithi077 ceph-mon[35865]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T14:51:03.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:03 smithi077 ceph-mon[35865]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T14:51:03.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:03 smithi077 ceph-mon[35865]: pgmap v283: 97 pgs: 21 active+clean, 24 creating+peering, 52 unknown; 578 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-03-18T14:51:03.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:03 smithi077 ceph-mon[35865]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T14:51:03.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:03 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3047970011' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:51:03.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:03 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:03 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:03 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:03 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:03 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:03 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:03 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:51:03.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:03 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:51:03.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:03 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:51:03.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:03 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2537678269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:51:04.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.009+0000 7f0d69ffb640 1 -- 172.21.15.76:0/2702788205 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f0d640be1c0 con 0x7f0d6c102290 2024-03-18T14:51:04.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.011+0000 7f0d73189640 1 -- 172.21.15.76:0/2702788205 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0d5c0a2800 msgr2=0x7f0d5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:04.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.011+0000 7f0d73189640 1 --2- 172.21.15.76:0/2702788205 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0d5c0a2800 0x7f0d5c0a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0d60005db0 tx=0x7f0d60005d40 comp rx=0 tx=0).stop 2024-03-18T14:51:04.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.011+0000 7f0d73189640 1 -- 172.21.15.76:0/2702788205 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c102290 msgr2=0x7f0d6c19ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:04.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.011+0000 7f0d73189640 1 --2- 172.21.15.76:0/2702788205 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c102290 0x7f0d6c19ee30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0d6402c700 tx=0x7f0d64032f20 comp rx=0 tx=0).stop 2024-03-18T14:51:04.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.012+0000 7f0d73189640 1 -- 172.21.15.76:0/2702788205 shutdown_connections 2024-03-18T14:51:04.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.012+0000 7f0d73189640 1 --2- 172.21.15.76:0/2702788205 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0d5c0a2800 0x7f0d5c0a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:04.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.012+0000 7f0d73189640 1 --2- 172.21.15.76:0/2702788205 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c102290 0x7f0d6c19ee30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:04.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.012+0000 7f0d73189640 1 --2- 172.21.15.76:0/2702788205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d6c1018d0 0x7f0d6c19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:04.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.012+0000 7f0d73189640 1 --2- 172.21.15.76:0/2702788205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d6c1006e0 0x7f0d6c069510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:04.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.012+0000 7f0d73189640 1 -- 172.21.15.76:0/2702788205 >> 172.21.15.76:0/2702788205 conn(0x7f0d6c0fbc70 msgr2=0x7f0d6c104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:04.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.012+0000 7f0d73189640 1 -- 172.21.15.76:0/2702788205 shutdown_connections 2024-03-18T14:51:04.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.012+0000 7f0d73189640 1 -- 172.21.15.76:0/2702788205 wait complete. 2024-03-18T14:51:04.025 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-18T14:51:04.025 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-18T14:51:04.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.133+0000 7ffb543e6640 1 -- 172.21.15.76:0/1947451141 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb4c100700 msgr2=0x7ffb4c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:04.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.133+0000 7ffb543e6640 1 --2- 172.21.15.76:0/1947451141 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb4c100700 0x7ffb4c100b20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ffb44009900 tx=0x7ffb4402bc30 comp rx=0 tx=0).stop 2024-03-18T14:51:04.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.134+0000 7ffb543e6640 1 -- 172.21.15.76:0/1947451141 shutdown_connections 2024-03-18T14:51:04.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.134+0000 7ffb543e6640 1 --2- 172.21.15.76:0/1947451141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb4c1018f0 0x7ffb4c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:04.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.134+0000 7ffb543e6640 1 --2- 172.21.15.76:0/1947451141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb4c1022b0 0x7ffb4c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:04.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.134+0000 7ffb543e6640 1 --2- 172.21.15.76:0/1947451141 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb4c100700 0x7ffb4c100b20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:04.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.134+0000 7ffb543e6640 1 -- 172.21.15.76:0/1947451141 >> 172.21.15.76:0/1947451141 conn(0x7ffb4c0fbc90 msgr2=0x7ffb4c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:04.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.134+0000 7ffb543e6640 1 -- 172.21.15.76:0/1947451141 shutdown_connections 2024-03-18T14:51:04.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.134+0000 7ffb543e6640 1 -- 172.21.15.76:0/1947451141 wait complete. 2024-03-18T14:51:04.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.134+0000 7ffb543e6640 1 Processor -- start 2024-03-18T14:51:04.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.134+0000 7ffb543e6640 1 -- start start 2024-03-18T14:51:04.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.134+0000 7ffb543e6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb4c100700 0x7ffb4c06a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:04.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.134+0000 7ffb543e6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb4c1018f0 0x7ffb4c19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:04.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.134+0000 7ffb543e6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb4c1022b0 0x7ffb4c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:04.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.134+0000 7ffb543e6640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffb4c10c3d0 con 0x7ffb4c1022b0 2024-03-18T14:51:04.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.134+0000 7ffb543e6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffb4c10c250 con 0x7ffb4c100700 2024-03-18T14:51:04.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.134+0000 7ffb543e6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffb4c10c550 con 0x7ffb4c1018f0 2024-03-18T14:51:04.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb52983640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb4c1022b0 0x7ffb4c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:04.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb52983640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb4c1022b0 0x7ffb4c19ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58094/0 (socket says 172.21.15.76:58094) 2024-03-18T14:51:04.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb52983640 1 -- 172.21.15.76:0/924961750 learned_addr learned my addr 172.21.15.76:0/924961750 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:04.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb52182640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb4c100700 0x7ffb4c06a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:04.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb51981640 1 --2- 172.21.15.76:0/924961750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb4c1018f0 0x7ffb4c19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:04.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb52983640 1 -- 172.21.15.76:0/924961750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb4c1018f0 msgr2=0x7ffb4c19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:04.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb52983640 1 --2- 172.21.15.76:0/924961750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb4c1018f0 0x7ffb4c19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:04.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb52983640 1 -- 172.21.15.76:0/924961750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb4c100700 msgr2=0x7ffb4c06a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:04.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb52983640 1 --2- 172.21.15.76:0/924961750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb4c100700 0x7ffb4c06a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:04.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb52983640 1 -- 172.21.15.76:0/924961750 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb4c19f7f0 con 0x7ffb4c1022b0 2024-03-18T14:51:04.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb51981640 1 --2- 172.21.15.76:0/924961750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb4c1018f0 0x7ffb4c19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:51:04.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb52182640 1 --2- 172.21.15.76:0/924961750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb4c100700 0x7ffb4c06a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:04.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb52983640 1 --2- 172.21.15.76:0/924961750 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb4c1022b0 0x7ffb4c19ef30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c00eb10 tx=0x7ffb3c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:04.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb4b7fe640 1 -- 172.21.15.76:0/924961750 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c004b20 con 0x7ffb4c1022b0 2024-03-18T14:51:04.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb4b7fe640 1 -- 172.21.15.76:0/924961750 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb3c004cc0 con 0x7ffb4c1022b0 2024-03-18T14:51:04.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb543e6640 1 -- 172.21.15.76:0/924961750 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb4c19fae0 con 0x7ffb4c1022b0 2024-03-18T14:51:04.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb4b7fe640 1 -- 172.21.15.76:0/924961750 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c010760 con 0x7ffb4c1022b0 2024-03-18T14:51:04.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.135+0000 7ffb543e6640 1 -- 172.21.15.76:0/924961750 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb4c1a0030 con 0x7ffb4c1022b0 2024-03-18T14:51:04.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.136+0000 7ffb543e6640 1 -- 172.21.15.76:0/924961750 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb4c107a90 con 0x7ffb4c1022b0 2024-03-18T14:51:04.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.137+0000 7ffb4b7fe640 1 -- 172.21.15.76:0/924961750 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ffb3c010900 con 0x7ffb4c1022b0 2024-03-18T14:51:04.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.138+0000 7ffb4b7fe640 1 --2- 172.21.15.76:0/924961750 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffb340a2880 0x7ffb340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:04.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.138+0000 7ffb52182640 1 --2- 172.21.15.76:0/924961750 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffb340a2880 0x7ffb340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:04.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.138+0000 7ffb4b7fe640 1 -- 172.21.15.76:0/924961750 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7ffb3c0cf0e0 con 0x7ffb4c1022b0 2024-03-18T14:51:04.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.138+0000 7ffb52182640 1 --2- 172.21.15.76:0/924961750 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffb340a2880 0x7ffb340a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ffb44000c00 tx=0x7ffb44005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:04.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.141+0000 7ffb4b7fe640 1 -- 172.21.15.76:0/924961750 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffb3c053740 con 0x7ffb4c1022b0 2024-03-18T14:51:04.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:04.357+0000 7ffb543e6640 1 -- 172.21.15.76:0/924961750 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7ffb4c02d400 con 0x7ffb4c1022b0 2024-03-18T14:51:04.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:04 smithi076 ceph-mon[32205]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T14:51:04.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:04 smithi076 ceph-mon[32205]: Cluster is now healthy 2024-03-18T14:51:04.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:04 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] up:active 2024-03-18T14:51:04.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:04 smithi076 ceph-mon[32205]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T14:51:04.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:04 smithi076 ceph-mon[32205]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T14:51:04.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:04 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2702788205' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T14:51:04.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:04 smithi174 ceph-mon[36787]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T14:51:04.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:04 smithi174 ceph-mon[36787]: Cluster is now healthy 2024-03-18T14:51:04.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:04 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] up:active 2024-03-18T14:51:04.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:04 smithi174 ceph-mon[36787]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T14:51:04.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:04 smithi174 ceph-mon[36787]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T14:51:04.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:04 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2702788205' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T14:51:04.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:04 smithi077 ceph-mon[35865]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T14:51:04.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:04 smithi077 ceph-mon[35865]: Cluster is now healthy 2024-03-18T14:51:04.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:04 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] up:active 2024-03-18T14:51:04.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:04 smithi077 ceph-mon[35865]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T14:51:04.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:04 smithi077 ceph-mon[35865]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T14:51:04.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:04 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2702788205' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T14:51:05.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.010+0000 7ffb4b7fe640 1 -- 172.21.15.76:0/924961750 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7ffb3c09c2d0 con 0x7ffb4c1022b0 2024-03-18T14:51:05.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.013+0000 7ffb543e6640 1 -- 172.21.15.76:0/924961750 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffb340a2880 msgr2=0x7ffb340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:05.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.013+0000 7ffb543e6640 1 --2- 172.21.15.76:0/924961750 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffb340a2880 0x7ffb340a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ffb44000c00 tx=0x7ffb44005fd0 comp rx=0 tx=0).stop 2024-03-18T14:51:05.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.013+0000 7ffb543e6640 1 -- 172.21.15.76:0/924961750 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb4c1022b0 msgr2=0x7ffb4c19ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:05.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.013+0000 7ffb543e6640 1 --2- 172.21.15.76:0/924961750 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb4c1022b0 0x7ffb4c19ef30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c00eb10 tx=0x7ffb3c00c2d0 comp rx=0 tx=0).stop 2024-03-18T14:51:05.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.013+0000 7ffb543e6640 1 -- 172.21.15.76:0/924961750 shutdown_connections 2024-03-18T14:51:05.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.013+0000 7ffb543e6640 1 --2- 172.21.15.76:0/924961750 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffb340a2880 0x7ffb340a4d40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:05.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.013+0000 7ffb543e6640 1 --2- 172.21.15.76:0/924961750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb4c1018f0 0x7ffb4c19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:05.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.013+0000 7ffb543e6640 1 --2- 172.21.15.76:0/924961750 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb4c1022b0 0x7ffb4c19ef30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:05.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.013+0000 7ffb543e6640 1 --2- 172.21.15.76:0/924961750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb4c100700 0x7ffb4c06a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:05.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.013+0000 7ffb543e6640 1 -- 172.21.15.76:0/924961750 >> 172.21.15.76:0/924961750 conn(0x7ffb4c0fbc90 msgr2=0x7ffb4c104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:05.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.013+0000 7ffb543e6640 1 -- 172.21.15.76:0/924961750 shutdown_connections 2024-03-18T14:51:05.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.013+0000 7ffb543e6640 1 -- 172.21.15.76:0/924961750 wait complete. 2024-03-18T14:51:05.029 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-18T14:51:05.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.134+0000 7f1c4f28a640 1 -- 172.21.15.76:0/2244076455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c481029e0 msgr2=0x7f1c48102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:05.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.134+0000 7f1c4f28a640 1 --2- 172.21.15.76:0/2244076455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c481029e0 0x7f1c48102e60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009900 tx=0x7f1c3802bc30 comp rx=0 tx=0).stop 2024-03-18T14:51:05.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.134+0000 7f1c4f28a640 1 -- 172.21.15.76:0/2244076455 shutdown_connections 2024-03-18T14:51:05.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.134+0000 7f1c4f28a640 1 --2- 172.21.15.76:0/2244076455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c481033a0 0x7f1c48109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:05.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.134+0000 7f1c4f28a640 1 --2- 172.21.15.76:0/2244076455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c481029e0 0x7f1c48102e60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:05.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.134+0000 7f1c4f28a640 1 --2- 172.21.15.76:0/2244076455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c481017f0 0x7f1c48101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:05.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.134+0000 7f1c4f28a640 1 -- 172.21.15.76:0/2244076455 >> 172.21.15.76:0/2244076455 conn(0x7f1c480fcd80 msgr2=0x7f1c480ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:05.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.134+0000 7f1c4f28a640 1 -- 172.21.15.76:0/2244076455 shutdown_connections 2024-03-18T14:51:05.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.134+0000 7f1c4f28a640 1 -- 172.21.15.76:0/2244076455 wait complete. 2024-03-18T14:51:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.134+0000 7f1c4f28a640 1 Processor -- start 2024-03-18T14:51:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4f28a640 1 -- start start 2024-03-18T14:51:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4f28a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c481017f0 0x7f1c48197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4f28a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c481029e0 0x7f1c48198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4f28a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c481033a0 0x7f1c4819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4f28a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4810d450 con 0x7f1c481033a0 2024-03-18T14:51:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4f28a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4810d2d0 con 0x7f1c481017f0 2024-03-18T14:51:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4f28a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4810d5d0 con 0x7f1c481029e0 2024-03-18T14:51:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4d827640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c481033a0 0x7f1c4819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4d827640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c481033a0 0x7f1c4819f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58110/0 (socket says 172.21.15.76:58110) 2024-03-18T14:51:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4d827640 1 -- 172.21.15.76:0/1119538490 learned_addr learned my addr 172.21.15.76:0/1119538490 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4c825640 1 --2- 172.21.15.76:0/1119538490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c481029e0 0x7f1c48198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4d026640 1 --2- 172.21.15.76:0/1119538490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c481017f0 0x7f1c48197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4d827640 1 -- 172.21.15.76:0/1119538490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c481029e0 msgr2=0x7f1c48198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4d827640 1 --2- 172.21.15.76:0/1119538490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c481029e0 0x7f1c48198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4d827640 1 -- 172.21.15.76:0/1119538490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c481017f0 msgr2=0x7f1c48197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4d827640 1 --2- 172.21.15.76:0/1119538490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c481017f0 0x7f1c48197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4d827640 1 -- 172.21.15.76:0/1119538490 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c4819f700 con 0x7f1c481033a0 2024-03-18T14:51:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4d827640 1 --2- 172.21.15.76:0/1119538490 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c481033a0 0x7f1c4819f090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000eaa0 tx=0x7f1c4000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c3e7fc640 1 -- 172.21.15.76:0/1119538490 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4000cdb0 con 0x7f1c481033a0 2024-03-18T14:51:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c3e7fc640 1 -- 172.21.15.76:0/1119538490 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c40005020 con 0x7f1c481033a0 2024-03-18T14:51:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c3e7fc640 1 -- 172.21.15.76:0/1119538490 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c400116b0 con 0x7f1c481033a0 2024-03-18T14:51:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.135+0000 7f1c4f28a640 1 -- 172.21.15.76:0/1119538490 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c4819f9f0 con 0x7f1c481033a0 2024-03-18T14:51:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.136+0000 7f1c4f28a640 1 -- 172.21.15.76:0/1119538490 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c4819fec0 con 0x7f1c481033a0 2024-03-18T14:51:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.137+0000 7f1c4f28a640 1 -- 172.21.15.76:0/1119538490 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c48108be0 con 0x7f1c481033a0 2024-03-18T14:51:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.138+0000 7f1c3e7fc640 1 -- 172.21.15.76:0/1119538490 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1c40005560 con 0x7f1c481033a0 2024-03-18T14:51:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.138+0000 7f1c3e7fc640 1 --2- 172.21.15.76:0/1119538490 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1c340a2800 0x7f1c340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.138+0000 7f1c4d026640 1 --2- 172.21.15.76:0/1119538490 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1c340a2800 0x7f1c340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.139+0000 7f1c3e7fc640 1 -- 172.21.15.76:0/1119538490 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f1c400ceac0 con 0x7f1c481033a0 2024-03-18T14:51:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.139+0000 7f1c4d026640 1 --2- 172.21.15.76:0/1119538490 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1c340a2800 0x7f1c340a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1c48102840 tx=0x7f1c44005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:05.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.143+0000 7f1c3e7fc640 1 -- 172.21.15.76:0/1119538490 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1c40016030 con 0x7f1c481033a0 2024-03-18T14:51:05.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:05.357+0000 7f1c4f28a640 1 -- 172.21.15.76:0/1119538490 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f1c4802d290 con 0x7f1c481033a0 2024-03-18T14:51:05.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:05 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2702788205' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T14:51:05.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:05 smithi076 ceph-mon[32205]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T14:51:05.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:05 smithi076 ceph-mon[32205]: pgmap v285: 97 pgs: 40 active+clean, 24 creating+peering, 33 unknown; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 927 B/s wr, 2 op/s 2024-03-18T14:51:05.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:05 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/924961750' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-18T14:51:05.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:05 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2702788205' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T14:51:05.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:05 smithi174 ceph-mon[36787]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T14:51:05.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:05 smithi174 ceph-mon[36787]: pgmap v285: 97 pgs: 40 active+clean, 24 creating+peering, 33 unknown; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 927 B/s wr, 2 op/s 2024-03-18T14:51:05.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:05 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/924961750' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-18T14:51:05.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:05 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2702788205' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T14:51:05.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:05 smithi077 ceph-mon[35865]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T14:51:05.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:05 smithi077 ceph-mon[35865]: pgmap v285: 97 pgs: 40 active+clean, 24 creating+peering, 33 unknown; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 927 B/s wr, 2 op/s 2024-03-18T14:51:05.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:05 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/924961750' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-18T14:51:06.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.017+0000 7f1c3e7fc640 1 -- 172.21.15.76:0/1119538490 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f1c4009b660 con 0x7f1c481033a0 2024-03-18T14:51:06.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.020+0000 7f1c4f28a640 1 -- 172.21.15.76:0/1119538490 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1c340a2800 msgr2=0x7f1c340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:06.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.020+0000 7f1c4f28a640 1 --2- 172.21.15.76:0/1119538490 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1c340a2800 0x7f1c340a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1c48102840 tx=0x7f1c44005f90 comp rx=0 tx=0).stop 2024-03-18T14:51:06.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.020+0000 7f1c4f28a640 1 -- 172.21.15.76:0/1119538490 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c481033a0 msgr2=0x7f1c4819f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:06.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.020+0000 7f1c4f28a640 1 --2- 172.21.15.76:0/1119538490 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c481033a0 0x7f1c4819f090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000eaa0 tx=0x7f1c4000ef60 comp rx=0 tx=0).stop 2024-03-18T14:51:06.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.020+0000 7f1c4f28a640 1 -- 172.21.15.76:0/1119538490 shutdown_connections 2024-03-18T14:51:06.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.020+0000 7f1c4f28a640 1 --2- 172.21.15.76:0/1119538490 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1c340a2800 0x7f1c340a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:06.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.020+0000 7f1c4f28a640 1 --2- 172.21.15.76:0/1119538490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c481029e0 0x7f1c48198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:06.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.020+0000 7f1c4f28a640 1 --2- 172.21.15.76:0/1119538490 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c481033a0 0x7f1c4819f090 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:06.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.020+0000 7f1c4f28a640 1 --2- 172.21.15.76:0/1119538490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c481017f0 0x7f1c48197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.020+0000 7f1c4f28a640 1 -- 172.21.15.76:0/1119538490 >> 172.21.15.76:0/1119538490 conn(0x7f1c480fcd80 msgr2=0x7f1c48105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.020+0000 7f1c4f28a640 1 -- 172.21.15.76:0/1119538490 shutdown_connections 2024-03-18T14:51:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.020+0000 7f1c4f28a640 1 -- 172.21.15.76:0/1119538490 wait complete. 2024-03-18T14:51:06.038 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-18T14:51:06.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.147+0000 7fb4f2f9d640 1 -- 172.21.15.76:0/2935710985 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ec1005c0 msgr2=0x7fb4ec1009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:06.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.147+0000 7fb4f2f9d640 1 --2- 172.21.15.76:0/2935710985 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ec1005c0 0x7fb4ec1009e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8009900 tx=0x7fb4e802bc30 comp rx=0 tx=0).stop 2024-03-18T14:51:06.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.147+0000 7fb4f2f9d640 1 -- 172.21.15.76:0/2935710985 shutdown_connections 2024-03-18T14:51:06.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.147+0000 7fb4f2f9d640 1 --2- 172.21.15.76:0/2935710985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4ec1017b0 0x7fb4ec101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:06.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.147+0000 7fb4f2f9d640 1 --2- 172.21.15.76:0/2935710985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4ec102170 0x7fb4ec108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.147+0000 7fb4f2f9d640 1 --2- 172.21.15.76:0/2935710985 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ec1005c0 0x7fb4ec1009e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.147+0000 7fb4f2f9d640 1 -- 172.21.15.76:0/2935710985 >> 172.21.15.76:0/2935710985 conn(0x7fb4ec0fbb30 msgr2=0x7fb4ec0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.147+0000 7fb4f2f9d640 1 -- 172.21.15.76:0/2935710985 shutdown_connections 2024-03-18T14:51:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.147+0000 7fb4f2f9d640 1 -- 172.21.15.76:0/2935710985 wait complete. 2024-03-18T14:51:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f2f9d640 1 Processor -- start 2024-03-18T14:51:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f2f9d640 1 -- start start 2024-03-18T14:51:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f2f9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4ec1005c0 0x7fb4ec06a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f2f9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4ec1017b0 0x7fb4ec19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f2f9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ec102170 0x7fb4ec19ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f2f9d640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ec10c230 con 0x7fb4ec102170 2024-03-18T14:51:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f2f9d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ec10c0b0 con 0x7fb4ec1017b0 2024-03-18T14:51:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f2f9d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ec10c3b0 con 0x7fb4ec1005c0 2024-03-18T14:51:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f279c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ec102170 0x7fb4ec19ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f279c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ec102170 0x7fb4ec19ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58136/0 (socket says 172.21.15.76:58136) 2024-03-18T14:51:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f279c640 1 -- 172.21.15.76:0/2488485640 learned_addr learned my addr 172.21.15.76:0/2488485640 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f179a640 1 --2- 172.21.15.76:0/2488485640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4ec1017b0 0x7fb4ec19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f1f9b640 1 --2- 172.21.15.76:0/2488485640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4ec1005c0 0x7fb4ec06a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f279c640 1 -- 172.21.15.76:0/2488485640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4ec1005c0 msgr2=0x7fb4ec06a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f279c640 1 --2- 172.21.15.76:0/2488485640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4ec1005c0 0x7fb4ec06a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f279c640 1 -- 172.21.15.76:0/2488485640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4ec1017b0 msgr2=0x7fb4ec19c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f279c640 1 --2- 172.21.15.76:0/2488485640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4ec1017b0 0x7fb4ec19c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f279c640 1 -- 172.21.15.76:0/2488485640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4ec19f600 con 0x7fb4ec102170 2024-03-18T14:51:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.148+0000 7fb4f279c640 1 --2- 172.21.15.76:0/2488485640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ec102170 0x7fb4ec19ed40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb4e400eb10 tx=0x7fb4e400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.149+0000 7fb4e2ffd640 1 -- 172.21.15.76:0/2488485640 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4004b20 con 0x7fb4ec102170 2024-03-18T14:51:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.149+0000 7fb4e2ffd640 1 -- 172.21.15.76:0/2488485640 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4e4004cc0 con 0x7fb4ec102170 2024-03-18T14:51:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.149+0000 7fb4f2f9d640 1 -- 172.21.15.76:0/2488485640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4ec19f8f0 con 0x7fb4ec102170 2024-03-18T14:51:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.149+0000 7fb4e2ffd640 1 -- 172.21.15.76:0/2488485640 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4010760 con 0x7fb4ec102170 2024-03-18T14:51:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.149+0000 7fb4f2f9d640 1 -- 172.21.15.76:0/2488485640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4ec19fdf0 con 0x7fb4ec102170 2024-03-18T14:51:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.150+0000 7fb4f2f9d640 1 -- 172.21.15.76:0/2488485640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4ec1078e0 con 0x7fb4ec102170 2024-03-18T14:51:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.151+0000 7fb4e2ffd640 1 -- 172.21.15.76:0/2488485640 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb4e4005020 con 0x7fb4ec102170 2024-03-18T14:51:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.151+0000 7fb4e2ffd640 1 --2- 172.21.15.76:0/2488485640 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb4d80a2800 0x7fb4d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.151+0000 7fb4e2ffd640 1 -- 172.21.15.76:0/2488485640 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fb4e40cee80 con 0x7fb4ec102170 2024-03-18T14:51:06.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.151+0000 7fb4f1f9b640 1 --2- 172.21.15.76:0/2488485640 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb4d80a2800 0x7fb4d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:06.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.152+0000 7fb4f1f9b640 1 --2- 172.21.15.76:0/2488485640 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb4d80a2800 0x7fb4d80a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8000c00 tx=0x7fb4e8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:06.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.156+0000 7fb4e2ffd640 1 -- 172.21.15.76:0/2488485640 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb4e409c070 con 0x7fb4ec102170 2024-03-18T14:51:06.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:06.374+0000 7fb4f2f9d640 1 -- 172.21.15.76:0/2488485640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fb4ec02d400 con 0x7fb4ec102170 2024-03-18T14:51:06.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:06 smithi076 ceph-mon[32205]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T14:51:06.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:06 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/924961750' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-18T14:51:06.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:06 smithi076 ceph-mon[32205]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T14:51:06.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:06 smithi076 ceph-mon[32205]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T14:51:06.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:06 smithi076 ceph-mon[32205]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T14:51:06.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:06 smithi076 ceph-mon[32205]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T14:51:06.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:06 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1119538490' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T14:51:06.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:06 smithi174 ceph-mon[36787]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T14:51:06.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:06 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/924961750' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-18T14:51:06.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:06 smithi174 ceph-mon[36787]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T14:51:06.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:06 smithi174 ceph-mon[36787]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T14:51:06.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:06 smithi174 ceph-mon[36787]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T14:51:06.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:06 smithi174 ceph-mon[36787]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T14:51:06.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:06 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1119538490' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T14:51:06.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:06 smithi077 ceph-mon[35865]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T14:51:06.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:06 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/924961750' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-18T14:51:06.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:06 smithi077 ceph-mon[35865]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T14:51:06.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:06 smithi077 ceph-mon[35865]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T14:51:06.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:06 smithi077 ceph-mon[35865]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T14:51:06.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:06 smithi077 ceph-mon[35865]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T14:51:06.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:06 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1119538490' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T14:51:07.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.027+0000 7fb4e2ffd640 1 -- 172.21.15.76:0/2488485640 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7fb4e409b970 con 0x7fb4ec102170 2024-03-18T14:51:07.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.030+0000 7fb4f2f9d640 1 -- 172.21.15.76:0/2488485640 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb4d80a2800 msgr2=0x7fb4d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:07.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.030+0000 7fb4f2f9d640 1 --2- 172.21.15.76:0/2488485640 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb4d80a2800 0x7fb4d80a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8000c00 tx=0x7fb4e8005fd0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.030+0000 7fb4f2f9d640 1 -- 172.21.15.76:0/2488485640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ec102170 msgr2=0x7fb4ec19ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:07.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.030+0000 7fb4f2f9d640 1 --2- 172.21.15.76:0/2488485640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ec102170 0x7fb4ec19ed40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb4e400eb10 tx=0x7fb4e400c2d0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.030+0000 7fb4f2f9d640 1 -- 172.21.15.76:0/2488485640 shutdown_connections 2024-03-18T14:51:07.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.030+0000 7fb4f2f9d640 1 --2- 172.21.15.76:0/2488485640 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb4d80a2800 0x7fb4d80a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.030+0000 7fb4f2f9d640 1 --2- 172.21.15.76:0/2488485640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ec102170 0x7fb4ec19ed40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.030+0000 7fb4f2f9d640 1 --2- 172.21.15.76:0/2488485640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4ec1017b0 0x7fb4ec19c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.030+0000 7fb4f2f9d640 1 --2- 172.21.15.76:0/2488485640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4ec1005c0 0x7fb4ec06a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.030+0000 7fb4f2f9d640 1 -- 172.21.15.76:0/2488485640 >> 172.21.15.76:0/2488485640 conn(0x7fb4ec0fbb30 msgr2=0x7fb4ec104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:07.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.030+0000 7fb4f2f9d640 1 -- 172.21.15.76:0/2488485640 shutdown_connections 2024-03-18T14:51:07.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.030+0000 7fb4f2f9d640 1 -- 172.21.15.76:0/2488485640 wait complete. 2024-03-18T14:51:07.046 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-03-18T14:51:07.046 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-18T14:51:07.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.156+0000 7fc1e0812640 1 -- 172.21.15.76:0/8138805 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1dc1022a0 msgr2=0x7fc1dc108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:07.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.156+0000 7fc1e0812640 1 --2- 172.21.15.76:0/8138805 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1dc1022a0 0x7fc1dc108b30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4009ca0 tx=0x7fc1d402c0f0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.156+0000 7fc1e0812640 1 -- 172.21.15.76:0/8138805 shutdown_connections 2024-03-18T14:51:07.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.156+0000 7fc1e0812640 1 --2- 172.21.15.76:0/8138805 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1dc1022a0 0x7fc1dc108b30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.156+0000 7fc1e0812640 1 --2- 172.21.15.76:0/8138805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1dc1018e0 0x7fc1dc101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.156+0000 7fc1e0812640 1 --2- 172.21.15.76:0/8138805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1dc1006f0 0x7fc1dc100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.156+0000 7fc1e0812640 1 -- 172.21.15.76:0/8138805 >> 172.21.15.76:0/8138805 conn(0x7fc1dc0fbc80 msgr2=0x7fc1dc0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:07.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.156+0000 7fc1e0812640 1 -- 172.21.15.76:0/8138805 shutdown_connections 2024-03-18T14:51:07.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.156+0000 7fc1e0812640 1 -- 172.21.15.76:0/8138805 wait complete. 2024-03-18T14:51:07.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1e0812640 1 Processor -- start 2024-03-18T14:51:07.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1e0812640 1 -- start start 2024-03-18T14:51:07.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1e0812640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1dc1006f0 0x7fc1dc197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:07.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1e0812640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1dc1018e0 0x7fc1dc198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:07.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1e0812640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1dc1022a0 0x7fc1dc19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:07.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1e0812640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc10c350 con 0x7fc1dc1006f0 2024-03-18T14:51:07.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1e0812640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc10c1d0 con 0x7fc1dc1022a0 2024-03-18T14:51:07.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1e0812640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc10c4d0 con 0x7fc1dc1018e0 2024-03-18T14:51:07.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1da59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1dc1006f0 0x7fc1dc197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:07.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1da59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1dc1006f0 0x7fc1dc197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58154/0 (socket says 172.21.15.76:58154) 2024-03-18T14:51:07.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1da59c640 1 -- 172.21.15.76:0/3714528815 learned_addr learned my addr 172.21.15.76:0/3714528815 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:07.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1dad9d640 1 --2- 172.21.15.76:0/3714528815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1dc1022a0 0x7fc1dc19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:07.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1d9d9b640 1 --2- 172.21.15.76:0/3714528815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1dc1018e0 0x7fc1dc198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:07.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1da59c640 1 -- 172.21.15.76:0/3714528815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1dc1018e0 msgr2=0x7fc1dc198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:07.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1da59c640 1 --2- 172.21.15.76:0/3714528815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1dc1018e0 0x7fc1dc198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1da59c640 1 -- 172.21.15.76:0/3714528815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1dc1022a0 msgr2=0x7fc1dc19f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:07.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1da59c640 1 --2- 172.21.15.76:0/3714528815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1dc1022a0 0x7fc1dc19f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1da59c640 1 -- 172.21.15.76:0/3714528815 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1dc19f700 con 0x7fc1dc1006f0 2024-03-18T14:51:07.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1d9d9b640 1 --2- 172.21.15.76:0/3714528815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1dc1018e0 0x7fc1dc198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:51:07.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1da59c640 1 --2- 172.21.15.76:0/3714528815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1dc1006f0 0x7fc1dc197ae0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc1d000e9c0 tx=0x7fc1d000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:07.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1cb7fe640 1 -- 172.21.15.76:0/3714528815 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d000cd80 con 0x7fc1dc1006f0 2024-03-18T14:51:07.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1cb7fe640 1 -- 172.21.15.76:0/3714528815 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1d00035f0 con 0x7fc1dc1006f0 2024-03-18T14:51:07.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1e0812640 1 -- 172.21.15.76:0/3714528815 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1dc19f9f0 con 0x7fc1dc1006f0 2024-03-18T14:51:07.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.157+0000 7fc1cb7fe640 1 -- 172.21.15.76:0/3714528815 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d0010690 con 0x7fc1dc1006f0 2024-03-18T14:51:07.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.158+0000 7fc1e0812640 1 -- 172.21.15.76:0/3714528815 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1dc19ff40 con 0x7fc1dc1006f0 2024-03-18T14:51:07.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.159+0000 7fc1e0812640 1 -- 172.21.15.76:0/3714528815 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1dc068a20 con 0x7fc1dc1006f0 2024-03-18T14:51:07.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.160+0000 7fc1cb7fe640 1 -- 172.21.15.76:0/3714528815 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc1d00037a0 con 0x7fc1dc1006f0 2024-03-18T14:51:07.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.160+0000 7fc1cb7fe640 1 --2- 172.21.15.76:0/3714528815 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc1c40a2800 0x7fc1c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:07.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.160+0000 7fc1d9d9b640 1 --2- 172.21.15.76:0/3714528815 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc1c40a2800 0x7fc1c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:07.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.160+0000 7fc1cb7fe640 1 -- 172.21.15.76:0/3714528815 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fc1d00ced70 con 0x7fc1dc1006f0 2024-03-18T14:51:07.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.161+0000 7fc1d9d9b640 1 --2- 172.21.15.76:0/3714528815 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc1c40a2800 0x7fc1c40a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc198ff0 tx=0x7fc1cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:07.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.165+0000 7fc1cb7fe640 1 -- 172.21.15.76:0/3714528815 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc1d009bf60 con 0x7fc1dc1006f0 2024-03-18T14:51:07.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.367+0000 7fc1e0812640 1 -- 172.21.15.76:0/3714528815 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc1dc0fdde0 con 0x7fc1c40a2800 2024-03-18T14:51:07.374 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:07 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1119538490' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T14:51:07.374 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:07 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] up:active 2024-03-18T14:51:07.374 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:07 smithi076 ceph-mon[32205]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T14:51:07.374 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:07 smithi076 ceph-mon[32205]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T14:51:07.374 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:07 smithi076 ceph-mon[32205]: Cluster is now healthy 2024-03-18T14:51:07.374 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:07 smithi076 ceph-mon[32205]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T14:51:07.374 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:07 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-18T14:51:07.374 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:07 smithi076 ceph-mon[32205]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-18T14:51:07.374 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:07 smithi076 ceph-mon[32205]: pgmap v286: 97 pgs: 97 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s wr, 12 op/s 2024-03-18T14:51:07.374 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:07 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2488485640' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T14:51:07.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.386+0000 7fc1cb7fe640 1 -- 172.21.15.76:0/3714528815 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc1dc0fdde0 con 0x7fc1c40a2800 2024-03-18T14:51:07.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.388+0000 7fc1e0812640 1 -- 172.21.15.76:0/3714528815 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc1c40a2800 msgr2=0x7fc1c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:07.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.388+0000 7fc1e0812640 1 --2- 172.21.15.76:0/3714528815 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc1c40a2800 0x7fc1c40a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc198ff0 tx=0x7fc1cc005f90 comp rx=0 tx=0).stop 2024-03-18T14:51:07.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.388+0000 7fc1e0812640 1 -- 172.21.15.76:0/3714528815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1dc1006f0 msgr2=0x7fc1dc197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:07.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.388+0000 7fc1e0812640 1 --2- 172.21.15.76:0/3714528815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1dc1006f0 0x7fc1dc197ae0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc1d000e9c0 tx=0x7fc1d000ee80 comp rx=0 tx=0).stop 2024-03-18T14:51:07.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.388+0000 7fc1e0812640 1 -- 172.21.15.76:0/3714528815 shutdown_connections 2024-03-18T14:51:07.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.388+0000 7fc1e0812640 1 --2- 172.21.15.76:0/3714528815 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc1c40a2800 0x7fc1c40a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.388+0000 7fc1e0812640 1 --2- 172.21.15.76:0/3714528815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1dc1018e0 0x7fc1dc198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.388+0000 7fc1e0812640 1 --2- 172.21.15.76:0/3714528815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1dc1022a0 0x7fc1dc19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.388+0000 7fc1e0812640 1 --2- 172.21.15.76:0/3714528815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1dc1006f0 0x7fc1dc197ae0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.388+0000 7fc1e0812640 1 -- 172.21.15.76:0/3714528815 >> 172.21.15.76:0/3714528815 conn(0x7fc1dc0fbc80 msgr2=0x7fc1dc0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:07.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.388+0000 7fc1e0812640 1 -- 172.21.15.76:0/3714528815 shutdown_connections 2024-03-18T14:51:07.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.388+0000 7fc1e0812640 1 -- 172.21.15.76:0/3714528815 wait complete. 2024-03-18T14:51:07.400 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-18T14:51:07.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:07 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1119538490' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T14:51:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:07 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] up:active 2024-03-18T14:51:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:07 smithi174 ceph-mon[36787]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T14:51:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:07 smithi174 ceph-mon[36787]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T14:51:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:07 smithi174 ceph-mon[36787]: Cluster is now healthy 2024-03-18T14:51:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:07 smithi174 ceph-mon[36787]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T14:51:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:07 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-18T14:51:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:07 smithi174 ceph-mon[36787]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-18T14:51:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:07 smithi174 ceph-mon[36787]: pgmap v286: 97 pgs: 97 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s wr, 12 op/s 2024-03-18T14:51:07.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:07 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2488485640' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T14:51:07.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:07 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1119538490' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T14:51:07.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:07 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] up:active 2024-03-18T14:51:07.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:07 smithi077 ceph-mon[35865]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T14:51:07.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:07 smithi077 ceph-mon[35865]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T14:51:07.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:07 smithi077 ceph-mon[35865]: Cluster is now healthy 2024-03-18T14:51:07.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:07 smithi077 ceph-mon[35865]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T14:51:07.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:07 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-18T14:51:07.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:07 smithi077 ceph-mon[35865]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-18T14:51:07.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:07 smithi077 ceph-mon[35865]: pgmap v286: 97 pgs: 97 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s wr, 12 op/s 2024-03-18T14:51:07.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:07 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2488485640' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T14:51:07.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.505+0000 7f249adc4640 1 -- 172.21.15.76:0/1064690700 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2494104bb0 msgr2=0x7f2494106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:07.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.505+0000 7f249adc4640 1 --2- 172.21.15.76:0/1064690700 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2494104bb0 0x7f2494106fa0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2480009900 tx=0x7f248002bc30 comp rx=0 tx=0).stop 2024-03-18T14:51:07.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.505+0000 7f249adc4640 1 -- 172.21.15.76:0/1064690700 shutdown_connections 2024-03-18T14:51:07.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.505+0000 7f249adc4640 1 --2- 172.21.15.76:0/1064690700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24941074e0 0x7f2494109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.505+0000 7f249adc4640 1 --2- 172.21.15.76:0/1064690700 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2494104bb0 0x7f2494106fa0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.505+0000 7f249adc4640 1 --2- 172.21.15.76:0/1064690700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2494102280 0x7f2494104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.505+0000 7f249adc4640 1 -- 172.21.15.76:0/1064690700 >> 172.21.15.76:0/1064690700 conn(0x7f24940fbc90 msgr2=0x7f24940fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:07.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.505+0000 7f249adc4640 1 -- 172.21.15.76:0/1064690700 shutdown_connections 2024-03-18T14:51:07.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.505+0000 7f249adc4640 1 -- 172.21.15.76:0/1064690700 wait complete. 2024-03-18T14:51:07.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f249adc4640 1 Processor -- start 2024-03-18T14:51:07.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f249adc4640 1 -- start start 2024-03-18T14:51:07.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f249adc4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2494102280 0x7f2494101bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:07.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f249adc4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2494104bb0 0x7f2494100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:07.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f249adc4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24941074e0 0x7f2494100770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:07.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f249adc4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f249410d490 con 0x7f2494104bb0 2024-03-18T14:51:07.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f249adc4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f249410d310 con 0x7f24941074e0 2024-03-18T14:51:07.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f249adc4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f249410d610 con 0x7f2494102280 2024-03-18T14:51:07.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f2493fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2494104bb0 0x7f2494100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:07.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f2493fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2494104bb0 0x7f2494100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58196/0 (socket says 172.21.15.76:58196) 2024-03-18T14:51:07.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f2493fff640 1 -- 172.21.15.76:0/3936511381 learned_addr learned my addr 172.21.15.76:0/3936511381 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:07.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f2498b60640 1 --2- 172.21.15.76:0/3936511381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2494102280 0x7f2494101bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:07.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f2499361640 1 --2- 172.21.15.76:0/3936511381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24941074e0 0x7f2494100770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:07.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f2493fff640 1 -- 172.21.15.76:0/3936511381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2494102280 msgr2=0x7f2494101bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:07.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f2493fff640 1 --2- 172.21.15.76:0/3936511381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2494102280 0x7f2494101bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f2493fff640 1 -- 172.21.15.76:0/3936511381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24941074e0 msgr2=0x7f2494100770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:07.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f2493fff640 1 --2- 172.21.15.76:0/3936511381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24941074e0 0x7f2494100770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f2493fff640 1 -- 172.21.15.76:0/3936511381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2494101000 con 0x7f2494104bb0 2024-03-18T14:51:07.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.506+0000 7f2493fff640 1 --2- 172.21.15.76:0/3936511381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2494104bb0 0x7f2494100230 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2480000c00 tx=0x7f248002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:07.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.507+0000 7f2491ffb640 1 -- 172.21.15.76:0/3936511381 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2480041070 con 0x7f2494104bb0 2024-03-18T14:51:07.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.507+0000 7f2491ffb640 1 -- 172.21.15.76:0/3936511381 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2480004450 con 0x7f2494104bb0 2024-03-18T14:51:07.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.507+0000 7f249adc4640 1 -- 172.21.15.76:0/3936511381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24941a1bf0 con 0x7f2494104bb0 2024-03-18T14:51:07.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.507+0000 7f2491ffb640 1 -- 172.21.15.76:0/3936511381 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2480034440 con 0x7f2494104bb0 2024-03-18T14:51:07.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.507+0000 7f249adc4640 1 -- 172.21.15.76:0/3936511381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24941a2140 con 0x7f2494104bb0 2024-03-18T14:51:07.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.508+0000 7f249adc4640 1 -- 172.21.15.76:0/3936511381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24940fd780 con 0x7f2494104bb0 2024-03-18T14:51:07.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.509+0000 7f2491ffb640 1 -- 172.21.15.76:0/3936511381 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2480044070 con 0x7f2494104bb0 2024-03-18T14:51:07.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.509+0000 7f2491ffb640 1 --2- 172.21.15.76:0/3936511381 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f24840a2800 0x7f24840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:07.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.509+0000 7f2498b60640 1 --2- 172.21.15.76:0/3936511381 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f24840a2800 0x7f24840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:07.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.509+0000 7f2491ffb640 1 -- 172.21.15.76:0/3936511381 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f24800f03c0 con 0x7f2494104bb0 2024-03-18T14:51:07.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.510+0000 7f2498b60640 1 --2- 172.21.15.76:0/3936511381 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f24840a2800 0x7f24840a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2488009bd0 tx=0x7f2488005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:07.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.514+0000 7f2491ffb640 1 -- 172.21.15.76:0/3936511381 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f24800bcff0 con 0x7f2494104bb0 2024-03-18T14:51:07.661 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:51:07 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:51:07.372+0000 7fa61f6f1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:51:07.661 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:51:07 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:51:07.372+0000 7fa61f6f1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:51:07.661 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:51:07 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:51:07.372+0000 7fa61f6f1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:51:07.661 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:51:07 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:51:07.372+0000 7fa61f6f1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:51:07.661 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:51:07 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:51:07.372+0000 7fa61f6f1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:51:07.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.717+0000 7f249adc4640 1 -- 172.21.15.76:0/3936511381 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f2494061cb0 con 0x7f24840a2800 2024-03-18T14:51:07.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.736+0000 7f2491ffb640 1 -- 172.21.15.76:0/3936511381 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2494061cb0 con 0x7f24840a2800 2024-03-18T14:51:07.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.738+0000 7f249adc4640 1 -- 172.21.15.76:0/3936511381 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f24840a2800 msgr2=0x7f24840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:07.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.738+0000 7f249adc4640 1 --2- 172.21.15.76:0/3936511381 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f24840a2800 0x7f24840a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2488009bd0 tx=0x7f2488005f90 comp rx=0 tx=0).stop 2024-03-18T14:51:07.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.738+0000 7f249adc4640 1 -- 172.21.15.76:0/3936511381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2494104bb0 msgr2=0x7f2494100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:07.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.738+0000 7f249adc4640 1 --2- 172.21.15.76:0/3936511381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2494104bb0 0x7f2494100230 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2480000c00 tx=0x7f248002c640 comp rx=0 tx=0).stop 2024-03-18T14:51:07.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.738+0000 7f249adc4640 1 -- 172.21.15.76:0/3936511381 shutdown_connections 2024-03-18T14:51:07.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.738+0000 7f249adc4640 1 --2- 172.21.15.76:0/3936511381 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f24840a2800 0x7f24840a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.738+0000 7f249adc4640 1 --2- 172.21.15.76:0/3936511381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24941074e0 0x7f2494100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.738+0000 7f249adc4640 1 --2- 172.21.15.76:0/3936511381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2494104bb0 0x7f2494100230 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.738+0000 7f249adc4640 1 --2- 172.21.15.76:0/3936511381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2494102280 0x7f2494101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.738+0000 7f249adc4640 1 -- 172.21.15.76:0/3936511381 >> 172.21.15.76:0/3936511381 conn(0x7f24940fbc90 msgr2=0x7f2494102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:07.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.738+0000 7f249adc4640 1 -- 172.21.15.76:0/3936511381 shutdown_connections 2024-03-18T14:51:07.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.738+0000 7f249adc4640 1 -- 172.21.15.76:0/3936511381 wait complete. 2024-03-18T14:51:07.749 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-18T14:51:07.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.855+0000 7fb9eaaf1640 1 -- 172.21.15.76:0/4177278229 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9e41029e0 msgr2=0x7fb9e4102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:07.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.855+0000 7fb9eaaf1640 1 --2- 172.21.15.76:0/4177278229 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9e41029e0 0x7fb9e4102e60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0009900 tx=0x7fb9d002bc30 comp rx=0 tx=0).stop 2024-03-18T14:51:07.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.855+0000 7fb9eaaf1640 1 -- 172.21.15.76:0/4177278229 shutdown_connections 2024-03-18T14:51:07.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.855+0000 7fb9eaaf1640 1 --2- 172.21.15.76:0/4177278229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e41033a0 0x7fb9e4109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.855+0000 7fb9eaaf1640 1 --2- 172.21.15.76:0/4177278229 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9e41029e0 0x7fb9e4102e60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.855+0000 7fb9eaaf1640 1 --2- 172.21.15.76:0/4177278229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9e41017f0 0x7fb9e4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.855+0000 7fb9eaaf1640 1 -- 172.21.15.76:0/4177278229 >> 172.21.15.76:0/4177278229 conn(0x7fb9e40fcd80 msgr2=0x7fb9e40ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:07.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.855+0000 7fb9eaaf1640 1 -- 172.21.15.76:0/4177278229 shutdown_connections 2024-03-18T14:51:07.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.855+0000 7fb9eaaf1640 1 -- 172.21.15.76:0/4177278229 wait complete. 2024-03-18T14:51:07.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.855+0000 7fb9eaaf1640 1 Processor -- start 2024-03-18T14:51:07.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9eaaf1640 1 -- start start 2024-03-18T14:51:07.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9eaaf1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e41017f0 0x7fb9e4197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:07.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9eaaf1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9e41029e0 0x7fb9e4198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:07.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9eaaf1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9e41033a0 0x7fb9e419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:07.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9eaaf1640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e410d470 con 0x7fb9e41029e0 2024-03-18T14:51:07.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9eaaf1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e410d2f0 con 0x7fb9e41017f0 2024-03-18T14:51:07.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9eaaf1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e410d5f0 con 0x7fb9e41033a0 2024-03-18T14:51:07.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9e3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9e41029e0 0x7fb9e4198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:07.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9e3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9e41029e0 0x7fb9e4198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58212/0 (socket says 172.21.15.76:58212) 2024-03-18T14:51:07.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9e3fff640 1 -- 172.21.15.76:0/3839191723 learned_addr learned my addr 172.21.15.76:0/3839191723 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:07.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9e888d640 1 --2- 172.21.15.76:0/3839191723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e41017f0 0x7fb9e4197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:07.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9e908e640 1 --2- 172.21.15.76:0/3839191723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9e41033a0 0x7fb9e419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:07.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9e3fff640 1 -- 172.21.15.76:0/3839191723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9e41033a0 msgr2=0x7fb9e419f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:07.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9e3fff640 1 --2- 172.21.15.76:0/3839191723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9e41033a0 0x7fb9e419f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9e3fff640 1 -- 172.21.15.76:0/3839191723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e41017f0 msgr2=0x7fb9e4197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:07.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9e3fff640 1 --2- 172.21.15.76:0/3839191723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e41017f0 0x7fb9e4197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:07.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9e3fff640 1 -- 172.21.15.76:0/3839191723 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e419f730 con 0x7fb9e41029e0 2024-03-18T14:51:07.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9e3fff640 1 --2- 172.21.15.76:0/3839191723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9e41029e0 0x7fb9e4198050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0009230 tx=0x7fb9d002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:07.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9e1ffb640 1 -- 172.21.15.76:0/3839191723 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0041070 con 0x7fb9e41029e0 2024-03-18T14:51:07.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9e1ffb640 1 -- 172.21.15.76:0/3839191723 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9d0004450 con 0x7fb9e41029e0 2024-03-18T14:51:07.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9eaaf1640 1 -- 172.21.15.76:0/3839191723 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9e419f9c0 con 0x7fb9e41029e0 2024-03-18T14:51:07.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.856+0000 7fb9e1ffb640 1 -- 172.21.15.76:0/3839191723 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0003e20 con 0x7fb9e41029e0 2024-03-18T14:51:07.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.857+0000 7fb9eaaf1640 1 -- 172.21.15.76:0/3839191723 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9e419feb0 con 0x7fb9e41029e0 2024-03-18T14:51:07.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.858+0000 7fb9eaaf1640 1 -- 172.21.15.76:0/3839191723 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9e4108b90 con 0x7fb9e41029e0 2024-03-18T14:51:07.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.859+0000 7fb9e1ffb640 1 -- 172.21.15.76:0/3839191723 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb9d0044070 con 0x7fb9e41029e0 2024-03-18T14:51:07.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.859+0000 7fb9e1ffb640 1 --2- 172.21.15.76:0/3839191723 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb9d40a2800 0x7fb9d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:07.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.859+0000 7fb9e888d640 1 --2- 172.21.15.76:0/3839191723 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb9d40a2800 0x7fb9d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:07.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.859+0000 7fb9e1ffb640 1 -- 172.21.15.76:0/3839191723 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fb9d00f0810 con 0x7fb9e41029e0 2024-03-18T14:51:07.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.860+0000 7fb9e888d640 1 --2- 172.21.15.76:0/3839191723 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb9d40a2800 0x7fb9d40a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4102840 tx=0x7fb9d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:07.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:07.863+0000 7fb9e1ffb640 1 -- 172.21.15.76:0/3839191723 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb9d00bd3b0 con 0x7fb9e41029e0 2024-03-18T14:51:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.069+0000 7fb9eaaf1640 1 -- 172.21.15.76:0/3839191723 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fb9e4061cb0 con 0x7fb9d40a2800 2024-03-18T14:51:08.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.073+0000 7fb9e1ffb640 1 -- 172.21.15.76:0/3839191723 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fb9e4061cb0 con 0x7fb9d40a2800 2024-03-18T14:51:08.074 INFO:teuthology.orchestra.run.smithi076.stdout:/volumes/qa/sv_0/6b829921-7ec0-435a-8f95-43d9026acfb6 2024-03-18T14:51:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.075+0000 7fb9eaaf1640 1 -- 172.21.15.76:0/3839191723 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb9d40a2800 msgr2=0x7fb9d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.075+0000 7fb9eaaf1640 1 --2- 172.21.15.76:0/3839191723 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb9d40a2800 0x7fb9d40a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4102840 tx=0x7fb9d8005f90 comp rx=0 tx=0).stop 2024-03-18T14:51:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.075+0000 7fb9eaaf1640 1 -- 172.21.15.76:0/3839191723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9e41029e0 msgr2=0x7fb9e4198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.075+0000 7fb9eaaf1640 1 --2- 172.21.15.76:0/3839191723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9e41029e0 0x7fb9e4198050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0009230 tx=0x7fb9d002c650 comp rx=0 tx=0).stop 2024-03-18T14:51:08.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.075+0000 7fb9eaaf1640 1 -- 172.21.15.76:0/3839191723 shutdown_connections 2024-03-18T14:51:08.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.075+0000 7fb9eaaf1640 1 --2- 172.21.15.76:0/3839191723 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fb9d40a2800 0x7fb9d40a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.075+0000 7fb9eaaf1640 1 --2- 172.21.15.76:0/3839191723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9e41033a0 0x7fb9e419f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.075+0000 7fb9eaaf1640 1 --2- 172.21.15.76:0/3839191723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9e41029e0 0x7fb9e4198050 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.075+0000 7fb9eaaf1640 1 --2- 172.21.15.76:0/3839191723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e41017f0 0x7fb9e4197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.075+0000 7fb9eaaf1640 1 -- 172.21.15.76:0/3839191723 >> 172.21.15.76:0/3839191723 conn(0x7fb9e40fcd80 msgr2=0x7fb9e4105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:08.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.075+0000 7fb9eaaf1640 1 -- 172.21.15.76:0/3839191723 shutdown_connections 2024-03-18T14:51:08.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.075+0000 7fb9eaaf1640 1 -- 172.21.15.76:0/3839191723 wait complete. 2024-03-18T14:51:08.087 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-18T14:51:08.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.192+0000 7fc1dc680640 1 -- 172.21.15.76:0/2808725046 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1d4100700 msgr2=0x7fc1d4100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:08.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.192+0000 7fc1dc680640 1 --2- 172.21.15.76:0/2808725046 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1d4100700 0x7fc1d4100b20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0009900 tx=0x7fc1d002bc30 comp rx=0 tx=0).stop 2024-03-18T14:51:08.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.192+0000 7fc1dc680640 1 -- 172.21.15.76:0/2808725046 shutdown_connections 2024-03-18T14:51:08.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.192+0000 7fc1dc680640 1 --2- 172.21.15.76:0/2808725046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1d41018f0 0x7fc1d4101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.192+0000 7fc1dc680640 1 --2- 172.21.15.76:0/2808725046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1d41022b0 0x7fc1d4108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.192+0000 7fc1dc680640 1 --2- 172.21.15.76:0/2808725046 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1d4100700 0x7fc1d4100b20 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.192+0000 7fc1dc680640 1 -- 172.21.15.76:0/2808725046 >> 172.21.15.76:0/2808725046 conn(0x7fc1d40fbc90 msgr2=0x7fc1d40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:08.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.192+0000 7fc1dc680640 1 -- 172.21.15.76:0/2808725046 shutdown_connections 2024-03-18T14:51:08.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.192+0000 7fc1dc680640 1 -- 172.21.15.76:0/2808725046 wait complete. 2024-03-18T14:51:08.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.193+0000 7fc1dc680640 1 Processor -- start 2024-03-18T14:51:08.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.193+0000 7fc1dc680640 1 -- start start 2024-03-18T14:51:08.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.193+0000 7fc1dc680640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1d4100700 0x7fc1d4197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:08.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.193+0000 7fc1dc680640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1d41018f0 0x7fc1d4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:08.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.193+0000 7fc1dc680640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1d41022b0 0x7fc1d419f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:08.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.193+0000 7fc1dc680640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d410d4a0 con 0x7fc1d41018f0 2024-03-18T14:51:08.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.193+0000 7fc1dc680640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d410d320 con 0x7fc1d41022b0 2024-03-18T14:51:08.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.193+0000 7fc1dc680640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d410d620 con 0x7fc1d4100700 2024-03-18T14:51:08.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.193+0000 7fc1d9c1b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1d41018f0 0x7fc1d4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:08.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.193+0000 7fc1d9c1b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1d41018f0 0x7fc1d4198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58222/0 (socket says 172.21.15.76:58222) 2024-03-18T14:51:08.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.193+0000 7fc1d9c1b640 1 -- 172.21.15.76:0/2781521639 learned_addr learned my addr 172.21.15.76:0/2781521639 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:08.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.193+0000 7fc1da41c640 1 --2- 172.21.15.76:0/2781521639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1d4100700 0x7fc1d4197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:08.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.193+0000 7fc1dac1d640 1 --2- 172.21.15.76:0/2781521639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1d41022b0 0x7fc1d419f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:08.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.194+0000 7fc1d9c1b640 1 -- 172.21.15.76:0/2781521639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1d4100700 msgr2=0x7fc1d4197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:08.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.194+0000 7fc1d9c1b640 1 --2- 172.21.15.76:0/2781521639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1d4100700 0x7fc1d4197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.194+0000 7fc1d9c1b640 1 -- 172.21.15.76:0/2781521639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1d41022b0 msgr2=0x7fc1d419f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:08.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.194+0000 7fc1d9c1b640 1 --2- 172.21.15.76:0/2781521639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1d41022b0 0x7fc1d419f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.194+0000 7fc1d9c1b640 1 -- 172.21.15.76:0/2781521639 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d419f770 con 0x7fc1d41018f0 2024-03-18T14:51:08.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.194+0000 7fc1da41c640 1 --2- 172.21.15.76:0/2781521639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1d4100700 0x7fc1d4197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:51:08.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.194+0000 7fc1d9c1b640 1 --2- 172.21.15.76:0/2781521639 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1d41018f0 0x7fc1d4198090 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400e9f0 tx=0x7fc1c400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:08.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.194+0000 7fc1cb7fe640 1 -- 172.21.15.76:0/2781521639 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c400cd80 con 0x7fc1d41018f0 2024-03-18T14:51:08.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.194+0000 7fc1cb7fe640 1 -- 172.21.15.76:0/2781521639 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c4005500 con 0x7fc1d41018f0 2024-03-18T14:51:08.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.194+0000 7fc1dc680640 1 -- 172.21.15.76:0/2781521639 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1d419fa60 con 0x7fc1d41018f0 2024-03-18T14:51:08.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.194+0000 7fc1cb7fe640 1 -- 172.21.15.76:0/2781521639 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4010690 con 0x7fc1d41018f0 2024-03-18T14:51:08.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.194+0000 7fc1dc680640 1 -- 172.21.15.76:0/2781521639 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1d419ff60 con 0x7fc1d41018f0 2024-03-18T14:51:08.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.195+0000 7fc1dc680640 1 -- 172.21.15.76:0/2781521639 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1d4107ac0 con 0x7fc1d41018f0 2024-03-18T14:51:08.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.196+0000 7fc1cb7fe640 1 -- 172.21.15.76:0/2781521639 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc1c40046c0 con 0x7fc1d41018f0 2024-03-18T14:51:08.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.197+0000 7fc1cb7fe640 1 --2- 172.21.15.76:0/2781521639 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc1c00a2800 0x7fc1c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:08.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.197+0000 7fc1da41c640 1 --2- 172.21.15.76:0/2781521639 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc1c00a2800 0x7fc1c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:08.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.197+0000 7fc1cb7fe640 1 -- 172.21.15.76:0/2781521639 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fc1c40cef90 con 0x7fc1d41018f0 2024-03-18T14:51:08.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.197+0000 7fc1da41c640 1 --2- 172.21.15.76:0/2781521639 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc1c00a2800 0x7fc1c00a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc1d002c7b0 tx=0x7fc1d0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:08.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.200+0000 7fc1cb7fe640 1 -- 172.21.15.76:0/2781521639 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc1c4053740 con 0x7fc1d41018f0 2024-03-18T14:51:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:08 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2488485640' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T14:51:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:08 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] up:active 2024-03-18T14:51:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:08 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-18T14:51:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:08 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:51:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:08 smithi077 ceph-mon[35865]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:51:08.227 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 14:51:07 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-b[66028]: 2024-03-18T14:51:07.908+0000 7f572e14b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T14:51:08.227 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 14:51:07 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-b[66028]: 2024-03-18T14:51:07.908+0000 7f572e14b700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-18T14:51:08.228 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 14:51:07 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-b[66028]: 2024-03-18T14:51:07.908+0000 7f572e14b700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-18T14:51:08.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.402+0000 7fc1dc680640 1 -- 172.21.15.76:0/2781521639 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc1d4061c20 con 0x7fc1c00a2800 2024-03-18T14:51:08.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:08 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2488485640' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T14:51:08.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:08 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] up:active 2024-03-18T14:51:08.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:08 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-18T14:51:08.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:08 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:51:08.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:08 smithi076 ceph-mon[32205]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:51:08.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.421+0000 7fc1cb7fe640 1 -- 172.21.15.76:0/2781521639 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc1d4061c20 con 0x7fc1c00a2800 2024-03-18T14:51:08.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.423+0000 7fc1dc680640 1 -- 172.21.15.76:0/2781521639 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc1c00a2800 msgr2=0x7fc1c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:08.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.423+0000 7fc1dc680640 1 --2- 172.21.15.76:0/2781521639 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc1c00a2800 0x7fc1c00a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc1d002c7b0 tx=0x7fc1d0005fd0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.423+0000 7fc1dc680640 1 -- 172.21.15.76:0/2781521639 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1d41018f0 msgr2=0x7fc1d4198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:08.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.423+0000 7fc1dc680640 1 --2- 172.21.15.76:0/2781521639 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1d41018f0 0x7fc1d4198090 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400e9f0 tx=0x7fc1c400edf0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.423+0000 7fc1dc680640 1 -- 172.21.15.76:0/2781521639 shutdown_connections 2024-03-18T14:51:08.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.423+0000 7fc1dc680640 1 --2- 172.21.15.76:0/2781521639 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc1c00a2800 0x7fc1c00a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.423+0000 7fc1dc680640 1 --2- 172.21.15.76:0/2781521639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1d41022b0 0x7fc1d419f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.423+0000 7fc1dc680640 1 --2- 172.21.15.76:0/2781521639 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1d41018f0 0x7fc1d4198090 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.423+0000 7fc1dc680640 1 --2- 172.21.15.76:0/2781521639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1d4100700 0x7fc1d4197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.423+0000 7fc1dc680640 1 -- 172.21.15.76:0/2781521639 >> 172.21.15.76:0/2781521639 conn(0x7fc1d40fbc90 msgr2=0x7fc1d4104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:08.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.423+0000 7fc1dc680640 1 -- 172.21.15.76:0/2781521639 shutdown_connections 2024-03-18T14:51:08.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.423+0000 7fc1dc680640 1 -- 172.21.15.76:0/2781521639 wait complete. 2024-03-18T14:51:08.435 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-18T14:51:08.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:08 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2488485640' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T14:51:08.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:08 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] up:active 2024-03-18T14:51:08.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:08 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-18T14:51:08.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:08 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:51:08.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:08 smithi174 ceph-mon[36787]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:51:08.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.541+0000 7f2d90412640 1 -- 172.21.15.76:0/3138147827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d881006e0 msgr2=0x7f2d88100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:08.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.541+0000 7f2d90412640 1 --2- 172.21.15.76:0/3138147827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d881006e0 0x7f2d88100b00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2d84009900 tx=0x7f2d8402bc30 comp rx=0 tx=0).stop 2024-03-18T14:51:08.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.541+0000 7f2d90412640 1 -- 172.21.15.76:0/3138147827 shutdown_connections 2024-03-18T14:51:08.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.541+0000 7f2d90412640 1 --2- 172.21.15.76:0/3138147827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d881018d0 0x7f2d88101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.541+0000 7f2d90412640 1 --2- 172.21.15.76:0/3138147827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d88102290 0x7f2d88108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.541+0000 7f2d90412640 1 --2- 172.21.15.76:0/3138147827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d881006e0 0x7f2d88100b00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.541+0000 7f2d90412640 1 -- 172.21.15.76:0/3138147827 >> 172.21.15.76:0/3138147827 conn(0x7f2d880fbc70 msgr2=0x7f2d880fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.541+0000 7f2d90412640 1 -- 172.21.15.76:0/3138147827 shutdown_connections 2024-03-18T14:51:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.541+0000 7f2d90412640 1 -- 172.21.15.76:0/3138147827 wait complete. 2024-03-18T14:51:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d90412640 1 Processor -- start 2024-03-18T14:51:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d90412640 1 -- start start 2024-03-18T14:51:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d90412640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d881006e0 0x7f2d88197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d90412640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d881018d0 0x7f2d88198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d90412640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d88102290 0x7f2d8819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d90412640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8810d480 con 0x7f2d881006e0 2024-03-18T14:51:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d90412640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8810d300 con 0x7f2d881018d0 2024-03-18T14:51:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d90412640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8810d600 con 0x7f2d88102290 2024-03-18T14:51:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d8e1ae640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d881006e0 0x7f2d88197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d8e1ae640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d881006e0 0x7f2d88197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40634/0 (socket says 172.21.15.76:40634) 2024-03-18T14:51:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d8e1ae640 1 -- 172.21.15.76:0/3445250642 learned_addr learned my addr 172.21.15.76:0/3445250642 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d8e9af640 1 --2- 172.21.15.76:0/3445250642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d88102290 0x7f2d8819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d8d9ad640 1 --2- 172.21.15.76:0/3445250642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d881018d0 0x7f2d88198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d8e1ae640 1 -- 172.21.15.76:0/3445250642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d88102290 msgr2=0x7f2d8819f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d8e1ae640 1 --2- 172.21.15.76:0/3445250642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d88102290 0x7f2d8819f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d8e1ae640 1 -- 172.21.15.76:0/3445250642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d881018d0 msgr2=0x7f2d88198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d8e1ae640 1 --2- 172.21.15.76:0/3445250642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d881018d0 0x7f2d88198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d8e1ae640 1 -- 172.21.15.76:0/3445250642 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8819f720 con 0x7f2d881006e0 2024-03-18T14:51:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.542+0000 7f2d8e1ae640 1 --2- 172.21.15.76:0/3445250642 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d881006e0 0x7f2d88197b00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2d84000c00 tx=0x7f2d8402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.543+0000 7f2d7f7fe640 1 -- 172.21.15.76:0/3445250642 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d84041070 con 0x7f2d881006e0 2024-03-18T14:51:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.543+0000 7f2d7f7fe640 1 -- 172.21.15.76:0/3445250642 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d84004450 con 0x7f2d881006e0 2024-03-18T14:51:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.543+0000 7f2d90412640 1 -- 172.21.15.76:0/3445250642 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d8819f9b0 con 0x7f2d881006e0 2024-03-18T14:51:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.543+0000 7f2d7f7fe640 1 -- 172.21.15.76:0/3445250642 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d84034440 con 0x7f2d881006e0 2024-03-18T14:51:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.543+0000 7f2d90412640 1 -- 172.21.15.76:0/3445250642 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d8819fea0 con 0x7f2d881006e0 2024-03-18T14:51:08.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.544+0000 7f2d90412640 1 -- 172.21.15.76:0/3445250642 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d88107a80 con 0x7f2d881006e0 2024-03-18T14:51:08.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.545+0000 7f2d7f7fe640 1 -- 172.21.15.76:0/3445250642 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2d84044070 con 0x7f2d881006e0 2024-03-18T14:51:08.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.545+0000 7f2d7f7fe640 1 --2- 172.21.15.76:0/3445250642 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2d740a2800 0x7f2d740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:08.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.545+0000 7f2d8d9ad640 1 --2- 172.21.15.76:0/3445250642 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2d740a2800 0x7f2d740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:08.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.546+0000 7f2d7f7fe640 1 -- 172.21.15.76:0/3445250642 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f2d840f03d0 con 0x7f2d881006e0 2024-03-18T14:51:08.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.546+0000 7f2d8d9ad640 1 --2- 172.21.15.76:0/3445250642 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2d740a2800 0x7f2d740a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2d88199010 tx=0x7f2d78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:08.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.550+0000 7f2d7f7fe640 1 -- 172.21.15.76:0/3445250642 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2d840bd000 con 0x7f2d881006e0 2024-03-18T14:51:08.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.767+0000 7f2d90412640 1 -- 172.21.15.76:0/3445250642 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f2d88061cb0 con 0x7f2d740a2800 2024-03-18T14:51:08.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.770+0000 7f2d7f7fe640 1 -- 172.21.15.76:0/3445250642 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f2d88061cb0 con 0x7f2d740a2800 2024-03-18T14:51:08.771 INFO:teuthology.orchestra.run.smithi076.stdout:/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a 2024-03-18T14:51:08.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.772+0000 7f2d90412640 1 -- 172.21.15.76:0/3445250642 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2d740a2800 msgr2=0x7f2d740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:08.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.772+0000 7f2d90412640 1 --2- 172.21.15.76:0/3445250642 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2d740a2800 0x7f2d740a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2d88199010 tx=0x7f2d78005f90 comp rx=0 tx=0).stop 2024-03-18T14:51:08.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.772+0000 7f2d90412640 1 -- 172.21.15.76:0/3445250642 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d881006e0 msgr2=0x7f2d88197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:08.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.772+0000 7f2d90412640 1 --2- 172.21.15.76:0/3445250642 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d881006e0 0x7f2d88197b00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2d84000c00 tx=0x7f2d8402c640 comp rx=0 tx=0).stop 2024-03-18T14:51:08.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.772+0000 7f2d90412640 1 -- 172.21.15.76:0/3445250642 shutdown_connections 2024-03-18T14:51:08.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.772+0000 7f2d90412640 1 --2- 172.21.15.76:0/3445250642 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2d740a2800 0x7f2d740a4cc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.772+0000 7f2d90412640 1 --2- 172.21.15.76:0/3445250642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d88102290 0x7f2d8819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.772+0000 7f2d90412640 1 --2- 172.21.15.76:0/3445250642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d881018d0 0x7f2d88198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.772+0000 7f2d90412640 1 --2- 172.21.15.76:0/3445250642 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d881006e0 0x7f2d88197b00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.772+0000 7f2d90412640 1 -- 172.21.15.76:0/3445250642 >> 172.21.15.76:0/3445250642 conn(0x7f2d880fbc70 msgr2=0x7f2d88104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:08.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.772+0000 7f2d90412640 1 -- 172.21.15.76:0/3445250642 shutdown_connections 2024-03-18T14:51:08.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.772+0000 7f2d90412640 1 -- 172.21.15.76:0/3445250642 wait complete. 2024-03-18T14:51:08.784 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:51:08.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.885+0000 7fecd2537640 1 -- 172.21.15.76:0/2759728933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feccc1022b0 msgr2=0x7feccc108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:08.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.885+0000 7fecd2537640 1 --2- 172.21.15.76:0/2759728933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feccc1022b0 0x7feccc108b40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fecc4009be0 tx=0x7fecc402bef0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.885+0000 7fecd2537640 1 -- 172.21.15.76:0/2759728933 shutdown_connections 2024-03-18T14:51:08.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.885+0000 7fecd2537640 1 --2- 172.21.15.76:0/2759728933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feccc1018f0 0x7feccc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.885+0000 7fecd2537640 1 --2- 172.21.15.76:0/2759728933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feccc1022b0 0x7feccc108b40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.885+0000 7fecd2537640 1 --2- 172.21.15.76:0/2759728933 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feccc100700 0x7feccc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.885+0000 7fecd2537640 1 -- 172.21.15.76:0/2759728933 >> 172.21.15.76:0/2759728933 conn(0x7feccc0fbc70 msgr2=0x7feccc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:08.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.885+0000 7fecd2537640 1 -- 172.21.15.76:0/2759728933 shutdown_connections 2024-03-18T14:51:08.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.885+0000 7fecd2537640 1 -- 172.21.15.76:0/2759728933 wait complete. 2024-03-18T14:51:08.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.885+0000 7fecd2537640 1 Processor -- start 2024-03-18T14:51:08.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.885+0000 7fecd2537640 1 -- start start 2024-03-18T14:51:08.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.885+0000 7fecd2537640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feccc100700 0x7feccc197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:08.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecd2537640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feccc1018f0 0x7feccc197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:08.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecd2537640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feccc1022b0 0x7feccc19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:08.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecd2537640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7feccc10c360 con 0x7feccc1022b0 2024-03-18T14:51:08.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecd2537640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7feccc10c1e0 con 0x7feccc100700 2024-03-18T14:51:08.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecd2537640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feccc10c4e0 con 0x7feccc1018f0 2024-03-18T14:51:08.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecd0ad4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feccc1022b0 0x7feccc19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:08.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecd0ad4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feccc1022b0 0x7feccc19f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40648/0 (socket says 172.21.15.76:40648) 2024-03-18T14:51:08.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecd0ad4640 1 -- 172.21.15.76:0/598636241 learned_addr learned my addr 172.21.15.76:0/598636241 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:08.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7feccbfff640 1 --2- 172.21.15.76:0/598636241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feccc100700 0x7feccc197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:08.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7feccb7fe640 1 --2- 172.21.15.76:0/598636241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feccc1018f0 0x7feccc197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:08.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecd0ad4640 1 -- 172.21.15.76:0/598636241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feccc1018f0 msgr2=0x7feccc197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:08.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecd0ad4640 1 --2- 172.21.15.76:0/598636241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feccc1018f0 0x7feccc197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecd0ad4640 1 -- 172.21.15.76:0/598636241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feccc100700 msgr2=0x7feccc197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:08.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecd0ad4640 1 --2- 172.21.15.76:0/598636241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feccc100700 0x7feccc197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:08.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecd0ad4640 1 -- 172.21.15.76:0/598636241 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feccc19f6c0 con 0x7feccc1022b0 2024-03-18T14:51:08.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecd0ad4640 1 --2- 172.21.15.76:0/598636241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feccc1022b0 0x7feccc19f050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fecc4036040 tx=0x7fecc40337b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:08.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecc97fa640 1 -- 172.21.15.76:0/598636241 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc4043020 con 0x7feccc1022b0 2024-03-18T14:51:08.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecd2537640 1 -- 172.21.15.76:0/598636241 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feccc19f950 con 0x7feccc1022b0 2024-03-18T14:51:08.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecc97fa640 1 -- 172.21.15.76:0/598636241 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecc402ca60 con 0x7feccc1022b0 2024-03-18T14:51:08.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecc97fa640 1 -- 172.21.15.76:0/598636241 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc4034690 con 0x7feccc1022b0 2024-03-18T14:51:08.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.886+0000 7fecd2537640 1 -- 172.21.15.76:0/598636241 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feccc19fea0 con 0x7feccc1022b0 2024-03-18T14:51:08.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.887+0000 7fecd2537640 1 -- 172.21.15.76:0/598636241 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feccc0685d0 con 0x7feccc1022b0 2024-03-18T14:51:08.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.888+0000 7fecc97fa640 1 -- 172.21.15.76:0/598636241 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fecc4046020 con 0x7feccc1022b0 2024-03-18T14:51:08.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.889+0000 7fecc97fa640 1 --2- 172.21.15.76:0/598636241 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fecb40a2800 0x7fecb40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:08.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.889+0000 7feccbfff640 1 --2- 172.21.15.76:0/598636241 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fecb40a2800 0x7fecb40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:08.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.889+0000 7fecc97fa640 1 -- 172.21.15.76:0/598636241 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fecc40f0f10 con 0x7feccc1022b0 2024-03-18T14:51:08.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.889+0000 7feccbfff640 1 --2- 172.21.15.76:0/598636241 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fecb40a2800 0x7fecb40a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7feccc101750 tx=0x7fecc0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:08.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:08.893+0000 7fecc97fa640 1 -- 172.21.15.76:0/598636241 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fecc404d020 con 0x7feccc1022b0 2024-03-18T14:51:09.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.043+0000 7fecc97fa640 1 -- 172.21.15.76:0/598636241 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fecc40ba070 con 0x7feccc1022b0 2024-03-18T14:51:09.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.115+0000 7fecd2537640 1 -- 172.21.15.76:0/598636241 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feccc1a0280 con 0x7feccc1022b0 2024-03-18T14:51:09.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.116+0000 7fecc97fa640 1 -- 172.21.15.76:0/598636241 <== mon.0 v2:172.21.15.76:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11863 (secure 0 0 0) 0x7fecc40be100 con 0x7feccc1022b0 2024-03-18T14:51:09.117 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:51:09.117 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/2801069369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":2801069369},{"type":"v1","addr":"172.21.15.76:6839","nonce":2801069369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2645229232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2645229232},{"type":"v1","addr":"172.21.15.76:6841","nonce":2645229232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T14:51:01.293009+0000","modified":"2024-03-18T14:51:07.032425+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24490,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6839/399810652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":399810652},{"type":"v1","addr":"172.21.15.174:6839","nonce":399810652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1685041985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1685041985},{"type":"v1","addr":"172.21.15.77:6839","nonce":1685041985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24463,24490,24524]},"id":1}]} 2024-03-18T14:51:09.117 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 23 2024-03-18T14:51:09.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.118+0000 7fecd2537640 1 -- 172.21.15.76:0/598636241 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fecb40a2800 msgr2=0x7fecb40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:09.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.118+0000 7fecd2537640 1 --2- 172.21.15.76:0/598636241 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fecb40a2800 0x7fecb40a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7feccc101750 tx=0x7fecc0009260 comp rx=0 tx=0).stop 2024-03-18T14:51:09.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.118+0000 7fecd2537640 1 -- 172.21.15.76:0/598636241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feccc1022b0 msgr2=0x7feccc19f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:09.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.118+0000 7fecd2537640 1 --2- 172.21.15.76:0/598636241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feccc1022b0 0x7feccc19f050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fecc4036040 tx=0x7fecc40337b0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.118+0000 7fecd2537640 1 -- 172.21.15.76:0/598636241 shutdown_connections 2024-03-18T14:51:09.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.118+0000 7fecd2537640 1 --2- 172.21.15.76:0/598636241 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fecb40a2800 0x7fecb40a4cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.118+0000 7fecd2537640 1 --2- 172.21.15.76:0/598636241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feccc1018f0 0x7feccc197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.118+0000 7fecd2537640 1 --2- 172.21.15.76:0/598636241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feccc1022b0 0x7feccc19f050 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.118+0000 7fecd2537640 1 --2- 172.21.15.76:0/598636241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feccc100700 0x7feccc197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.118+0000 7fecd2537640 1 -- 172.21.15.76:0/598636241 >> 172.21.15.76:0/598636241 conn(0x7feccc0fbc70 msgr2=0x7feccc0fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:09.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.118+0000 7fecd2537640 1 -- 172.21.15.76:0/598636241 shutdown_connections 2024-03-18T14:51:09.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.118+0000 7fecd2537640 1 -- 172.21.15.76:0/598636241 wait complete. 2024-03-18T14:51:09.130 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:51:09.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.231+0000 7f5dfeb89640 1 -- 172.21.15.76:0/2559457008 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5df8101a00 msgr2=0x7f5df8108320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:09.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.231+0000 7f5dfeb89640 1 --2- 172.21.15.76:0/2559457008 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5df8101a00 0x7f5df8108320 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5de0009900 tx=0x7f5de002bc30 comp rx=0 tx=0).stop 2024-03-18T14:51:09.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.231+0000 7f5dfeb89640 1 -- 172.21.15.76:0/2559457008 shutdown_connections 2024-03-18T14:51:09.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.231+0000 7f5dfeb89640 1 --2- 172.21.15.76:0/2559457008 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5df8101a00 0x7f5df8108320 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.231+0000 7f5dfeb89640 1 --2- 172.21.15.76:0/2559457008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5df8101040 0x7f5df81014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.231+0000 7f5dfeb89640 1 --2- 172.21.15.76:0/2559457008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5df80ffe60 0x7f5df8100280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.231+0000 7f5dfeb89640 1 -- 172.21.15.76:0/2559457008 >> 172.21.15.76:0/2559457008 conn(0x7f5df80fb420 msgr2=0x7f5df80fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:09.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.231+0000 7f5dfeb89640 1 -- 172.21.15.76:0/2559457008 shutdown_connections 2024-03-18T14:51:09.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.231+0000 7f5dfeb89640 1 -- 172.21.15.76:0/2559457008 wait complete. 2024-03-18T14:51:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.231+0000 7f5dfeb89640 1 Processor -- start 2024-03-18T14:51:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.231+0000 7f5dfeb89640 1 -- start start 2024-03-18T14:51:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5dfeb89640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5df80ffe60 0x7f5df8196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5dfeb89640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5df8101040 0x7f5df8196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5dfeb89640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5df8101a00 0x7f5df819dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5dfeb89640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5df810c3d0 con 0x7f5df80ffe60 2024-03-18T14:51:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5dfeb89640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5df810c250 con 0x7f5df8101040 2024-03-18T14:51:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5dfeb89640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5df810c550 con 0x7f5df8101a00 2024-03-18T14:51:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5dfc925640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5df80ffe60 0x7f5df8196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5dfc925640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5df80ffe60 0x7f5df8196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40660/0 (socket says 172.21.15.76:40660) 2024-03-18T14:51:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5dfc925640 1 -- 172.21.15.76:0/1327407993 learned_addr learned my addr 172.21.15.76:0/1327407993 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5df7fff640 1 --2- 172.21.15.76:0/1327407993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5df8101040 0x7f5df8196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5dfd126640 1 --2- 172.21.15.76:0/1327407993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5df8101a00 0x7f5df819dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5dfc925640 1 -- 172.21.15.76:0/1327407993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5df8101a00 msgr2=0x7f5df819dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5dfc925640 1 --2- 172.21.15.76:0/1327407993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5df8101a00 0x7f5df819dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5dfc925640 1 -- 172.21.15.76:0/1327407993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5df8101040 msgr2=0x7f5df8196f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:09.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5dfc925640 1 --2- 172.21.15.76:0/1327407993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5df8101040 0x7f5df8196f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5dfc925640 1 -- 172.21.15.76:0/1327407993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5df819e630 con 0x7f5df80ffe60 2024-03-18T14:51:09.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5df7fff640 1 --2- 172.21.15.76:0/1327407993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5df8101040 0x7f5df8196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:09.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5dfc925640 1 --2- 172.21.15.76:0/1327407993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5df80ffe60 0x7f5df8196a10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5df000eb70 tx=0x7f5df000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:09.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5df5ffb640 1 -- 172.21.15.76:0/1327407993 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df0003b70 con 0x7f5df80ffe60 2024-03-18T14:51:09.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5df5ffb640 1 -- 172.21.15.76:0/1327407993 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5df000ee70 con 0x7f5df80ffe60 2024-03-18T14:51:09.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5dfeb89640 1 -- 172.21.15.76:0/1327407993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5df819e920 con 0x7f5df80ffe60 2024-03-18T14:51:09.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.232+0000 7f5df5ffb640 1 -- 172.21.15.76:0/1327407993 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df0010830 con 0x7f5df80ffe60 2024-03-18T14:51:09.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.233+0000 7f5dfeb89640 1 -- 172.21.15.76:0/1327407993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5df819ee70 con 0x7f5df80ffe60 2024-03-18T14:51:09.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.234+0000 7f5dfeb89640 1 -- 172.21.15.76:0/1327407993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5df8107200 con 0x7f5df80ffe60 2024-03-18T14:51:09.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.234+0000 7f5df5ffb640 1 -- 172.21.15.76:0/1327407993 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5df0004020 con 0x7f5df80ffe60 2024-03-18T14:51:09.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.235+0000 7f5df5ffb640 1 --2- 172.21.15.76:0/1327407993 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5de40a2910 0x7f5de40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:09.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.235+0000 7f5df7fff640 1 --2- 172.21.15.76:0/1327407993 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5de40a2910 0x7f5de40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:09.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.235+0000 7f5df5ffb640 1 -- 172.21.15.76:0/1327407993 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f5df00cf030 con 0x7f5df80ffe60 2024-03-18T14:51:09.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.235+0000 7f5df7fff640 1 --2- 172.21.15.76:0/1327407993 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5de40a2910 0x7f5de40a4dd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5df8197f20 tx=0x7f5de8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:09.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.239+0000 7f5df5ffb640 1 -- 172.21.15.76:0/1327407993 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5df009c120 con 0x7f5df80ffe60 2024-03-18T14:51:09.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:09 smithi076 ceph-mon[32205]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:51:09.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:09 smithi076 ceph-mon[32205]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:51:09.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:09 smithi076 ceph-mon[32205]: pgmap v287: 97 pgs: 97 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 690 B/s rd, 3.6 KiB/s wr, 14 op/s 2024-03-18T14:51:09.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:09 smithi076 ceph-mon[32205]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:51:09.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.422+0000 7f5dfeb89640 1 -- 172.21.15.76:0/1327407993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5df802d1b0 con 0x7f5df80ffe60 2024-03-18T14:51:09.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.424+0000 7f5df5ffb640 1 -- 172.21.15.76:0/1327407993 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22192 (secure 0 0 0) 0x7f5df009ba20 con 0x7f5df80ffe60 2024-03-18T14:51:09.425 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:51:09.425 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":86,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:51:02.338985+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:51:09.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.426+0000 7f5dfeb89640 1 -- 172.21.15.76:0/1327407993 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5de40a2910 msgr2=0x7f5de40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:09.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.426+0000 7f5dfeb89640 1 --2- 172.21.15.76:0/1327407993 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5de40a2910 0x7f5de40a4dd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5df8197f20 tx=0x7f5de8005f90 comp rx=0 tx=0).stop 2024-03-18T14:51:09.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.426+0000 7f5dfeb89640 1 -- 172.21.15.76:0/1327407993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5df80ffe60 msgr2=0x7f5df8196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:09.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.426+0000 7f5dfeb89640 1 --2- 172.21.15.76:0/1327407993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5df80ffe60 0x7f5df8196a10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5df000eb70 tx=0x7f5df000c6a0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.426+0000 7f5dfeb89640 1 -- 172.21.15.76:0/1327407993 shutdown_connections 2024-03-18T14:51:09.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.426+0000 7f5dfeb89640 1 --2- 172.21.15.76:0/1327407993 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5de40a2910 0x7f5de40a4dd0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.426+0000 7f5dfeb89640 1 --2- 172.21.15.76:0/1327407993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5df8101a00 0x7f5df819dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.426+0000 7f5dfeb89640 1 --2- 172.21.15.76:0/1327407993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5df8101040 0x7f5df8196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.426+0000 7f5dfeb89640 1 --2- 172.21.15.76:0/1327407993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5df80ffe60 0x7f5df8196a10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.426+0000 7f5dfeb89640 1 -- 172.21.15.76:0/1327407993 >> 172.21.15.76:0/1327407993 conn(0x7f5df80fb420 msgr2=0x7f5df81043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:09.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.426+0000 7f5dfeb89640 1 -- 172.21.15.76:0/1327407993 shutdown_connections 2024-03-18T14:51:09.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.426+0000 7f5dfeb89640 1 -- 172.21.15.76:0/1327407993 wait complete. 2024-03-18T14:51:09.438 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T14:51:09.438 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:51:09.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:09 smithi174 ceph-mon[36787]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:51:09.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:09 smithi174 ceph-mon[36787]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:51:09.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:09 smithi174 ceph-mon[36787]: pgmap v287: 97 pgs: 97 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 690 B/s rd, 3.6 KiB/s wr, 14 op/s 2024-03-18T14:51:09.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:09 smithi174 ceph-mon[36787]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:51:09.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:09 smithi077 ceph-mon[35865]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:51:09.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:09 smithi077 ceph-mon[35865]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:51:09.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:09 smithi077 ceph-mon[35865]: pgmap v287: 97 pgs: 97 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 690 B/s rd, 3.6 KiB/s wr, 14 op/s 2024-03-18T14:51:09.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:09 smithi077 ceph-mon[35865]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:51:09.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.539+0000 7f11dea85640 1 -- 172.21.15.76:0/4183738939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81033b0 msgr2=0x7f11d8109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:09.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.539+0000 7f11dea85640 1 --2- 172.21.15.76:0/4183738939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81033b0 0x7f11d8109c40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f11d0009900 tx=0x7f11d002bc30 comp rx=0 tx=0).stop 2024-03-18T14:51:09.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.539+0000 7f11dea85640 1 -- 172.21.15.76:0/4183738939 shutdown_connections 2024-03-18T14:51:09.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.539+0000 7f11dea85640 1 --2- 172.21.15.76:0/4183738939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81033b0 0x7f11d8109c40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.539+0000 7f11dea85640 1 --2- 172.21.15.76:0/4183738939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11d81029f0 0x7f11d8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.539+0000 7f11dea85640 1 --2- 172.21.15.76:0/4183738939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11d8101800 0x7f11d8101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.539+0000 7f11dea85640 1 -- 172.21.15.76:0/4183738939 >> 172.21.15.76:0/4183738939 conn(0x7f11d80fcd90 msgr2=0x7f11d80ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.539+0000 7f11dea85640 1 -- 172.21.15.76:0/4183738939 shutdown_connections 2024-03-18T14:51:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.539+0000 7f11dea85640 1 -- 172.21.15.76:0/4183738939 wait complete. 2024-03-18T14:51:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11dea85640 1 Processor -- start 2024-03-18T14:51:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11dea85640 1 -- start start 2024-03-18T14:51:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11dea85640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11d8101800 0x7f11d8197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11dea85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11d81029f0 0x7f11d8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11dea85640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81033b0 0x7f11d819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11dea85640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f11d810d480 con 0x7f11d81033b0 2024-03-18T14:51:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11dea85640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f11d810d300 con 0x7f11d81029f0 2024-03-18T14:51:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11dea85640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f11d810d600 con 0x7f11d8101800 2024-03-18T14:51:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11dd022640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81033b0 0x7f11d819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11dd022640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81033b0 0x7f11d819f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40688/0 (socket says 172.21.15.76:40688) 2024-03-18T14:51:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11dd022640 1 -- 172.21.15.76:0/1489847305 learned_addr learned my addr 172.21.15.76:0/1489847305 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11dc821640 1 --2- 172.21.15.76:0/1489847305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11d8101800 0x7f11d8197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11d7fff640 1 --2- 172.21.15.76:0/1489847305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11d81029f0 0x7f11d8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11dd022640 1 -- 172.21.15.76:0/1489847305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11d8101800 msgr2=0x7f11d8197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11dd022640 1 --2- 172.21.15.76:0/1489847305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11d8101800 0x7f11d8197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11dd022640 1 -- 172.21.15.76:0/1489847305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11d81029f0 msgr2=0x7f11d8197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11dd022640 1 --2- 172.21.15.76:0/1489847305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11d81029f0 0x7f11d8197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.540+0000 7f11dd022640 1 -- 172.21.15.76:0/1489847305 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d819f6b0 con 0x7f11d81033b0 2024-03-18T14:51:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.541+0000 7f11dd022640 1 --2- 172.21.15.76:0/1489847305 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81033b0 0x7f11d819f040 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f11d0005ab0 tx=0x7f11d002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.541+0000 7f11d5ffb640 1 -- 172.21.15.76:0/1489847305 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0041070 con 0x7f11d81033b0 2024-03-18T14:51:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.541+0000 7f11dea85640 1 -- 172.21.15.76:0/1489847305 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d819f940 con 0x7f11d81033b0 2024-03-18T14:51:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.541+0000 7f11d5ffb640 1 -- 172.21.15.76:0/1489847305 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11d0004450 con 0x7f11d81033b0 2024-03-18T14:51:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.541+0000 7f11d5ffb640 1 -- 172.21.15.76:0/1489847305 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0003e60 con 0x7f11d81033b0 2024-03-18T14:51:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.541+0000 7f11dea85640 1 -- 172.21.15.76:0/1489847305 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d819fe80 con 0x7f11d81033b0 2024-03-18T14:51:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.542+0000 7f11dea85640 1 -- 172.21.15.76:0/1489847305 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d8108b20 con 0x7f11d81033b0 2024-03-18T14:51:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.543+0000 7f11d5ffb640 1 -- 172.21.15.76:0/1489847305 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f11d0044070 con 0x7f11d81033b0 2024-03-18T14:51:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.543+0000 7f11d5ffb640 1 --2- 172.21.15.76:0/1489847305 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f11c40a2890 0x7f11c40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.544+0000 7f11dc821640 1 --2- 172.21.15.76:0/1489847305 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f11c40a2890 0x7f11c40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.544+0000 7f11d5ffb640 1 -- 172.21.15.76:0/1489847305 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f11d00f0540 con 0x7f11d81033b0 2024-03-18T14:51:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.544+0000 7f11dc821640 1 --2- 172.21.15.76:0/1489847305 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f11c40a2890 0x7f11c40a4d50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f11d8102850 tx=0x7f11c8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.548+0000 7f11d5ffb640 1 -- 172.21.15.76:0/1489847305 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f11d00bd0f0 con 0x7f11d81033b0 2024-03-18T14:51:09.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.722+0000 7f11dea85640 1 -- 172.21.15.76:0/1489847305 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f11d8061c20 con 0x7f11c40a2890 2024-03-18T14:51:09.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.730+0000 7f11d5ffb640 1 -- 172.21.15.76:0/1489847305 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258899 (secure 0 0 0) 0x7f11d8061c20 con 0x7f11c40a2890 2024-03-18T14:51:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:51:09.735 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-03-18T14:51:09.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.733+0000 7f11dea85640 1 -- 172.21.15.76:0/1489847305 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f11c40a2890 msgr2=0x7f11c40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:09.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.733+0000 7f11dea85640 1 --2- 172.21.15.76:0/1489847305 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f11c40a2890 0x7f11c40a4d50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f11d8102850 tx=0x7f11c8009500 comp rx=0 tx=0).stop 2024-03-18T14:51:09.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.733+0000 7f11dea85640 1 -- 172.21.15.76:0/1489847305 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81033b0 msgr2=0x7f11d819f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:09.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.733+0000 7f11dea85640 1 --2- 172.21.15.76:0/1489847305 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81033b0 0x7f11d819f040 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f11d0005ab0 tx=0x7f11d002c6c0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.733+0000 7f11dea85640 1 -- 172.21.15.76:0/1489847305 shutdown_connections 2024-03-18T14:51:09.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.733+0000 7f11dea85640 1 --2- 172.21.15.76:0/1489847305 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f11c40a2890 0x7f11c40a4d50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.733+0000 7f11dea85640 1 --2- 172.21.15.76:0/1489847305 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81033b0 0x7f11d819f040 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.733+0000 7f11dea85640 1 --2- 172.21.15.76:0/1489847305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11d81029f0 0x7f11d8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.733+0000 7f11dea85640 1 --2- 172.21.15.76:0/1489847305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11d8101800 0x7f11d8197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.733+0000 7f11dea85640 1 -- 172.21.15.76:0/1489847305 >> 172.21.15.76:0/1489847305 conn(0x7f11d80fcd90 msgr2=0x7f11d8105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:09.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.733+0000 7f11dea85640 1 -- 172.21.15.76:0/1489847305 shutdown_connections 2024-03-18T14:51:09.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.733+0000 7f11dea85640 1 -- 172.21.15.76:0/1489847305 wait complete. 2024-03-18T14:51:09.745 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-03-18T14:51:08.330357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594766,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":142,"num_read_kb":124,"num_write":285,"num_write_kb":2965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368628,"kb_used_data":11100,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124573196,"statfs":{"total":1151940427776,"available":1151562952704,"internally_reserved":0,"allocated":11366400,"data_stored":7086752,"data_compressed":107082,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4398,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":104,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.412889"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212862+0000","last_change":"2024-03-18T14:51:01.365948+0000","last_active":"2024-03-18T14:51:03.212862+0000","last_peered":"2024-03-18T14:51:03.212862+0000","last_clean":"2024-03-18T14:51:03.212862+0000","last_became_active":"2024-03-18T14:51:01.364600+0000","last_became_peered":"2024-03-18T14:51:01.364600+0000","last_unstale":"2024-03-18T14:51:03.212862+0000","last_undegraded":"2024-03-18T14:51:03.212862+0000","last_fullsized":"2024-03-18T14:51:03.212862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:14:45.198618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212439+0000","last_change":"2024-03-18T14:51:01.360186+0000","last_active":"2024-03-18T14:51:03.212439+0000","last_peered":"2024-03-18T14:51:03.212439+0000","last_clean":"2024-03-18T14:51:03.212439+0000","last_became_active":"2024-03-18T14:51:01.359862+0000","last_became_peered":"2024-03-18T14:51:01.359862+0000","last_unstale":"2024-03-18T14:51:03.212439+0000","last_undegraded":"2024-03-18T14:51:03.212439+0000","last_fullsized":"2024-03-18T14:51:03.212439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:46.206823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212349+0000","last_change":"2024-03-18T14:51:01.364635+0000","last_active":"2024-03-18T14:51:03.212349+0000","last_peered":"2024-03-18T14:51:03.212349+0000","last_clean":"2024-03-18T14:51:03.212349+0000","last_became_active":"2024-03-18T14:51:01.364320+0000","last_became_peered":"2024-03-18T14:51:01.364320+0000","last_unstale":"2024-03-18T14:51:03.212349+0000","last_undegraded":"2024-03-18T14:51:03.212349+0000","last_fullsized":"2024-03-18T14:51:03.212349+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:08:20.227639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212416+0000","last_change":"2024-03-18T14:51:01.365103+0000","last_active":"2024-03-18T14:51:03.212416+0000","last_peered":"2024-03-18T14:51:03.212416+0000","last_clean":"2024-03-18T14:51:03.212416+0000","last_became_active":"2024-03-18T14:51:01.364750+0000","last_became_peered":"2024-03-18T14:51:01.364750+0000","last_unstale":"2024-03-18T14:51:03.212416+0000","last_undegraded":"2024-03-18T14:51:03.212416+0000","last_fullsized":"2024-03-18T14:51:03.212416+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:47:33.849019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.092728+0000","last_change":"2024-03-18T14:51:01.356839+0000","last_active":"2024-03-18T14:51:03.092728+0000","last_peered":"2024-03-18T14:51:03.092728+0000","last_clean":"2024-03-18T14:51:03.092728+0000","last_became_active":"2024-03-18T14:51:01.356531+0000","last_became_peered":"2024-03-18T14:51:01.356531+0000","last_unstale":"2024-03-18T14:51:03.092728+0000","last_undegraded":"2024-03-18T14:51:03.092728+0000","last_fullsized":"2024-03-18T14:51:03.092728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:28.499573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212558+0000","last_change":"2024-03-18T14:51:01.362912+0000","last_active":"2024-03-18T14:51:03.212558+0000","last_peered":"2024-03-18T14:51:03.212558+0000","last_clean":"2024-03-18T14:51:03.212558+0000","last_became_active":"2024-03-18T14:51:01.362590+0000","last_became_peered":"2024-03-18T14:51:01.362590+0000","last_unstale":"2024-03-18T14:51:03.212558+0000","last_undegraded":"2024-03-18T14:51:03.212558+0000","last_fullsized":"2024-03-18T14:51:03.212558+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:49.199616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212123+0000","last_change":"2024-03-18T14:51:01.358652+0000","last_active":"2024-03-18T14:51:03.212123+0000","last_peered":"2024-03-18T14:51:03.212123+0000","last_clean":"2024-03-18T14:51:03.212123+0000","last_became_active":"2024-03-18T14:51:01.358312+0000","last_became_peered":"2024-03-18T14:51:01.358312+0000","last_unstale":"2024-03-18T14:51:03.212123+0000","last_undegraded":"2024-03-18T14:51:03.212123+0000","last_fullsized":"2024-03-18T14:51:03.212123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:40.150240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.342639+0000","last_change":"2024-03-18T14:51:01.360082+0000","last_active":"2024-03-18T14:51:02.342639+0000","last_peered":"2024-03-18T14:51:02.342639+0000","last_clean":"2024-03-18T14:51:02.342639+0000","last_became_active":"2024-03-18T14:51:01.359769+0000","last_became_peered":"2024-03-18T14:51:01.359769+0000","last_unstale":"2024-03-18T14:51:02.342639+0000","last_undegraded":"2024-03-18T14:51:02.342639+0000","last_fullsized":"2024-03-18T14:51:02.342639+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:53:27.081546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345212+0000","last_change":"2024-03-18T14:51:01.363947+0000","last_active":"2024-03-18T14:51:02.345212+0000","last_peered":"2024-03-18T14:51:02.345212+0000","last_clean":"2024-03-18T14:51:02.345212+0000","last_became_active":"2024-03-18T14:51:01.363106+0000","last_became_peered":"2024-03-18T14:51:01.363106+0000","last_unstale":"2024-03-18T14:51:02.345212+0000","last_undegraded":"2024-03-18T14:51:02.345212+0000","last_fullsized":"2024-03-18T14:51:02.345212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:24:50.478426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213560+0000","last_change":"2024-03-18T14:51:01.357950+0000","last_active":"2024-03-18T14:51:03.213560+0000","last_peered":"2024-03-18T14:51:03.213560+0000","last_clean":"2024-03-18T14:51:03.213560+0000","last_became_active":"2024-03-18T14:51:01.357651+0000","last_became_peered":"2024-03-18T14:51:01.357651+0000","last_unstale":"2024-03-18T14:51:03.213560+0000","last_undegraded":"2024-03-18T14:51:03.213560+0000","last_fullsized":"2024-03-18T14:51:03.213560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:49:01.376008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213271+0000","last_change":"2024-03-18T14:51:01.360069+0000","last_active":"2024-03-18T14:51:03.213271+0000","last_peered":"2024-03-18T14:51:03.213271+0000","last_clean":"2024-03-18T14:51:03.213271+0000","last_became_active":"2024-03-18T14:51:01.358960+0000","last_became_peered":"2024-03-18T14:51:01.358960+0000","last_unstale":"2024-03-18T14:51:03.213271+0000","last_undegraded":"2024-03-18T14:51:03.213271+0000","last_fullsized":"2024-03-18T14:51:03.213271+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:48:42.197329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213054+0000","last_change":"2024-03-18T14:51:01.361200+0000","last_active":"2024-03-18T14:51:03.213054+0000","last_peered":"2024-03-18T14:51:03.213054+0000","last_clean":"2024-03-18T14:51:03.213054+0000","last_became_active":"2024-03-18T14:51:01.360879+0000","last_became_peered":"2024-03-18T14:51:01.360879+0000","last_unstale":"2024-03-18T14:51:03.213054+0000","last_undegraded":"2024-03-18T14:51:03.213054+0000","last_fullsized":"2024-03-18T14:51:03.213054+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:05.335627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213085+0000","last_change":"2024-03-18T14:51:01.352441+0000","last_active":"2024-03-18T14:51:03.213085+0000","last_peered":"2024-03-18T14:51:03.213085+0000","last_clean":"2024-03-18T14:51:03.213085+0000","last_became_active":"2024-03-18T14:51:01.352118+0000","last_became_peered":"2024-03-18T14:51:01.352118+0000","last_unstale":"2024-03-18T14:51:03.213085+0000","last_undegraded":"2024-03-18T14:51:03.213085+0000","last_fullsized":"2024-03-18T14:51:03.213085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:45:42.781439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.211890+0000","last_change":"2024-03-18T14:51:01.361257+0000","last_active":"2024-03-18T14:51:03.211890+0000","last_peered":"2024-03-18T14:51:03.211890+0000","last_clean":"2024-03-18T14:51:03.211890+0000","last_became_active":"2024-03-18T14:51:01.360898+0000","last_became_peered":"2024-03-18T14:51:01.360898+0000","last_unstale":"2024-03-18T14:51:03.211890+0000","last_undegraded":"2024-03-18T14:51:03.211890+0000","last_fullsized":"2024-03-18T14:51:03.211890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:38:26.200678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.211851+0000","last_change":"2024-03-18T14:51:01.366131+0000","last_active":"2024-03-18T14:51:03.211851+0000","last_peered":"2024-03-18T14:51:03.211851+0000","last_clean":"2024-03-18T14:51:03.211851+0000","last_became_active":"2024-03-18T14:51:01.365796+0000","last_became_peered":"2024-03-18T14:51:01.365796+0000","last_unstale":"2024-03-18T14:51:03.211851+0000","last_undegraded":"2024-03-18T14:51:03.211851+0000","last_fullsized":"2024-03-18T14:51:03.211851+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:57:29.526032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.344720+0000","last_change":"2024-03-18T14:51:01.357223+0000","last_active":"2024-03-18T14:51:02.344720+0000","last_peered":"2024-03-18T14:51:02.344720+0000","last_clean":"2024-03-18T14:51:02.344720+0000","last_became_active":"2024-03-18T14:51:01.356934+0000","last_became_peered":"2024-03-18T14:51:01.356934+0000","last_unstale":"2024-03-18T14:51:02.344720+0000","last_undegraded":"2024-03-18T14:51:02.344720+0000","last_fullsized":"2024-03-18T14:51:02.344720+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:28.683605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213063+0000","last_change":"2024-03-18T14:51:01.366414+0000","last_active":"2024-03-18T14:51:03.213063+0000","last_peered":"2024-03-18T14:51:03.213063+0000","last_clean":"2024-03-18T14:51:03.213063+0000","last_became_active":"2024-03-18T14:51:01.365534+0000","last_became_peered":"2024-03-18T14:51:01.365534+0000","last_unstale":"2024-03-18T14:51:03.213063+0000","last_undegraded":"2024-03-18T14:51:03.213063+0000","last_fullsized":"2024-03-18T14:51:03.213063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:28:51.568662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212484+0000","last_change":"2024-03-18T14:51:01.354084+0000","last_active":"2024-03-18T14:51:03.212484+0000","last_peered":"2024-03-18T14:51:03.212484+0000","last_clean":"2024-03-18T14:51:03.212484+0000","last_became_active":"2024-03-18T14:51:01.353599+0000","last_became_peered":"2024-03-18T14:51:01.353599+0000","last_unstale":"2024-03-18T14:51:03.212484+0000","last_undegraded":"2024-03-18T14:51:03.212484+0000","last_fullsized":"2024-03-18T14:51:03.212484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:40.502783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212538+0000","last_change":"2024-03-18T14:51:01.353391+0000","last_active":"2024-03-18T14:51:03.212538+0000","last_peered":"2024-03-18T14:51:03.212538+0000","last_clean":"2024-03-18T14:51:03.212538+0000","last_became_active":"2024-03-18T14:51:01.353084+0000","last_became_peered":"2024-03-18T14:51:01.353084+0000","last_unstale":"2024-03-18T14:51:03.212538+0000","last_undegraded":"2024-03-18T14:51:03.212538+0000","last_fullsized":"2024-03-18T14:51:03.212538+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:18:02.774589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:06.036513+0000","last_change":"2024-03-18T14:51:00.942617+0000","last_active":"2024-03-18T14:51:06.036513+0000","last_peered":"2024-03-18T14:51:06.036513+0000","last_clean":"2024-03-18T14:51:06.036513+0000","last_became_active":"2024-03-18T14:51:00.942276+0000","last_became_peered":"2024-03-18T14:51:00.942276+0000","last_unstale":"2024-03-18T14:51:06.036513+0000","last_undegraded":"2024-03-18T14:51:06.036513+0000","last_fullsized":"2024-03-18T14:51:06.036513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:03:05.984066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.343269+0000","last_change":"2024-03-18T14:51:01.363887+0000","last_active":"2024-03-18T14:51:02.343269+0000","last_peered":"2024-03-18T14:51:02.343269+0000","last_clean":"2024-03-18T14:51:02.343269+0000","last_became_active":"2024-03-18T14:51:01.361359+0000","last_became_peered":"2024-03-18T14:51:01.361359+0000","last_unstale":"2024-03-18T14:51:02.343269+0000","last_undegraded":"2024-03-18T14:51:02.343269+0000","last_fullsized":"2024-03-18T14:51:02.343269+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:22.996930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.343494+0000","last_change":"2024-03-18T14:51:00.940392+0000","last_active":"2024-03-18T14:51:02.343494+0000","last_peered":"2024-03-18T14:51:02.343494+0000","last_clean":"2024-03-18T14:51:02.343494+0000","last_became_active":"2024-03-18T14:51:00.940055+0000","last_became_peered":"2024-03-18T14:51:00.940055+0000","last_unstale":"2024-03-18T14:51:02.343494+0000","last_undegraded":"2024-03-18T14:51:02.343494+0000","last_fullsized":"2024-03-18T14:51:02.343494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:57:28.523549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212843+0000","last_change":"2024-03-18T14:51:01.361458+0000","last_active":"2024-03-18T14:51:03.212843+0000","last_peered":"2024-03-18T14:51:03.212843+0000","last_clean":"2024-03-18T14:51:03.212843+0000","last_became_active":"2024-03-18T14:51:01.361105+0000","last_became_peered":"2024-03-18T14:51:01.361105+0000","last_unstale":"2024-03-18T14:51:03.212843+0000","last_undegraded":"2024-03-18T14:51:03.212843+0000","last_fullsized":"2024-03-18T14:51:03.212843+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:22:59.410080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.093033+0000","last_change":"2024-03-18T14:51:00.938440+0000","last_active":"2024-03-18T14:51:03.093033+0000","last_peered":"2024-03-18T14:51:03.093033+0000","last_clean":"2024-03-18T14:51:03.093033+0000","last_became_active":"2024-03-18T14:51:00.938046+0000","last_became_peered":"2024-03-18T14:51:00.938046+0000","last_unstale":"2024-03-18T14:51:03.093033+0000","last_undegraded":"2024-03-18T14:51:03.093033+0000","last_fullsized":"2024-03-18T14:51:03.093033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:25:43.414853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.344839+0000","last_change":"2024-03-18T14:51:01.366305+0000","last_active":"2024-03-18T14:51:02.344839+0000","last_peered":"2024-03-18T14:51:02.344839+0000","last_clean":"2024-03-18T14:51:02.344839+0000","last_became_active":"2024-03-18T14:51:01.364626+0000","last_became_peered":"2024-03-18T14:51:01.364626+0000","last_unstale":"2024-03-18T14:51:02.344839+0000","last_undegraded":"2024-03-18T14:51:02.344839+0000","last_fullsized":"2024-03-18T14:51:02.344839+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:06.212030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.092844+0000","last_change":"2024-03-18T14:51:00.932514+0000","last_active":"2024-03-18T14:51:03.092844+0000","last_peered":"2024-03-18T14:51:03.092844+0000","last_clean":"2024-03-18T14:51:03.092844+0000","last_became_active":"2024-03-18T14:51:00.932183+0000","last_became_peered":"2024-03-18T14:51:00.932183+0000","last_unstale":"2024-03-18T14:51:03.092844+0000","last_undegraded":"2024-03-18T14:51:03.092844+0000","last_fullsized":"2024-03-18T14:51:03.092844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:36:37.230185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345376+0000","last_change":"2024-03-18T14:51:01.358939+0000","last_active":"2024-03-18T14:51:02.345376+0000","last_peered":"2024-03-18T14:51:02.345376+0000","last_clean":"2024-03-18T14:51:02.345376+0000","last_became_active":"2024-03-18T14:51:01.358601+0000","last_became_peered":"2024-03-18T14:51:01.358601+0000","last_unstale":"2024-03-18T14:51:02.345376+0000","last_undegraded":"2024-03-18T14:51:02.345376+0000","last_fullsized":"2024-03-18T14:51:02.345376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:40:14.876143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.029051+0000","last_change":"2024-03-18T14:51:00.945116+0000","last_active":"2024-03-18T14:51:05.029051+0000","last_peered":"2024-03-18T14:51:05.029051+0000","last_clean":"2024-03-18T14:51:05.029051+0000","last_became_active":"2024-03-18T14:51:00.944791+0000","last_became_peered":"2024-03-18T14:51:00.944791+0000","last_unstale":"2024-03-18T14:51:05.029051+0000","last_undegraded":"2024-03-18T14:51:05.029051+0000","last_fullsized":"2024-03-18T14:51:05.029051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:06.637940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.092793+0000","last_change":"2024-03-18T14:51:01.358682+0000","last_active":"2024-03-18T14:51:03.092793+0000","last_peered":"2024-03-18T14:51:03.092793+0000","last_clean":"2024-03-18T14:51:03.092793+0000","last_became_active":"2024-03-18T14:51:01.358378+0000","last_became_peered":"2024-03-18T14:51:01.358378+0000","last_unstale":"2024-03-18T14:51:03.092793+0000","last_undegraded":"2024-03-18T14:51:03.092793+0000","last_fullsized":"2024-03-18T14:51:03.092793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:42.697944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:06.035904+0000","last_change":"2024-03-18T14:51:00.943976+0000","last_active":"2024-03-18T14:51:06.035904+0000","last_peered":"2024-03-18T14:51:06.035904+0000","last_clean":"2024-03-18T14:51:06.035904+0000","last_became_active":"2024-03-18T14:51:00.943614+0000","last_became_peered":"2024-03-18T14:51:00.943614+0000","last_unstale":"2024-03-18T14:51:06.035904+0000","last_undegraded":"2024-03-18T14:51:06.035904+0000","last_fullsized":"2024-03-18T14:51:06.035904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:07:00.744708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345093+0000","last_change":"2024-03-18T14:51:01.365316+0000","last_active":"2024-03-18T14:51:02.345093+0000","last_peered":"2024-03-18T14:51:02.345093+0000","last_clean":"2024-03-18T14:51:02.345093+0000","last_became_active":"2024-03-18T14:51:01.365019+0000","last_became_peered":"2024-03-18T14:51:01.365019+0000","last_unstale":"2024-03-18T14:51:02.345093+0000","last_undegraded":"2024-03-18T14:51:02.345093+0000","last_fullsized":"2024-03-18T14:51:02.345093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:39:59.040907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212226+0000","last_change":"2024-03-18T14:51:00.936923+0000","last_active":"2024-03-18T14:51:03.212226+0000","last_peered":"2024-03-18T14:51:03.212226+0000","last_clean":"2024-03-18T14:51:03.212226+0000","last_became_active":"2024-03-18T14:51:00.936591+0000","last_became_peered":"2024-03-18T14:51:00.936591+0000","last_unstale":"2024-03-18T14:51:03.212226+0000","last_undegraded":"2024-03-18T14:51:03.212226+0000","last_fullsized":"2024-03-18T14:51:03.212226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:48:46.423033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.342666+0000","last_change":"2024-03-18T14:51:01.356827+0000","last_active":"2024-03-18T14:51:02.342666+0000","last_peered":"2024-03-18T14:51:02.342666+0000","last_clean":"2024-03-18T14:51:02.342666+0000","last_became_active":"2024-03-18T14:51:01.356148+0000","last_became_peered":"2024-03-18T14:51:01.356148+0000","last_unstale":"2024-03-18T14:51:02.342666+0000","last_undegraded":"2024-03-18T14:51:02.342666+0000","last_fullsized":"2024-03-18T14:51:02.342666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:18.477890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.344906+0000","last_change":"2024-03-18T14:51:00.934024+0000","last_active":"2024-03-18T14:51:02.344906+0000","last_peered":"2024-03-18T14:51:02.344906+0000","last_clean":"2024-03-18T14:51:02.344906+0000","last_became_active":"2024-03-18T14:51:00.933739+0000","last_became_peered":"2024-03-18T14:51:00.933739+0000","last_unstale":"2024-03-18T14:51:02.344906+0000","last_undegraded":"2024-03-18T14:51:02.344906+0000","last_fullsized":"2024-03-18T14:51:02.344906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:03.967260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212216+0000","last_change":"2024-03-18T14:51:01.356519+0000","last_active":"2024-03-18T14:51:03.212216+0000","last_peered":"2024-03-18T14:51:03.212216+0000","last_clean":"2024-03-18T14:51:03.212216+0000","last_became_active":"2024-03-18T14:51:01.356193+0000","last_became_peered":"2024-03-18T14:51:01.356193+0000","last_unstale":"2024-03-18T14:51:03.212216+0000","last_undegraded":"2024-03-18T14:51:03.212216+0000","last_fullsized":"2024-03-18T14:51:03.212216+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:34:59.633241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.026438+0000","last_change":"2024-03-18T14:51:00.940189+0000","last_active":"2024-03-18T14:51:05.026438+0000","last_peered":"2024-03-18T14:51:05.026438+0000","last_clean":"2024-03-18T14:51:05.026438+0000","last_became_active":"2024-03-18T14:51:00.939820+0000","last_became_peered":"2024-03-18T14:51:00.939820+0000","last_unstale":"2024-03-18T14:51:05.026438+0000","last_undegraded":"2024-03-18T14:51:05.026438+0000","last_fullsized":"2024-03-18T14:51:05.026438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:44:09.749747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345071+0000","last_change":"2024-03-18T14:51:01.355756+0000","last_active":"2024-03-18T14:51:02.345071+0000","last_peered":"2024-03-18T14:51:02.345071+0000","last_clean":"2024-03-18T14:51:02.345071+0000","last_became_active":"2024-03-18T14:51:01.352871+0000","last_became_peered":"2024-03-18T14:51:01.352871+0000","last_unstale":"2024-03-18T14:51:02.345071+0000","last_undegraded":"2024-03-18T14:51:02.345071+0000","last_fullsized":"2024-03-18T14:51:02.345071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:20:26.533066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.026367+0000","last_change":"2024-03-18T14:51:00.941908+0000","last_active":"2024-03-18T14:51:05.026367+0000","last_peered":"2024-03-18T14:51:05.026367+0000","last_clean":"2024-03-18T14:51:05.026367+0000","last_became_active":"2024-03-18T14:51:00.940978+0000","last_became_peered":"2024-03-18T14:51:00.940978+0000","last_unstale":"2024-03-18T14:51:05.026367+0000","last_undegraded":"2024-03-18T14:51:05.026367+0000","last_fullsized":"2024-03-18T14:51:05.026367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:52.705243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212743+0000","last_change":"2024-03-18T14:51:01.363462+0000","last_active":"2024-03-18T14:51:03.212743+0000","last_peered":"2024-03-18T14:51:03.212743+0000","last_clean":"2024-03-18T14:51:03.212743+0000","last_became_active":"2024-03-18T14:51:01.362262+0000","last_became_peered":"2024-03-18T14:51:01.362262+0000","last_unstale":"2024-03-18T14:51:03.212743+0000","last_undegraded":"2024-03-18T14:51:03.212743+0000","last_fullsized":"2024-03-18T14:51:03.212743+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:07:18.952876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345331+0000","last_change":"2024-03-18T14:51:00.936050+0000","last_active":"2024-03-18T14:51:02.345331+0000","last_peered":"2024-03-18T14:51:02.345331+0000","last_clean":"2024-03-18T14:51:02.345331+0000","last_became_active":"2024-03-18T14:51:00.935676+0000","last_became_peered":"2024-03-18T14:51:00.935676+0000","last_unstale":"2024-03-18T14:51:02.345331+0000","last_undegraded":"2024-03-18T14:51:02.345331+0000","last_fullsized":"2024-03-18T14:51:02.345331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:29.423110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345221+0000","last_change":"2024-03-18T14:51:01.364295+0000","last_active":"2024-03-18T14:51:02.345221+0000","last_peered":"2024-03-18T14:51:02.345221+0000","last_clean":"2024-03-18T14:51:02.345221+0000","last_became_active":"2024-03-18T14:51:01.363916+0000","last_became_peered":"2024-03-18T14:51:01.363916+0000","last_unstale":"2024-03-18T14:51:02.345221+0000","last_undegraded":"2024-03-18T14:51:02.345221+0000","last_fullsized":"2024-03-18T14:51:02.345221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:21:17.905819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:07.039768+0000","last_change":"2024-03-18T14:51:00.936364+0000","last_active":"2024-03-18T14:51:07.039768+0000","last_peered":"2024-03-18T14:51:07.039768+0000","last_clean":"2024-03-18T14:51:07.039768+0000","last_became_active":"2024-03-18T14:51:00.936056+0000","last_became_peered":"2024-03-18T14:51:00.936056+0000","last_unstale":"2024-03-18T14:51:07.039768+0000","last_undegraded":"2024-03-18T14:51:07.039768+0000","last_fullsized":"2024-03-18T14:51:07.039768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:48:19.044660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.092980+0000","last_change":"2024-03-18T14:51:01.362426+0000","last_active":"2024-03-18T14:51:03.092980+0000","last_peered":"2024-03-18T14:51:03.092980+0000","last_clean":"2024-03-18T14:51:03.092980+0000","last_became_active":"2024-03-18T14:51:01.362130+0000","last_became_peered":"2024-03-18T14:51:01.362130+0000","last_unstale":"2024-03-18T14:51:03.092980+0000","last_undegraded":"2024-03-18T14:51:03.092980+0000","last_fullsized":"2024-03-18T14:51:03.092980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:14.545997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:06.034524+0000","last_change":"2024-03-18T14:51:00.934028+0000","last_active":"2024-03-18T14:51:06.034524+0000","last_peered":"2024-03-18T14:51:06.034524+0000","last_clean":"2024-03-18T14:51:06.034524+0000","last_became_active":"2024-03-18T14:51:00.933594+0000","last_became_peered":"2024-03-18T14:51:00.933594+0000","last_unstale":"2024-03-18T14:51:06.034524+0000","last_undegraded":"2024-03-18T14:51:06.034524+0000","last_fullsized":"2024-03-18T14:51:06.034524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:55:08.675533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212583+0000","last_change":"2024-03-18T14:51:01.353544+0000","last_active":"2024-03-18T14:51:03.212583+0000","last_peered":"2024-03-18T14:51:03.212583+0000","last_clean":"2024-03-18T14:51:03.212583+0000","last_became_active":"2024-03-18T14:51:01.353241+0000","last_became_peered":"2024-03-18T14:51:01.353241+0000","last_unstale":"2024-03-18T14:51:03.212583+0000","last_undegraded":"2024-03-18T14:51:03.212583+0000","last_fullsized":"2024-03-18T14:51:03.212583+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:45:27.617681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.026529+0000","last_change":"2024-03-18T14:51:00.944492+0000","last_active":"2024-03-18T14:51:05.026529+0000","last_peered":"2024-03-18T14:51:05.026529+0000","last_clean":"2024-03-18T14:51:05.026529+0000","last_became_active":"2024-03-18T14:51:00.944165+0000","last_became_peered":"2024-03-18T14:51:00.944165+0000","last_unstale":"2024-03-18T14:51:05.026529+0000","last_undegraded":"2024-03-18T14:51:05.026529+0000","last_fullsized":"2024-03-18T14:51:05.026529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:47:05.968966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213320+0000","last_change":"2024-03-18T14:51:01.356782+0000","last_active":"2024-03-18T14:51:03.213320+0000","last_peered":"2024-03-18T14:51:03.213320+0000","last_clean":"2024-03-18T14:51:03.213320+0000","last_became_active":"2024-03-18T14:51:01.356242+0000","last_became_peered":"2024-03-18T14:51:01.356242+0000","last_unstale":"2024-03-18T14:51:03.213320+0000","last_undegraded":"2024-03-18T14:51:03.213320+0000","last_fullsized":"2024-03-18T14:51:03.213320+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:56:07.662545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212656+0000","last_change":"2024-03-18T14:51:00.937873+0000","last_active":"2024-03-18T14:51:03.212656+0000","last_peered":"2024-03-18T14:51:03.212656+0000","last_clean":"2024-03-18T14:51:03.212656+0000","last_became_active":"2024-03-18T14:51:00.937562+0000","last_became_peered":"2024-03-18T14:51:00.937562+0000","last_unstale":"2024-03-18T14:51:03.212656+0000","last_undegraded":"2024-03-18T14:51:03.212656+0000","last_fullsized":"2024-03-18T14:51:03.212656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:36:44.342414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212245+0000","last_change":"2024-03-18T14:51:01.358681+0000","last_active":"2024-03-18T14:51:03.212245+0000","last_peered":"2024-03-18T14:51:03.212245+0000","last_clean":"2024-03-18T14:51:03.212245+0000","last_became_active":"2024-03-18T14:51:01.358374+0000","last_became_peered":"2024-03-18T14:51:01.358374+0000","last_unstale":"2024-03-18T14:51:03.212245+0000","last_undegraded":"2024-03-18T14:51:03.212245+0000","last_fullsized":"2024-03-18T14:51:03.212245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:43:47.804271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'10","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.026917+0000","last_change":"2024-03-18T14:51:00.935639+0000","last_active":"2024-03-18T14:51:05.026917+0000","last_peered":"2024-03-18T14:51:05.026917+0000","last_clean":"2024-03-18T14:51:05.026917+0000","last_became_active":"2024-03-18T14:51:00.935334+0000","last_became_peered":"2024-03-18T14:51:00.935334+0000","last_unstale":"2024-03-18T14:51:05.026917+0000","last_undegraded":"2024-03-18T14:51:05.026917+0000","last_fullsized":"2024-03-18T14:51:05.026917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:46.966812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345011+0000","last_change":"2024-03-18T14:51:01.364755+0000","last_active":"2024-03-18T14:51:02.345011+0000","last_peered":"2024-03-18T14:51:02.345011+0000","last_clean":"2024-03-18T14:51:02.345011+0000","last_became_active":"2024-03-18T14:51:01.364448+0000","last_became_peered":"2024-03-18T14:51:01.364448+0000","last_unstale":"2024-03-18T14:51:02.345011+0000","last_undegraded":"2024-03-18T14:51:02.345011+0000","last_fullsized":"2024-03-18T14:51:02.345011+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:37:49.480787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.093047+0000","last_change":"2024-03-18T14:51:00.933271+0000","last_active":"2024-03-18T14:51:03.093047+0000","last_peered":"2024-03-18T14:51:03.093047+0000","last_clean":"2024-03-18T14:51:03.093047+0000","last_became_active":"2024-03-18T14:51:00.932928+0000","last_became_peered":"2024-03-18T14:51:00.932928+0000","last_unstale":"2024-03-18T14:51:03.093047+0000","last_undegraded":"2024-03-18T14:51:03.093047+0000","last_fullsized":"2024-03-18T14:51:03.093047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:11:55.281763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212703+0000","last_change":"2024-03-18T14:51:01.362911+0000","last_active":"2024-03-18T14:51:03.212703+0000","last_peered":"2024-03-18T14:51:03.212703+0000","last_clean":"2024-03-18T14:51:03.212703+0000","last_became_active":"2024-03-18T14:51:01.362598+0000","last_became_peered":"2024-03-18T14:51:01.362598+0000","last_unstale":"2024-03-18T14:51:03.212703+0000","last_undegraded":"2024-03-18T14:51:03.212703+0000","last_fullsized":"2024-03-18T14:51:03.212703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:22:58.627753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212158+0000","last_change":"2024-03-18T14:51:00.935554+0000","last_active":"2024-03-18T14:51:03.212158+0000","last_peered":"2024-03-18T14:51:03.212158+0000","last_clean":"2024-03-18T14:51:03.212158+0000","last_became_active":"2024-03-18T14:51:00.935184+0000","last_became_peered":"2024-03-18T14:51:00.935184+0000","last_unstale":"2024-03-18T14:51:03.212158+0000","last_undegraded":"2024-03-18T14:51:03.212158+0000","last_fullsized":"2024-03-18T14:51:03.212158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:21:13.973570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213198+0000","last_change":"2024-03-18T14:51:01.355990+0000","last_active":"2024-03-18T14:51:03.213198+0000","last_peered":"2024-03-18T14:51:03.213198+0000","last_clean":"2024-03-18T14:51:03.213198+0000","last_became_active":"2024-03-18T14:51:01.355692+0000","last_became_peered":"2024-03-18T14:51:01.355692+0000","last_unstale":"2024-03-18T14:51:03.213198+0000","last_undegraded":"2024-03-18T14:51:03.213198+0000","last_fullsized":"2024-03-18T14:51:03.213198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:27:19.742684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212425+0000","last_change":"2024-03-18T14:51:00.943947+0000","last_active":"2024-03-18T14:51:03.212425+0000","last_peered":"2024-03-18T14:51:03.212425+0000","last_clean":"2024-03-18T14:51:03.212425+0000","last_became_active":"2024-03-18T14:51:00.943565+0000","last_became_peered":"2024-03-18T14:51:00.943565+0000","last_unstale":"2024-03-18T14:51:03.212425+0000","last_undegraded":"2024-03-18T14:51:03.212425+0000","last_fullsized":"2024-03-18T14:51:03.212425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:33:06.103148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212859+0000","last_change":"2024-03-18T14:51:01.363221+0000","last_active":"2024-03-18T14:51:03.212859+0000","last_peered":"2024-03-18T14:51:03.212859+0000","last_clean":"2024-03-18T14:51:03.212859+0000","last_became_active":"2024-03-18T14:51:01.362848+0000","last_became_peered":"2024-03-18T14:51:01.362848+0000","last_unstale":"2024-03-18T14:51:03.212859+0000","last_undegraded":"2024-03-18T14:51:03.212859+0000","last_fullsized":"2024-03-18T14:51:03.212859+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:04.139365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345140+0000","last_change":"2024-03-18T14:51:01.356237+0000","last_active":"2024-03-18T14:51:02.345140+0000","last_peered":"2024-03-18T14:51:02.345140+0000","last_clean":"2024-03-18T14:51:02.345140+0000","last_became_active":"2024-03-18T14:51:01.355922+0000","last_became_peered":"2024-03-18T14:51:01.355922+0000","last_unstale":"2024-03-18T14:51:02.345140+0000","last_undegraded":"2024-03-18T14:51:02.345140+0000","last_fullsized":"2024-03-18T14:51:02.345140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:04:50.491752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212172+0000","last_change":"2024-03-18T14:51:00.936058+0000","last_active":"2024-03-18T14:51:03.212172+0000","last_peered":"2024-03-18T14:51:03.212172+0000","last_clean":"2024-03-18T14:51:03.212172+0000","last_became_active":"2024-03-18T14:51:00.935731+0000","last_became_peered":"2024-03-18T14:51:00.935731+0000","last_unstale":"2024-03-18T14:51:03.212172+0000","last_undegraded":"2024-03-18T14:51:03.212172+0000","last_fullsized":"2024-03-18T14:51:03.212172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:47:49.023943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.342875+0000","last_change":"2024-03-18T14:51:01.359742+0000","last_active":"2024-03-18T14:51:02.342875+0000","last_peered":"2024-03-18T14:51:02.342875+0000","last_clean":"2024-03-18T14:51:02.342875+0000","last_became_active":"2024-03-18T14:51:01.358507+0000","last_became_peered":"2024-03-18T14:51:01.358507+0000","last_unstale":"2024-03-18T14:51:02.342875+0000","last_undegraded":"2024-03-18T14:51:02.342875+0000","last_fullsized":"2024-03-18T14:51:02.342875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:17.129695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.026576+0000","last_change":"2024-03-18T14:51:00.934593+0000","last_active":"2024-03-18T14:51:05.026576+0000","last_peered":"2024-03-18T14:51:05.026576+0000","last_clean":"2024-03-18T14:51:05.026576+0000","last_became_active":"2024-03-18T14:51:00.934253+0000","last_became_peered":"2024-03-18T14:51:00.934253+0000","last_unstale":"2024-03-18T14:51:05.026576+0000","last_undegraded":"2024-03-18T14:51:05.026576+0000","last_fullsized":"2024-03-18T14:51:05.026576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:43:26.323162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":347,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.211833+0000","last_change":"2024-03-18T14:49:51.158965+0000","last_active":"2024-03-18T14:51:03.211833+0000","last_peered":"2024-03-18T14:51:03.211833+0000","last_clean":"2024-03-18T14:51:03.211833+0000","last_became_active":"2024-03-18T14:49:51.158580+0000","last_became_peered":"2024-03-18T14:49:51.158580+0000","last_unstale":"2024-03-18T14:51:03.211833+0000","last_undegraded":"2024-03-18T14:51:03.211833+0000","last_fullsized":"2024-03-18T14:51:03.211833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:46:27.210244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:46:27.210244+0000","last_clean_scrub_stamp":"2024-03-18T14:46:27.210244+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:37:08.104617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212902+0000","last_change":"2024-03-18T14:51:01.359972+0000","last_active":"2024-03-18T14:51:03.212902+0000","last_peered":"2024-03-18T14:51:03.212902+0000","last_clean":"2024-03-18T14:51:03.212902+0000","last_became_active":"2024-03-18T14:51:01.359430+0000","last_became_peered":"2024-03-18T14:51:01.359430+0000","last_unstale":"2024-03-18T14:51:03.212902+0000","last_undegraded":"2024-03-18T14:51:03.212902+0000","last_fullsized":"2024-03-18T14:51:03.212902+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:00:14.099499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212428+0000","last_change":"2024-03-18T14:51:00.934196+0000","last_active":"2024-03-18T14:51:03.212428+0000","last_peered":"2024-03-18T14:51:03.212428+0000","last_clean":"2024-03-18T14:51:03.212428+0000","last_became_active":"2024-03-18T14:51:00.933875+0000","last_became_peered":"2024-03-18T14:51:00.933875+0000","last_unstale":"2024-03-18T14:51:03.212428+0000","last_undegraded":"2024-03-18T14:51:03.212428+0000","last_fullsized":"2024-03-18T14:51:03.212428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:56.797227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.092942+0000","last_change":"2024-03-18T14:51:00.941862+0000","last_active":"2024-03-18T14:51:03.092942+0000","last_peered":"2024-03-18T14:51:03.092942+0000","last_clean":"2024-03-18T14:51:03.092942+0000","last_became_active":"2024-03-18T14:51:00.941536+0000","last_became_peered":"2024-03-18T14:51:00.941536+0000","last_unstale":"2024-03-18T14:51:03.092942+0000","last_undegraded":"2024-03-18T14:51:03.092942+0000","last_fullsized":"2024-03-18T14:51:03.092942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:56.364552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213093+0000","last_change":"2024-03-18T14:51:01.352404+0000","last_active":"2024-03-18T14:51:03.213093+0000","last_peered":"2024-03-18T14:51:03.213093+0000","last_clean":"2024-03-18T14:51:03.213093+0000","last_became_active":"2024-03-18T14:51:01.352077+0000","last_became_peered":"2024-03-18T14:51:01.352077+0000","last_unstale":"2024-03-18T14:51:03.213093+0000","last_undegraded":"2024-03-18T14:51:03.213093+0000","last_fullsized":"2024-03-18T14:51:03.213093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:30:38.050712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:06.034970+0000","last_change":"2024-03-18T14:51:00.935564+0000","last_active":"2024-03-18T14:51:06.034970+0000","last_peered":"2024-03-18T14:51:06.034970+0000","last_clean":"2024-03-18T14:51:06.034970+0000","last_became_active":"2024-03-18T14:51:00.935268+0000","last_became_peered":"2024-03-18T14:51:00.935268+0000","last_unstale":"2024-03-18T14:51:06.034970+0000","last_undegraded":"2024-03-18T14:51:06.034970+0000","last_fullsized":"2024-03-18T14:51:06.034970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:22:33.504184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.343508+0000","last_change":"2024-03-18T14:51:01.356096+0000","last_active":"2024-03-18T14:51:02.343508+0000","last_peered":"2024-03-18T14:51:02.343508+0000","last_clean":"2024-03-18T14:51:02.343508+0000","last_became_active":"2024-03-18T14:51:01.355688+0000","last_became_peered":"2024-03-18T14:51:01.355688+0000","last_unstale":"2024-03-18T14:51:02.343508+0000","last_undegraded":"2024-03-18T14:51:02.343508+0000","last_fullsized":"2024-03-18T14:51:02.343508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:04:38.678482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.026742+0000","last_change":"2024-03-18T14:51:00.942610+0000","last_active":"2024-03-18T14:51:05.026742+0000","last_peered":"2024-03-18T14:51:05.026742+0000","last_clean":"2024-03-18T14:51:05.026742+0000","last_became_active":"2024-03-18T14:51:00.942203+0000","last_became_peered":"2024-03-18T14:51:00.942203+0000","last_unstale":"2024-03-18T14:51:05.026742+0000","last_undegraded":"2024-03-18T14:51:05.026742+0000","last_fullsized":"2024-03-18T14:51:05.026742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:17:26.324282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212588+0000","last_change":"2024-03-18T14:51:01.362931+0000","last_active":"2024-03-18T14:51:03.212588+0000","last_peered":"2024-03-18T14:51:03.212588+0000","last_clean":"2024-03-18T14:51:03.212588+0000","last_became_active":"2024-03-18T14:51:01.362624+0000","last_became_peered":"2024-03-18T14:51:01.362624+0000","last_unstale":"2024-03-18T14:51:03.212588+0000","last_undegraded":"2024-03-18T14:51:03.212588+0000","last_fullsized":"2024-03-18T14:51:03.212588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:40:52.512871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.025211+0000","last_change":"2024-03-18T14:51:00.938971+0000","last_active":"2024-03-18T14:51:05.025211+0000","last_peered":"2024-03-18T14:51:05.025211+0000","last_clean":"2024-03-18T14:51:05.025211+0000","last_became_active":"2024-03-18T14:51:00.938645+0000","last_became_peered":"2024-03-18T14:51:00.938645+0000","last_unstale":"2024-03-18T14:51:05.025211+0000","last_undegraded":"2024-03-18T14:51:05.025211+0000","last_fullsized":"2024-03-18T14:51:05.025211+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:18:54.999927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.092674+0000","last_change":"2024-03-18T14:51:01.359510+0000","last_active":"2024-03-18T14:51:03.092674+0000","last_peered":"2024-03-18T14:51:03.092674+0000","last_clean":"2024-03-18T14:51:03.092674+0000","last_became_active":"2024-03-18T14:51:01.359192+0000","last_became_peered":"2024-03-18T14:51:01.359192+0000","last_unstale":"2024-03-18T14:51:03.092674+0000","last_undegraded":"2024-03-18T14:51:03.092674+0000","last_fullsized":"2024-03-18T14:51:03.092674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:11:52.942825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.026414+0000","last_change":"2024-03-18T14:51:00.940194+0000","last_active":"2024-03-18T14:51:05.026414+0000","last_peered":"2024-03-18T14:51:05.026414+0000","last_clean":"2024-03-18T14:51:05.026414+0000","last_became_active":"2024-03-18T14:51:00.939861+0000","last_became_peered":"2024-03-18T14:51:00.939861+0000","last_unstale":"2024-03-18T14:51:05.026414+0000","last_undegraded":"2024-03-18T14:51:05.026414+0000","last_fullsized":"2024-03-18T14:51:05.026414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:06:30.314102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212316+0000","last_change":"2024-03-18T14:51:01.355098+0000","last_active":"2024-03-18T14:51:03.212316+0000","last_peered":"2024-03-18T14:51:03.212316+0000","last_clean":"2024-03-18T14:51:03.212316+0000","last_became_active":"2024-03-18T14:51:01.354452+0000","last_became_peered":"2024-03-18T14:51:01.354452+0000","last_unstale":"2024-03-18T14:51:03.212316+0000","last_undegraded":"2024-03-18T14:51:03.212316+0000","last_fullsized":"2024-03-18T14:51:03.212316+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:40:54.650294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212368+0000","last_change":"2024-03-18T14:51:00.934585+0000","last_active":"2024-03-18T14:51:03.212368+0000","last_peered":"2024-03-18T14:51:03.212368+0000","last_clean":"2024-03-18T14:51:03.212368+0000","last_became_active":"2024-03-18T14:51:00.933660+0000","last_became_peered":"2024-03-18T14:51:00.933660+0000","last_unstale":"2024-03-18T14:51:03.212368+0000","last_undegraded":"2024-03-18T14:51:03.212368+0000","last_fullsized":"2024-03-18T14:51:03.212368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:29:01.144836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.343188+0000","last_change":"2024-03-18T14:51:01.357920+0000","last_active":"2024-03-18T14:51:02.343188+0000","last_peered":"2024-03-18T14:51:02.343188+0000","last_clean":"2024-03-18T14:51:02.343188+0000","last_became_active":"2024-03-18T14:51:01.357565+0000","last_became_peered":"2024-03-18T14:51:01.357565+0000","last_unstale":"2024-03-18T14:51:02.343188+0000","last_undegraded":"2024-03-18T14:51:02.343188+0000","last_fullsized":"2024-03-18T14:51:02.343188+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:39:57.880178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.027002+0000","last_change":"2024-03-18T14:51:00.937887+0000","last_active":"2024-03-18T14:51:05.027002+0000","last_peered":"2024-03-18T14:51:05.027002+0000","last_clean":"2024-03-18T14:51:05.027002+0000","last_became_active":"2024-03-18T14:51:00.937588+0000","last_became_peered":"2024-03-18T14:51:00.937588+0000","last_unstale":"2024-03-18T14:51:05.027002+0000","last_undegraded":"2024-03-18T14:51:05.027002+0000","last_fullsized":"2024-03-18T14:51:05.027002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:43:37.558412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212616+0000","last_change":"2024-03-18T14:51:01.354175+0000","last_active":"2024-03-18T14:51:03.212616+0000","last_peered":"2024-03-18T14:51:03.212616+0000","last_clean":"2024-03-18T14:51:03.212616+0000","last_became_active":"2024-03-18T14:51:01.353839+0000","last_became_peered":"2024-03-18T14:51:01.353839+0000","last_unstale":"2024-03-18T14:51:03.212616+0000","last_undegraded":"2024-03-18T14:51:03.212616+0000","last_fullsized":"2024-03-18T14:51:03.212616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:21:52.372938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:06.036037+0000","last_change":"2024-03-18T14:51:00.941600+0000","last_active":"2024-03-18T14:51:06.036037+0000","last_peered":"2024-03-18T14:51:06.036037+0000","last_clean":"2024-03-18T14:51:06.036037+0000","last_became_active":"2024-03-18T14:51:00.941291+0000","last_became_peered":"2024-03-18T14:51:00.941291+0000","last_unstale":"2024-03-18T14:51:06.036037+0000","last_undegraded":"2024-03-18T14:51:06.036037+0000","last_fullsized":"2024-03-18T14:51:06.036037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:04:11.142029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213191+0000","last_change":"2024-03-18T14:51:01.359058+0000","last_active":"2024-03-18T14:51:03.213191+0000","last_peered":"2024-03-18T14:51:03.213191+0000","last_clean":"2024-03-18T14:51:03.213191+0000","last_became_active":"2024-03-18T14:51:01.358437+0000","last_became_peered":"2024-03-18T14:51:01.358437+0000","last_unstale":"2024-03-18T14:51:03.213191+0000","last_undegraded":"2024-03-18T14:51:03.213191+0000","last_fullsized":"2024-03-18T14:51:03.213191+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:39:58.945080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345181+0000","last_change":"2024-03-18T14:51:00.933647+0000","last_active":"2024-03-18T14:51:02.345181+0000","last_peered":"2024-03-18T14:51:02.345181+0000","last_clean":"2024-03-18T14:51:02.345181+0000","last_became_active":"2024-03-18T14:51:00.933320+0000","last_became_peered":"2024-03-18T14:51:00.933320+0000","last_unstale":"2024-03-18T14:51:02.345181+0000","last_undegraded":"2024-03-18T14:51:02.345181+0000","last_fullsized":"2024-03-18T14:51:02.345181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:16:46.498845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212920+0000","last_change":"2024-03-18T14:51:01.359874+0000","last_active":"2024-03-18T14:51:03.212920+0000","last_peered":"2024-03-18T14:51:03.212920+0000","last_clean":"2024-03-18T14:51:03.212920+0000","last_became_active":"2024-03-18T14:51:01.359335+0000","last_became_peered":"2024-03-18T14:51:01.359335+0000","last_unstale":"2024-03-18T14:51:03.212920+0000","last_undegraded":"2024-03-18T14:51:03.212920+0000","last_fullsized":"2024-03-18T14:51:03.212920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:03.583402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212929+0000","last_change":"2024-03-18T14:51:00.935755+0000","last_active":"2024-03-18T14:51:03.212929+0000","last_peered":"2024-03-18T14:51:03.212929+0000","last_clean":"2024-03-18T14:51:03.212929+0000","last_became_active":"2024-03-18T14:51:00.935421+0000","last_became_peered":"2024-03-18T14:51:00.935421+0000","last_unstale":"2024-03-18T14:51:03.212929+0000","last_undegraded":"2024-03-18T14:51:03.212929+0000","last_fullsized":"2024-03-18T14:51:03.212929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:46:11.680972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213443+0000","last_change":"2024-03-18T14:51:01.355605+0000","last_active":"2024-03-18T14:51:03.213443+0000","last_peered":"2024-03-18T14:51:03.213443+0000","last_clean":"2024-03-18T14:51:03.213443+0000","last_became_active":"2024-03-18T14:51:01.355295+0000","last_became_peered":"2024-03-18T14:51:01.355295+0000","last_unstale":"2024-03-18T14:51:03.213443+0000","last_undegraded":"2024-03-18T14:51:03.213443+0000","last_fullsized":"2024-03-18T14:51:03.213443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:15:20.466314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212467+0000","last_change":"2024-03-18T14:51:01.359704+0000","last_active":"2024-03-18T14:51:03.212467+0000","last_peered":"2024-03-18T14:51:03.212467+0000","last_clean":"2024-03-18T14:51:03.212467+0000","last_became_active":"2024-03-18T14:51:01.359367+0000","last_became_peered":"2024-03-18T14:51:01.359367+0000","last_unstale":"2024-03-18T14:51:03.212467+0000","last_undegraded":"2024-03-18T14:51:03.212467+0000","last_fullsized":"2024-03-18T14:51:03.212467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:26.933993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212607+0000","last_change":"2024-03-18T14:51:01.357440+0000","last_active":"2024-03-18T14:51:03.212607+0000","last_peered":"2024-03-18T14:51:03.212607+0000","last_clean":"2024-03-18T14:51:03.212607+0000","last_became_active":"2024-03-18T14:51:01.356537+0000","last_became_peered":"2024-03-18T14:51:01.356537+0000","last_unstale":"2024-03-18T14:51:03.212607+0000","last_undegraded":"2024-03-18T14:51:03.212607+0000","last_fullsized":"2024-03-18T14:51:03.212607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:01.389053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212231+0000","last_change":"2024-03-18T14:51:01.355678+0000","last_active":"2024-03-18T14:51:03.212231+0000","last_peered":"2024-03-18T14:51:03.212231+0000","last_clean":"2024-03-18T14:51:03.212231+0000","last_became_active":"2024-03-18T14:51:01.355197+0000","last_became_peered":"2024-03-18T14:51:01.355197+0000","last_unstale":"2024-03-18T14:51:03.212231+0000","last_undegraded":"2024-03-18T14:51:03.212231+0000","last_fullsized":"2024-03-18T14:51:03.212231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:34.194834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.211933+0000","last_change":"2024-03-18T14:51:01.363964+0000","last_active":"2024-03-18T14:51:03.211933+0000","last_peered":"2024-03-18T14:51:03.211933+0000","last_clean":"2024-03-18T14:51:03.211933+0000","last_became_active":"2024-03-18T14:51:01.363590+0000","last_became_peered":"2024-03-18T14:51:01.363590+0000","last_unstale":"2024-03-18T14:51:03.211933+0000","last_undegraded":"2024-03-18T14:51:03.211933+0000","last_fullsized":"2024-03-18T14:51:03.211933+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:59.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213137+0000","last_change":"2024-03-18T14:51:01.361599+0000","last_active":"2024-03-18T14:51:03.213137+0000","last_peered":"2024-03-18T14:51:03.213137+0000","last_clean":"2024-03-18T14:51:03.213137+0000","last_became_active":"2024-03-18T14:51:01.361280+0000","last_became_peered":"2024-03-18T14:51:01.361280+0000","last_unstale":"2024-03-18T14:51:03.213137+0000","last_undegraded":"2024-03-18T14:51:03.213137+0000","last_fullsized":"2024-03-18T14:51:03.213137+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:52:50.572257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212612+0000","last_change":"2024-03-18T14:51:01.360495+0000","last_active":"2024-03-18T14:51:03.212612+0000","last_peered":"2024-03-18T14:51:03.212612+0000","last_clean":"2024-03-18T14:51:03.212612+0000","last_became_active":"2024-03-18T14:51:01.360146+0000","last_became_peered":"2024-03-18T14:51:01.360146+0000","last_unstale":"2024-03-18T14:51:03.212612+0000","last_undegraded":"2024-03-18T14:51:03.212612+0000","last_fullsized":"2024-03-18T14:51:03.212612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:52.048137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212815+0000","last_change":"2024-03-18T14:51:01.357650+0000","last_active":"2024-03-18T14:51:03.212815+0000","last_peered":"2024-03-18T14:51:03.212815+0000","last_clean":"2024-03-18T14:51:03.212815+0000","last_became_active":"2024-03-18T14:51:01.356265+0000","last_became_peered":"2024-03-18T14:51:01.356265+0000","last_unstale":"2024-03-18T14:51:03.212815+0000","last_undegraded":"2024-03-18T14:51:03.212815+0000","last_fullsized":"2024-03-18T14:51:03.212815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:49.328404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.343356+0000","last_change":"2024-03-18T14:51:01.363753+0000","last_active":"2024-03-18T14:51:02.343356+0000","last_peered":"2024-03-18T14:51:02.343356+0000","last_clean":"2024-03-18T14:51:02.343356+0000","last_became_active":"2024-03-18T14:51:01.363411+0000","last_became_peered":"2024-03-18T14:51:01.363411+0000","last_unstale":"2024-03-18T14:51:02.343356+0000","last_undegraded":"2024-03-18T14:51:02.343356+0000","last_fullsized":"2024-03-18T14:51:02.343356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:55.452209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345116+0000","last_change":"2024-03-18T14:51:01.360723+0000","last_active":"2024-03-18T14:51:02.345116+0000","last_peered":"2024-03-18T14:51:02.345116+0000","last_clean":"2024-03-18T14:51:02.345116+0000","last_became_active":"2024-03-18T14:51:01.360130+0000","last_became_peered":"2024-03-18T14:51:01.360130+0000","last_unstale":"2024-03-18T14:51:02.345116+0000","last_undegraded":"2024-03-18T14:51:02.345116+0000","last_fullsized":"2024-03-18T14:51:02.345116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:51.634011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.342707+0000","last_change":"2024-03-18T14:51:01.356462+0000","last_active":"2024-03-18T14:51:02.342707+0000","last_peered":"2024-03-18T14:51:02.342707+0000","last_clean":"2024-03-18T14:51:02.342707+0000","last_became_active":"2024-03-18T14:51:01.355395+0000","last_became_peered":"2024-03-18T14:51:01.355395+0000","last_unstale":"2024-03-18T14:51:02.342707+0000","last_undegraded":"2024-03-18T14:51:02.342707+0000","last_fullsized":"2024-03-18T14:51:02.342707+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:37:15.326354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345048+0000","last_change":"2024-03-18T14:51:01.358626+0000","last_active":"2024-03-18T14:51:02.345048+0000","last_peered":"2024-03-18T14:51:02.345048+0000","last_clean":"2024-03-18T14:51:02.345048+0000","last_became_active":"2024-03-18T14:51:01.358293+0000","last_became_peered":"2024-03-18T14:51:01.358293+0000","last_unstale":"2024-03-18T14:51:02.345048+0000","last_undegraded":"2024-03-18T14:51:02.345048+0000","last_fullsized":"2024-03-18T14:51:02.345048+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:52:41.139885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212980+0000","last_change":"2024-03-18T14:51:01.355276+0000","last_active":"2024-03-18T14:51:03.212980+0000","last_peered":"2024-03-18T14:51:03.212980+0000","last_clean":"2024-03-18T14:51:03.212980+0000","last_became_active":"2024-03-18T14:51:01.354661+0000","last_became_peered":"2024-03-18T14:51:01.354661+0000","last_unstale":"2024-03-18T14:51:03.212980+0000","last_undegraded":"2024-03-18T14:51:03.212980+0000","last_fullsized":"2024-03-18T14:51:03.212980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:28.569055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212498+0000","last_change":"2024-03-18T14:51:01.363101+0000","last_active":"2024-03-18T14:51:03.212498+0000","last_peered":"2024-03-18T14:51:03.212498+0000","last_clean":"2024-03-18T14:51:03.212498+0000","last_became_active":"2024-03-18T14:51:01.362576+0000","last_became_peered":"2024-03-18T14:51:01.362576+0000","last_unstale":"2024-03-18T14:51:03.212498+0000","last_undegraded":"2024-03-18T14:51:03.212498+0000","last_fullsized":"2024-03-18T14:51:03.212498+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:42:32.109523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280632,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4398,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":104,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":31567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350989,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":921600,"data_stored":441572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":1035130,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776152,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":442390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037788,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":929792,"data_stored":442176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299425,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":1033144,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462950,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":921600,"data_stored":441572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":449784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":925696,"data_stored":441662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":946176,"data_stored":443840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":933888,"data_stored":441766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888113,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":925696,"data_stored":441594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724587,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":983040,"data_stored":1032122,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":20895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:51:09.749 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:51:09.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.887+0000 7f8938f48640 1 -- 172.21.15.76:0/3674298358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8934100700 msgr2=0x7f8934100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:09.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.887+0000 7f8938f48640 1 --2- 172.21.15.76:0/3674298358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8934100700 0x7f8934100b20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8924009900 tx=0x7f892402bc30 comp rx=0 tx=0).stop 2024-03-18T14:51:09.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.887+0000 7f8938f48640 1 -- 172.21.15.76:0/3674298358 shutdown_connections 2024-03-18T14:51:09.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.887+0000 7f8938f48640 1 --2- 172.21.15.76:0/3674298358 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89341018f0 0x7f8934101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.887+0000 7f8938f48640 1 --2- 172.21.15.76:0/3674298358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89341022b0 0x7f8934108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.887+0000 7f8938f48640 1 --2- 172.21.15.76:0/3674298358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8934100700 0x7f8934100b20 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.887+0000 7f8938f48640 1 -- 172.21.15.76:0/3674298358 >> 172.21.15.76:0/3674298358 conn(0x7f89340fbc70 msgr2=0x7f89340fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:09.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.887+0000 7f8938f48640 1 -- 172.21.15.76:0/3674298358 shutdown_connections 2024-03-18T14:51:09.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.887+0000 7f8938f48640 1 -- 172.21.15.76:0/3674298358 wait complete. 2024-03-18T14:51:09.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f8938f48640 1 Processor -- start 2024-03-18T14:51:09.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f8938f48640 1 -- start start 2024-03-18T14:51:09.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f8938f48640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8934100700 0x7f8934104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:09.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f8938f48640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89341018f0 0x7f8934102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:09.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f8938f48640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89341022b0 0x7f8934103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:09.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f8938f48640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f893410c370 con 0x7f89341022b0 2024-03-18T14:51:09.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f8938f48640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f893410c1f0 con 0x7f8934100700 2024-03-18T14:51:09.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f8938f48640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f893410c4f0 con 0x7f89341018f0 2024-03-18T14:51:09.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f8932d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89341022b0 0x7f8934103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:09.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f8932d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89341022b0 0x7f8934103210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40708/0 (socket says 172.21.15.76:40708) 2024-03-18T14:51:09.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f8932d9d640 1 -- 172.21.15.76:0/2633937428 learned_addr learned my addr 172.21.15.76:0/2633937428 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:09.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f893259c640 1 --2- 172.21.15.76:0/2633937428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8934100700 0x7f8934104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:09.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f892bfff640 1 --2- 172.21.15.76:0/2633937428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89341018f0 0x7f8934102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:09.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f8932d9d640 1 -- 172.21.15.76:0/2633937428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89341018f0 msgr2=0x7f8934102cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:09.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f8932d9d640 1 --2- 172.21.15.76:0/2633937428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89341018f0 0x7f8934102cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f8932d9d640 1 -- 172.21.15.76:0/2633937428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8934100700 msgr2=0x7f8934104690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:09.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f8932d9d640 1 --2- 172.21.15.76:0/2633937428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8934100700 0x7f8934104690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:09.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.888+0000 7f8932d9d640 1 -- 172.21.15.76:0/2633937428 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8934103aa0 con 0x7f89341022b0 2024-03-18T14:51:09.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.889+0000 7f8932d9d640 1 --2- 172.21.15.76:0/2633937428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89341022b0 0x7f8934103210 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f892c00efd0 tx=0x7f892c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:09.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.889+0000 7f892b7fe640 1 -- 172.21.15.76:0/2633937428 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c003b70 con 0x7f89341022b0 2024-03-18T14:51:09.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.889+0000 7f8938f48640 1 -- 172.21.15.76:0/2633937428 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89341a1cb0 con 0x7f89341022b0 2024-03-18T14:51:09.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.889+0000 7f892b7fe640 1 -- 172.21.15.76:0/2633937428 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f892c003d10 con 0x7f89341022b0 2024-03-18T14:51:09.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.889+0000 7f892b7fe640 1 -- 172.21.15.76:0/2633937428 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c0109f0 con 0x7f89341022b0 2024-03-18T14:51:09.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.889+0000 7f8938f48640 1 -- 172.21.15.76:0/2633937428 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89341a21b0 con 0x7f89341022b0 2024-03-18T14:51:09.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.890+0000 7f8938f48640 1 -- 172.21.15.76:0/2633937428 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89341a23e0 con 0x7f89341022b0 2024-03-18T14:51:09.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.891+0000 7f892b7fe640 1 -- 172.21.15.76:0/2633937428 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f892c004020 con 0x7f89341022b0 2024-03-18T14:51:09.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.891+0000 7f892b7fe640 1 --2- 172.21.15.76:0/2633937428 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89200a2890 0x7f89200a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:09.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.891+0000 7f893259c640 1 --2- 172.21.15.76:0/2633937428 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89200a2890 0x7f89200a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:09.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.891+0000 7f892b7fe640 1 -- 172.21.15.76:0/2633937428 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f892c0ceee0 con 0x7f89341022b0 2024-03-18T14:51:09.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.892+0000 7f893259c640 1 --2- 172.21.15.76:0/2633937428 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89200a2890 0x7f89200a4d50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8924000c00 tx=0x7f8924005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:09.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:09.896+0000 7f892b7fe640 1 -- 172.21.15.76:0/2633937428 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f892c09bfd0 con 0x7f89341022b0 2024-03-18T14:51:10.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:10.069+0000 7f8938f48640 1 -- 172.21.15.76:0/2633937428 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f89340fde90 con 0x7f89200a2890 2024-03-18T14:51:10.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:10.076+0000 7f892b7fe640 1 -- 172.21.15.76:0/2633937428 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258899 (secure 0 0 0) 0x7f89340fde90 con 0x7f89200a2890 2024-03-18T14:51:10.080 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:51:10.081 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-03-18T14:51:10.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:10.079+0000 7f8938f48640 1 -- 172.21.15.76:0/2633937428 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89200a2890 msgr2=0x7f89200a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:10.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:10.079+0000 7f8938f48640 1 --2- 172.21.15.76:0/2633937428 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89200a2890 0x7f89200a4d50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8924000c00 tx=0x7f8924005fd0 comp rx=0 tx=0).stop 2024-03-18T14:51:10.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:10.079+0000 7f8938f48640 1 -- 172.21.15.76:0/2633937428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89341022b0 msgr2=0x7f8934103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:10.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:10.079+0000 7f8938f48640 1 --2- 172.21.15.76:0/2633937428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89341022b0 0x7f8934103210 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f892c00efd0 tx=0x7f892c00c6a0 comp rx=0 tx=0).stop 2024-03-18T14:51:10.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:10.079+0000 7f8938f48640 1 -- 172.21.15.76:0/2633937428 shutdown_connections 2024-03-18T14:51:10.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:10.079+0000 7f8938f48640 1 --2- 172.21.15.76:0/2633937428 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89200a2890 0x7f89200a4d50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:10.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:10.079+0000 7f8938f48640 1 --2- 172.21.15.76:0/2633937428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89341018f0 0x7f8934102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:10.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:10.079+0000 7f8938f48640 1 --2- 172.21.15.76:0/2633937428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89341022b0 0x7f8934103210 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:10.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:10.079+0000 7f8938f48640 1 --2- 172.21.15.76:0/2633937428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8934100700 0x7f8934104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:10.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:10.079+0000 7f8938f48640 1 -- 172.21.15.76:0/2633937428 >> 172.21.15.76:0/2633937428 conn(0x7f89340fbc70 msgr2=0x7f89340fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:10.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:10.080+0000 7f8938f48640 1 -- 172.21.15.76:0/2633937428 shutdown_connections 2024-03-18T14:51:10.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:10.080+0000 7f8938f48640 1 -- 172.21.15.76:0/2633937428 wait complete. 2024-03-18T14:51:10.091 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-03-18T14:51:08.330357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594766,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":142,"num_read_kb":124,"num_write":285,"num_write_kb":2965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368628,"kb_used_data":11100,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124573196,"statfs":{"total":1151940427776,"available":1151562952704,"internally_reserved":0,"allocated":11366400,"data_stored":7086752,"data_compressed":107082,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4398,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":104,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.412889"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212862+0000","last_change":"2024-03-18T14:51:01.365948+0000","last_active":"2024-03-18T14:51:03.212862+0000","last_peered":"2024-03-18T14:51:03.212862+0000","last_clean":"2024-03-18T14:51:03.212862+0000","last_became_active":"2024-03-18T14:51:01.364600+0000","last_became_peered":"2024-03-18T14:51:01.364600+0000","last_unstale":"2024-03-18T14:51:03.212862+0000","last_undegraded":"2024-03-18T14:51:03.212862+0000","last_fullsized":"2024-03-18T14:51:03.212862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:14:45.198618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212439+0000","last_change":"2024-03-18T14:51:01.360186+0000","last_active":"2024-03-18T14:51:03.212439+0000","last_peered":"2024-03-18T14:51:03.212439+0000","last_clean":"2024-03-18T14:51:03.212439+0000","last_became_active":"2024-03-18T14:51:01.359862+0000","last_became_peered":"2024-03-18T14:51:01.359862+0000","last_unstale":"2024-03-18T14:51:03.212439+0000","last_undegraded":"2024-03-18T14:51:03.212439+0000","last_fullsized":"2024-03-18T14:51:03.212439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:46.206823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212349+0000","last_change":"2024-03-18T14:51:01.364635+0000","last_active":"2024-03-18T14:51:03.212349+0000","last_peered":"2024-03-18T14:51:03.212349+0000","last_clean":"2024-03-18T14:51:03.212349+0000","last_became_active":"2024-03-18T14:51:01.364320+0000","last_became_peered":"2024-03-18T14:51:01.364320+0000","last_unstale":"2024-03-18T14:51:03.212349+0000","last_undegraded":"2024-03-18T14:51:03.212349+0000","last_fullsized":"2024-03-18T14:51:03.212349+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:08:20.227639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212416+0000","last_change":"2024-03-18T14:51:01.365103+0000","last_active":"2024-03-18T14:51:03.212416+0000","last_peered":"2024-03-18T14:51:03.212416+0000","last_clean":"2024-03-18T14:51:03.212416+0000","last_became_active":"2024-03-18T14:51:01.364750+0000","last_became_peered":"2024-03-18T14:51:01.364750+0000","last_unstale":"2024-03-18T14:51:03.212416+0000","last_undegraded":"2024-03-18T14:51:03.212416+0000","last_fullsized":"2024-03-18T14:51:03.212416+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:47:33.849019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.092728+0000","last_change":"2024-03-18T14:51:01.356839+0000","last_active":"2024-03-18T14:51:03.092728+0000","last_peered":"2024-03-18T14:51:03.092728+0000","last_clean":"2024-03-18T14:51:03.092728+0000","last_became_active":"2024-03-18T14:51:01.356531+0000","last_became_peered":"2024-03-18T14:51:01.356531+0000","last_unstale":"2024-03-18T14:51:03.092728+0000","last_undegraded":"2024-03-18T14:51:03.092728+0000","last_fullsized":"2024-03-18T14:51:03.092728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:28.499573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212558+0000","last_change":"2024-03-18T14:51:01.362912+0000","last_active":"2024-03-18T14:51:03.212558+0000","last_peered":"2024-03-18T14:51:03.212558+0000","last_clean":"2024-03-18T14:51:03.212558+0000","last_became_active":"2024-03-18T14:51:01.362590+0000","last_became_peered":"2024-03-18T14:51:01.362590+0000","last_unstale":"2024-03-18T14:51:03.212558+0000","last_undegraded":"2024-03-18T14:51:03.212558+0000","last_fullsized":"2024-03-18T14:51:03.212558+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:49.199616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212123+0000","last_change":"2024-03-18T14:51:01.358652+0000","last_active":"2024-03-18T14:51:03.212123+0000","last_peered":"2024-03-18T14:51:03.212123+0000","last_clean":"2024-03-18T14:51:03.212123+0000","last_became_active":"2024-03-18T14:51:01.358312+0000","last_became_peered":"2024-03-18T14:51:01.358312+0000","last_unstale":"2024-03-18T14:51:03.212123+0000","last_undegraded":"2024-03-18T14:51:03.212123+0000","last_fullsized":"2024-03-18T14:51:03.212123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:40.150240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.342639+0000","last_change":"2024-03-18T14:51:01.360082+0000","last_active":"2024-03-18T14:51:02.342639+0000","last_peered":"2024-03-18T14:51:02.342639+0000","last_clean":"2024-03-18T14:51:02.342639+0000","last_became_active":"2024-03-18T14:51:01.359769+0000","last_became_peered":"2024-03-18T14:51:01.359769+0000","last_unstale":"2024-03-18T14:51:02.342639+0000","last_undegraded":"2024-03-18T14:51:02.342639+0000","last_fullsized":"2024-03-18T14:51:02.342639+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:53:27.081546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345212+0000","last_change":"2024-03-18T14:51:01.363947+0000","last_active":"2024-03-18T14:51:02.345212+0000","last_peered":"2024-03-18T14:51:02.345212+0000","last_clean":"2024-03-18T14:51:02.345212+0000","last_became_active":"2024-03-18T14:51:01.363106+0000","last_became_peered":"2024-03-18T14:51:01.363106+0000","last_unstale":"2024-03-18T14:51:02.345212+0000","last_undegraded":"2024-03-18T14:51:02.345212+0000","last_fullsized":"2024-03-18T14:51:02.345212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:24:50.478426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213560+0000","last_change":"2024-03-18T14:51:01.357950+0000","last_active":"2024-03-18T14:51:03.213560+0000","last_peered":"2024-03-18T14:51:03.213560+0000","last_clean":"2024-03-18T14:51:03.213560+0000","last_became_active":"2024-03-18T14:51:01.357651+0000","last_became_peered":"2024-03-18T14:51:01.357651+0000","last_unstale":"2024-03-18T14:51:03.213560+0000","last_undegraded":"2024-03-18T14:51:03.213560+0000","last_fullsized":"2024-03-18T14:51:03.213560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:49:01.376008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213271+0000","last_change":"2024-03-18T14:51:01.360069+0000","last_active":"2024-03-18T14:51:03.213271+0000","last_peered":"2024-03-18T14:51:03.213271+0000","last_clean":"2024-03-18T14:51:03.213271+0000","last_became_active":"2024-03-18T14:51:01.358960+0000","last_became_peered":"2024-03-18T14:51:01.358960+0000","last_unstale":"2024-03-18T14:51:03.213271+0000","last_undegraded":"2024-03-18T14:51:03.213271+0000","last_fullsized":"2024-03-18T14:51:03.213271+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:48:42.197329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213054+0000","last_change":"2024-03-18T14:51:01.361200+0000","last_active":"2024-03-18T14:51:03.213054+0000","last_peered":"2024-03-18T14:51:03.213054+0000","last_clean":"2024-03-18T14:51:03.213054+0000","last_became_active":"2024-03-18T14:51:01.360879+0000","last_became_peered":"2024-03-18T14:51:01.360879+0000","last_unstale":"2024-03-18T14:51:03.213054+0000","last_undegraded":"2024-03-18T14:51:03.213054+0000","last_fullsized":"2024-03-18T14:51:03.213054+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:05.335627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213085+0000","last_change":"2024-03-18T14:51:01.352441+0000","last_active":"2024-03-18T14:51:03.213085+0000","last_peered":"2024-03-18T14:51:03.213085+0000","last_clean":"2024-03-18T14:51:03.213085+0000","last_became_active":"2024-03-18T14:51:01.352118+0000","last_became_peered":"2024-03-18T14:51:01.352118+0000","last_unstale":"2024-03-18T14:51:03.213085+0000","last_undegraded":"2024-03-18T14:51:03.213085+0000","last_fullsized":"2024-03-18T14:51:03.213085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:45:42.781439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.211890+0000","last_change":"2024-03-18T14:51:01.361257+0000","last_active":"2024-03-18T14:51:03.211890+0000","last_peered":"2024-03-18T14:51:03.211890+0000","last_clean":"2024-03-18T14:51:03.211890+0000","last_became_active":"2024-03-18T14:51:01.360898+0000","last_became_peered":"2024-03-18T14:51:01.360898+0000","last_unstale":"2024-03-18T14:51:03.211890+0000","last_undegraded":"2024-03-18T14:51:03.211890+0000","last_fullsized":"2024-03-18T14:51:03.211890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:38:26.200678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.211851+0000","last_change":"2024-03-18T14:51:01.366131+0000","last_active":"2024-03-18T14:51:03.211851+0000","last_peered":"2024-03-18T14:51:03.211851+0000","last_clean":"2024-03-18T14:51:03.211851+0000","last_became_active":"2024-03-18T14:51:01.365796+0000","last_became_peered":"2024-03-18T14:51:01.365796+0000","last_unstale":"2024-03-18T14:51:03.211851+0000","last_undegraded":"2024-03-18T14:51:03.211851+0000","last_fullsized":"2024-03-18T14:51:03.211851+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:57:29.526032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.344720+0000","last_change":"2024-03-18T14:51:01.357223+0000","last_active":"2024-03-18T14:51:02.344720+0000","last_peered":"2024-03-18T14:51:02.344720+0000","last_clean":"2024-03-18T14:51:02.344720+0000","last_became_active":"2024-03-18T14:51:01.356934+0000","last_became_peered":"2024-03-18T14:51:01.356934+0000","last_unstale":"2024-03-18T14:51:02.344720+0000","last_undegraded":"2024-03-18T14:51:02.344720+0000","last_fullsized":"2024-03-18T14:51:02.344720+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:28.683605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213063+0000","last_change":"2024-03-18T14:51:01.366414+0000","last_active":"2024-03-18T14:51:03.213063+0000","last_peered":"2024-03-18T14:51:03.213063+0000","last_clean":"2024-03-18T14:51:03.213063+0000","last_became_active":"2024-03-18T14:51:01.365534+0000","last_became_peered":"2024-03-18T14:51:01.365534+0000","last_unstale":"2024-03-18T14:51:03.213063+0000","last_undegraded":"2024-03-18T14:51:03.213063+0000","last_fullsized":"2024-03-18T14:51:03.213063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:28:51.568662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212484+0000","last_change":"2024-03-18T14:51:01.354084+0000","last_active":"2024-03-18T14:51:03.212484+0000","last_peered":"2024-03-18T14:51:03.212484+0000","last_clean":"2024-03-18T14:51:03.212484+0000","last_became_active":"2024-03-18T14:51:01.353599+0000","last_became_peered":"2024-03-18T14:51:01.353599+0000","last_unstale":"2024-03-18T14:51:03.212484+0000","last_undegraded":"2024-03-18T14:51:03.212484+0000","last_fullsized":"2024-03-18T14:51:03.212484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:40.502783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212538+0000","last_change":"2024-03-18T14:51:01.353391+0000","last_active":"2024-03-18T14:51:03.212538+0000","last_peered":"2024-03-18T14:51:03.212538+0000","last_clean":"2024-03-18T14:51:03.212538+0000","last_became_active":"2024-03-18T14:51:01.353084+0000","last_became_peered":"2024-03-18T14:51:01.353084+0000","last_unstale":"2024-03-18T14:51:03.212538+0000","last_undegraded":"2024-03-18T14:51:03.212538+0000","last_fullsized":"2024-03-18T14:51:03.212538+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:18:02.774589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:06.036513+0000","last_change":"2024-03-18T14:51:00.942617+0000","last_active":"2024-03-18T14:51:06.036513+0000","last_peered":"2024-03-18T14:51:06.036513+0000","last_clean":"2024-03-18T14:51:06.036513+0000","last_became_active":"2024-03-18T14:51:00.942276+0000","last_became_peered":"2024-03-18T14:51:00.942276+0000","last_unstale":"2024-03-18T14:51:06.036513+0000","last_undegraded":"2024-03-18T14:51:06.036513+0000","last_fullsized":"2024-03-18T14:51:06.036513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:03:05.984066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.343269+0000","last_change":"2024-03-18T14:51:01.363887+0000","last_active":"2024-03-18T14:51:02.343269+0000","last_peered":"2024-03-18T14:51:02.343269+0000","last_clean":"2024-03-18T14:51:02.343269+0000","last_became_active":"2024-03-18T14:51:01.361359+0000","last_became_peered":"2024-03-18T14:51:01.361359+0000","last_unstale":"2024-03-18T14:51:02.343269+0000","last_undegraded":"2024-03-18T14:51:02.343269+0000","last_fullsized":"2024-03-18T14:51:02.343269+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:22.996930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.343494+0000","last_change":"2024-03-18T14:51:00.940392+0000","last_active":"2024-03-18T14:51:02.343494+0000","last_peered":"2024-03-18T14:51:02.343494+0000","last_clean":"2024-03-18T14:51:02.343494+0000","last_became_active":"2024-03-18T14:51:00.940055+0000","last_became_peered":"2024-03-18T14:51:00.940055+0000","last_unstale":"2024-03-18T14:51:02.343494+0000","last_undegraded":"2024-03-18T14:51:02.343494+0000","last_fullsized":"2024-03-18T14:51:02.343494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:57:28.523549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212843+0000","last_change":"2024-03-18T14:51:01.361458+0000","last_active":"2024-03-18T14:51:03.212843+0000","last_peered":"2024-03-18T14:51:03.212843+0000","last_clean":"2024-03-18T14:51:03.212843+0000","last_became_active":"2024-03-18T14:51:01.361105+0000","last_became_peered":"2024-03-18T14:51:01.361105+0000","last_unstale":"2024-03-18T14:51:03.212843+0000","last_undegraded":"2024-03-18T14:51:03.212843+0000","last_fullsized":"2024-03-18T14:51:03.212843+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:22:59.410080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.093033+0000","last_change":"2024-03-18T14:51:00.938440+0000","last_active":"2024-03-18T14:51:03.093033+0000","last_peered":"2024-03-18T14:51:03.093033+0000","last_clean":"2024-03-18T14:51:03.093033+0000","last_became_active":"2024-03-18T14:51:00.938046+0000","last_became_peered":"2024-03-18T14:51:00.938046+0000","last_unstale":"2024-03-18T14:51:03.093033+0000","last_undegraded":"2024-03-18T14:51:03.093033+0000","last_fullsized":"2024-03-18T14:51:03.093033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:25:43.414853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.344839+0000","last_change":"2024-03-18T14:51:01.366305+0000","last_active":"2024-03-18T14:51:02.344839+0000","last_peered":"2024-03-18T14:51:02.344839+0000","last_clean":"2024-03-18T14:51:02.344839+0000","last_became_active":"2024-03-18T14:51:01.364626+0000","last_became_peered":"2024-03-18T14:51:01.364626+0000","last_unstale":"2024-03-18T14:51:02.344839+0000","last_undegraded":"2024-03-18T14:51:02.344839+0000","last_fullsized":"2024-03-18T14:51:02.344839+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:06.212030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.092844+0000","last_change":"2024-03-18T14:51:00.932514+0000","last_active":"2024-03-18T14:51:03.092844+0000","last_peered":"2024-03-18T14:51:03.092844+0000","last_clean":"2024-03-18T14:51:03.092844+0000","last_became_active":"2024-03-18T14:51:00.932183+0000","last_became_peered":"2024-03-18T14:51:00.932183+0000","last_unstale":"2024-03-18T14:51:03.092844+0000","last_undegraded":"2024-03-18T14:51:03.092844+0000","last_fullsized":"2024-03-18T14:51:03.092844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:36:37.230185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345376+0000","last_change":"2024-03-18T14:51:01.358939+0000","last_active":"2024-03-18T14:51:02.345376+0000","last_peered":"2024-03-18T14:51:02.345376+0000","last_clean":"2024-03-18T14:51:02.345376+0000","last_became_active":"2024-03-18T14:51:01.358601+0000","last_became_peered":"2024-03-18T14:51:01.358601+0000","last_unstale":"2024-03-18T14:51:02.345376+0000","last_undegraded":"2024-03-18T14:51:02.345376+0000","last_fullsized":"2024-03-18T14:51:02.345376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:40:14.876143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.029051+0000","last_change":"2024-03-18T14:51:00.945116+0000","last_active":"2024-03-18T14:51:05.029051+0000","last_peered":"2024-03-18T14:51:05.029051+0000","last_clean":"2024-03-18T14:51:05.029051+0000","last_became_active":"2024-03-18T14:51:00.944791+0000","last_became_peered":"2024-03-18T14:51:00.944791+0000","last_unstale":"2024-03-18T14:51:05.029051+0000","last_undegraded":"2024-03-18T14:51:05.029051+0000","last_fullsized":"2024-03-18T14:51:05.029051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:06.637940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.092793+0000","last_change":"2024-03-18T14:51:01.358682+0000","last_active":"2024-03-18T14:51:03.092793+0000","last_peered":"2024-03-18T14:51:03.092793+0000","last_clean":"2024-03-18T14:51:03.092793+0000","last_became_active":"2024-03-18T14:51:01.358378+0000","last_became_peered":"2024-03-18T14:51:01.358378+0000","last_unstale":"2024-03-18T14:51:03.092793+0000","last_undegraded":"2024-03-18T14:51:03.092793+0000","last_fullsized":"2024-03-18T14:51:03.092793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:42.697944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:06.035904+0000","last_change":"2024-03-18T14:51:00.943976+0000","last_active":"2024-03-18T14:51:06.035904+0000","last_peered":"2024-03-18T14:51:06.035904+0000","last_clean":"2024-03-18T14:51:06.035904+0000","last_became_active":"2024-03-18T14:51:00.943614+0000","last_became_peered":"2024-03-18T14:51:00.943614+0000","last_unstale":"2024-03-18T14:51:06.035904+0000","last_undegraded":"2024-03-18T14:51:06.035904+0000","last_fullsized":"2024-03-18T14:51:06.035904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:07:00.744708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345093+0000","last_change":"2024-03-18T14:51:01.365316+0000","last_active":"2024-03-18T14:51:02.345093+0000","last_peered":"2024-03-18T14:51:02.345093+0000","last_clean":"2024-03-18T14:51:02.345093+0000","last_became_active":"2024-03-18T14:51:01.365019+0000","last_became_peered":"2024-03-18T14:51:01.365019+0000","last_unstale":"2024-03-18T14:51:02.345093+0000","last_undegraded":"2024-03-18T14:51:02.345093+0000","last_fullsized":"2024-03-18T14:51:02.345093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:39:59.040907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212226+0000","last_change":"2024-03-18T14:51:00.936923+0000","last_active":"2024-03-18T14:51:03.212226+0000","last_peered":"2024-03-18T14:51:03.212226+0000","last_clean":"2024-03-18T14:51:03.212226+0000","last_became_active":"2024-03-18T14:51:00.936591+0000","last_became_peered":"2024-03-18T14:51:00.936591+0000","last_unstale":"2024-03-18T14:51:03.212226+0000","last_undegraded":"2024-03-18T14:51:03.212226+0000","last_fullsized":"2024-03-18T14:51:03.212226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:48:46.423033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.342666+0000","last_change":"2024-03-18T14:51:01.356827+0000","last_active":"2024-03-18T14:51:02.342666+0000","last_peered":"2024-03-18T14:51:02.342666+0000","last_clean":"2024-03-18T14:51:02.342666+0000","last_became_active":"2024-03-18T14:51:01.356148+0000","last_became_peered":"2024-03-18T14:51:01.356148+0000","last_unstale":"2024-03-18T14:51:02.342666+0000","last_undegraded":"2024-03-18T14:51:02.342666+0000","last_fullsized":"2024-03-18T14:51:02.342666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:18.477890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.344906+0000","last_change":"2024-03-18T14:51:00.934024+0000","last_active":"2024-03-18T14:51:02.344906+0000","last_peered":"2024-03-18T14:51:02.344906+0000","last_clean":"2024-03-18T14:51:02.344906+0000","last_became_active":"2024-03-18T14:51:00.933739+0000","last_became_peered":"2024-03-18T14:51:00.933739+0000","last_unstale":"2024-03-18T14:51:02.344906+0000","last_undegraded":"2024-03-18T14:51:02.344906+0000","last_fullsized":"2024-03-18T14:51:02.344906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:03.967260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212216+0000","last_change":"2024-03-18T14:51:01.356519+0000","last_active":"2024-03-18T14:51:03.212216+0000","last_peered":"2024-03-18T14:51:03.212216+0000","last_clean":"2024-03-18T14:51:03.212216+0000","last_became_active":"2024-03-18T14:51:01.356193+0000","last_became_peered":"2024-03-18T14:51:01.356193+0000","last_unstale":"2024-03-18T14:51:03.212216+0000","last_undegraded":"2024-03-18T14:51:03.212216+0000","last_fullsized":"2024-03-18T14:51:03.212216+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:34:59.633241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.026438+0000","last_change":"2024-03-18T14:51:00.940189+0000","last_active":"2024-03-18T14:51:05.026438+0000","last_peered":"2024-03-18T14:51:05.026438+0000","last_clean":"2024-03-18T14:51:05.026438+0000","last_became_active":"2024-03-18T14:51:00.939820+0000","last_became_peered":"2024-03-18T14:51:00.939820+0000","last_unstale":"2024-03-18T14:51:05.026438+0000","last_undegraded":"2024-03-18T14:51:05.026438+0000","last_fullsized":"2024-03-18T14:51:05.026438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:44:09.749747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345071+0000","last_change":"2024-03-18T14:51:01.355756+0000","last_active":"2024-03-18T14:51:02.345071+0000","last_peered":"2024-03-18T14:51:02.345071+0000","last_clean":"2024-03-18T14:51:02.345071+0000","last_became_active":"2024-03-18T14:51:01.352871+0000","last_became_peered":"2024-03-18T14:51:01.352871+0000","last_unstale":"2024-03-18T14:51:02.345071+0000","last_undegraded":"2024-03-18T14:51:02.345071+0000","last_fullsized":"2024-03-18T14:51:02.345071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:20:26.533066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.026367+0000","last_change":"2024-03-18T14:51:00.941908+0000","last_active":"2024-03-18T14:51:05.026367+0000","last_peered":"2024-03-18T14:51:05.026367+0000","last_clean":"2024-03-18T14:51:05.026367+0000","last_became_active":"2024-03-18T14:51:00.940978+0000","last_became_peered":"2024-03-18T14:51:00.940978+0000","last_unstale":"2024-03-18T14:51:05.026367+0000","last_undegraded":"2024-03-18T14:51:05.026367+0000","last_fullsized":"2024-03-18T14:51:05.026367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:52.705243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212743+0000","last_change":"2024-03-18T14:51:01.363462+0000","last_active":"2024-03-18T14:51:03.212743+0000","last_peered":"2024-03-18T14:51:03.212743+0000","last_clean":"2024-03-18T14:51:03.212743+0000","last_became_active":"2024-03-18T14:51:01.362262+0000","last_became_peered":"2024-03-18T14:51:01.362262+0000","last_unstale":"2024-03-18T14:51:03.212743+0000","last_undegraded":"2024-03-18T14:51:03.212743+0000","last_fullsized":"2024-03-18T14:51:03.212743+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:07:18.952876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345331+0000","last_change":"2024-03-18T14:51:00.936050+0000","last_active":"2024-03-18T14:51:02.345331+0000","last_peered":"2024-03-18T14:51:02.345331+0000","last_clean":"2024-03-18T14:51:02.345331+0000","last_became_active":"2024-03-18T14:51:00.935676+0000","last_became_peered":"2024-03-18T14:51:00.935676+0000","last_unstale":"2024-03-18T14:51:02.345331+0000","last_undegraded":"2024-03-18T14:51:02.345331+0000","last_fullsized":"2024-03-18T14:51:02.345331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:29.423110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345221+0000","last_change":"2024-03-18T14:51:01.364295+0000","last_active":"2024-03-18T14:51:02.345221+0000","last_peered":"2024-03-18T14:51:02.345221+0000","last_clean":"2024-03-18T14:51:02.345221+0000","last_became_active":"2024-03-18T14:51:01.363916+0000","last_became_peered":"2024-03-18T14:51:01.363916+0000","last_unstale":"2024-03-18T14:51:02.345221+0000","last_undegraded":"2024-03-18T14:51:02.345221+0000","last_fullsized":"2024-03-18T14:51:02.345221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:21:17.905819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:07.039768+0000","last_change":"2024-03-18T14:51:00.936364+0000","last_active":"2024-03-18T14:51:07.039768+0000","last_peered":"2024-03-18T14:51:07.039768+0000","last_clean":"2024-03-18T14:51:07.039768+0000","last_became_active":"2024-03-18T14:51:00.936056+0000","last_became_peered":"2024-03-18T14:51:00.936056+0000","last_unstale":"2024-03-18T14:51:07.039768+0000","last_undegraded":"2024-03-18T14:51:07.039768+0000","last_fullsized":"2024-03-18T14:51:07.039768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:48:19.044660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.092980+0000","last_change":"2024-03-18T14:51:01.362426+0000","last_active":"2024-03-18T14:51:03.092980+0000","last_peered":"2024-03-18T14:51:03.092980+0000","last_clean":"2024-03-18T14:51:03.092980+0000","last_became_active":"2024-03-18T14:51:01.362130+0000","last_became_peered":"2024-03-18T14:51:01.362130+0000","last_unstale":"2024-03-18T14:51:03.092980+0000","last_undegraded":"2024-03-18T14:51:03.092980+0000","last_fullsized":"2024-03-18T14:51:03.092980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:14.545997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:06.034524+0000","last_change":"2024-03-18T14:51:00.934028+0000","last_active":"2024-03-18T14:51:06.034524+0000","last_peered":"2024-03-18T14:51:06.034524+0000","last_clean":"2024-03-18T14:51:06.034524+0000","last_became_active":"2024-03-18T14:51:00.933594+0000","last_became_peered":"2024-03-18T14:51:00.933594+0000","last_unstale":"2024-03-18T14:51:06.034524+0000","last_undegraded":"2024-03-18T14:51:06.034524+0000","last_fullsized":"2024-03-18T14:51:06.034524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:55:08.675533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212583+0000","last_change":"2024-03-18T14:51:01.353544+0000","last_active":"2024-03-18T14:51:03.212583+0000","last_peered":"2024-03-18T14:51:03.212583+0000","last_clean":"2024-03-18T14:51:03.212583+0000","last_became_active":"2024-03-18T14:51:01.353241+0000","last_became_peered":"2024-03-18T14:51:01.353241+0000","last_unstale":"2024-03-18T14:51:03.212583+0000","last_undegraded":"2024-03-18T14:51:03.212583+0000","last_fullsized":"2024-03-18T14:51:03.212583+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:45:27.617681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.026529+0000","last_change":"2024-03-18T14:51:00.944492+0000","last_active":"2024-03-18T14:51:05.026529+0000","last_peered":"2024-03-18T14:51:05.026529+0000","last_clean":"2024-03-18T14:51:05.026529+0000","last_became_active":"2024-03-18T14:51:00.944165+0000","last_became_peered":"2024-03-18T14:51:00.944165+0000","last_unstale":"2024-03-18T14:51:05.026529+0000","last_undegraded":"2024-03-18T14:51:05.026529+0000","last_fullsized":"2024-03-18T14:51:05.026529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:47:05.968966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213320+0000","last_change":"2024-03-18T14:51:01.356782+0000","last_active":"2024-03-18T14:51:03.213320+0000","last_peered":"2024-03-18T14:51:03.213320+0000","last_clean":"2024-03-18T14:51:03.213320+0000","last_became_active":"2024-03-18T14:51:01.356242+0000","last_became_peered":"2024-03-18T14:51:01.356242+0000","last_unstale":"2024-03-18T14:51:03.213320+0000","last_undegraded":"2024-03-18T14:51:03.213320+0000","last_fullsized":"2024-03-18T14:51:03.213320+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:56:07.662545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212656+0000","last_change":"2024-03-18T14:51:00.937873+0000","last_active":"2024-03-18T14:51:03.212656+0000","last_peered":"2024-03-18T14:51:03.212656+0000","last_clean":"2024-03-18T14:51:03.212656+0000","last_became_active":"2024-03-18T14:51:00.937562+0000","last_became_peered":"2024-03-18T14:51:00.937562+0000","last_unstale":"2024-03-18T14:51:03.212656+0000","last_undegraded":"2024-03-18T14:51:03.212656+0000","last_fullsized":"2024-03-18T14:51:03.212656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:36:44.342414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212245+0000","last_change":"2024-03-18T14:51:01.358681+0000","last_active":"2024-03-18T14:51:03.212245+0000","last_peered":"2024-03-18T14:51:03.212245+0000","last_clean":"2024-03-18T14:51:03.212245+0000","last_became_active":"2024-03-18T14:51:01.358374+0000","last_became_peered":"2024-03-18T14:51:01.358374+0000","last_unstale":"2024-03-18T14:51:03.212245+0000","last_undegraded":"2024-03-18T14:51:03.212245+0000","last_fullsized":"2024-03-18T14:51:03.212245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:43:47.804271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'10","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.026917+0000","last_change":"2024-03-18T14:51:00.935639+0000","last_active":"2024-03-18T14:51:05.026917+0000","last_peered":"2024-03-18T14:51:05.026917+0000","last_clean":"2024-03-18T14:51:05.026917+0000","last_became_active":"2024-03-18T14:51:00.935334+0000","last_became_peered":"2024-03-18T14:51:00.935334+0000","last_unstale":"2024-03-18T14:51:05.026917+0000","last_undegraded":"2024-03-18T14:51:05.026917+0000","last_fullsized":"2024-03-18T14:51:05.026917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:46.966812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345011+0000","last_change":"2024-03-18T14:51:01.364755+0000","last_active":"2024-03-18T14:51:02.345011+0000","last_peered":"2024-03-18T14:51:02.345011+0000","last_clean":"2024-03-18T14:51:02.345011+0000","last_became_active":"2024-03-18T14:51:01.364448+0000","last_became_peered":"2024-03-18T14:51:01.364448+0000","last_unstale":"2024-03-18T14:51:02.345011+0000","last_undegraded":"2024-03-18T14:51:02.345011+0000","last_fullsized":"2024-03-18T14:51:02.345011+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:37:49.480787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.093047+0000","last_change":"2024-03-18T14:51:00.933271+0000","last_active":"2024-03-18T14:51:03.093047+0000","last_peered":"2024-03-18T14:51:03.093047+0000","last_clean":"2024-03-18T14:51:03.093047+0000","last_became_active":"2024-03-18T14:51:00.932928+0000","last_became_peered":"2024-03-18T14:51:00.932928+0000","last_unstale":"2024-03-18T14:51:03.093047+0000","last_undegraded":"2024-03-18T14:51:03.093047+0000","last_fullsized":"2024-03-18T14:51:03.093047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:11:55.281763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212703+0000","last_change":"2024-03-18T14:51:01.362911+0000","last_active":"2024-03-18T14:51:03.212703+0000","last_peered":"2024-03-18T14:51:03.212703+0000","last_clean":"2024-03-18T14:51:03.212703+0000","last_became_active":"2024-03-18T14:51:01.362598+0000","last_became_peered":"2024-03-18T14:51:01.362598+0000","last_unstale":"2024-03-18T14:51:03.212703+0000","last_undegraded":"2024-03-18T14:51:03.212703+0000","last_fullsized":"2024-03-18T14:51:03.212703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:22:58.627753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212158+0000","last_change":"2024-03-18T14:51:00.935554+0000","last_active":"2024-03-18T14:51:03.212158+0000","last_peered":"2024-03-18T14:51:03.212158+0000","last_clean":"2024-03-18T14:51:03.212158+0000","last_became_active":"2024-03-18T14:51:00.935184+0000","last_became_peered":"2024-03-18T14:51:00.935184+0000","last_unstale":"2024-03-18T14:51:03.212158+0000","last_undegraded":"2024-03-18T14:51:03.212158+0000","last_fullsized":"2024-03-18T14:51:03.212158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:21:13.973570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213198+0000","last_change":"2024-03-18T14:51:01.355990+0000","last_active":"2024-03-18T14:51:03.213198+0000","last_peered":"2024-03-18T14:51:03.213198+0000","last_clean":"2024-03-18T14:51:03.213198+0000","last_became_active":"2024-03-18T14:51:01.355692+0000","last_became_peered":"2024-03-18T14:51:01.355692+0000","last_unstale":"2024-03-18T14:51:03.213198+0000","last_undegraded":"2024-03-18T14:51:03.213198+0000","last_fullsized":"2024-03-18T14:51:03.213198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:27:19.742684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212425+0000","last_change":"2024-03-18T14:51:00.943947+0000","last_active":"2024-03-18T14:51:03.212425+0000","last_peered":"2024-03-18T14:51:03.212425+0000","last_clean":"2024-03-18T14:51:03.212425+0000","last_became_active":"2024-03-18T14:51:00.943565+0000","last_became_peered":"2024-03-18T14:51:00.943565+0000","last_unstale":"2024-03-18T14:51:03.212425+0000","last_undegraded":"2024-03-18T14:51:03.212425+0000","last_fullsized":"2024-03-18T14:51:03.212425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:33:06.103148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212859+0000","last_change":"2024-03-18T14:51:01.363221+0000","last_active":"2024-03-18T14:51:03.212859+0000","last_peered":"2024-03-18T14:51:03.212859+0000","last_clean":"2024-03-18T14:51:03.212859+0000","last_became_active":"2024-03-18T14:51:01.362848+0000","last_became_peered":"2024-03-18T14:51:01.362848+0000","last_unstale":"2024-03-18T14:51:03.212859+0000","last_undegraded":"2024-03-18T14:51:03.212859+0000","last_fullsized":"2024-03-18T14:51:03.212859+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:04.139365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345140+0000","last_change":"2024-03-18T14:51:01.356237+0000","last_active":"2024-03-18T14:51:02.345140+0000","last_peered":"2024-03-18T14:51:02.345140+0000","last_clean":"2024-03-18T14:51:02.345140+0000","last_became_active":"2024-03-18T14:51:01.355922+0000","last_became_peered":"2024-03-18T14:51:01.355922+0000","last_unstale":"2024-03-18T14:51:02.345140+0000","last_undegraded":"2024-03-18T14:51:02.345140+0000","last_fullsized":"2024-03-18T14:51:02.345140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:04:50.491752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212172+0000","last_change":"2024-03-18T14:51:00.936058+0000","last_active":"2024-03-18T14:51:03.212172+0000","last_peered":"2024-03-18T14:51:03.212172+0000","last_clean":"2024-03-18T14:51:03.212172+0000","last_became_active":"2024-03-18T14:51:00.935731+0000","last_became_peered":"2024-03-18T14:51:00.935731+0000","last_unstale":"2024-03-18T14:51:03.212172+0000","last_undegraded":"2024-03-18T14:51:03.212172+0000","last_fullsized":"2024-03-18T14:51:03.212172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:47:49.023943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.342875+0000","last_change":"2024-03-18T14:51:01.359742+0000","last_active":"2024-03-18T14:51:02.342875+0000","last_peered":"2024-03-18T14:51:02.342875+0000","last_clean":"2024-03-18T14:51:02.342875+0000","last_became_active":"2024-03-18T14:51:01.358507+0000","last_became_peered":"2024-03-18T14:51:01.358507+0000","last_unstale":"2024-03-18T14:51:02.342875+0000","last_undegraded":"2024-03-18T14:51:02.342875+0000","last_fullsized":"2024-03-18T14:51:02.342875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:17.129695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.026576+0000","last_change":"2024-03-18T14:51:00.934593+0000","last_active":"2024-03-18T14:51:05.026576+0000","last_peered":"2024-03-18T14:51:05.026576+0000","last_clean":"2024-03-18T14:51:05.026576+0000","last_became_active":"2024-03-18T14:51:00.934253+0000","last_became_peered":"2024-03-18T14:51:00.934253+0000","last_unstale":"2024-03-18T14:51:05.026576+0000","last_undegraded":"2024-03-18T14:51:05.026576+0000","last_fullsized":"2024-03-18T14:51:05.026576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:43:26.323162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":347,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.211833+0000","last_change":"2024-03-18T14:49:51.158965+0000","last_active":"2024-03-18T14:51:03.211833+0000","last_peered":"2024-03-18T14:51:03.211833+0000","last_clean":"2024-03-18T14:51:03.211833+0000","last_became_active":"2024-03-18T14:49:51.158580+0000","last_became_peered":"2024-03-18T14:49:51.158580+0000","last_unstale":"2024-03-18T14:51:03.211833+0000","last_undegraded":"2024-03-18T14:51:03.211833+0000","last_fullsized":"2024-03-18T14:51:03.211833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:46:27.210244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:46:27.210244+0000","last_clean_scrub_stamp":"2024-03-18T14:46:27.210244+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:37:08.104617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212902+0000","last_change":"2024-03-18T14:51:01.359972+0000","last_active":"2024-03-18T14:51:03.212902+0000","last_peered":"2024-03-18T14:51:03.212902+0000","last_clean":"2024-03-18T14:51:03.212902+0000","last_became_active":"2024-03-18T14:51:01.359430+0000","last_became_peered":"2024-03-18T14:51:01.359430+0000","last_unstale":"2024-03-18T14:51:03.212902+0000","last_undegraded":"2024-03-18T14:51:03.212902+0000","last_fullsized":"2024-03-18T14:51:03.212902+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:00:14.099499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212428+0000","last_change":"2024-03-18T14:51:00.934196+0000","last_active":"2024-03-18T14:51:03.212428+0000","last_peered":"2024-03-18T14:51:03.212428+0000","last_clean":"2024-03-18T14:51:03.212428+0000","last_became_active":"2024-03-18T14:51:00.933875+0000","last_became_peered":"2024-03-18T14:51:00.933875+0000","last_unstale":"2024-03-18T14:51:03.212428+0000","last_undegraded":"2024-03-18T14:51:03.212428+0000","last_fullsized":"2024-03-18T14:51:03.212428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:56.797227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.092942+0000","last_change":"2024-03-18T14:51:00.941862+0000","last_active":"2024-03-18T14:51:03.092942+0000","last_peered":"2024-03-18T14:51:03.092942+0000","last_clean":"2024-03-18T14:51:03.092942+0000","last_became_active":"2024-03-18T14:51:00.941536+0000","last_became_peered":"2024-03-18T14:51:00.941536+0000","last_unstale":"2024-03-18T14:51:03.092942+0000","last_undegraded":"2024-03-18T14:51:03.092942+0000","last_fullsized":"2024-03-18T14:51:03.092942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:56.364552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213093+0000","last_change":"2024-03-18T14:51:01.352404+0000","last_active":"2024-03-18T14:51:03.213093+0000","last_peered":"2024-03-18T14:51:03.213093+0000","last_clean":"2024-03-18T14:51:03.213093+0000","last_became_active":"2024-03-18T14:51:01.352077+0000","last_became_peered":"2024-03-18T14:51:01.352077+0000","last_unstale":"2024-03-18T14:51:03.213093+0000","last_undegraded":"2024-03-18T14:51:03.213093+0000","last_fullsized":"2024-03-18T14:51:03.213093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:30:38.050712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:06.034970+0000","last_change":"2024-03-18T14:51:00.935564+0000","last_active":"2024-03-18T14:51:06.034970+0000","last_peered":"2024-03-18T14:51:06.034970+0000","last_clean":"2024-03-18T14:51:06.034970+0000","last_became_active":"2024-03-18T14:51:00.935268+0000","last_became_peered":"2024-03-18T14:51:00.935268+0000","last_unstale":"2024-03-18T14:51:06.034970+0000","last_undegraded":"2024-03-18T14:51:06.034970+0000","last_fullsized":"2024-03-18T14:51:06.034970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:22:33.504184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.343508+0000","last_change":"2024-03-18T14:51:01.356096+0000","last_active":"2024-03-18T14:51:02.343508+0000","last_peered":"2024-03-18T14:51:02.343508+0000","last_clean":"2024-03-18T14:51:02.343508+0000","last_became_active":"2024-03-18T14:51:01.355688+0000","last_became_peered":"2024-03-18T14:51:01.355688+0000","last_unstale":"2024-03-18T14:51:02.343508+0000","last_undegraded":"2024-03-18T14:51:02.343508+0000","last_fullsized":"2024-03-18T14:51:02.343508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:04:38.678482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.026742+0000","last_change":"2024-03-18T14:51:00.942610+0000","last_active":"2024-03-18T14:51:05.026742+0000","last_peered":"2024-03-18T14:51:05.026742+0000","last_clean":"2024-03-18T14:51:05.026742+0000","last_became_active":"2024-03-18T14:51:00.942203+0000","last_became_peered":"2024-03-18T14:51:00.942203+0000","last_unstale":"2024-03-18T14:51:05.026742+0000","last_undegraded":"2024-03-18T14:51:05.026742+0000","last_fullsized":"2024-03-18T14:51:05.026742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:17:26.324282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212588+0000","last_change":"2024-03-18T14:51:01.362931+0000","last_active":"2024-03-18T14:51:03.212588+0000","last_peered":"2024-03-18T14:51:03.212588+0000","last_clean":"2024-03-18T14:51:03.212588+0000","last_became_active":"2024-03-18T14:51:01.362624+0000","last_became_peered":"2024-03-18T14:51:01.362624+0000","last_unstale":"2024-03-18T14:51:03.212588+0000","last_undegraded":"2024-03-18T14:51:03.212588+0000","last_fullsized":"2024-03-18T14:51:03.212588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:40:52.512871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.025211+0000","last_change":"2024-03-18T14:51:00.938971+0000","last_active":"2024-03-18T14:51:05.025211+0000","last_peered":"2024-03-18T14:51:05.025211+0000","last_clean":"2024-03-18T14:51:05.025211+0000","last_became_active":"2024-03-18T14:51:00.938645+0000","last_became_peered":"2024-03-18T14:51:00.938645+0000","last_unstale":"2024-03-18T14:51:05.025211+0000","last_undegraded":"2024-03-18T14:51:05.025211+0000","last_fullsized":"2024-03-18T14:51:05.025211+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:18:54.999927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.092674+0000","last_change":"2024-03-18T14:51:01.359510+0000","last_active":"2024-03-18T14:51:03.092674+0000","last_peered":"2024-03-18T14:51:03.092674+0000","last_clean":"2024-03-18T14:51:03.092674+0000","last_became_active":"2024-03-18T14:51:01.359192+0000","last_became_peered":"2024-03-18T14:51:01.359192+0000","last_unstale":"2024-03-18T14:51:03.092674+0000","last_undegraded":"2024-03-18T14:51:03.092674+0000","last_fullsized":"2024-03-18T14:51:03.092674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:11:52.942825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.026414+0000","last_change":"2024-03-18T14:51:00.940194+0000","last_active":"2024-03-18T14:51:05.026414+0000","last_peered":"2024-03-18T14:51:05.026414+0000","last_clean":"2024-03-18T14:51:05.026414+0000","last_became_active":"2024-03-18T14:51:00.939861+0000","last_became_peered":"2024-03-18T14:51:00.939861+0000","last_unstale":"2024-03-18T14:51:05.026414+0000","last_undegraded":"2024-03-18T14:51:05.026414+0000","last_fullsized":"2024-03-18T14:51:05.026414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:06:30.314102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212316+0000","last_change":"2024-03-18T14:51:01.355098+0000","last_active":"2024-03-18T14:51:03.212316+0000","last_peered":"2024-03-18T14:51:03.212316+0000","last_clean":"2024-03-18T14:51:03.212316+0000","last_became_active":"2024-03-18T14:51:01.354452+0000","last_became_peered":"2024-03-18T14:51:01.354452+0000","last_unstale":"2024-03-18T14:51:03.212316+0000","last_undegraded":"2024-03-18T14:51:03.212316+0000","last_fullsized":"2024-03-18T14:51:03.212316+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:40:54.650294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212368+0000","last_change":"2024-03-18T14:51:00.934585+0000","last_active":"2024-03-18T14:51:03.212368+0000","last_peered":"2024-03-18T14:51:03.212368+0000","last_clean":"2024-03-18T14:51:03.212368+0000","last_became_active":"2024-03-18T14:51:00.933660+0000","last_became_peered":"2024-03-18T14:51:00.933660+0000","last_unstale":"2024-03-18T14:51:03.212368+0000","last_undegraded":"2024-03-18T14:51:03.212368+0000","last_fullsized":"2024-03-18T14:51:03.212368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:29:01.144836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.343188+0000","last_change":"2024-03-18T14:51:01.357920+0000","last_active":"2024-03-18T14:51:02.343188+0000","last_peered":"2024-03-18T14:51:02.343188+0000","last_clean":"2024-03-18T14:51:02.343188+0000","last_became_active":"2024-03-18T14:51:01.357565+0000","last_became_peered":"2024-03-18T14:51:01.357565+0000","last_unstale":"2024-03-18T14:51:02.343188+0000","last_undegraded":"2024-03-18T14:51:02.343188+0000","last_fullsized":"2024-03-18T14:51:02.343188+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:39:57.880178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:05.027002+0000","last_change":"2024-03-18T14:51:00.937887+0000","last_active":"2024-03-18T14:51:05.027002+0000","last_peered":"2024-03-18T14:51:05.027002+0000","last_clean":"2024-03-18T14:51:05.027002+0000","last_became_active":"2024-03-18T14:51:00.937588+0000","last_became_peered":"2024-03-18T14:51:00.937588+0000","last_unstale":"2024-03-18T14:51:05.027002+0000","last_undegraded":"2024-03-18T14:51:05.027002+0000","last_fullsized":"2024-03-18T14:51:05.027002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:43:37.558412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212616+0000","last_change":"2024-03-18T14:51:01.354175+0000","last_active":"2024-03-18T14:51:03.212616+0000","last_peered":"2024-03-18T14:51:03.212616+0000","last_clean":"2024-03-18T14:51:03.212616+0000","last_became_active":"2024-03-18T14:51:01.353839+0000","last_became_peered":"2024-03-18T14:51:01.353839+0000","last_unstale":"2024-03-18T14:51:03.212616+0000","last_undegraded":"2024-03-18T14:51:03.212616+0000","last_fullsized":"2024-03-18T14:51:03.212616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:21:52.372938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:06.036037+0000","last_change":"2024-03-18T14:51:00.941600+0000","last_active":"2024-03-18T14:51:06.036037+0000","last_peered":"2024-03-18T14:51:06.036037+0000","last_clean":"2024-03-18T14:51:06.036037+0000","last_became_active":"2024-03-18T14:51:00.941291+0000","last_became_peered":"2024-03-18T14:51:00.941291+0000","last_unstale":"2024-03-18T14:51:06.036037+0000","last_undegraded":"2024-03-18T14:51:06.036037+0000","last_fullsized":"2024-03-18T14:51:06.036037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:04:11.142029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213191+0000","last_change":"2024-03-18T14:51:01.359058+0000","last_active":"2024-03-18T14:51:03.213191+0000","last_peered":"2024-03-18T14:51:03.213191+0000","last_clean":"2024-03-18T14:51:03.213191+0000","last_became_active":"2024-03-18T14:51:01.358437+0000","last_became_peered":"2024-03-18T14:51:01.358437+0000","last_unstale":"2024-03-18T14:51:03.213191+0000","last_undegraded":"2024-03-18T14:51:03.213191+0000","last_fullsized":"2024-03-18T14:51:03.213191+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:39:58.945080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345181+0000","last_change":"2024-03-18T14:51:00.933647+0000","last_active":"2024-03-18T14:51:02.345181+0000","last_peered":"2024-03-18T14:51:02.345181+0000","last_clean":"2024-03-18T14:51:02.345181+0000","last_became_active":"2024-03-18T14:51:00.933320+0000","last_became_peered":"2024-03-18T14:51:00.933320+0000","last_unstale":"2024-03-18T14:51:02.345181+0000","last_undegraded":"2024-03-18T14:51:02.345181+0000","last_fullsized":"2024-03-18T14:51:02.345181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:16:46.498845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212920+0000","last_change":"2024-03-18T14:51:01.359874+0000","last_active":"2024-03-18T14:51:03.212920+0000","last_peered":"2024-03-18T14:51:03.212920+0000","last_clean":"2024-03-18T14:51:03.212920+0000","last_became_active":"2024-03-18T14:51:01.359335+0000","last_became_peered":"2024-03-18T14:51:01.359335+0000","last_unstale":"2024-03-18T14:51:03.212920+0000","last_undegraded":"2024-03-18T14:51:03.212920+0000","last_fullsized":"2024-03-18T14:51:03.212920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:03.583402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212929+0000","last_change":"2024-03-18T14:51:00.935755+0000","last_active":"2024-03-18T14:51:03.212929+0000","last_peered":"2024-03-18T14:51:03.212929+0000","last_clean":"2024-03-18T14:51:03.212929+0000","last_became_active":"2024-03-18T14:51:00.935421+0000","last_became_peered":"2024-03-18T14:51:00.935421+0000","last_unstale":"2024-03-18T14:51:03.212929+0000","last_undegraded":"2024-03-18T14:51:03.212929+0000","last_fullsized":"2024-03-18T14:51:03.212929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:46:11.680972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213443+0000","last_change":"2024-03-18T14:51:01.355605+0000","last_active":"2024-03-18T14:51:03.213443+0000","last_peered":"2024-03-18T14:51:03.213443+0000","last_clean":"2024-03-18T14:51:03.213443+0000","last_became_active":"2024-03-18T14:51:01.355295+0000","last_became_peered":"2024-03-18T14:51:01.355295+0000","last_unstale":"2024-03-18T14:51:03.213443+0000","last_undegraded":"2024-03-18T14:51:03.213443+0000","last_fullsized":"2024-03-18T14:51:03.213443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:15:20.466314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212467+0000","last_change":"2024-03-18T14:51:01.359704+0000","last_active":"2024-03-18T14:51:03.212467+0000","last_peered":"2024-03-18T14:51:03.212467+0000","last_clean":"2024-03-18T14:51:03.212467+0000","last_became_active":"2024-03-18T14:51:01.359367+0000","last_became_peered":"2024-03-18T14:51:01.359367+0000","last_unstale":"2024-03-18T14:51:03.212467+0000","last_undegraded":"2024-03-18T14:51:03.212467+0000","last_fullsized":"2024-03-18T14:51:03.212467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:26.933993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212607+0000","last_change":"2024-03-18T14:51:01.357440+0000","last_active":"2024-03-18T14:51:03.212607+0000","last_peered":"2024-03-18T14:51:03.212607+0000","last_clean":"2024-03-18T14:51:03.212607+0000","last_became_active":"2024-03-18T14:51:01.356537+0000","last_became_peered":"2024-03-18T14:51:01.356537+0000","last_unstale":"2024-03-18T14:51:03.212607+0000","last_undegraded":"2024-03-18T14:51:03.212607+0000","last_fullsized":"2024-03-18T14:51:03.212607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:01.389053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212231+0000","last_change":"2024-03-18T14:51:01.355678+0000","last_active":"2024-03-18T14:51:03.212231+0000","last_peered":"2024-03-18T14:51:03.212231+0000","last_clean":"2024-03-18T14:51:03.212231+0000","last_became_active":"2024-03-18T14:51:01.355197+0000","last_became_peered":"2024-03-18T14:51:01.355197+0000","last_unstale":"2024-03-18T14:51:03.212231+0000","last_undegraded":"2024-03-18T14:51:03.212231+0000","last_fullsized":"2024-03-18T14:51:03.212231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:34.194834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.211933+0000","last_change":"2024-03-18T14:51:01.363964+0000","last_active":"2024-03-18T14:51:03.211933+0000","last_peered":"2024-03-18T14:51:03.211933+0000","last_clean":"2024-03-18T14:51:03.211933+0000","last_became_active":"2024-03-18T14:51:01.363590+0000","last_became_peered":"2024-03-18T14:51:01.363590+0000","last_unstale":"2024-03-18T14:51:03.211933+0000","last_undegraded":"2024-03-18T14:51:03.211933+0000","last_fullsized":"2024-03-18T14:51:03.211933+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:59.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.213137+0000","last_change":"2024-03-18T14:51:01.361599+0000","last_active":"2024-03-18T14:51:03.213137+0000","last_peered":"2024-03-18T14:51:03.213137+0000","last_clean":"2024-03-18T14:51:03.213137+0000","last_became_active":"2024-03-18T14:51:01.361280+0000","last_became_peered":"2024-03-18T14:51:01.361280+0000","last_unstale":"2024-03-18T14:51:03.213137+0000","last_undegraded":"2024-03-18T14:51:03.213137+0000","last_fullsized":"2024-03-18T14:51:03.213137+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:52:50.572257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212612+0000","last_change":"2024-03-18T14:51:01.360495+0000","last_active":"2024-03-18T14:51:03.212612+0000","last_peered":"2024-03-18T14:51:03.212612+0000","last_clean":"2024-03-18T14:51:03.212612+0000","last_became_active":"2024-03-18T14:51:01.360146+0000","last_became_peered":"2024-03-18T14:51:01.360146+0000","last_unstale":"2024-03-18T14:51:03.212612+0000","last_undegraded":"2024-03-18T14:51:03.212612+0000","last_fullsized":"2024-03-18T14:51:03.212612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:52.048137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212815+0000","last_change":"2024-03-18T14:51:01.357650+0000","last_active":"2024-03-18T14:51:03.212815+0000","last_peered":"2024-03-18T14:51:03.212815+0000","last_clean":"2024-03-18T14:51:03.212815+0000","last_became_active":"2024-03-18T14:51:01.356265+0000","last_became_peered":"2024-03-18T14:51:01.356265+0000","last_unstale":"2024-03-18T14:51:03.212815+0000","last_undegraded":"2024-03-18T14:51:03.212815+0000","last_fullsized":"2024-03-18T14:51:03.212815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:49.328404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.343356+0000","last_change":"2024-03-18T14:51:01.363753+0000","last_active":"2024-03-18T14:51:02.343356+0000","last_peered":"2024-03-18T14:51:02.343356+0000","last_clean":"2024-03-18T14:51:02.343356+0000","last_became_active":"2024-03-18T14:51:01.363411+0000","last_became_peered":"2024-03-18T14:51:01.363411+0000","last_unstale":"2024-03-18T14:51:02.343356+0000","last_undegraded":"2024-03-18T14:51:02.343356+0000","last_fullsized":"2024-03-18T14:51:02.343356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:55.452209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345116+0000","last_change":"2024-03-18T14:51:01.360723+0000","last_active":"2024-03-18T14:51:02.345116+0000","last_peered":"2024-03-18T14:51:02.345116+0000","last_clean":"2024-03-18T14:51:02.345116+0000","last_became_active":"2024-03-18T14:51:01.360130+0000","last_became_peered":"2024-03-18T14:51:01.360130+0000","last_unstale":"2024-03-18T14:51:02.345116+0000","last_undegraded":"2024-03-18T14:51:02.345116+0000","last_fullsized":"2024-03-18T14:51:02.345116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:51.634011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.342707+0000","last_change":"2024-03-18T14:51:01.356462+0000","last_active":"2024-03-18T14:51:02.342707+0000","last_peered":"2024-03-18T14:51:02.342707+0000","last_clean":"2024-03-18T14:51:02.342707+0000","last_became_active":"2024-03-18T14:51:01.355395+0000","last_became_peered":"2024-03-18T14:51:01.355395+0000","last_unstale":"2024-03-18T14:51:02.342707+0000","last_undegraded":"2024-03-18T14:51:02.342707+0000","last_fullsized":"2024-03-18T14:51:02.342707+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:37:15.326354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:02.345048+0000","last_change":"2024-03-18T14:51:01.358626+0000","last_active":"2024-03-18T14:51:02.345048+0000","last_peered":"2024-03-18T14:51:02.345048+0000","last_clean":"2024-03-18T14:51:02.345048+0000","last_became_active":"2024-03-18T14:51:01.358293+0000","last_became_peered":"2024-03-18T14:51:01.358293+0000","last_unstale":"2024-03-18T14:51:02.345048+0000","last_undegraded":"2024-03-18T14:51:02.345048+0000","last_fullsized":"2024-03-18T14:51:02.345048+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:52:41.139885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212980+0000","last_change":"2024-03-18T14:51:01.355276+0000","last_active":"2024-03-18T14:51:03.212980+0000","last_peered":"2024-03-18T14:51:03.212980+0000","last_clean":"2024-03-18T14:51:03.212980+0000","last_became_active":"2024-03-18T14:51:01.354661+0000","last_became_peered":"2024-03-18T14:51:01.354661+0000","last_unstale":"2024-03-18T14:51:03.212980+0000","last_undegraded":"2024-03-18T14:51:03.212980+0000","last_fullsized":"2024-03-18T14:51:03.212980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:28.569055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T14:51:03.212498+0000","last_change":"2024-03-18T14:51:01.363101+0000","last_active":"2024-03-18T14:51:03.212498+0000","last_peered":"2024-03-18T14:51:03.212498+0000","last_clean":"2024-03-18T14:51:03.212498+0000","last_became_active":"2024-03-18T14:51:01.362576+0000","last_became_peered":"2024-03-18T14:51:01.362576+0000","last_unstale":"2024-03-18T14:51:03.212498+0000","last_undegraded":"2024-03-18T14:51:03.212498+0000","last_fullsized":"2024-03-18T14:51:03.212498+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:42:32.109523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280632,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4398,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":104,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":31567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350989,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":921600,"data_stored":441572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":1035130,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776152,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":442390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037788,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":929792,"data_stored":442176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299425,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":1033144,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462950,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":921600,"data_stored":441572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":449784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":925696,"data_stored":441662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":946176,"data_stored":443840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":933888,"data_stored":441766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888113,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":925696,"data_stored":441594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724587,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":983040,"data_stored":1032122,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":20895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:51:10.097 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T14:51:10.098 INFO:tasks.cephadm:Setting up client nodes... 2024-03-18T14:51:10.098 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-18T14:51:10.368 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:10.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:10 smithi076 ceph-mon[32205]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:51:10.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:10 smithi076 ceph-mon[32205]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-18T14:51:10.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:10 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/598636241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:51:10.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:10 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1327407993' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:51:10.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:10 smithi174 ceph-mon[36787]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:51:10.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:10 smithi174 ceph-mon[36787]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-18T14:51:10.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:10 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/598636241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:51:10.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:10 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1327407993' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:51:10.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:10 smithi077 ceph-mon[35865]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:51:10.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:10 smithi077 ceph-mon[35865]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-18T14:51:10.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:10 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/598636241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:51:10.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:10 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1327407993' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:51:11.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:11 smithi076 ceph-mon[32205]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:51:11.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:11 smithi076 ceph-mon[32205]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:51:11.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:11 smithi076 ceph-mon[32205]: pgmap v288: 97 pgs: 97 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3.9 KiB/s wr, 13 op/s 2024-03-18T14:51:11.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:11 smithi174 ceph-mon[36787]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:51:11.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:11 smithi174 ceph-mon[36787]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:51:11.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:11 smithi174 ceph-mon[36787]: pgmap v288: 97 pgs: 97 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3.9 KiB/s wr, 13 op/s 2024-03-18T14:51:11.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:11 smithi077 ceph-mon[35865]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:51:11.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:11 smithi077 ceph-mon[35865]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:51:11.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:11 smithi077 ceph-mon[35865]: pgmap v288: 97 pgs: 97 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3.9 KiB/s wr, 13 op/s 2024-03-18T14:51:12.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.082+0000 7f9ae846b700 1 -- 172.21.15.76:0/2649500190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ae0108000 msgr2=0x7f9ae010a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:12.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.082+0000 7f9ae846b700 1 --2- 172.21.15.76:0/2649500190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ae0108000 0x7f9ae010a470 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8009990 tx=0x7f9ad8009ca0 comp rx=0 tx=0).stop 2024-03-18T14:51:12.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.082+0000 7f9ae846b700 1 -- 172.21.15.76:0/2649500190 shutdown_connections 2024-03-18T14:51:12.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.082+0000 7f9ae846b700 1 --2- 172.21.15.76:0/2649500190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ae0108000 0x7f9ae010a470 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:12.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.082+0000 7f9ae846b700 1 --2- 172.21.15.76:0/2649500190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ae0102da0 0x7f9ae0105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:12.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.082+0000 7f9ae846b700 1 --2- 172.21.15.76:0/2649500190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae01056d0 0x7f9ae0107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:12.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.082+0000 7f9ae846b700 1 -- 172.21.15.76:0/2649500190 >> 172.21.15.76:0/2649500190 conn(0x7f9ae00fc7d0 msgr2=0x7f9ae00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:12.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.082+0000 7f9ae846b700 1 -- 172.21.15.76:0/2649500190 shutdown_connections 2024-03-18T14:51:12.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.082+0000 7f9ae846b700 1 -- 172.21.15.76:0/2649500190 wait complete. 2024-03-18T14:51:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.083+0000 7f9ae846b700 1 Processor -- start 2024-03-18T14:51:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.083+0000 7f9ae846b700 1 -- start start 2024-03-18T14:51:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.083+0000 7f9ae846b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0102da0 0x7f9ae019a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.083+0000 7f9ae846b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ae01056d0 0x7f9ae019ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.083+0000 7f9ae846b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ae0108000 0x7f9ae01a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.083+0000 7f9ae846b700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae010df20 con 0x7f9ae0108000 2024-03-18T14:51:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.083+0000 7f9ae846b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae010dda0 con 0x7f9ae0102da0 2024-03-18T14:51:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.083+0000 7f9ae846b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae010e0a0 con 0x7f9ae01056d0 2024-03-18T14:51:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.083+0000 7f9ae6a08700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ae0108000 0x7f9ae01a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.083+0000 7f9ae6207700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0102da0 0x7f9ae019a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.083+0000 7f9ae6a08700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ae0108000 0x7f9ae01a1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40718/0 (socket says 172.21.15.76:40718) 2024-03-18T14:51:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.083+0000 7f9ae6a08700 1 -- 172.21.15.76:0/3122303334 learned_addr learned my addr 172.21.15.76:0/3122303334 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.083+0000 7f9ae5a06700 1 --2- 172.21.15.76:0/3122303334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ae01056d0 0x7f9ae019ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.084+0000 7f9ae6207700 1 -- 172.21.15.76:0/3122303334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ae01056d0 msgr2=0x7f9ae019ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:12.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.084+0000 7f9ae6207700 1 --2- 172.21.15.76:0/3122303334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ae01056d0 0x7f9ae019ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:12.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.084+0000 7f9ae6207700 1 -- 172.21.15.76:0/3122303334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ae0108000 msgr2=0x7f9ae01a1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:12.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.084+0000 7f9ae6207700 1 --2- 172.21.15.76:0/3122303334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ae0108000 0x7f9ae01a1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:12.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.084+0000 7f9ae6207700 1 -- 172.21.15.76:0/3122303334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae01a2240 con 0x7f9ae0102da0 2024-03-18T14:51:12.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.084+0000 7f9ae6a08700 1 --2- 172.21.15.76:0/3122303334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ae0108000 0x7f9ae01a1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:12.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.084+0000 7f9ae6207700 1 --2- 172.21.15.76:0/3122303334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0102da0 0x7f9ae019a5e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9adc00ef10 tx=0x7f9adc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:12.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.084+0000 7f9ad77fe700 1 -- 172.21.15.76:0/3122303334 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc00c9c0 con 0x7f9ae0102da0 2024-03-18T14:51:12.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.084+0000 7f9ad77fe700 1 -- 172.21.15.76:0/3122303334 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9adc019070 con 0x7f9ae0102da0 2024-03-18T14:51:12.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.084+0000 7f9ad77fe700 1 -- 172.21.15.76:0/3122303334 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc0146e0 con 0x7f9ae0102da0 2024-03-18T14:51:12.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.084+0000 7f9ae846b700 1 -- 172.21.15.76:0/3122303334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ae01a24d0 con 0x7f9ae0102da0 2024-03-18T14:51:12.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.084+0000 7f9ae846b700 1 -- 172.21.15.76:0/3122303334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ae01a2a10 con 0x7f9ae0102da0 2024-03-18T14:51:12.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.085+0000 7f9ae846b700 1 -- 172.21.15.76:0/3122303334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ae00507e0 con 0x7f9ae0102da0 2024-03-18T14:51:12.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.086+0000 7f9ad77fe700 1 -- 172.21.15.76:0/3122303334 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f9adc012050 con 0x7f9ae0102da0 2024-03-18T14:51:12.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.087+0000 7f9ad77fe700 1 --2- 172.21.15.76:0/3122303334 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9acc0a29d0 0x7f9acc0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:12.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.087+0000 7f9ae5a06700 1 --2- 172.21.15.76:0/3122303334 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9acc0a29d0 0x7f9acc0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:12.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.087+0000 7f9ad77fe700 1 -- 172.21.15.76:0/3122303334 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f9adc0cf130 con 0x7f9ae0102da0 2024-03-18T14:51:12.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.087+0000 7f9ae5a06700 1 --2- 172.21.15.76:0/3122303334 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9acc0a29d0 0x7f9acc0a4e90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0009c60 tx=0x7f9ad0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:12.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.091+0000 7f9ad77fe700 1 -- 172.21.15.76:0/3122303334 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9adc010040 con 0x7f9ae0102da0 2024-03-18T14:51:12.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.366+0000 7f9ae846b700 1 -- 172.21.15.76:0/3122303334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f9ae0062ee0 con 0x7f9ae0102da0 2024-03-18T14:51:12.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.370+0000 7f9ad77fe700 1 -- 172.21.15.76:0/3122303334 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) ==== 170+0+59 (secure 0 0 0) 0x7f9adc09c120 con 0x7f9ae0102da0 2024-03-18T14:51:12.371 INFO:teuthology.orchestra.run.smithi076.stdout:[client.0] 2024-03-18T14:51:12.371 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQDgVPhl47zsFRAA6JQksHm/RdXrJzGEWcp4Ag== 2024-03-18T14:51:12.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.372+0000 7f9ae846b700 1 -- 172.21.15.76:0/3122303334 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9acc0a29d0 msgr2=0x7f9acc0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:12.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.372+0000 7f9ae846b700 1 --2- 172.21.15.76:0/3122303334 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9acc0a29d0 0x7f9acc0a4e90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0009c60 tx=0x7f9ad0009260 comp rx=0 tx=0).stop 2024-03-18T14:51:12.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.372+0000 7f9ae846b700 1 -- 172.21.15.76:0/3122303334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0102da0 msgr2=0x7f9ae019a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:12.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.372+0000 7f9ae846b700 1 --2- 172.21.15.76:0/3122303334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0102da0 0x7f9ae019a5e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9adc00ef10 tx=0x7f9adc00c3a0 comp rx=0 tx=0).stop 2024-03-18T14:51:12.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.372+0000 7f9ae846b700 1 -- 172.21.15.76:0/3122303334 shutdown_connections 2024-03-18T14:51:12.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.372+0000 7f9ae846b700 1 --2- 172.21.15.76:0/3122303334 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9acc0a29d0 0x7f9acc0a4e90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:12.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.372+0000 7f9ae846b700 1 --2- 172.21.15.76:0/3122303334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ae0108000 0x7f9ae01a1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:12.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.372+0000 7f9ae846b700 1 --2- 172.21.15.76:0/3122303334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0102da0 0x7f9ae019a5e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:12.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.372+0000 7f9ae846b700 1 --2- 172.21.15.76:0/3122303334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ae01056d0 0x7f9ae019ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:12.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.372+0000 7f9ae846b700 1 -- 172.21.15.76:0/3122303334 >> 172.21.15.76:0/3122303334 conn(0x7f9ae00fc7d0 msgr2=0x7f9ae0102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:12.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.372+0000 7f9ae846b700 1 -- 172.21.15.76:0/3122303334 shutdown_connections 2024-03-18T14:51:12.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:12.372+0000 7f9ae846b700 1 -- 172.21.15.76:0/3122303334 wait complete. 2024-03-18T14:51:12.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:12 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3122303334' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T14:51:12.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:12 smithi076 ceph-mon[32205]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T14:51:12.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:12 smithi076 ceph-mon[32205]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T14:51:12.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:12 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3122303334' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T14:51:12.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:12 smithi174 ceph-mon[36787]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T14:51:12.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:12 smithi174 ceph-mon[36787]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T14:51:12.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:12 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3122303334' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T14:51:12.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:12 smithi077 ceph-mon[35865]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T14:51:12.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:12 smithi077 ceph-mon[35865]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T14:51:13.135 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T14:51:13.135 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-18T14:51:13.135 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-18T14:51:13.180 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-18T14:51:13.181 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-18T14:51:13.181 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-18T14:51:13.460 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:13.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:13 smithi174 ceph-mon[36787]: pgmap v289: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.6 KiB/s wr, 18 op/s 2024-03-18T14:51:13.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:13 smithi077 ceph-mon[35865]: pgmap v289: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.6 KiB/s wr, 18 op/s 2024-03-18T14:51:13.728 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:13 smithi076 ceph-mon[32205]: pgmap v289: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.6 KiB/s wr, 18 op/s 2024-03-18T14:51:15.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.193+0000 7f1f7a551700 1 -- 172.21.15.76:0/2542440800 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f74102f60 msgr2=0x7f1f74106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:15.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.193+0000 7f1f7a551700 1 --2- 172.21.15.76:0/2542440800 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f74102f60 0x7f1f74106f00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1f68009990 tx=0x7f1f68009ca0 comp rx=0 tx=0).stop 2024-03-18T14:51:15.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.193+0000 7f1f7a551700 1 -- 172.21.15.76:0/2542440800 shutdown_connections 2024-03-18T14:51:15.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.193+0000 7f1f7a551700 1 --2- 172.21.15.76:0/2542440800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f741075e0 0x7f1f7410f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:15.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.193+0000 7f1f7a551700 1 --2- 172.21.15.76:0/2542440800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f741025b0 0x7f1f74102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:15.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.193+0000 7f1f7a551700 1 --2- 172.21.15.76:0/2542440800 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f74102f60 0x7f1f74106f00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:15.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.193+0000 7f1f7a551700 1 -- 172.21.15.76:0/2542440800 >> 172.21.15.76:0/2542440800 conn(0x7f1f74079360 msgr2=0x7f1f74079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:15.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.193+0000 7f1f7a551700 1 -- 172.21.15.76:0/2542440800 shutdown_connections 2024-03-18T14:51:15.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.193+0000 7f1f7a551700 1 -- 172.21.15.76:0/2542440800 wait complete. 2024-03-18T14:51:15.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.194+0000 7f1f7a551700 1 Processor -- start 2024-03-18T14:51:15.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.194+0000 7f1f7a551700 1 -- start start 2024-03-18T14:51:15.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.194+0000 7f1f7a551700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f741025b0 0x7f1f741a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:15.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.194+0000 7f1f7a551700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f74102f60 0x7f1f741a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:15.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.194+0000 7f1f7a551700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f741075e0 0x7f1f7419aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:15.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.194+0000 7f1f7a551700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1f74112580 con 0x7f1f741075e0 2024-03-18T14:51:15.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.194+0000 7f1f7a551700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1f74112400 con 0x7f1f741025b0 2024-03-18T14:51:15.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.194+0000 7f1f7a551700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1f74112700 con 0x7f1f74102f60 2024-03-18T14:51:15.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.194+0000 7f1f78aee700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f741075e0 0x7f1f7419aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:15.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.194+0000 7f1f73fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f741025b0 0x7f1f741a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:15.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.194+0000 7f1f78aee700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f741075e0 0x7f1f7419aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40740/0 (socket says 172.21.15.76:40740) 2024-03-18T14:51:15.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.194+0000 7f1f78aee700 1 -- 172.21.15.76:0/622222259 learned_addr learned my addr 172.21.15.76:0/622222259 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:15.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.194+0000 7f1f737fe700 1 --2- 172.21.15.76:0/622222259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f74102f60 0x7f1f741a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:15.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.195+0000 7f1f78aee700 1 -- 172.21.15.76:0/622222259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f74102f60 msgr2=0x7f1f741a0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:15.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.195+0000 7f1f78aee700 1 --2- 172.21.15.76:0/622222259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f74102f60 0x7f1f741a0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:15.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.195+0000 7f1f78aee700 1 -- 172.21.15.76:0/622222259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f741025b0 msgr2=0x7f1f741a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:15.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.195+0000 7f1f78aee700 1 --2- 172.21.15.76:0/622222259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f741025b0 0x7f1f741a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:15.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.195+0000 7f1f78aee700 1 -- 172.21.15.76:0/622222259 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7419b230 con 0x7f1f741075e0 2024-03-18T14:51:15.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.195+0000 7f1f78aee700 1 --2- 172.21.15.76:0/622222259 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f741075e0 0x7f1f7419aad0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00ef30 tx=0x7f1f6c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:15.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.195+0000 7f1f717fa700 1 -- 172.21.15.76:0/622222259 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c0092e0 con 0x7f1f741075e0 2024-03-18T14:51:15.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.195+0000 7f1f717fa700 1 -- 172.21.15.76:0/622222259 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f6c004bb0 con 0x7f1f741075e0 2024-03-18T14:51:15.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.195+0000 7f1f7a551700 1 -- 172.21.15.76:0/622222259 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f7419b4c0 con 0x7f1f741075e0 2024-03-18T14:51:15.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.195+0000 7f1f717fa700 1 -- 172.21.15.76:0/622222259 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c004280 con 0x7f1f741075e0 2024-03-18T14:51:15.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.195+0000 7f1f7a551700 1 -- 172.21.15.76:0/622222259 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f741a7790 con 0x7f1f741075e0 2024-03-18T14:51:15.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.196+0000 7f1f7a551700 1 -- 172.21.15.76:0/622222259 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f74068f90 con 0x7f1f741075e0 2024-03-18T14:51:15.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.197+0000 7f1f717fa700 1 -- 172.21.15.76:0/622222259 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f1f6c01b030 con 0x7f1f741075e0 2024-03-18T14:51:15.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.198+0000 7f1f717fa700 1 --2- 172.21.15.76:0/622222259 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1f5c0a2920 0x7f1f5c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:15.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.198+0000 7f1f73fff700 1 --2- 172.21.15.76:0/622222259 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1f5c0a2920 0x7f1f5c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:15.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.198+0000 7f1f717fa700 1 -- 172.21.15.76:0/622222259 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f1f6c0ce490 con 0x7f1f741075e0 2024-03-18T14:51:15.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.198+0000 7f1f73fff700 1 --2- 172.21.15.76:0/622222259 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1f5c0a2920 0x7f1f5c0a4de0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1f74108780 tx=0x7f1f64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:15.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.202+0000 7f1f717fa700 1 -- 172.21.15.76:0/622222259 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1f6c022020 con 0x7f1f741075e0 2024-03-18T14:51:15.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.456+0000 7f1f7a551700 1 -- 172.21.15.76:0/622222259 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f1f741a7960 con 0x7f1f741075e0 2024-03-18T14:51:15.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.461+0000 7f1f717fa700 1 -- 172.21.15.76:0/622222259 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262659 (secure 0 0 0) 0x7f1f6c010040 con 0x7f1f741075e0 2024-03-18T14:51:15.466 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:51:15.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.464+0000 7f1f7a551700 1 -- 172.21.15.76:0/622222259 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1f5c0a2920 msgr2=0x7f1f5c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:15.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.464+0000 7f1f7a551700 1 --2- 172.21.15.76:0/622222259 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1f5c0a2920 0x7f1f5c0a4de0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1f74108780 tx=0x7f1f64009260 comp rx=0 tx=0).stop 2024-03-18T14:51:15.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.464+0000 7f1f7a551700 1 -- 172.21.15.76:0/622222259 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f741075e0 msgr2=0x7f1f7419aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:15.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.464+0000 7f1f7a551700 1 --2- 172.21.15.76:0/622222259 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f741075e0 0x7f1f7419aad0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00ef30 tx=0x7f1f6c00c3a0 comp rx=0 tx=0).stop 2024-03-18T14:51:15.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.464+0000 7f1f7a551700 1 -- 172.21.15.76:0/622222259 shutdown_connections 2024-03-18T14:51:15.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.464+0000 7f1f7a551700 1 --2- 172.21.15.76:0/622222259 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1f5c0a2920 0x7f1f5c0a4de0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:15.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.464+0000 7f1f7a551700 1 --2- 172.21.15.76:0/622222259 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f741075e0 0x7f1f7419aad0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:15.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.464+0000 7f1f7a551700 1 --2- 172.21.15.76:0/622222259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f741025b0 0x7f1f741a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:15.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.464+0000 7f1f7a551700 1 --2- 172.21.15.76:0/622222259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f74102f60 0x7f1f741a0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:15.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.464+0000 7f1f7a551700 1 -- 172.21.15.76:0/622222259 >> 172.21.15.76:0/622222259 conn(0x7f1f74079360 msgr2=0x7f1f74101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:15.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.464+0000 7f1f7a551700 1 -- 172.21.15.76:0/622222259 shutdown_connections 2024-03-18T14:51:15.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:15.464+0000 7f1f7a551700 1 -- 172.21.15.76:0/622222259 wait complete. 2024-03-18T14:51:15.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:15 smithi076 ceph-mon[32205]: pgmap v290: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.5 KiB/s wr, 15 op/s 2024-03-18T14:51:15.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:15 smithi174 ceph-mon[36787]: pgmap v290: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.5 KiB/s wr, 15 op/s 2024-03-18T14:51:15.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:15 smithi077 ceph-mon[35865]: pgmap v290: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.5 KiB/s wr, 15 op/s 2024-03-18T14:51:16.166 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":3010549595},{"type":"v1","addr":"172.21.15.76:6801","nonce":3010549595}]},"active_addr":"172.21.15.76:6801/3010549595","active_change":"2024-03-18T14:44:06.251919+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.76:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":788250061}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":2082819586}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":880938501}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":3349561522}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":2640453375}]}]} 2024-03-18T14:51:16.171 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-18T14:51:16.171 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-18T14:51:16.172 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T14:51:16.425 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:16.681 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:16 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/622222259' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T14:51:16.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:16 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/622222259' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T14:51:16.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:16 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/622222259' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T14:51:17.615 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:17 smithi076 ceph-mon[32205]: pgmap v291: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 6.4 KiB/s wr, 16 op/s 2024-03-18T14:51:17.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:17 smithi174 ceph-mon[36787]: pgmap v291: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 6.4 KiB/s wr, 16 op/s 2024-03-18T14:51:17.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:17 smithi077 ceph-mon[35865]: pgmap v291: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 6.4 KiB/s wr, 16 op/s 2024-03-18T14:51:18.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.159+0000 7ff4fd58a700 1 -- 172.21.15.76:0/3102948351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff4f8101420 msgr2=0x7ff4f8101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:18.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.159+0000 7ff4fd58a700 1 --2- 172.21.15.76:0/3102948351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff4f8101420 0x7ff4f8101800 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8009990 tx=0x7ff4e8009ca0 comp rx=0 tx=0).stop 2024-03-18T14:51:18.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.160+0000 7ff4fd58a700 1 -- 172.21.15.76:0/3102948351 shutdown_connections 2024-03-18T14:51:18.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.160+0000 7ff4fd58a700 1 --2- 172.21.15.76:0/3102948351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4f81064e0 0x7ff4f810e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:18.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.160+0000 7ff4fd58a700 1 --2- 172.21.15.76:0/3102948351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff4f8101420 0x7ff4f8101800 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:18.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.160+0000 7ff4fd58a700 1 --2- 172.21.15.76:0/3102948351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4f8101dd0 0x7ff4f8105e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:18.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.160+0000 7ff4fd58a700 1 -- 172.21.15.76:0/3102948351 >> 172.21.15.76:0/3102948351 conn(0x7ff4f8078140 msgr2=0x7ff4f8078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:18.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.160+0000 7ff4fd58a700 1 -- 172.21.15.76:0/3102948351 shutdown_connections 2024-03-18T14:51:18.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.160+0000 7ff4fd58a700 1 -- 172.21.15.76:0/3102948351 wait complete. 2024-03-18T14:51:18.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.160+0000 7ff4fd58a700 1 Processor -- start 2024-03-18T14:51:18.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.160+0000 7ff4fd58a700 1 -- start start 2024-03-18T14:51:18.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4fd58a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4f8101420 0x7ff4f8102ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:18.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4fd58a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4f8101dd0 0x7ff4f81033e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4fd58a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff4f81064e0 0x7ff4f8103920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4fd58a700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f81114c0 con 0x7ff4f81064e0 2024-03-18T14:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4fd58a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f8111340 con 0x7ff4f8101dd0 2024-03-18T14:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4fd58a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f8111640 con 0x7ff4f8101420 2024-03-18T14:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4f77fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff4f81064e0 0x7ff4f8103920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4f67fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4f8101dd0 0x7ff4f81033e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4f6ffd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4f8101420 0x7ff4f8102ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4f67fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4f8101dd0 0x7ff4f81033e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:47010/0 (socket says 172.21.15.76:47010) 2024-03-18T14:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4f67fc700 1 -- 172.21.15.76:0/3198635586 learned_addr learned my addr 172.21.15.76:0/3198635586 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4f77fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff4f81064e0 0x7ff4f8103920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40770/0 (socket says 172.21.15.76:40770) 2024-03-18T14:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4f67fc700 1 -- 172.21.15.76:0/3198635586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4f8101420 msgr2=0x7ff4f8102ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4f67fc700 1 --2- 172.21.15.76:0/3198635586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4f8101420 0x7ff4f8102ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4f67fc700 1 -- 172.21.15.76:0/3198635586 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff4f81064e0 msgr2=0x7ff4f8103920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4f67fc700 1 --2- 172.21.15.76:0/3198635586 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff4f81064e0 0x7ff4f8103920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4f67fc700 1 -- 172.21.15.76:0/3198635586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f81a6f90 con 0x7ff4f8101dd0 2024-03-18T14:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4f77fe700 1 --2- 172.21.15.76:0/3198635586 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff4f81064e0 0x7ff4f8103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4f67fc700 1 --2- 172.21.15.76:0/3198635586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4f8101dd0 0x7ff4f81033e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec00d950 tx=0x7ff4ec00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4dffff700 1 -- 172.21.15.76:0/3198635586 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec017070 con 0x7ff4f8101dd0 2024-03-18T14:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4fd58a700 1 -- 172.21.15.76:0/3198635586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4f81a7280 con 0x7ff4f8101dd0 2024-03-18T14:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4dffff700 1 -- 172.21.15.76:0/3198635586 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4ec005020 con 0x7ff4f8101dd0 2024-03-18T14:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4fd58a700 1 -- 172.21.15.76:0/3198635586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4f81a77c0 con 0x7ff4f8101dd0 2024-03-18T14:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.161+0000 7ff4dffff700 1 -- 172.21.15.76:0/3198635586 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec00fbc0 con 0x7ff4f8101dd0 2024-03-18T14:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.162+0000 7ff4fd58a700 1 -- 172.21.15.76:0/3198635586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4f81028e0 con 0x7ff4f8101dd0 2024-03-18T14:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.164+0000 7ff4dffff700 1 -- 172.21.15.76:0/3198635586 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ff4ec00fd60 con 0x7ff4f8101dd0 2024-03-18T14:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.164+0000 7ff4dffff700 1 --2- 172.21.15.76:0/3198635586 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff4e00a2920 0x7ff4e00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.164+0000 7ff4f6ffd700 1 --2- 172.21.15.76:0/3198635586 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff4e00a2920 0x7ff4e00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.164+0000 7ff4dffff700 1 -- 172.21.15.76:0/3198635586 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7ff4ec0ce5e0 con 0x7ff4f8101dd0 2024-03-18T14:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.165+0000 7ff4f6ffd700 1 --2- 172.21.15.76:0/3198635586 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff4e00a2920 0x7ff4e00a4de0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800b5c0 tx=0x7ff4e8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:18.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.172+0000 7ff4dffff700 1 -- 172.21.15.76:0/3198635586 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff4ec09b6d0 con 0x7ff4f8101dd0 2024-03-18T14:51:18.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.368+0000 7ff4fd58a700 1 -- 172.21.15.76:0/3198635586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff4f802db70 con 0x7ff4f8101dd0 2024-03-18T14:51:18.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.370+0000 7ff4dffff700 1 -- 172.21.15.76:0/3198635586 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22192 (secure 0 0 0) 0x7ff4ec09afd0 con 0x7ff4f8101dd0 2024-03-18T14:51:18.371 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:51:18.371 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":86,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:51:02.338985+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:51:18.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.373+0000 7ff4fd58a700 1 -- 172.21.15.76:0/3198635586 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff4e00a2920 msgr2=0x7ff4e00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:18.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.373+0000 7ff4fd58a700 1 --2- 172.21.15.76:0/3198635586 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff4e00a2920 0x7ff4e00a4de0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800b5c0 tx=0x7ff4e8005960 comp rx=0 tx=0).stop 2024-03-18T14:51:18.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.373+0000 7ff4fd58a700 1 -- 172.21.15.76:0/3198635586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4f8101dd0 msgr2=0x7ff4f81033e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:18.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.373+0000 7ff4fd58a700 1 --2- 172.21.15.76:0/3198635586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4f8101dd0 0x7ff4f81033e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec00d950 tx=0x7ff4ec00dc60 comp rx=0 tx=0).stop 2024-03-18T14:51:18.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.373+0000 7ff4fd58a700 1 -- 172.21.15.76:0/3198635586 shutdown_connections 2024-03-18T14:51:18.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.373+0000 7ff4fd58a700 1 --2- 172.21.15.76:0/3198635586 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff4e00a2920 0x7ff4e00a4de0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:18.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.373+0000 7ff4fd58a700 1 --2- 172.21.15.76:0/3198635586 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff4f81064e0 0x7ff4f8103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:18.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.373+0000 7ff4fd58a700 1 --2- 172.21.15.76:0/3198635586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4f8101420 0x7ff4f8102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:18.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.373+0000 7ff4fd58a700 1 --2- 172.21.15.76:0/3198635586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4f8101dd0 0x7ff4f81033e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:18.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.373+0000 7ff4fd58a700 1 -- 172.21.15.76:0/3198635586 >> 172.21.15.76:0/3198635586 conn(0x7ff4f8078140 msgr2=0x7ff4f806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:18.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.373+0000 7ff4fd58a700 1 -- 172.21.15.76:0/3198635586 shutdown_connections 2024-03-18T14:51:18.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:18.373+0000 7ff4fd58a700 1 -- 172.21.15.76:0/3198635586 wait complete. 2024-03-18T14:51:18.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:18 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3198635586' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:51:18.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:18 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3198635586' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:51:18.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:18 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3198635586' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:51:19.068 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-18T14:51:19.069 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T14:51:19.312 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:19.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:19 smithi174 ceph-mon[36787]: pgmap v292: 97 pgs: 97 active+clean; 624 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.4 KiB/s wr, 13 op/s 2024-03-18T14:51:19.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:19 smithi077 ceph-mon[35865]: pgmap v292: 97 pgs: 97 active+clean; 624 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.4 KiB/s wr, 13 op/s 2024-03-18T14:51:19.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:19 smithi076 ceph-mon[32205]: pgmap v292: 97 pgs: 97 active+clean; 624 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.4 KiB/s wr, 13 op/s 2024-03-18T14:51:21.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.013+0000 7f012856c700 1 -- 172.21.15.76:0/3673761996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0120101b70 msgr2=0x7f012010e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:21.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.013+0000 7f012856c700 1 --2- 172.21.15.76:0/3673761996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0120101b70 0x7f012010e050 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f011800b0a0 tx=0x7f011800b3b0 comp rx=0 tx=0).stop 2024-03-18T14:51:21.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.013+0000 7f012856c700 1 -- 172.21.15.76:0/3673761996 shutdown_connections 2024-03-18T14:51:21.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.013+0000 7f012856c700 1 --2- 172.21.15.76:0/3673761996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0120101b70 0x7f012010e050 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:21.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.013+0000 7f012856c700 1 --2- 172.21.15.76:0/3673761996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0120107210 0x7f01201075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:21.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.013+0000 7f012856c700 1 --2- 172.21.15.76:0/3673761996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01201011d0 0x7f0120101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:21.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.013+0000 7f012856c700 1 -- 172.21.15.76:0/3673761996 >> 172.21.15.76:0/3673761996 conn(0x7f0120078100 msgr2=0x7f0120078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:21.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.013+0000 7f012856c700 1 -- 172.21.15.76:0/3673761996 shutdown_connections 2024-03-18T14:51:21.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.013+0000 7f012856c700 1 -- 172.21.15.76:0/3673761996 wait complete. 2024-03-18T14:51:21.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.014+0000 7f012856c700 1 Processor -- start 2024-03-18T14:51:21.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.014+0000 7f012856c700 1 -- start start 2024-03-18T14:51:21.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.014+0000 7f012856c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01201011d0 0x7f012019d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:21.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.014+0000 7f012856c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0120101b70 0x7f012019db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:21.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.014+0000 7f012856c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0120107210 0x7f0120197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:21.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.014+0000 7f012856c700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0120111460 con 0x7f0120107210 2024-03-18T14:51:21.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.014+0000 7f012856c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f01201112e0 con 0x7f01201011d0 2024-03-18T14:51:21.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.014+0000 7f012856c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f01201115e0 con 0x7f0120101b70 2024-03-18T14:51:21.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.015+0000 7f0126b09700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0120107210 0x7f0120197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:21.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.015+0000 7f0126308700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01201011d0 0x7f012019d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:21.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.015+0000 7f0126b09700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0120107210 0x7f0120197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58084/0 (socket says 172.21.15.76:58084) 2024-03-18T14:51:21.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.015+0000 7f0125b07700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0120101b70 0x7f012019db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:21.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.015+0000 7f0126b09700 1 -- 172.21.15.76:0/2837617340 learned_addr learned my addr 172.21.15.76:0/2837617340 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:21.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.015+0000 7f0126b09700 1 -- 172.21.15.76:0/2837617340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0120101b70 msgr2=0x7f012019db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:21.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.015+0000 7f0126b09700 1 --2- 172.21.15.76:0/2837617340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0120101b70 0x7f012019db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:21.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.015+0000 7f0126b09700 1 -- 172.21.15.76:0/2837617340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01201011d0 msgr2=0x7f012019d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:21.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.015+0000 7f0126b09700 1 --2- 172.21.15.76:0/2837617340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01201011d0 0x7f012019d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:21.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.015+0000 7f0126b09700 1 -- 172.21.15.76:0/2837617340 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0120198000 con 0x7f0120107210 2024-03-18T14:51:21.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.015+0000 7f0126308700 1 --2- 172.21.15.76:0/2837617340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01201011d0 0x7f012019d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:21.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.015+0000 7f0126b09700 1 --2- 172.21.15.76:0/2837617340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0120107210 0x7f0120197720 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f011800b780 tx=0x7f0118004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:21.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.015+0000 7f01177fe700 1 -- 172.21.15.76:0/2837617340 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0118020070 con 0x7f0120107210 2024-03-18T14:51:21.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.015+0000 7f01177fe700 1 -- 172.21.15.76:0/2837617340 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0118007b80 con 0x7f0120107210 2024-03-18T14:51:21.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.015+0000 7f012856c700 1 -- 172.21.15.76:0/2837617340 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0120198290 con 0x7f0120107210 2024-03-18T14:51:21.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.015+0000 7f01177fe700 1 -- 172.21.15.76:0/2837617340 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01180052a0 con 0x7f0120107210 2024-03-18T14:51:21.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.015+0000 7f012856c700 1 -- 172.21.15.76:0/2837617340 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f012006a150 con 0x7f0120107210 2024-03-18T14:51:21.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.016+0000 7f012856c700 1 -- 172.21.15.76:0/2837617340 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01201985f0 con 0x7f0120107210 2024-03-18T14:51:21.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.018+0000 7f01177fe700 1 -- 172.21.15.76:0/2837617340 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f011802b020 con 0x7f0120107210 2024-03-18T14:51:21.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.019+0000 7f01177fe700 1 --2- 172.21.15.76:0/2837617340 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f010c0a29a0 0x7f010c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:21.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.019+0000 7f0126308700 1 --2- 172.21.15.76:0/2837617340 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f010c0a29a0 0x7f010c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:21.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.019+0000 7f01177fe700 1 -- 172.21.15.76:0/2837617340 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f011800f020 con 0x7f0120107210 2024-03-18T14:51:21.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.019+0000 7f0126308700 1 --2- 172.21.15.76:0/2837617340 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f010c0a29a0 0x7f010c0a4e60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f011c009fd0 tx=0x7f011c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:21.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.022+0000 7f01177fe700 1 -- 172.21.15.76:0/2837617340 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0118032080 con 0x7f0120107210 2024-03-18T14:51:21.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.218+0000 7f012856c700 1 -- 172.21.15.76:0/2837617340 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0120198c50 con 0x7f0120107210 2024-03-18T14:51:21.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.220+0000 7f01177fe700 1 -- 172.21.15.76:0/2837617340 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22192 (secure 0 0 0) 0x7f011809e660 con 0x7f0120107210 2024-03-18T14:51:21.221 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:51:21.221 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":86,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:51:02.338985+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:51:21.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.222+0000 7f012856c700 1 -- 172.21.15.76:0/2837617340 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f010c0a29a0 msgr2=0x7f010c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:21.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.222+0000 7f012856c700 1 --2- 172.21.15.76:0/2837617340 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f010c0a29a0 0x7f010c0a4e60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f011c009fd0 tx=0x7f011c009260 comp rx=0 tx=0).stop 2024-03-18T14:51:21.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.222+0000 7f012856c700 1 -- 172.21.15.76:0/2837617340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0120107210 msgr2=0x7f0120197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:21.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.222+0000 7f012856c700 1 --2- 172.21.15.76:0/2837617340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0120107210 0x7f0120197720 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f011800b780 tx=0x7f0118004d00 comp rx=0 tx=0).stop 2024-03-18T14:51:21.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.222+0000 7f012856c700 1 -- 172.21.15.76:0/2837617340 shutdown_connections 2024-03-18T14:51:21.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.222+0000 7f012856c700 1 --2- 172.21.15.76:0/2837617340 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f010c0a29a0 0x7f010c0a4e60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:21.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.222+0000 7f012856c700 1 --2- 172.21.15.76:0/2837617340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0120107210 0x7f0120197720 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:21.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.222+0000 7f012856c700 1 --2- 172.21.15.76:0/2837617340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01201011d0 0x7f012019d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:21.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.222+0000 7f012856c700 1 --2- 172.21.15.76:0/2837617340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0120101b70 0x7f012019db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:21.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.222+0000 7f012856c700 1 -- 172.21.15.76:0/2837617340 >> 172.21.15.76:0/2837617340 conn(0x7f0120078100 msgr2=0x7f01200ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:21.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.222+0000 7f012856c700 1 -- 172.21.15.76:0/2837617340 shutdown_connections 2024-03-18T14:51:21.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:21.222+0000 7f012856c700 1 -- 172.21.15.76:0/2837617340 wait complete. 2024-03-18T14:51:21.600 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:21 smithi076 ceph-mon[32205]: pgmap v293: 97 pgs: 97 active+clean; 624 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.2 KiB/s wr, 10 op/s 2024-03-18T14:51:21.600 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:21 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2837617340' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:51:21.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:21 smithi174 ceph-mon[36787]: pgmap v293: 97 pgs: 97 active+clean; 624 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.2 KiB/s wr, 10 op/s 2024-03-18T14:51:21.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:21 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2837617340' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:51:21.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:21 smithi077 ceph-mon[35865]: pgmap v293: 97 pgs: 97 active+clean; 624 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.2 KiB/s wr, 10 op/s 2024-03-18T14:51:21.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:21 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2837617340' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:51:21.940 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-18T14:51:21.940 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-18T14:51:21.940 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-18T14:51:21.941 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-18T14:51:21.941 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-18T14:51:21.942 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-18T14:51:21.942 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-18T14:51:21.943 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-18T14:51:21.943 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-18T14:51:21.943 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-18T14:51:21.944 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-18T14:51:21.944 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-18T14:51:22.737 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:22.737 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:22.737 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:22.741 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:22.742 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:22.742 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:22.743 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:22.747 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:22.747 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:22.753 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:22.753 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:22.757 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:23.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:23 smithi174 ceph-mon[36787]: pgmap v294: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-03-18T14:51:23.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:23 smithi076 ceph-mon[32205]: pgmap v294: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-03-18T14:51:23.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:23 smithi077 ceph-mon[35865]: pgmap v294: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-03-18T14:51:25.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:25 smithi174 ceph-mon[36787]: pgmap v295: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-18T14:51:25.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:25 smithi077 ceph-mon[35865]: pgmap v295: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-18T14:51:25.872 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:25 smithi076 ceph-mon[32205]: pgmap v295: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-18T14:51:27.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:27 smithi174 ceph-mon[36787]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-18T14:51:27.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:27 smithi077 ceph-mon[35865]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-18T14:51:27.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:27 smithi076 ceph-mon[32205]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-18T14:51:29.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:29 smithi174 ceph-mon[36787]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-18T14:51:29.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:29 smithi077 ceph-mon[35865]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-18T14:51:29.727 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:29 smithi076 ceph-mon[32205]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-18T14:51:30.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.545+0000 7f9232284700 1 -- 172.21.15.76:0/3898174814 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f922c1008c0 msgr2=0x7f922c100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:30.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.545+0000 7f9232284700 1 --2- 172.21.15.76:0/3898174814 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f922c1008c0 0x7f922c100d40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9220009990 tx=0x7f9220009ca0 comp rx=0 tx=0).stop 2024-03-18T14:51:30.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.545+0000 7f9232284700 1 -- 172.21.15.76:0/3898174814 shutdown_connections 2024-03-18T14:51:30.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.545+0000 7f9232284700 1 --2- 172.21.15.76:0/3898174814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f922c101280 0x7f922c10f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:30.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.545+0000 7f9232284700 1 --2- 172.21.15.76:0/3898174814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f922c107340 0x7f922c107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:30.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.545+0000 7f9232284700 1 --2- 172.21.15.76:0/3898174814 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f922c1008c0 0x7f922c100d40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:30.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.545+0000 7f9232284700 1 -- 172.21.15.76:0/3898174814 >> 172.21.15.76:0/3898174814 conn(0x7f922c0777d0 msgr2=0x7f922c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:30.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.545+0000 7f9232284700 1 -- 172.21.15.76:0/3898174814 shutdown_connections 2024-03-18T14:51:30.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.545+0000 7f9232284700 1 -- 172.21.15.76:0/3898174814 wait complete. 2024-03-18T14:51:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.545+0000 7f9232284700 1 Processor -- start 2024-03-18T14:51:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.545+0000 7f9232284700 1 -- start start 2024-03-18T14:51:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f9232284700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f922c1008c0 0x7f922c19ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f9232284700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f922c101280 0x7f922c19b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f9232284700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f922c107340 0x7f922c19f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f9232284700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f922c112720 con 0x7f922c101280 2024-03-18T14:51:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f9232284700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f922c1125a0 con 0x7f922c107340 2024-03-18T14:51:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f9232284700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f922c1128a0 con 0x7f922c1008c0 2024-03-18T14:51:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f922b7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f922c101280 0x7f922c19b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f922b7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f922c101280 0x7f922c19b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45142/0 (socket says 172.21.15.76:45142) 2024-03-18T14:51:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f922b7fe700 1 -- 172.21.15.76:0/4232591258 learned_addr learned my addr 172.21.15.76:0/4232591258 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f922bfff700 1 --2- 172.21.15.76:0/4232591258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f922c1008c0 0x7f922c19ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f9230821700 1 --2- 172.21.15.76:0/4232591258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f922c107340 0x7f922c19f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f922b7fe700 1 -- 172.21.15.76:0/4232591258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f922c1008c0 msgr2=0x7f922c19ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f922b7fe700 1 --2- 172.21.15.76:0/4232591258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f922c1008c0 0x7f922c19ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f922b7fe700 1 -- 172.21.15.76:0/4232591258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f922c107340 msgr2=0x7f922c19f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f922b7fe700 1 --2- 172.21.15.76:0/4232591258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f922c107340 0x7f922c19f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f922b7fe700 1 -- 172.21.15.76:0/4232591258 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f922c19fda0 con 0x7f922c101280 2024-03-18T14:51:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f922bfff700 1 --2- 172.21.15.76:0/4232591258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f922c1008c0 0x7f922c19ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f9230821700 1 --2- 172.21.15.76:0/4232591258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f922c107340 0x7f922c19f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.546+0000 7f922b7fe700 1 --2- 172.21.15.76:0/4232591258 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f922c101280 0x7f922c19b2d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9220000c00 tx=0x7f922000bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.547+0000 7f92297fa700 1 -- 172.21.15.76:0/4232591258 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9220027070 con 0x7f922c101280 2024-03-18T14:51:30.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.547+0000 7f92297fa700 1 -- 172.21.15.76:0/4232591258 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9220003d70 con 0x7f922c101280 2024-03-18T14:51:30.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.547+0000 7f9232284700 1 -- 172.21.15.76:0/4232591258 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f922c1a0030 con 0x7f922c101280 2024-03-18T14:51:30.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.547+0000 7f92297fa700 1 -- 172.21.15.76:0/4232591258 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922000f460 con 0x7f922c101280 2024-03-18T14:51:30.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.547+0000 7f9232284700 1 -- 172.21.15.76:0/4232591258 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f922c1a7910 con 0x7f922c101280 2024-03-18T14:51:30.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.548+0000 7f9232284700 1 -- 172.21.15.76:0/4232591258 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f922c10e810 con 0x7f922c101280 2024-03-18T14:51:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.549+0000 7f92297fa700 1 -- 172.21.15.76:0/4232591258 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f922002a070 con 0x7f922c101280 2024-03-18T14:51:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.549+0000 7f92297fa700 1 --2- 172.21.15.76:0/4232591258 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f92140a2920 0x7f92140a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.550+0000 7f922bfff700 1 --2- 172.21.15.76:0/4232591258 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f92140a2920 0x7f92140a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.550+0000 7f92297fa700 1 -- 172.21.15.76:0/4232591258 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f92200cfde0 con 0x7f922c101280 2024-03-18T14:51:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.550+0000 7f92297fa700 1 --2- 172.21.15.76:0/4232591258 >> [v2:172.21.15.174:6816/2665844766,v1:172.21.15.174:6817/2665844766] conn(0x7f92140ac100 0x7f92140ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.550+0000 7f92297fa700 1 -- 172.21.15.76:0/4232591258 --> [v2:172.21.15.174:6816/2665844766,v1:172.21.15.174:6817/2665844766] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f92200056d0 con 0x7f92140ac100 2024-03-18T14:51:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.550+0000 7f92297fa700 1 -- 172.21.15.76:0/4232591258 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f92200d02c0 con 0x7f922c101280 2024-03-18T14:51:30.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.550+0000 7f922bfff700 1 --2- 172.21.15.76:0/4232591258 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f92140a2920 0x7f92140a4de0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f921c011490 tx=0x7f921c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:30.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.550+0000 7f9230821700 1 --2- 172.21.15.76:0/4232591258 >> [v2:172.21.15.174:6816/2665844766,v1:172.21.15.174:6817/2665844766] conn(0x7f92140ac100 0x7f92140ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:30.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.550+0000 7f9230821700 1 --2- 172.21.15.76:0/4232591258 >> [v2:172.21.15.174:6816/2665844766,v1:172.21.15.174:6817/2665844766] conn(0x7f92140ac100 0x7f92140ae520 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:30.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.551+0000 7f92297fa700 1 -- 172.21.15.76:0/4232591258 <== osd.8 v2:172.21.15.174:6816/2665844766 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f92200056d0 con 0x7f92140ac100 2024-03-18T14:51:30.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.567+0000 7f9232284700 1 -- 172.21.15.76:0/4232591258 --> [v2:172.21.15.174:6816/2665844766,v1:172.21.15.174:6817/2665844766] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f922c103be0 con 0x7f92140ac100 2024-03-18T14:51:30.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.567+0000 7f92297fa700 1 -- 172.21.15.76:0/4232591258 <== osd.8 v2:172.21.15.174:6816/2665844766 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f922c103be0 con 0x7f92140ac100 2024-03-18T14:51:30.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.568+0000 7f9232284700 1 -- 172.21.15.76:0/4232591258 >> [v2:172.21.15.174:6816/2665844766,v1:172.21.15.174:6817/2665844766] conn(0x7f92140ac100 msgr2=0x7f92140ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:30.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.568+0000 7f9232284700 1 --2- 172.21.15.76:0/4232591258 >> [v2:172.21.15.174:6816/2665844766,v1:172.21.15.174:6817/2665844766] conn(0x7f92140ac100 0x7f92140ae520 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:30.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.568+0000 7f9232284700 1 -- 172.21.15.76:0/4232591258 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f92140a2920 msgr2=0x7f92140a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:30.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.568+0000 7f9232284700 1 --2- 172.21.15.76:0/4232591258 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f92140a2920 0x7f92140a4de0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f921c011490 tx=0x7f921c009260 comp rx=0 tx=0).stop 2024-03-18T14:51:30.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.568+0000 7f9232284700 1 -- 172.21.15.76:0/4232591258 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f922c101280 msgr2=0x7f922c19b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:30.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.568+0000 7f9232284700 1 --2- 172.21.15.76:0/4232591258 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f922c101280 0x7f922c19b2d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9220000c00 tx=0x7f922000bab0 comp rx=0 tx=0).stop 2024-03-18T14:51:30.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.568+0000 7f9230821700 1 -- 172.21.15.76:0/4232591258 reap_dead start 2024-03-18T14:51:30.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.568+0000 7f9232284700 1 -- 172.21.15.76:0/4232591258 shutdown_connections 2024-03-18T14:51:30.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.568+0000 7f9232284700 1 -- 172.21.15.76:0/4232591258 >> 172.21.15.76:0/4232591258 conn(0x7f922c0777d0 msgr2=0x7f922c1056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:30.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.568+0000 7f9232284700 1 -- 172.21.15.76:0/4232591258 shutdown_connections 2024-03-18T14:51:30.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:30.568+0000 7f9232284700 1 -- 172.21.15.76:0/4232591258 wait complete. 2024-03-18T14:51:31.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:31 smithi076 ceph-mon[32205]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-18T14:51:31.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:31 smithi174 ceph-mon[36787]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-18T14:51:31.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:31 smithi077 ceph-mon[35865]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-18T14:51:32.822 INFO:teuthology.orchestra.run.smithi076.stdout:257698037794 2024-03-18T14:51:32.823 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-18T14:51:33.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:33 smithi174 ceph-mon[36787]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-18T14:51:33.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:33 smithi077 ceph-mon[35865]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-18T14:51:33.811 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:33 smithi076 ceph-mon[32205]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-18T14:51:35.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:35 smithi174 ceph-mon[36787]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:51:35.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:35 smithi077 ceph-mon[35865]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:51:35.765 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:35 smithi076 ceph-mon[32205]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:51:37.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:37 smithi174 ceph-mon[36787]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:51:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:37 smithi077 ceph-mon[35865]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:51:37.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:37 smithi076 ceph-mon[32205]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:51:38.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.745+0000 7f25a88cf700 1 -- 172.21.15.76:0/3461475624 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25a406b420 msgr2=0x7f25a4106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:38.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.745+0000 7f25a88cf700 1 --2- 172.21.15.76:0/3461475624 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25a406b420 0x7f25a4106f20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f25940099c0 tx=0x7f2594009cd0 comp rx=0 tx=0).stop 2024-03-18T14:51:38.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.745+0000 7f25a88cf700 1 -- 172.21.15.76:0/3461475624 shutdown_connections 2024-03-18T14:51:38.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.745+0000 7f25a88cf700 1 --2- 172.21.15.76:0/3461475624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25a4107650 0x7f25a410f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:38.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.745+0000 7f25a88cf700 1 --2- 172.21.15.76:0/3461475624 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25a406aa70 0x7f25a406ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:38.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.745+0000 7f25a88cf700 1 --2- 172.21.15.76:0/3461475624 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25a406b420 0x7f25a4106f20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:38.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.745+0000 7f25a88cf700 1 -- 172.21.15.76:0/3461475624 >> 172.21.15.76:0/3461475624 conn(0x7f25a4077220 msgr2=0x7f25a4077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:38.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.745+0000 7f25a88cf700 1 -- 172.21.15.76:0/3461475624 shutdown_connections 2024-03-18T14:51:38.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.745+0000 7f25a88cf700 1 -- 172.21.15.76:0/3461475624 wait complete. 2024-03-18T14:51:38.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.746+0000 7f25a88cf700 1 Processor -- start 2024-03-18T14:51:38.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.746+0000 7f25a88cf700 1 -- start start 2024-03-18T14:51:38.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.746+0000 7f25a88cf700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25a406aa70 0x7f25a419e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:38.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.746+0000 7f25a88cf700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25a406b420 0x7f25a419ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:38.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.746+0000 7f25a88cf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25a4107650 0x7f25a4198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:38.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.746+0000 7f25a88cf700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f25a41125a0 con 0x7f25a406aa70 2024-03-18T14:51:38.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.746+0000 7f25a88cf700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f25a4112420 con 0x7f25a4107650 2024-03-18T14:51:38.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.746+0000 7f25a88cf700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f25a4112720 con 0x7f25a406b420 2024-03-18T14:51:38.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.747+0000 7f25a259c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25a406aa70 0x7f25a419e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:38.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.747+0000 7f25a259c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25a406aa70 0x7f25a419e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37812/0 (socket says 172.21.15.76:37812) 2024-03-18T14:51:38.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.747+0000 7f25a259c700 1 -- 172.21.15.76:0/3932289845 learned_addr learned my addr 172.21.15.76:0/3932289845 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:38.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.747+0000 7f25a1d9b700 1 --2- 172.21.15.76:0/3932289845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25a406b420 0x7f25a419ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:38.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.747+0000 7f25a2d9d700 1 --2- 172.21.15.76:0/3932289845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25a4107650 0x7f25a4198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:38.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.747+0000 7f25a259c700 1 -- 172.21.15.76:0/3932289845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25a406b420 msgr2=0x7f25a419ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:38.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.747+0000 7f25a259c700 1 --2- 172.21.15.76:0/3932289845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25a406b420 0x7f25a419ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:38.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.747+0000 7f25a259c700 1 -- 172.21.15.76:0/3932289845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25a4107650 msgr2=0x7f25a4198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:38.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.747+0000 7f25a259c700 1 --2- 172.21.15.76:0/3932289845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25a4107650 0x7f25a4198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:38.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.747+0000 7f25a259c700 1 -- 172.21.15.76:0/3932289845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a41990a0 con 0x7f25a406aa70 2024-03-18T14:51:38.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.747+0000 7f25a259c700 1 --2- 172.21.15.76:0/3932289845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25a406aa70 0x7f25a419e780 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f259800e9a0 tx=0x7f259800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:38.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.747+0000 7f25937fe700 1 -- 172.21.15.76:0/3932289845 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259800ca10 con 0x7f25a406aa70 2024-03-18T14:51:38.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.747+0000 7f25a88cf700 1 -- 172.21.15.76:0/3932289845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a4199390 con 0x7f25a406aa70 2024-03-18T14:51:38.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.747+0000 7f25937fe700 1 -- 172.21.15.76:0/3932289845 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f259800cbb0 con 0x7f25a406aa70 2024-03-18T14:51:38.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.747+0000 7f25937fe700 1 -- 172.21.15.76:0/3932289845 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2598018680 con 0x7f25a406aa70 2024-03-18T14:51:38.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.747+0000 7f25a88cf700 1 -- 172.21.15.76:0/3932289845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a41a5650 con 0x7f25a406aa70 2024-03-18T14:51:38.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.748+0000 7f25a88cf700 1 -- 172.21.15.76:0/3932289845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f25a40629e0 con 0x7f25a406aa70 2024-03-18T14:51:38.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.749+0000 7f25937fe700 1 -- 172.21.15.76:0/3932289845 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2598010430 con 0x7f25a406aa70 2024-03-18T14:51:38.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.750+0000 7f25937fe700 1 --2- 172.21.15.76:0/3932289845 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f258c0a2920 0x7f258c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:38.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.750+0000 7f25a1d9b700 1 --2- 172.21.15.76:0/3932289845 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f258c0a2920 0x7f258c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:38.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.750+0000 7f25937fe700 1 -- 172.21.15.76:0/3932289845 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f25980cf490 con 0x7f25a406aa70 2024-03-18T14:51:38.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.750+0000 7f25937fe700 1 --2- 172.21.15.76:0/3932289845 >> [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439] conn(0x7f258c0ac100 0x7f258c0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:38.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.750+0000 7f25937fe700 1 -- 172.21.15.76:0/3932289845 --> [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f259801bec0 con 0x7f258c0ac100 2024-03-18T14:51:38.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.750+0000 7f25937fe700 1 -- 172.21.15.76:0/3932289845 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f25980cf970 con 0x7f25a406aa70 2024-03-18T14:51:38.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.750+0000 7f25a2d9d700 1 --2- 172.21.15.76:0/3932289845 >> [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439] conn(0x7f258c0ac100 0x7f258c0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:38.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.750+0000 7f25a1d9b700 1 --2- 172.21.15.76:0/3932289845 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f258c0a2920 0x7f258c0a4de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2594009990 tx=0x7f2594005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:38.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.751+0000 7f25a2d9d700 1 --2- 172.21.15.76:0/3932289845 >> [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439] conn(0x7f258c0ac100 0x7f258c0ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:38.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.751+0000 7f25937fe700 1 -- 172.21.15.76:0/3932289845 <== osd.3 v2:172.21.15.76:6810/140675439 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f259801bec0 con 0x7f258c0ac100 2024-03-18T14:51:38.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.767+0000 7f25a88cf700 1 -- 172.21.15.76:0/3932289845 --> [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f25a4062e60 con 0x7f258c0ac100 2024-03-18T14:51:38.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.767+0000 7f25937fe700 1 -- 172.21.15.76:0/3932289845 <== osd.3 v2:172.21.15.76:6810/140675439 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f25a4062e60 con 0x7f258c0ac100 2024-03-18T14:51:38.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.768+0000 7f25a88cf700 1 -- 172.21.15.76:0/3932289845 >> [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439] conn(0x7f258c0ac100 msgr2=0x7f258c0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:38.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.768+0000 7f25a88cf700 1 --2- 172.21.15.76:0/3932289845 >> [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439] conn(0x7f258c0ac100 0x7f258c0ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:38.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.768+0000 7f25a88cf700 1 -- 172.21.15.76:0/3932289845 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f258c0a2920 msgr2=0x7f258c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:38.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.768+0000 7f25a88cf700 1 --2- 172.21.15.76:0/3932289845 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f258c0a2920 0x7f258c0a4de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2594009990 tx=0x7f2594005fb0 comp rx=0 tx=0).stop 2024-03-18T14:51:38.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.768+0000 7f25a88cf700 1 -- 172.21.15.76:0/3932289845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25a406aa70 msgr2=0x7f25a419e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:38.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.768+0000 7f25a88cf700 1 --2- 172.21.15.76:0/3932289845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25a406aa70 0x7f25a419e780 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f259800e9a0 tx=0x7f259800ecb0 comp rx=0 tx=0).stop 2024-03-18T14:51:38.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.768+0000 7f25a2d9d700 1 -- 172.21.15.76:0/3932289845 reap_dead start 2024-03-18T14:51:38.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.768+0000 7f25a88cf700 1 -- 172.21.15.76:0/3932289845 shutdown_connections 2024-03-18T14:51:38.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.768+0000 7f25a88cf700 1 -- 172.21.15.76:0/3932289845 >> 172.21.15.76:0/3932289845 conn(0x7f25a4077220 msgr2=0x7f25a40ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:38.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.768+0000 7f25a88cf700 1 -- 172.21.15.76:0/3932289845 shutdown_connections 2024-03-18T14:51:38.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:38.768+0000 7f25a88cf700 1 -- 172.21.15.76:0/3932289845 wait complete. 2024-03-18T14:51:39.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:39 smithi076 ceph-mon[32205]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T14:51:39.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:39 smithi174 ceph-mon[36787]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T14:51:39.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:39 smithi077 ceph-mon[35865]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T14:51:39.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.896+0000 7f1c6e26a700 1 -- 172.21.15.76:0/283331765 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c6806a9f0 msgr2=0x7f1c6806ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:39.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.896+0000 7f1c6e26a700 1 --2- 172.21.15.76:0/283331765 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c6806a9f0 0x7f1c6806ae50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1c580099c0 tx=0x7f1c58009cd0 comp rx=0 tx=0).stop 2024-03-18T14:51:39.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.896+0000 7f1c6e26a700 1 -- 172.21.15.76:0/283331765 shutdown_connections 2024-03-18T14:51:39.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.896+0000 7f1c6e26a700 1 --2- 172.21.15.76:0/283331765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c6806b390 0x7f1c6810f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:39.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.896+0000 7f1c6e26a700 1 --2- 172.21.15.76:0/283331765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c68108300 0x7f1c681086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:39.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.896+0000 7f1c6e26a700 1 --2- 172.21.15.76:0/283331765 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c6806a9f0 0x7f1c6806ae50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:39.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.896+0000 7f1c6e26a700 1 -- 172.21.15.76:0/283331765 >> 172.21.15.76:0/283331765 conn(0x7f1c68078170 msgr2=0x7f1c68078580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:39.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.896+0000 7f1c6e26a700 1 -- 172.21.15.76:0/283331765 shutdown_connections 2024-03-18T14:51:39.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.896+0000 7f1c6e26a700 1 -- 172.21.15.76:0/283331765 wait complete. 2024-03-18T14:51:39.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.897+0000 7f1c6e26a700 1 Processor -- start 2024-03-18T14:51:39.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.897+0000 7f1c6e26a700 1 -- start start 2024-03-18T14:51:39.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c6e26a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c6806a9f0 0x7f1c6819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:39.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c6e26a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c6806b390 0x7f1c6819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:39.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c6e26a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c68108300 0x7f1c681987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:39.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c6e26a700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1c68112560 con 0x7f1c68108300 2024-03-18T14:51:39.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c6e26a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1c681123e0 con 0x7f1c6806b390 2024-03-18T14:51:39.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c6e26a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1c681126e0 con 0x7f1c6806a9f0 2024-03-18T14:51:39.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c6c807700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c68108300 0x7f1c681987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:39.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c677fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c6806b390 0x7f1c6819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:39.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c6c807700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c68108300 0x7f1c681987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37832/0 (socket says 172.21.15.76:37832) 2024-03-18T14:51:39.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c67fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c6806a9f0 0x7f1c6819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:39.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c6c807700 1 -- 172.21.15.76:0/626367697 learned_addr learned my addr 172.21.15.76:0/626367697 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:39.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c677fe700 1 -- 172.21.15.76:0/626367697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c6806a9f0 msgr2=0x7f1c6819e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:39.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c677fe700 1 --2- 172.21.15.76:0/626367697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c6806a9f0 0x7f1c6819e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:39.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c677fe700 1 -- 172.21.15.76:0/626367697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c68108300 msgr2=0x7f1c681987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:39.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c677fe700 1 --2- 172.21.15.76:0/626367697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c68108300 0x7f1c681987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:39.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c677fe700 1 -- 172.21.15.76:0/626367697 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c68198fc0 con 0x7f1c6806b390 2024-03-18T14:51:39.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c67fff700 1 --2- 172.21.15.76:0/626367697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c6806a9f0 0x7f1c6819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:39.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c6c807700 1 --2- 172.21.15.76:0/626367697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c68108300 0x7f1c681987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:39.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c677fe700 1 --2- 172.21.15.76:0/626367697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c6806b390 0x7f1c6819eba0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1c580098e0 tx=0x7f1c58005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:39.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c657fa700 1 -- 172.21.15.76:0/626367697 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58027070 con 0x7f1c6806b390 2024-03-18T14:51:39.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c657fa700 1 -- 172.21.15.76:0/626367697 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c58003680 con 0x7f1c6806b390 2024-03-18T14:51:39.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c657fa700 1 -- 172.21.15.76:0/626367697 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58022e10 con 0x7f1c6806b390 2024-03-18T14:51:39.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c6e26a700 1 -- 172.21.15.76:0/626367697 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c68199250 con 0x7f1c6806b390 2024-03-18T14:51:39.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.898+0000 7f1c6e26a700 1 -- 172.21.15.76:0/626367697 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c681a54c0 con 0x7f1c6806b390 2024-03-18T14:51:39.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.899+0000 7f1c6e26a700 1 -- 172.21.15.76:0/626367697 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1c68062a60 con 0x7f1c6806b390 2024-03-18T14:51:39.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.900+0000 7f1c657fa700 1 -- 172.21.15.76:0/626367697 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f1c5802a070 con 0x7f1c6806b390 2024-03-18T14:51:39.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.901+0000 7f1c657fa700 1 --2- 172.21.15.76:0/626367697 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1c500a29a0 0x7f1c500a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:39.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.901+0000 7f1c67fff700 1 --2- 172.21.15.76:0/626367697 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1c500a29a0 0x7f1c500a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:39.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.901+0000 7f1c657fa700 1 -- 172.21.15.76:0/626367697 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f1c580cfce0 con 0x7f1c6806b390 2024-03-18T14:51:39.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.901+0000 7f1c657fa700 1 --2- 172.21.15.76:0/626367697 >> [v2:172.21.15.77:6808/1533069359,v1:172.21.15.77:6809/1533069359] conn(0x7f1c500ac180 0x7f1c500ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:39.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.901+0000 7f1c657fa700 1 -- 172.21.15.76:0/626367697 --> [v2:172.21.15.77:6808/1533069359,v1:172.21.15.77:6809/1533069359] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c5800b0b0 con 0x7f1c500ac180 2024-03-18T14:51:39.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.901+0000 7f1c657fa700 1 -- 172.21.15.76:0/626367697 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f1c580d0090 con 0x7f1c6806b390 2024-03-18T14:51:39.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.901+0000 7f1c6c807700 1 --2- 172.21.15.76:0/626367697 >> [v2:172.21.15.77:6808/1533069359,v1:172.21.15.77:6809/1533069359] conn(0x7f1c500ac180 0x7f1c500ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:39.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.901+0000 7f1c67fff700 1 --2- 172.21.15.76:0/626367697 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1c500a29a0 0x7f1c500a4e60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009fd0 tx=0x7f1c5c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:39.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.902+0000 7f1c6c807700 1 --2- 172.21.15.76:0/626367697 >> [v2:172.21.15.77:6808/1533069359,v1:172.21.15.77:6809/1533069359] conn(0x7f1c500ac180 0x7f1c500ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:39.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.902+0000 7f1c657fa700 1 -- 172.21.15.76:0/626367697 <== osd.4 v2:172.21.15.77:6808/1533069359 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c5800b0b0 con 0x7f1c500ac180 2024-03-18T14:51:39.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.919+0000 7f1c6e26a700 1 -- 172.21.15.76:0/626367697 --> [v2:172.21.15.77:6808/1533069359,v1:172.21.15.77:6809/1533069359] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1c68062e60 con 0x7f1c500ac180 2024-03-18T14:51:39.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.919+0000 7f1c657fa700 1 -- 172.21.15.76:0/626367697 <== osd.4 v2:172.21.15.77:6808/1533069359 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1c68062e60 con 0x7f1c500ac180 2024-03-18T14:51:39.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.919+0000 7f1c6e26a700 1 -- 172.21.15.76:0/626367697 >> [v2:172.21.15.77:6808/1533069359,v1:172.21.15.77:6809/1533069359] conn(0x7f1c500ac180 msgr2=0x7f1c500ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:39.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.919+0000 7f1c6e26a700 1 --2- 172.21.15.76:0/626367697 >> [v2:172.21.15.77:6808/1533069359,v1:172.21.15.77:6809/1533069359] conn(0x7f1c500ac180 0x7f1c500ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:39.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.919+0000 7f1c6e26a700 1 -- 172.21.15.76:0/626367697 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1c500a29a0 msgr2=0x7f1c500a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:39.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.919+0000 7f1c6e26a700 1 --2- 172.21.15.76:0/626367697 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1c500a29a0 0x7f1c500a4e60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009fd0 tx=0x7f1c5c009260 comp rx=0 tx=0).stop 2024-03-18T14:51:39.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.919+0000 7f1c6e26a700 1 -- 172.21.15.76:0/626367697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c6806b390 msgr2=0x7f1c6819eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:39.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.919+0000 7f1c6e26a700 1 --2- 172.21.15.76:0/626367697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c6806b390 0x7f1c6819eba0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1c580098e0 tx=0x7f1c58005f10 comp rx=0 tx=0).stop 2024-03-18T14:51:39.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.920+0000 7f1c6c807700 1 -- 172.21.15.76:0/626367697 reap_dead start 2024-03-18T14:51:39.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.920+0000 7f1c6e26a700 1 -- 172.21.15.76:0/626367697 shutdown_connections 2024-03-18T14:51:39.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.920+0000 7f1c6e26a700 1 -- 172.21.15.76:0/626367697 >> 172.21.15.76:0/626367697 conn(0x7f1c68078170 msgr2=0x7f1c680ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:39.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.920+0000 7f1c6e26a700 1 -- 172.21.15.76:0/626367697 shutdown_connections 2024-03-18T14:51:39.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:39.920+0000 7f1c6e26a700 1 -- 172.21.15.76:0/626367697 wait complete. 2024-03-18T14:51:40.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.106+0000 7f9db85e6700 1 -- 172.21.15.76:0/2060582355 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9db0101b70 msgr2=0x7f9db010e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:40.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.106+0000 7f9db85e6700 1 --2- 172.21.15.76:0/2060582355 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9db0101b70 0x7f9db010e050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9da800b0a0 tx=0x7f9da800b3b0 comp rx=0 tx=0).stop 2024-03-18T14:51:40.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.106+0000 7f9db85e6700 1 -- 172.21.15.76:0/2060582355 shutdown_connections 2024-03-18T14:51:40.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.106+0000 7f9db85e6700 1 --2- 172.21.15.76:0/2060582355 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9db0101b70 0x7f9db010e050 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:40.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.106+0000 7f9db85e6700 1 --2- 172.21.15.76:0/2060582355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db0107210 0x7f9db01075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:40.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.106+0000 7f9db85e6700 1 --2- 172.21.15.76:0/2060582355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9db01011d0 0x7f9db0101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:40.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.106+0000 7f9db85e6700 1 -- 172.21.15.76:0/2060582355 >> 172.21.15.76:0/2060582355 conn(0x7f9db0078100 msgr2=0x7f9db0078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:40.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.107+0000 7f9db85e6700 1 -- 172.21.15.76:0/2060582355 shutdown_connections 2024-03-18T14:51:40.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.107+0000 7f9db85e6700 1 -- 172.21.15.76:0/2060582355 wait complete. 2024-03-18T14:51:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.107+0000 7f9db85e6700 1 Processor -- start 2024-03-18T14:51:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.107+0000 7f9db85e6700 1 -- start start 2024-03-18T14:51:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.107+0000 7f9db85e6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db01011d0 0x7f9db0102e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.107+0000 7f9db85e6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9db0101b70 0x7f9db01033c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.107+0000 7f9db85e6700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9db0107210 0x7f9db0103900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.107+0000 7f9db85e6700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9db0111490 con 0x7f9db0107210 2024-03-18T14:51:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.107+0000 7f9db85e6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9db0111310 con 0x7f9db0101b70 2024-03-18T14:51:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.107+0000 7f9db85e6700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9db0111610 con 0x7f9db01011d0 2024-03-18T14:51:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.107+0000 7f9db6b83700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9db0107210 0x7f9db0103900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.108+0000 7f9db6382700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db01011d0 0x7f9db0102e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.108+0000 7f9db5b81700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9db0101b70 0x7f9db01033c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.108+0000 7f9db6382700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db01011d0 0x7f9db0102e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:53156/0 (socket says 172.21.15.76:53156) 2024-03-18T14:51:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.108+0000 7f9db6382700 1 -- 172.21.15.76:0/2082926170 learned_addr learned my addr 172.21.15.76:0/2082926170 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.108+0000 7f9db6382700 1 -- 172.21.15.76:0/2082926170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9db0101b70 msgr2=0x7f9db01033c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.108+0000 7f9db6382700 1 --2- 172.21.15.76:0/2082926170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9db0101b70 0x7f9db01033c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.108+0000 7f9db6382700 1 -- 172.21.15.76:0/2082926170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9db0107210 msgr2=0x7f9db0103900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.108+0000 7f9db6382700 1 --2- 172.21.15.76:0/2082926170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9db0107210 0x7f9db0103900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.108+0000 7f9db6382700 1 -- 172.21.15.76:0/2082926170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9db01a6f10 con 0x7f9db01011d0 2024-03-18T14:51:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.108+0000 7f9db5b81700 1 --2- 172.21.15.76:0/2082926170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9db0101b70 0x7f9db01033c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.108+0000 7f9db6b83700 1 --2- 172.21.15.76:0/2082926170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9db0107210 0x7f9db0103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:40.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.108+0000 7f9db6382700 1 --2- 172.21.15.76:0/2082926170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db01011d0 0x7f9db0102e80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9da000e9c0 tx=0x7f9da000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:40.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.108+0000 7f9da77fe700 1 -- 172.21.15.76:0/2082926170 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da000cbd0 con 0x7f9db01011d0 2024-03-18T14:51:40.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.108+0000 7f9da77fe700 1 -- 172.21.15.76:0/2082926170 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9da000ee60 con 0x7f9db01011d0 2024-03-18T14:51:40.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.108+0000 7f9db85e6700 1 -- 172.21.15.76:0/2082926170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9db01a7200 con 0x7f9db01011d0 2024-03-18T14:51:40.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.108+0000 7f9da77fe700 1 -- 172.21.15.76:0/2082926170 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da0005630 con 0x7f9db01011d0 2024-03-18T14:51:40.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.108+0000 7f9db85e6700 1 -- 172.21.15.76:0/2082926170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9db01a76f0 con 0x7f9db01011d0 2024-03-18T14:51:40.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.109+0000 7f9db85e6700 1 -- 172.21.15.76:0/2082926170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9db0105290 con 0x7f9db01011d0 2024-03-18T14:51:40.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.110+0000 7f9da77fe700 1 -- 172.21.15.76:0/2082926170 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f9da0004750 con 0x7f9db01011d0 2024-03-18T14:51:40.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.111+0000 7f9da77fe700 1 --2- 172.21.15.76:0/2082926170 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9d9c0a2920 0x7f9d9c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:40.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.111+0000 7f9db5b81700 1 --2- 172.21.15.76:0/2082926170 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9d9c0a2920 0x7f9d9c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:40.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.111+0000 7f9da77fe700 1 -- 172.21.15.76:0/2082926170 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f9da00cf190 con 0x7f9db01011d0 2024-03-18T14:51:40.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.111+0000 7f9da77fe700 1 --2- 172.21.15.76:0/2082926170 >> [v2:172.21.15.174:6824/1489039553,v1:172.21.15.174:6825/1489039553] conn(0x7f9d9c0ac100 0x7f9d9c0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:40.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.111+0000 7f9da77fe700 1 -- 172.21.15.76:0/2082926170 --> [v2:172.21.15.174:6824/1489039553,v1:172.21.15.174:6825/1489039553] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9da001fe50 con 0x7f9d9c0ac100 2024-03-18T14:51:40.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.111+0000 7f9da77fe700 1 -- 172.21.15.76:0/2082926170 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f9da00cf540 con 0x7f9db01011d0 2024-03-18T14:51:40.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.111+0000 7f9db5b81700 1 --2- 172.21.15.76:0/2082926170 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9d9c0a2920 0x7f9d9c0a4de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9dac009680 tx=0x7f9dac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:40.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.111+0000 7f9db6b83700 1 --2- 172.21.15.76:0/2082926170 >> [v2:172.21.15.174:6824/1489039553,v1:172.21.15.174:6825/1489039553] conn(0x7f9d9c0ac100 0x7f9d9c0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:40.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.112+0000 7f9db6b83700 1 --2- 172.21.15.76:0/2082926170 >> [v2:172.21.15.174:6824/1489039553,v1:172.21.15.174:6825/1489039553] conn(0x7f9d9c0ac100 0x7f9d9c0ae520 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:40.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.112+0000 7f9da77fe700 1 -- 172.21.15.76:0/2082926170 <== osd.11 v2:172.21.15.174:6824/1489039553 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9da001fe50 con 0x7f9d9c0ac100 2024-03-18T14:51:40.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.128+0000 7f9db85e6700 1 -- 172.21.15.76:0/2082926170 --> [v2:172.21.15.174:6824/1489039553,v1:172.21.15.174:6825/1489039553] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9db01028c0 con 0x7f9d9c0ac100 2024-03-18T14:51:40.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.129+0000 7f9da77fe700 1 -- 172.21.15.76:0/2082926170 <== osd.11 v2:172.21.15.174:6824/1489039553 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9db01028c0 con 0x7f9d9c0ac100 2024-03-18T14:51:40.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.129+0000 7f9db85e6700 1 -- 172.21.15.76:0/2082926170 >> [v2:172.21.15.174:6824/1489039553,v1:172.21.15.174:6825/1489039553] conn(0x7f9d9c0ac100 msgr2=0x7f9d9c0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:40.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.129+0000 7f9db85e6700 1 --2- 172.21.15.76:0/2082926170 >> [v2:172.21.15.174:6824/1489039553,v1:172.21.15.174:6825/1489039553] conn(0x7f9d9c0ac100 0x7f9d9c0ae520 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:40.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.129+0000 7f9db85e6700 1 -- 172.21.15.76:0/2082926170 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9d9c0a2920 msgr2=0x7f9d9c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:40.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.129+0000 7f9db85e6700 1 --2- 172.21.15.76:0/2082926170 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9d9c0a2920 0x7f9d9c0a4de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9dac009680 tx=0x7f9dac009260 comp rx=0 tx=0).stop 2024-03-18T14:51:40.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.129+0000 7f9db85e6700 1 -- 172.21.15.76:0/2082926170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db01011d0 msgr2=0x7f9db0102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:40.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.129+0000 7f9db85e6700 1 --2- 172.21.15.76:0/2082926170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db01011d0 0x7f9db0102e80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9da000e9c0 tx=0x7f9da000ecd0 comp rx=0 tx=0).stop 2024-03-18T14:51:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.129+0000 7f9db6b83700 1 -- 172.21.15.76:0/2082926170 reap_dead start 2024-03-18T14:51:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.129+0000 7f9db85e6700 1 -- 172.21.15.76:0/2082926170 shutdown_connections 2024-03-18T14:51:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.129+0000 7f9db85e6700 1 -- 172.21.15.76:0/2082926170 >> 172.21.15.76:0/2082926170 conn(0x7f9db0078100 msgr2=0x7f9db01004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.129+0000 7f9db85e6700 1 -- 172.21.15.76:0/2082926170 shutdown_connections 2024-03-18T14:51:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.129+0000 7f9db85e6700 1 -- 172.21.15.76:0/2082926170 wait complete. 2024-03-18T14:51:40.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.417+0000 7f9cc9fa5700 1 -- 172.21.15.76:0/4233402391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cc4102510 msgr2=0x7f9cc41028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:40.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.417+0000 7f9cc9fa5700 1 --2- 172.21.15.76:0/4233402391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cc4102510 0x7f9cc41028f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9cb40099c0 tx=0x7f9cb4009cd0 comp rx=0 tx=0).stop 2024-03-18T14:51:40.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.417+0000 7f9cc9fa5700 1 -- 172.21.15.76:0/4233402391 shutdown_connections 2024-03-18T14:51:40.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.417+0000 7f9cc9fa5700 1 --2- 172.21.15.76:0/4233402391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cc41075f0 0x7f9cc410f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:40.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.417+0000 7f9cc9fa5700 1 --2- 172.21.15.76:0/4233402391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cc4102510 0x7f9cc41028f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:40.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.417+0000 7f9cc9fa5700 1 --2- 172.21.15.76:0/4233402391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cc4102ec0 0x7f9cc4106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:40.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.417+0000 7f9cc9fa5700 1 -- 172.21.15.76:0/4233402391 >> 172.21.15.76:0/4233402391 conn(0x7f9cc40fdd80 msgr2=0x7f9cc41001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:40.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.417+0000 7f9cc9fa5700 1 -- 172.21.15.76:0/4233402391 shutdown_connections 2024-03-18T14:51:40.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.417+0000 7f9cc9fa5700 1 -- 172.21.15.76:0/4233402391 wait complete. 2024-03-18T14:51:40.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.418+0000 7f9cc9fa5700 1 Processor -- start 2024-03-18T14:51:40.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.418+0000 7f9cc9fa5700 1 -- start start 2024-03-18T14:51:40.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.418+0000 7f9cc9fa5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cc4102510 0x7f9cc40745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:40.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.418+0000 7f9cc9fa5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cc4102ec0 0x7f9cc4074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:40.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.418+0000 7f9cc9fa5700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cc41075f0 0x7f9cc4075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:40.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.418+0000 7f9cc9fa5700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc41125d0 con 0x7f9cc41075f0 2024-03-18T14:51:40.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.418+0000 7f9cc9fa5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc4112450 con 0x7f9cc4102510 2024-03-18T14:51:40.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.418+0000 7f9cc9fa5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc4112750 con 0x7f9cc4102ec0 2024-03-18T14:51:40.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc3fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cc41075f0 0x7f9cc4075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:40.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc3fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cc41075f0 0x7f9cc4075050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37880/0 (socket says 172.21.15.76:37880) 2024-03-18T14:51:40.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc3fff700 1 -- 172.21.15.76:0/641451660 learned_addr learned my addr 172.21.15.76:0/641451660 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:40.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc37fe700 1 --2- 172.21.15.76:0/641451660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cc4102510 0x7f9cc40745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:40.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc2ffd700 1 --2- 172.21.15.76:0/641451660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cc4102ec0 0x7f9cc4074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:40.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc3fff700 1 -- 172.21.15.76:0/641451660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cc4102ec0 msgr2=0x7f9cc4074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:40.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc3fff700 1 --2- 172.21.15.76:0/641451660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cc4102ec0 0x7f9cc4074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:40.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc3fff700 1 -- 172.21.15.76:0/641451660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cc4102510 msgr2=0x7f9cc40745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:40.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc3fff700 1 --2- 172.21.15.76:0/641451660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cc4102510 0x7f9cc40745d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:40.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc3fff700 1 -- 172.21.15.76:0/641451660 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc41a6f90 con 0x7f9cc41075f0 2024-03-18T14:51:40.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc2ffd700 1 --2- 172.21.15.76:0/641451660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cc4102ec0 0x7f9cc4074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:40.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc37fe700 1 --2- 172.21.15.76:0/641451660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cc4102510 0x7f9cc40745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:40.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc3fff700 1 --2- 172.21.15.76:0/641451660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cc41075f0 0x7f9cc4075050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00ef30 tx=0x7f9cbc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:40.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc0ff9700 1 -- 172.21.15.76:0/641451660 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc0092e0 con 0x7f9cc41075f0 2024-03-18T14:51:40.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc0ff9700 1 -- 172.21.15.76:0/641451660 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cbc004bb0 con 0x7f9cc41075f0 2024-03-18T14:51:40.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc9fa5700 1 -- 172.21.15.76:0/641451660 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cc41a7220 con 0x7f9cc41075f0 2024-03-18T14:51:40.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc0ff9700 1 -- 172.21.15.76:0/641451660 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc004280 con 0x7f9cc41075f0 2024-03-18T14:51:40.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.419+0000 7f9cc9fa5700 1 -- 172.21.15.76:0/641451660 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cc41a7760 con 0x7f9cc41075f0 2024-03-18T14:51:40.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.420+0000 7f9cc9fa5700 1 -- 172.21.15.76:0/641451660 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9cc40629e0 con 0x7f9cc41075f0 2024-03-18T14:51:40.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.421+0000 7f9cc0ff9700 1 -- 172.21.15.76:0/641451660 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f9cbc01b030 con 0x7f9cc41075f0 2024-03-18T14:51:40.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.422+0000 7f9cc0ff9700 1 --2- 172.21.15.76:0/641451660 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9cac0a29a0 0x7f9cac0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:40.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.422+0000 7f9cc37fe700 1 --2- 172.21.15.76:0/641451660 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9cac0a29a0 0x7f9cac0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:40.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.422+0000 7f9cc0ff9700 1 -- 172.21.15.76:0/641451660 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f9cbc0cedb0 con 0x7f9cc41075f0 2024-03-18T14:51:40.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.422+0000 7f9cc0ff9700 1 --2- 172.21.15.76:0/641451660 >> [v2:172.21.15.77:6816/3828133263,v1:172.21.15.77:6817/3828133263] conn(0x7f9cac0ac180 0x7f9cac0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:40.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.422+0000 7f9cc0ff9700 1 -- 172.21.15.76:0/641451660 --> [v2:172.21.15.77:6816/3828133263,v1:172.21.15.77:6817/3828133263] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9cbc01fe50 con 0x7f9cac0ac180 2024-03-18T14:51:40.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.422+0000 7f9cc0ff9700 1 -- 172.21.15.76:0/641451660 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f9cbc0cf290 con 0x7f9cc41075f0 2024-03-18T14:51:40.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.422+0000 7f9cc37fe700 1 --2- 172.21.15.76:0/641451660 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9cac0a29a0 0x7f9cac0a4e60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4000c00 tx=0x7f9cb4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:40.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.423+0000 7f9cc2ffd700 1 --2- 172.21.15.76:0/641451660 >> [v2:172.21.15.77:6816/3828133263,v1:172.21.15.77:6817/3828133263] conn(0x7f9cac0ac180 0x7f9cac0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:40.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.423+0000 7f9cc2ffd700 1 --2- 172.21.15.76:0/641451660 >> [v2:172.21.15.77:6816/3828133263,v1:172.21.15.77:6817/3828133263] conn(0x7f9cac0ac180 0x7f9cac0ae5a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:40.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.424+0000 7f9cc0ff9700 1 -- 172.21.15.76:0/641451660 <== osd.7 v2:172.21.15.77:6816/3828133263 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9cbc01fe50 con 0x7f9cac0ac180 2024-03-18T14:51:40.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.440+0000 7f9cc9fa5700 1 -- 172.21.15.76:0/641451660 --> [v2:172.21.15.77:6816/3828133263,v1:172.21.15.77:6817/3828133263] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9cc4062e60 con 0x7f9cac0ac180 2024-03-18T14:51:40.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.440+0000 7f9cc0ff9700 1 -- 172.21.15.76:0/641451660 <== osd.7 v2:172.21.15.77:6816/3828133263 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9cc4062e60 con 0x7f9cac0ac180 2024-03-18T14:51:40.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.440+0000 7f9cc9fa5700 1 -- 172.21.15.76:0/641451660 >> [v2:172.21.15.77:6816/3828133263,v1:172.21.15.77:6817/3828133263] conn(0x7f9cac0ac180 msgr2=0x7f9cac0ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:40.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.440+0000 7f9cc9fa5700 1 --2- 172.21.15.76:0/641451660 >> [v2:172.21.15.77:6816/3828133263,v1:172.21.15.77:6817/3828133263] conn(0x7f9cac0ac180 0x7f9cac0ae5a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:40.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.440+0000 7f9cc9fa5700 1 -- 172.21.15.76:0/641451660 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9cac0a29a0 msgr2=0x7f9cac0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:40.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.440+0000 7f9cc9fa5700 1 --2- 172.21.15.76:0/641451660 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f9cac0a29a0 0x7f9cac0a4e60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4000c00 tx=0x7f9cb4005fb0 comp rx=0 tx=0).stop 2024-03-18T14:51:40.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.440+0000 7f9cc9fa5700 1 -- 172.21.15.76:0/641451660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cc41075f0 msgr2=0x7f9cc4075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:40.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.440+0000 7f9cc9fa5700 1 --2- 172.21.15.76:0/641451660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cc41075f0 0x7f9cc4075050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00ef30 tx=0x7f9cbc00c3a0 comp rx=0 tx=0).stop 2024-03-18T14:51:40.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.441+0000 7f9cc3fff700 1 -- 172.21.15.76:0/641451660 reap_dead start 2024-03-18T14:51:40.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.441+0000 7f9cc9fa5700 1 -- 172.21.15.76:0/641451660 shutdown_connections 2024-03-18T14:51:40.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.441+0000 7f9cc9fa5700 1 -- 172.21.15.76:0/641451660 >> 172.21.15.76:0/641451660 conn(0x7f9cc40fdd80 msgr2=0x7f9cc40ffaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:40.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.441+0000 7f9cc9fa5700 1 -- 172.21.15.76:0/641451660 shutdown_connections 2024-03-18T14:51:40.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:40.441+0000 7f9cc9fa5700 1 -- 172.21.15.76:0/641451660 wait complete. 2024-03-18T14:51:41.586 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:41 smithi076 ceph-mon[32205]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-18T14:51:41.617 INFO:teuthology.orchestra.run.smithi076.stdout:111669149757 2024-03-18T14:51:41.617 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-18T14:51:41.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:41 smithi174 ceph-mon[36787]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-18T14:51:41.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:41 smithi077 ceph-mon[35865]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-18T14:51:41.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.810+0000 7f6c7929a700 1 -- 172.21.15.76:0/423858164 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c741011d0 msgr2=0x7f6c74101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:41.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.810+0000 7f6c7929a700 1 --2- 172.21.15.76:0/423858164 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c741011d0 0x7f6c74101630 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009990 tx=0x7f6c68009ca0 comp rx=0 tx=0).stop 2024-03-18T14:51:41.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.810+0000 7f6c7929a700 1 -- 172.21.15.76:0/423858164 shutdown_connections 2024-03-18T14:51:41.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.810+0000 7f6c7929a700 1 --2- 172.21.15.76:0/423858164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c74101b70 0x7f6c7410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:41.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.810+0000 7f6c7929a700 1 --2- 172.21.15.76:0/423858164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c74107210 0x7f6c741075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:41.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.810+0000 7f6c7929a700 1 --2- 172.21.15.76:0/423858164 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c741011d0 0x7f6c74101630 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:41.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.810+0000 7f6c7929a700 1 -- 172.21.15.76:0/423858164 >> 172.21.15.76:0/423858164 conn(0x7f6c74078100 msgr2=0x7f6c74078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:41.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.810+0000 7f6c7929a700 1 -- 172.21.15.76:0/423858164 shutdown_connections 2024-03-18T14:51:41.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.810+0000 7f6c7929a700 1 -- 172.21.15.76:0/423858164 wait complete. 2024-03-18T14:51:41.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.811+0000 7f6c7929a700 1 Processor -- start 2024-03-18T14:51:41.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.811+0000 7f6c7929a700 1 -- start start 2024-03-18T14:51:41.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.811+0000 7f6c7929a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c741011d0 0x7f6c74102e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:41.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.811+0000 7f6c7929a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c74101b70 0x7f6c741033c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:41.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.811+0000 7f6c7929a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c74107210 0x7f6c74103900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:41.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.811+0000 7f6c7929a700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6c74111490 con 0x7f6c741011d0 2024-03-18T14:51:41.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.811+0000 7f6c7929a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6c74111310 con 0x7f6c74107210 2024-03-18T14:51:41.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.811+0000 7f6c7929a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6c74111610 con 0x7f6c74101b70 2024-03-18T14:51:41.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.811+0000 7f6c72ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c741011d0 0x7f6c74102e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:41.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.811+0000 7f6c72ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c741011d0 0x7f6c74102e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37900/0 (socket says 172.21.15.76:37900) 2024-03-18T14:51:41.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.811+0000 7f6c72ffd700 1 -- 172.21.15.76:0/2497308899 learned_addr learned my addr 172.21.15.76:0/2497308899 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:41.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.811+0000 7f6c737fe700 1 --2- 172.21.15.76:0/2497308899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c74107210 0x7f6c74103900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:41.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.811+0000 7f6c727fc700 1 --2- 172.21.15.76:0/2497308899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c74101b70 0x7f6c741033c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:41.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.812+0000 7f6c72ffd700 1 -- 172.21.15.76:0/2497308899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c74101b70 msgr2=0x7f6c741033c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:41.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.812+0000 7f6c72ffd700 1 --2- 172.21.15.76:0/2497308899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c74101b70 0x7f6c741033c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:41.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.812+0000 7f6c72ffd700 1 -- 172.21.15.76:0/2497308899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c74107210 msgr2=0x7f6c74103900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:41.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.812+0000 7f6c72ffd700 1 --2- 172.21.15.76:0/2497308899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c74107210 0x7f6c74103900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:41.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.812+0000 7f6c72ffd700 1 -- 172.21.15.76:0/2497308899 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c741a6f10 con 0x7f6c741011d0 2024-03-18T14:51:41.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.812+0000 7f6c737fe700 1 --2- 172.21.15.76:0/2497308899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c74107210 0x7f6c74103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:41.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.812+0000 7f6c72ffd700 1 --2- 172.21.15.76:0/2497308899 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c741011d0 0x7f6c74102e80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00ef10 tx=0x7f6c6c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:41.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.812+0000 7f6c5bfff700 1 -- 172.21.15.76:0/2497308899 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c00c9c0 con 0x7f6c741011d0 2024-03-18T14:51:41.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.812+0000 7f6c7929a700 1 -- 172.21.15.76:0/2497308899 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c741a71a0 con 0x7f6c741011d0 2024-03-18T14:51:41.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.812+0000 7f6c5bfff700 1 -- 172.21.15.76:0/2497308899 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c6c019070 con 0x7f6c741011d0 2024-03-18T14:51:41.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.812+0000 7f6c5bfff700 1 -- 172.21.15.76:0/2497308899 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c007e60 con 0x7f6c741011d0 2024-03-18T14:51:41.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.812+0000 7f6c7929a700 1 -- 172.21.15.76:0/2497308899 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c741a7660 con 0x7f6c741011d0 2024-03-18T14:51:41.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.813+0000 7f6c7929a700 1 -- 172.21.15.76:0/2497308899 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6c74105290 con 0x7f6c741011d0 2024-03-18T14:51:41.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.814+0000 7f6c5bfff700 1 -- 172.21.15.76:0/2497308899 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6c6c012050 con 0x7f6c741011d0 2024-03-18T14:51:41.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.815+0000 7f6c5bfff700 1 --2- 172.21.15.76:0/2497308899 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6c5c0a2920 0x7f6c5c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:41.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.815+0000 7f6c727fc700 1 --2- 172.21.15.76:0/2497308899 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6c5c0a2920 0x7f6c5c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:41.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.815+0000 7f6c5bfff700 1 -- 172.21.15.76:0/2497308899 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f6c6c0cf400 con 0x7f6c741011d0 2024-03-18T14:51:41.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.815+0000 7f6c5bfff700 1 --2- 172.21.15.76:0/2497308899 >> [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352] conn(0x7f6c5c0ac100 0x7f6c5c0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:41.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.815+0000 7f6c5bfff700 1 -- 172.21.15.76:0/2497308899 --> [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6c6c00be50 con 0x7f6c5c0ac100 2024-03-18T14:51:41.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.815+0000 7f6c5bfff700 1 -- 172.21.15.76:0/2497308899 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f6c6c0cf8e0 con 0x7f6c741011d0 2024-03-18T14:51:41.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.815+0000 7f6c737fe700 1 --2- 172.21.15.76:0/2497308899 >> [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352] conn(0x7f6c5c0ac100 0x7f6c5c0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:41.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.815+0000 7f6c727fc700 1 --2- 172.21.15.76:0/2497308899 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6c5c0a2920 0x7f6c5c0a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6c6800b2e0 tx=0x7f6c68005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:41.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.815+0000 7f6c737fe700 1 --2- 172.21.15.76:0/2497308899 >> [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352] conn(0x7f6c5c0ac100 0x7f6c5c0ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:41.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.816+0000 7f6c5bfff700 1 -- 172.21.15.76:0/2497308899 <== osd.6 v2:172.21.15.76:6818/1891318352 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6c6c00be50 con 0x7f6c5c0ac100 2024-03-18T14:51:41.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.832+0000 7f6c7929a700 1 -- 172.21.15.76:0/2497308899 --> [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6c741028c0 con 0x7f6c5c0ac100 2024-03-18T14:51:41.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.832+0000 7f6c5bfff700 1 -- 172.21.15.76:0/2497308899 <== osd.6 v2:172.21.15.76:6818/1891318352 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6c741028c0 con 0x7f6c5c0ac100 2024-03-18T14:51:41.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.832+0000 7f6c7929a700 1 -- 172.21.15.76:0/2497308899 >> [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352] conn(0x7f6c5c0ac100 msgr2=0x7f6c5c0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:41.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.832+0000 7f6c7929a700 1 --2- 172.21.15.76:0/2497308899 >> [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352] conn(0x7f6c5c0ac100 0x7f6c5c0ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:41.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.832+0000 7f6c7929a700 1 -- 172.21.15.76:0/2497308899 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6c5c0a2920 msgr2=0x7f6c5c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:41.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.832+0000 7f6c7929a700 1 --2- 172.21.15.76:0/2497308899 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6c5c0a2920 0x7f6c5c0a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6c6800b2e0 tx=0x7f6c68005960 comp rx=0 tx=0).stop 2024-03-18T14:51:41.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.832+0000 7f6c7929a700 1 -- 172.21.15.76:0/2497308899 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c741011d0 msgr2=0x7f6c74102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:41.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.832+0000 7f6c7929a700 1 --2- 172.21.15.76:0/2497308899 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c741011d0 0x7f6c74102e80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00ef10 tx=0x7f6c6c00c3a0 comp rx=0 tx=0).stop 2024-03-18T14:51:41.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.833+0000 7f6c737fe700 1 -- 172.21.15.76:0/2497308899 reap_dead start 2024-03-18T14:51:41.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.833+0000 7f6c7929a700 1 -- 172.21.15.76:0/2497308899 shutdown_connections 2024-03-18T14:51:41.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.833+0000 7f6c7929a700 1 -- 172.21.15.76:0/2497308899 >> 172.21.15.76:0/2497308899 conn(0x7f6c74078100 msgr2=0x7f6c74100430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:41.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.833+0000 7f6c7929a700 1 -- 172.21.15.76:0/2497308899 shutdown_connections 2024-03-18T14:51:41.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:41.833+0000 7f6c7929a700 1 -- 172.21.15.76:0/2497308899 wait complete. 2024-03-18T14:51:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.844+0000 7f13daf1c700 1 -- 172.21.15.76:0/784350684 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d4069f80 msgr2=0x7f13d406a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.844+0000 7f13daf1c700 1 --2- 172.21.15.76:0/784350684 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d4069f80 0x7f13d406a3e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f13c00099c0 tx=0x7f13c0009cd0 comp rx=0 tx=0).stop 2024-03-18T14:51:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.844+0000 7f13daf1c700 1 -- 172.21.15.76:0/784350684 shutdown_connections 2024-03-18T14:51:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.844+0000 7f13daf1c700 1 --2- 172.21.15.76:0/784350684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13d406a920 0x7f13d410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.844+0000 7f13daf1c700 1 --2- 172.21.15.76:0/784350684 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13d4108320 0x7f13d4108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.844+0000 7f13daf1c700 1 --2- 172.21.15.76:0/784350684 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d4069f80 0x7f13d406a3e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.844+0000 7f13daf1c700 1 -- 172.21.15.76:0/784350684 >> 172.21.15.76:0/784350684 conn(0x7f13d4078100 msgr2=0x7f13d4078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.844+0000 7f13daf1c700 1 -- 172.21.15.76:0/784350684 shutdown_connections 2024-03-18T14:51:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.845+0000 7f13daf1c700 1 -- 172.21.15.76:0/784350684 wait complete. 2024-03-18T14:51:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.845+0000 7f13daf1c700 1 Processor -- start 2024-03-18T14:51:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.845+0000 7f13daf1c700 1 -- start start 2024-03-18T14:51:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.845+0000 7f13daf1c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d4069f80 0x7f13d41012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.845+0000 7f13daf1c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13d406a920 0x7f13d41017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.845+0000 7f13daf1c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13d4108320 0x7f13d4101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.845+0000 7f13daf1c700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f13d4112580 con 0x7f13d4069f80 2024-03-18T14:51:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.845+0000 7f13daf1c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f13d4112400 con 0x7f13d4108320 2024-03-18T14:51:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.845+0000 7f13daf1c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f13d4112700 con 0x7f13d406a920 2024-03-18T14:51:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.845+0000 7f13d8cb8700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d4069f80 0x7f13d41012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.845+0000 7f13d8cb8700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d4069f80 0x7f13d41012a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45166/0 (socket says 172.21.15.76:45166) 2024-03-18T14:51:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.845+0000 7f13d8cb8700 1 -- 172.21.15.76:0/1592705892 learned_addr learned my addr 172.21.15.76:0/1592705892 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.845+0000 7f13d3fff700 1 --2- 172.21.15.76:0/1592705892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13d406a920 0x7f13d41017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.846+0000 7f13d94b9700 1 --2- 172.21.15.76:0/1592705892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13d4108320 0x7f13d4101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.846+0000 7f13d8cb8700 1 -- 172.21.15.76:0/1592705892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13d406a920 msgr2=0x7f13d41017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.846+0000 7f13d8cb8700 1 --2- 172.21.15.76:0/1592705892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13d406a920 0x7f13d41017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.846+0000 7f13d8cb8700 1 -- 172.21.15.76:0/1592705892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13d4108320 msgr2=0x7f13d4101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.846+0000 7f13d8cb8700 1 --2- 172.21.15.76:0/1592705892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13d4108320 0x7f13d4101d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.846+0000 7f13d8cb8700 1 -- 172.21.15.76:0/1592705892 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d41a6fe0 con 0x7f13d4069f80 2024-03-18T14:51:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.846+0000 7f13d8cb8700 1 --2- 172.21.15.76:0/1592705892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d4069f80 0x7f13d41012a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f13c800e9c0 tx=0x7f13c800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.846+0000 7f13d1ffb700 1 -- 172.21.15.76:0/1592705892 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c800cbd0 con 0x7f13d4069f80 2024-03-18T14:51:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.846+0000 7f13d1ffb700 1 -- 172.21.15.76:0/1592705892 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13c800ee60 con 0x7f13d4069f80 2024-03-18T14:51:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.846+0000 7f13d1ffb700 1 -- 172.21.15.76:0/1592705892 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c8005630 con 0x7f13d4069f80 2024-03-18T14:51:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.846+0000 7f13daf1c700 1 -- 172.21.15.76:0/1592705892 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d41a72d0 con 0x7f13d4069f80 2024-03-18T14:51:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.846+0000 7f13daf1c700 1 -- 172.21.15.76:0/1592705892 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d41a7790 con 0x7f13d4069f80 2024-03-18T14:51:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.847+0000 7f13daf1c700 1 -- 172.21.15.76:0/1592705892 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f13d40629e0 con 0x7f13d4069f80 2024-03-18T14:51:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.848+0000 7f13d1ffb700 1 -- 172.21.15.76:0/1592705892 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f13c8004750 con 0x7f13d4069f80 2024-03-18T14:51:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.849+0000 7f13d1ffb700 1 --2- 172.21.15.76:0/1592705892 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f13c40a2920 0x7f13c40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.849+0000 7f13d3fff700 1 --2- 172.21.15.76:0/1592705892 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f13c40a2920 0x7f13c40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.849+0000 7f13d1ffb700 1 -- 172.21.15.76:0/1592705892 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f13c80cf060 con 0x7f13d4069f80 2024-03-18T14:51:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.849+0000 7f13d1ffb700 1 --2- 172.21.15.76:0/1592705892 >> [v2:172.21.15.174:6800/3812090370,v1:172.21.15.174:6801/3812090370] conn(0x7f13c40ac100 0x7f13c40ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.849+0000 7f13d1ffb700 1 -- 172.21.15.76:0/1592705892 --> [v2:172.21.15.174:6800/3812090370,v1:172.21.15.174:6801/3812090370] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f13c8004500 con 0x7f13c40ac100 2024-03-18T14:51:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.849+0000 7f13d1ffb700 1 -- 172.21.15.76:0/1592705892 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f13c80cf540 con 0x7f13d4069f80 2024-03-18T14:51:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.849+0000 7f13d94b9700 1 --2- 172.21.15.76:0/1592705892 >> [v2:172.21.15.174:6800/3812090370,v1:172.21.15.174:6801/3812090370] conn(0x7f13c40ac100 0x7f13c40ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.849+0000 7f13d3fff700 1 --2- 172.21.15.76:0/1592705892 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f13c40a2920 0x7f13c40a4de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f13d4102d30 tx=0x7f13c0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.850+0000 7f13d94b9700 1 --2- 172.21.15.76:0/1592705892 >> [v2:172.21.15.174:6800/3812090370,v1:172.21.15.174:6801/3812090370] conn(0x7f13c40ac100 0x7f13c40ae520 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.850+0000 7f13d1ffb700 1 -- 172.21.15.76:0/1592705892 <== osd.2 v2:172.21.15.174:6800/3812090370 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f13c8004500 con 0x7f13c40ac100 2024-03-18T14:51:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.866+0000 7f13daf1c700 1 -- 172.21.15.76:0/1592705892 --> [v2:172.21.15.174:6800/3812090370,v1:172.21.15.174:6801/3812090370] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f13d4062de0 con 0x7f13c40ac100 2024-03-18T14:51:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.866+0000 7f13d1ffb700 1 -- 172.21.15.76:0/1592705892 <== osd.2 v2:172.21.15.174:6800/3812090370 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f13d4062de0 con 0x7f13c40ac100 2024-03-18T14:51:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.867+0000 7f13daf1c700 1 -- 172.21.15.76:0/1592705892 >> [v2:172.21.15.174:6800/3812090370,v1:172.21.15.174:6801/3812090370] conn(0x7f13c40ac100 msgr2=0x7f13c40ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.867+0000 7f13daf1c700 1 --2- 172.21.15.76:0/1592705892 >> [v2:172.21.15.174:6800/3812090370,v1:172.21.15.174:6801/3812090370] conn(0x7f13c40ac100 0x7f13c40ae520 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.867+0000 7f13daf1c700 1 -- 172.21.15.76:0/1592705892 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f13c40a2920 msgr2=0x7f13c40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.867+0000 7f13daf1c700 1 --2- 172.21.15.76:0/1592705892 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f13c40a2920 0x7f13c40a4de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f13d4102d30 tx=0x7f13c0005fb0 comp rx=0 tx=0).stop 2024-03-18T14:51:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.867+0000 7f13daf1c700 1 -- 172.21.15.76:0/1592705892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d4069f80 msgr2=0x7f13d41012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.867+0000 7f13daf1c700 1 --2- 172.21.15.76:0/1592705892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13d4069f80 0x7f13d41012a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f13c800e9c0 tx=0x7f13c800ecd0 comp rx=0 tx=0).stop 2024-03-18T14:51:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.867+0000 7f13d94b9700 1 -- 172.21.15.76:0/1592705892 reap_dead start 2024-03-18T14:51:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.867+0000 7f13daf1c700 1 -- 172.21.15.76:0/1592705892 shutdown_connections 2024-03-18T14:51:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.867+0000 7f13daf1c700 1 -- 172.21.15.76:0/1592705892 >> 172.21.15.76:0/1592705892 conn(0x7f13d4078100 msgr2=0x7f13d40ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.867+0000 7f13daf1c700 1 -- 172.21.15.76:0/1592705892 shutdown_connections 2024-03-18T14:51:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:36.867+0000 7f13daf1c700 1 -- 172.21.15.76:0/1592705892 wait complete. 2024-03-18T14:51:42.652 INFO:teuthology.orchestra.run.smithi076.stdout:77309411394 2024-03-18T14:51:42.652 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-18T14:51:42.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:42 smithi076 ceph-mon[32205]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-18T14:51:42.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:42 smithi174 ceph-mon[36787]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-18T14:51:42.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:42 smithi077 ceph-mon[35865]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-18T14:51:43.285 INFO:teuthology.orchestra.run.smithi076.stdout:347892350997 2024-03-18T14:51:43.286 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-18T14:51:43.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.676+0000 7fc4da037700 1 -- 172.21.15.76:0/2055041016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4d4108300 msgr2=0x7fc4d41086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:43.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.676+0000 7fc4da037700 1 --2- 172.21.15.76:0/2055041016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4d4108300 0x7fc4d41086e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40099c0 tx=0x7fc4c4009cd0 comp rx=0 tx=0).stop 2024-03-18T14:51:43.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.676+0000 7fc4da037700 1 -- 172.21.15.76:0/2055041016 shutdown_connections 2024-03-18T14:51:43.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.676+0000 7fc4da037700 1 --2- 172.21.15.76:0/2055041016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d4102ce0 0x7fc4d410f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:43.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.676+0000 7fc4da037700 1 --2- 172.21.15.76:0/2055041016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4d4108300 0x7fc4d41086e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:43.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.676+0000 7fc4da037700 1 --2- 172.21.15.76:0/2055041016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4d4102320 0x7fc4d41027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:43.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.676+0000 7fc4da037700 1 -- 172.21.15.76:0/2055041016 >> 172.21.15.76:0/2055041016 conn(0x7fc4d4079400 msgr2=0x7fc4d4079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:43.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.677+0000 7fc4da037700 1 -- 172.21.15.76:0/2055041016 shutdown_connections 2024-03-18T14:51:43.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.677+0000 7fc4da037700 1 -- 172.21.15.76:0/2055041016 wait complete. 2024-03-18T14:51:43.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.677+0000 7fc4da037700 1 Processor -- start 2024-03-18T14:51:43.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.677+0000 7fc4da037700 1 -- start start 2024-03-18T14:51:43.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.677+0000 7fc4da037700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4d4102320 0x7fc4d419e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:43.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.677+0000 7fc4da037700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d4102ce0 0x7fc4d419ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:43.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.677+0000 7fc4da037700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4d4108300 0x7fc4d41988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:43.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.677+0000 7fc4da037700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d41125d0 con 0x7fc4d4108300 2024-03-18T14:51:43.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4da037700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d4112450 con 0x7fc4d4102ce0 2024-03-18T14:51:43.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4da037700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d4112750 con 0x7fc4d4102320 2024-03-18T14:51:43.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4d3fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4d4108300 0x7fc4d41988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:43.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4d37fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4d4102320 0x7fc4d419e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:43.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4d3fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4d4108300 0x7fc4d41988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37926/0 (socket says 172.21.15.76:37926) 2024-03-18T14:51:43.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4d37fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4d4102320 0x7fc4d419e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:53216/0 (socket says 172.21.15.76:53216) 2024-03-18T14:51:43.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4d3fff700 1 -- 172.21.15.76:0/4019034733 learned_addr learned my addr 172.21.15.76:0/4019034733 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:43.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4d2ffd700 1 --2- 172.21.15.76:0/4019034733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d4102ce0 0x7fc4d419ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:43.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4d2ffd700 1 -- 172.21.15.76:0/4019034733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4d4102320 msgr2=0x7fc4d419e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:43.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4d2ffd700 1 --2- 172.21.15.76:0/4019034733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4d4102320 0x7fc4d419e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:43.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4d2ffd700 1 -- 172.21.15.76:0/4019034733 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4d4108300 msgr2=0x7fc4d41988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:43.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4d2ffd700 1 --2- 172.21.15.76:0/4019034733 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4d4108300 0x7fc4d41988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:43.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4d2ffd700 1 -- 172.21.15.76:0/4019034733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4d4199010 con 0x7fc4d4102ce0 2024-03-18T14:51:43.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4d3fff700 1 --2- 172.21.15.76:0/4019034733 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4d4108300 0x7fc4d41988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:43.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4d37fe700 1 --2- 172.21.15.76:0/4019034733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4d4102320 0x7fc4d419e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:43.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4d2ffd700 1 --2- 172.21.15.76:0/4019034733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d4102ce0 0x7fc4d419ec30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800e9f0 tx=0x7fc4c800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:43.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4d0ff9700 1 -- 172.21.15.76:0/4019034733 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c800c9c0 con 0x7fc4d4102ce0 2024-03-18T14:51:43.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4d0ff9700 1 -- 172.21.15.76:0/4019034733 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4c800cb60 con 0x7fc4d4102ce0 2024-03-18T14:51:43.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4da037700 1 -- 172.21.15.76:0/4019034733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4d4199300 con 0x7fc4d4102ce0 2024-03-18T14:51:43.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4d0ff9700 1 -- 172.21.15.76:0/4019034733 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8018680 con 0x7fc4d4102ce0 2024-03-18T14:51:43.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.678+0000 7fc4da037700 1 -- 172.21.15.76:0/4019034733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4d41a55c0 con 0x7fc4d4102ce0 2024-03-18T14:51:43.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.679+0000 7fc4da037700 1 -- 172.21.15.76:0/4019034733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc4d4062a60 con 0x7fc4d4102ce0 2024-03-18T14:51:43.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.680+0000 7fc4d0ff9700 1 -- 172.21.15.76:0/4019034733 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc4c8010430 con 0x7fc4d4102ce0 2024-03-18T14:51:43.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.681+0000 7fc4d0ff9700 1 --2- 172.21.15.76:0/4019034733 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc4bc0a2920 0x7fc4bc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:43.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.681+0000 7fc4d37fe700 1 --2- 172.21.15.76:0/4019034733 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc4bc0a2920 0x7fc4bc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:43.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.681+0000 7fc4d0ff9700 1 -- 172.21.15.76:0/4019034733 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fc4c80cf100 con 0x7fc4d4102ce0 2024-03-18T14:51:43.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.681+0000 7fc4d0ff9700 1 --2- 172.21.15.76:0/4019034733 >> [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663] conn(0x7fc4bc0ac100 0x7fc4bc0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:43.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.681+0000 7fc4d0ff9700 1 -- 172.21.15.76:0/4019034733 --> [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc4c801bec0 con 0x7fc4bc0ac100 2024-03-18T14:51:43.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.681+0000 7fc4d0ff9700 1 -- 172.21.15.76:0/4019034733 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fc4c80cf4b0 con 0x7fc4d4102ce0 2024-03-18T14:51:43.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.681+0000 7fc4d3fff700 1 --2- 172.21.15.76:0/4019034733 >> [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663] conn(0x7fc4bc0ac100 0x7fc4bc0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:43.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.681+0000 7fc4d37fe700 1 --2- 172.21.15.76:0/4019034733 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc4bc0a2920 0x7fc4bc0a4de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4000c00 tx=0x7fc4c4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:43.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.682+0000 7fc4d3fff700 1 --2- 172.21.15.76:0/4019034733 >> [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663] conn(0x7fc4bc0ac100 0x7fc4bc0ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:43.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.682+0000 7fc4d0ff9700 1 -- 172.21.15.76:0/4019034733 <== osd.9 v2:172.21.15.76:6826/3937900663 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc4c801bec0 con 0x7fc4bc0ac100 2024-03-18T14:51:43.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.699+0000 7fc4da037700 1 -- 172.21.15.76:0/4019034733 --> [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc4d4062ee0 con 0x7fc4bc0ac100 2024-03-18T14:51:43.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.699+0000 7fc4d0ff9700 1 -- 172.21.15.76:0/4019034733 <== osd.9 v2:172.21.15.76:6826/3937900663 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc4d4062ee0 con 0x7fc4bc0ac100 2024-03-18T14:51:43.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.699+0000 7fc4da037700 1 -- 172.21.15.76:0/4019034733 >> [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663] conn(0x7fc4bc0ac100 msgr2=0x7fc4bc0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:43.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.699+0000 7fc4da037700 1 --2- 172.21.15.76:0/4019034733 >> [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663] conn(0x7fc4bc0ac100 0x7fc4bc0ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.699+0000 7fc4da037700 1 -- 172.21.15.76:0/4019034733 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc4bc0a2920 msgr2=0x7fc4bc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.699+0000 7fc4da037700 1 --2- 172.21.15.76:0/4019034733 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc4bc0a2920 0x7fc4bc0a4de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4000c00 tx=0x7fc4c4005fb0 comp rx=0 tx=0).stop 2024-03-18T14:51:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.699+0000 7fc4da037700 1 -- 172.21.15.76:0/4019034733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d4102ce0 msgr2=0x7fc4d419ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.699+0000 7fc4da037700 1 --2- 172.21.15.76:0/4019034733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d4102ce0 0x7fc4d419ec30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800e9f0 tx=0x7fc4c800ed00 comp rx=0 tx=0).stop 2024-03-18T14:51:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.699+0000 7fc4d3fff700 1 -- 172.21.15.76:0/4019034733 reap_dead start 2024-03-18T14:51:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.700+0000 7fc4da037700 1 -- 172.21.15.76:0/4019034733 shutdown_connections 2024-03-18T14:51:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.700+0000 7fc4da037700 1 -- 172.21.15.76:0/4019034733 >> 172.21.15.76:0/4019034733 conn(0x7fc4d4079400 msgr2=0x7fc4d4100fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.700+0000 7fc4da037700 1 -- 172.21.15.76:0/4019034733 shutdown_connections 2024-03-18T14:51:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:43.700+0000 7fc4da037700 1 -- 172.21.15.76:0/4019034733 wait complete. 2024-03-18T14:51:45.045 INFO:teuthology.orchestra.run.smithi076.stdout:292057776160 2024-03-18T14:51:45.045 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-18T14:51:45.497 INFO:teuthology.orchestra.run.smithi076.stdout:223338299432 2024-03-18T14:51:45.498 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-18T14:51:45.586 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:45 smithi076 ceph-mon[32205]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-18T14:51:45.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.694+0000 7f779ab8e700 1 -- 172.21.15.76:0/3854663088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794101b70 msgr2=0x7f779410e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:45.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.694+0000 7f779ab8e700 1 --2- 172.21.15.76:0/3854663088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794101b70 0x7f779410e050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f778c00b0a0 tx=0x7f778c00b3b0 comp rx=0 tx=0).stop 2024-03-18T14:51:45.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.694+0000 7f779ab8e700 1 -- 172.21.15.76:0/3854663088 shutdown_connections 2024-03-18T14:51:45.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.694+0000 7f779ab8e700 1 --2- 172.21.15.76:0/3854663088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794101b70 0x7f779410e050 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:45.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.694+0000 7f779ab8e700 1 --2- 172.21.15.76:0/3854663088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7794107210 0x7f77941075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:45.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.694+0000 7f779ab8e700 1 --2- 172.21.15.76:0/3854663088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77941011d0 0x7f7794101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:45.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.694+0000 7f779ab8e700 1 -- 172.21.15.76:0/3854663088 >> 172.21.15.76:0/3854663088 conn(0x7f7794078100 msgr2=0x7f7794078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:45.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.694+0000 7f779ab8e700 1 -- 172.21.15.76:0/3854663088 shutdown_connections 2024-03-18T14:51:45.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.694+0000 7f779ab8e700 1 -- 172.21.15.76:0/3854663088 wait complete. 2024-03-18T14:51:45.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.694+0000 7f779ab8e700 1 Processor -- start 2024-03-18T14:51:45.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.694+0000 7f779ab8e700 1 -- start start 2024-03-18T14:51:45.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f779ab8e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77941011d0 0x7f77941998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:45.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f779ab8e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7794101b70 0x7f7794199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:45.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f779ab8e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794107210 0x7f77941a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:45.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f779ab8e700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7794111490 con 0x7f7794107210 2024-03-18T14:51:45.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f779ab8e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7794111310 con 0x7f77941011d0 2024-03-18T14:51:45.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f779ab8e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7794111610 con 0x7f7794101b70 2024-03-18T14:51:45.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f7793fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7794101b70 0x7f7794199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:45.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f779912b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794107210 0x7f77941a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:45.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f779892a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77941011d0 0x7f77941998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:45.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f7793fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7794101b70 0x7f7794199df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:53244/0 (socket says 172.21.15.76:53244) 2024-03-18T14:51:45.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f7793fff700 1 -- 172.21.15.76:0/4192921746 learned_addr learned my addr 172.21.15.76:0/4192921746 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:45.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f779912b700 1 -- 172.21.15.76:0/4192921746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7794101b70 msgr2=0x7f7794199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:45.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f779912b700 1 --2- 172.21.15.76:0/4192921746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7794101b70 0x7f7794199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:45.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f779912b700 1 -- 172.21.15.76:0/4192921746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77941011d0 msgr2=0x7f77941998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:45.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f779912b700 1 --2- 172.21.15.76:0/4192921746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77941011d0 0x7f77941998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:45.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f779912b700 1 -- 172.21.15.76:0/4192921746 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77941a4e70 con 0x7f7794107210 2024-03-18T14:51:45.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f7793fff700 1 --2- 172.21.15.76:0/4192921746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7794101b70 0x7f7794199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:45.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f779912b700 1 --2- 172.21.15.76:0/4192921746 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794107210 0x7f77941a4930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f778c00b0a0 tx=0x7f778c004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:45.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f7791ffb700 1 -- 172.21.15.76:0/4192921746 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778c021070 con 0x7f7794107210 2024-03-18T14:51:45.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f7791ffb700 1 -- 172.21.15.76:0/4192921746 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f778c007b80 con 0x7f7794107210 2024-03-18T14:51:45.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f779ab8e700 1 -- 172.21.15.76:0/4192921746 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77941a5040 con 0x7f7794107210 2024-03-18T14:51:45.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.695+0000 7f7791ffb700 1 -- 172.21.15.76:0/4192921746 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778c005040 con 0x7f7794107210 2024-03-18T14:51:45.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.696+0000 7f779ab8e700 1 -- 172.21.15.76:0/4192921746 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77941a5540 con 0x7f7794107210 2024-03-18T14:51:45.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.697+0000 7f779ab8e700 1 -- 172.21.15.76:0/4192921746 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f77940629e0 con 0x7f7794107210 2024-03-18T14:51:45.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.698+0000 7f7791ffb700 1 -- 172.21.15.76:0/4192921746 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f778c024070 con 0x7f7794107210 2024-03-18T14:51:45.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.698+0000 7f7791ffb700 1 --2- 172.21.15.76:0/4192921746 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f77840a29a0 0x7f77840a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:45.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.698+0000 7f779892a700 1 --2- 172.21.15.76:0/4192921746 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f77840a29a0 0x7f77840a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:45.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.698+0000 7f7791ffb700 1 -- 172.21.15.76:0/4192921746 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f778c00f020 con 0x7f7794107210 2024-03-18T14:51:45.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.698+0000 7f7791ffb700 1 --2- 172.21.15.76:0/4192921746 >> [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861] conn(0x7f77840ac180 0x7f77840ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:45.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.699+0000 7f7791ffb700 1 -- 172.21.15.76:0/4192921746 --> [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f77840aec50 con 0x7f77840ac180 2024-03-18T14:51:45.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.699+0000 7f7791ffb700 1 -- 172.21.15.76:0/4192921746 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f778c0d03a0 con 0x7f7794107210 2024-03-18T14:51:45.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.699+0000 7f7793fff700 1 --2- 172.21.15.76:0/4192921746 >> [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861] conn(0x7f77840ac180 0x7f77840ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:45.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.699+0000 7f779892a700 1 --2- 172.21.15.76:0/4192921746 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f77840a29a0 0x7f77840a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7788009be0 tx=0x7f7788009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:45.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.699+0000 7f7793fff700 1 --2- 172.21.15.76:0/4192921746 >> [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861] conn(0x7f77840ac180 0x7f77840ae5a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:45.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.700+0000 7f7791ffb700 1 -- 172.21.15.76:0/4192921746 <== osd.0 v2:172.21.15.76:6802/2682004861 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77840aec50 con 0x7f77840ac180 2024-03-18T14:51:45.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:45 smithi174 ceph-mon[36787]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-18T14:51:45.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.718+0000 7f779ab8e700 1 -- 172.21.15.76:0/4192921746 --> [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7794062de0 con 0x7f77840ac180 2024-03-18T14:51:45.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.718+0000 7f7791ffb700 1 -- 172.21.15.76:0/4192921746 <== osd.0 v2:172.21.15.76:6802/2682004861 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7794062de0 con 0x7f77840ac180 2024-03-18T14:51:45.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.719+0000 7f779ab8e700 1 -- 172.21.15.76:0/4192921746 >> [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861] conn(0x7f77840ac180 msgr2=0x7f77840ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:45.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.719+0000 7f779ab8e700 1 --2- 172.21.15.76:0/4192921746 >> [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861] conn(0x7f77840ac180 0x7f77840ae5a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:45.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.719+0000 7f779ab8e700 1 -- 172.21.15.76:0/4192921746 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f77840a29a0 msgr2=0x7f77840a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:45.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.719+0000 7f779ab8e700 1 --2- 172.21.15.76:0/4192921746 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f77840a29a0 0x7f77840a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7788009be0 tx=0x7f7788009260 comp rx=0 tx=0).stop 2024-03-18T14:51:45.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.719+0000 7f779ab8e700 1 -- 172.21.15.76:0/4192921746 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794107210 msgr2=0x7f77941a4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:45.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.719+0000 7f779ab8e700 1 --2- 172.21.15.76:0/4192921746 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794107210 0x7f77941a4930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f778c00b0a0 tx=0x7f778c004d00 comp rx=0 tx=0).stop 2024-03-18T14:51:45.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.719+0000 7f779912b700 1 -- 172.21.15.76:0/4192921746 reap_dead start 2024-03-18T14:51:45.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.719+0000 7f779ab8e700 1 -- 172.21.15.76:0/4192921746 shutdown_connections 2024-03-18T14:51:45.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.719+0000 7f779ab8e700 1 -- 172.21.15.76:0/4192921746 >> 172.21.15.76:0/4192921746 conn(0x7f7794078100 msgr2=0x7f77940ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:45.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.719+0000 7f779ab8e700 1 -- 172.21.15.76:0/4192921746 shutdown_connections 2024-03-18T14:51:45.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:45.720+0000 7f779ab8e700 1 -- 172.21.15.76:0/4192921746 wait complete. 2024-03-18T14:51:45.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:45 smithi077 ceph-mon[35865]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-18T14:51:46.206 INFO:teuthology.orchestra.run.smithi076.stdout:201863462958 2024-03-18T14:51:46.207 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-18T14:51:46.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.388+0000 7f8ecdc68700 1 -- 172.21.15.76:0/120829598 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ec8106550 msgr2=0x7f8ec810e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:46.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.388+0000 7f8ecdc68700 1 --2- 172.21.15.76:0/120829598 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ec8106550 0x7f8ec810e120 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000b0d0 tx=0x7f8ec000b3e0 comp rx=0 tx=0).stop 2024-03-18T14:51:46.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.388+0000 7f8ecdc68700 1 -- 172.21.15.76:0/120829598 shutdown_connections 2024-03-18T14:51:46.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.388+0000 7f8ecdc68700 1 --2- 172.21.15.76:0/120829598 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ec8106550 0x7f8ec810e120 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:46.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.388+0000 7f8ecdc68700 1 --2- 172.21.15.76:0/120829598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ec8101c30 0x7f8ec8102010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:46.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.388+0000 7f8ecdc68700 1 --2- 172.21.15.76:0/120829598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ec81025e0 0x7f8ec8105e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:46.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.388+0000 7f8ecdc68700 1 -- 172.21.15.76:0/120829598 >> 172.21.15.76:0/120829598 conn(0x7f8ec80777d0 msgr2=0x7f8ec80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:46.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.388+0000 7f8ecdc68700 1 -- 172.21.15.76:0/120829598 shutdown_connections 2024-03-18T14:51:46.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.388+0000 7f8ecdc68700 1 -- 172.21.15.76:0/120829598 wait complete. 2024-03-18T14:51:46.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.389+0000 7f8ecdc68700 1 Processor -- start 2024-03-18T14:51:46.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.389+0000 7f8ecdc68700 1 -- start start 2024-03-18T14:51:46.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ecdc68700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ec8101c30 0x7f8ec819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:46.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ecdc68700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ec81025e0 0x7f8ec819ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:46.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ecdc68700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ec8106550 0x7f8ec81988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:46.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ecdc68700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec8111560 con 0x7f8ec8101c30 2024-03-18T14:51:46.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ecdc68700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec81113e0 con 0x7f8ec8106550 2024-03-18T14:51:46.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ecdc68700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec81116e0 con 0x7f8ec81025e0 2024-03-18T14:51:46.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ec77fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ec8101c30 0x7f8ec819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:46.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ec77fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ec8101c30 0x7f8ec819e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37962/0 (socket says 172.21.15.76:37962) 2024-03-18T14:51:46.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ec77fe700 1 -- 172.21.15.76:0/1083657096 learned_addr learned my addr 172.21.15.76:0/1083657096 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:46.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ec6ffd700 1 --2- 172.21.15.76:0/1083657096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ec81025e0 0x7f8ec819ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:46.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ec7fff700 1 --2- 172.21.15.76:0/1083657096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ec8106550 0x7f8ec81988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:46.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ec77fe700 1 -- 172.21.15.76:0/1083657096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ec81025e0 msgr2=0x7f8ec819ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:46.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ec77fe700 1 --2- 172.21.15.76:0/1083657096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ec81025e0 0x7f8ec819ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:46.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ec77fe700 1 -- 172.21.15.76:0/1083657096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ec8106550 msgr2=0x7f8ec81988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:46.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ec77fe700 1 --2- 172.21.15.76:0/1083657096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ec8106550 0x7f8ec81988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:46.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ec77fe700 1 -- 172.21.15.76:0/1083657096 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec8199030 con 0x7f8ec8101c30 2024-03-18T14:51:46.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ec7fff700 1 --2- 172.21.15.76:0/1083657096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ec8106550 0x7f8ec81988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:51:46.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ec6ffd700 1 --2- 172.21.15.76:0/1083657096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ec81025e0 0x7f8ec819ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:46.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.390+0000 7f8ec77fe700 1 --2- 172.21.15.76:0/1083657096 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ec8101c30 0x7f8ec819e6c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8eb800e9a0 tx=0x7f8eb800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:46.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.391+0000 7f8ec4ff9700 1 -- 172.21.15.76:0/1083657096 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb800cbd0 con 0x7f8ec8101c30 2024-03-18T14:51:46.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.391+0000 7f8ec4ff9700 1 -- 172.21.15.76:0/1083657096 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8eb800cd70 con 0x7f8ec8101c30 2024-03-18T14:51:46.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.391+0000 7f8ec4ff9700 1 -- 172.21.15.76:0/1083657096 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb8018750 con 0x7f8ec8101c30 2024-03-18T14:51:46.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.391+0000 7f8ecdc68700 1 -- 172.21.15.76:0/1083657096 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ec8199320 con 0x7f8ec8101c30 2024-03-18T14:51:46.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.391+0000 7f8ecdc68700 1 -- 172.21.15.76:0/1083657096 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ec81a5590 con 0x7f8ec8101c30 2024-03-18T14:51:46.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.392+0000 7f8ecdc68700 1 -- 172.21.15.76:0/1083657096 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8ec8103cb0 con 0x7f8ec8101c30 2024-03-18T14:51:46.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.393+0000 7f8ec4ff9700 1 -- 172.21.15.76:0/1083657096 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8eb8010430 con 0x7f8ec8101c30 2024-03-18T14:51:46.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.394+0000 7f8ec4ff9700 1 --2- 172.21.15.76:0/1083657096 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8eb00a2920 0x7f8eb00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:46.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.394+0000 7f8ec6ffd700 1 --2- 172.21.15.76:0/1083657096 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8eb00a2920 0x7f8eb00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:46.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.394+0000 7f8ec4ff9700 1 -- 172.21.15.76:0/1083657096 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f8eb80cf860 con 0x7f8ec8101c30 2024-03-18T14:51:46.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.395+0000 7f8ec4ff9700 1 --2- 172.21.15.76:0/1083657096 >> [v2:172.21.15.174:6808/1607563490,v1:172.21.15.174:6809/1607563490] conn(0x7f8eb00ac100 0x7f8eb00ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:46.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.395+0000 7f8ec4ff9700 1 -- 172.21.15.76:0/1083657096 --> [v2:172.21.15.174:6808/1607563490,v1:172.21.15.174:6809/1607563490] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8eb00aebd0 con 0x7f8eb00ac100 2024-03-18T14:51:46.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.395+0000 7f8ec4ff9700 1 -- 172.21.15.76:0/1083657096 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f8eb80cfd40 con 0x7f8ec8101c30 2024-03-18T14:51:46.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.395+0000 7f8ec6ffd700 1 --2- 172.21.15.76:0/1083657096 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8eb00a2920 0x7f8eb00a4de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8105d00 tx=0x7f8ebc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:46.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.395+0000 7f8ec7fff700 1 --2- 172.21.15.76:0/1083657096 >> [v2:172.21.15.174:6808/1607563490,v1:172.21.15.174:6809/1607563490] conn(0x7f8eb00ac100 0x7f8eb00ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:46.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.395+0000 7f8ec7fff700 1 --2- 172.21.15.76:0/1083657096 >> [v2:172.21.15.174:6808/1607563490,v1:172.21.15.174:6809/1607563490] conn(0x7f8eb00ac100 0x7f8eb00ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:46.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.396+0000 7f8ec4ff9700 1 -- 172.21.15.76:0/1083657096 <== osd.5 v2:172.21.15.174:6808/1607563490 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8eb00aebd0 con 0x7f8eb00ac100 2024-03-18T14:51:46.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.413+0000 7f8ecdc68700 1 -- 172.21.15.76:0/1083657096 --> [v2:172.21.15.174:6808/1607563490,v1:172.21.15.174:6809/1607563490] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8ec81029e0 con 0x7f8eb00ac100 2024-03-18T14:51:46.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.413+0000 7f8ec4ff9700 1 -- 172.21.15.76:0/1083657096 <== osd.5 v2:172.21.15.174:6808/1607563490 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8ec81029e0 con 0x7f8eb00ac100 2024-03-18T14:51:46.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.414+0000 7f8ecdc68700 1 -- 172.21.15.76:0/1083657096 >> [v2:172.21.15.174:6808/1607563490,v1:172.21.15.174:6809/1607563490] conn(0x7f8eb00ac100 msgr2=0x7f8eb00ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:46.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.414+0000 7f8ecdc68700 1 --2- 172.21.15.76:0/1083657096 >> [v2:172.21.15.174:6808/1607563490,v1:172.21.15.174:6809/1607563490] conn(0x7f8eb00ac100 0x7f8eb00ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:46.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.414+0000 7f8ecdc68700 1 -- 172.21.15.76:0/1083657096 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8eb00a2920 msgr2=0x7f8eb00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:46.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.414+0000 7f8ecdc68700 1 --2- 172.21.15.76:0/1083657096 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8eb00a2920 0x7f8eb00a4de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8105d00 tx=0x7f8ebc009260 comp rx=0 tx=0).stop 2024-03-18T14:51:46.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.414+0000 7f8ecdc68700 1 -- 172.21.15.76:0/1083657096 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ec8101c30 msgr2=0x7f8ec819e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:46.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.414+0000 7f8ecdc68700 1 --2- 172.21.15.76:0/1083657096 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ec8101c30 0x7f8ec819e6c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8eb800e9a0 tx=0x7f8eb800ecb0 comp rx=0 tx=0).stop 2024-03-18T14:51:46.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.414+0000 7f8ec7fff700 1 -- 172.21.15.76:0/1083657096 reap_dead start 2024-03-18T14:51:46.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.414+0000 7f8ecdc68700 1 -- 172.21.15.76:0/1083657096 shutdown_connections 2024-03-18T14:51:46.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.414+0000 7f8ecdc68700 1 -- 172.21.15.76:0/1083657096 >> 172.21.15.76:0/1083657096 conn(0x7f8ec80777d0 msgr2=0x7f8ec8103310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:46.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.414+0000 7f8ecdc68700 1 -- 172.21.15.76:0/1083657096 shutdown_connections 2024-03-18T14:51:46.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:46.414+0000 7f8ecdc68700 1 -- 172.21.15.76:0/1083657096 wait complete. 2024-03-18T14:51:47.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.132+0000 7fc5ce650700 1 -- 172.21.15.76:0/1809553247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5c8107660 msgr2=0x7fc5c810f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:47.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.132+0000 7fc5ce650700 1 --2- 172.21.15.76:0/1809553247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5c8107660 0x7fc5c810f1e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc5c000b0d0 tx=0x7fc5c000b3e0 comp rx=0 tx=0).stop 2024-03-18T14:51:47.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.132+0000 7fc5ce650700 1 -- 172.21.15.76:0/1809553247 shutdown_connections 2024-03-18T14:51:47.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.132+0000 7fc5ce650700 1 --2- 172.21.15.76:0/1809553247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5c8107660 0x7fc5c810f1e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:47.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.132+0000 7fc5ce650700 1 --2- 172.21.15.76:0/1809553247 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5c81025b0 0x7fc5c8102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:47.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.132+0000 7fc5ce650700 1 --2- 172.21.15.76:0/1809553247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5c8102f60 0x7fc5c8106f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:47.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.132+0000 7fc5ce650700 1 -- 172.21.15.76:0/1809553247 >> 172.21.15.76:0/1809553247 conn(0x7fc5c80793f0 msgr2=0x7fc5c8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:47.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.132+0000 7fc5ce650700 1 -- 172.21.15.76:0/1809553247 shutdown_connections 2024-03-18T14:51:47.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.132+0000 7fc5ce650700 1 -- 172.21.15.76:0/1809553247 wait complete. 2024-03-18T14:51:47.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.133+0000 7fc5ce650700 1 Processor -- start 2024-03-18T14:51:47.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.133+0000 7fc5ce650700 1 -- start start 2024-03-18T14:51:47.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.133+0000 7fc5ce650700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5c81025b0 0x7fc5c819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:47.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.133+0000 7fc5ce650700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5c8102f60 0x7fc5c819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:47.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.133+0000 7fc5ce650700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5c8107660 0x7fc5c81987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:47.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.133+0000 7fc5ce650700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c8112600 con 0x7fc5c81025b0 2024-03-18T14:51:47.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.133+0000 7fc5ce650700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c8112480 con 0x7fc5c8107660 2024-03-18T14:51:47.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.133+0000 7fc5ce650700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c8112780 con 0x7fc5c8102f60 2024-03-18T14:51:47.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.133+0000 7fc5c7fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5c81025b0 0x7fc5c819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:47.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.133+0000 7fc5c7fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5c81025b0 0x7fc5c819e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37982/0 (socket says 172.21.15.76:37982) 2024-03-18T14:51:47.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.133+0000 7fc5c7fff700 1 -- 172.21.15.76:0/2065208348 learned_addr learned my addr 172.21.15.76:0/2065208348 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:47.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.134+0000 7fc5c77fe700 1 --2- 172.21.15.76:0/2065208348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5c8102f60 0x7fc5c819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:47.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.134+0000 7fc5ccbed700 1 --2- 172.21.15.76:0/2065208348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5c8107660 0x7fc5c81987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:47.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.134+0000 7fc5c7fff700 1 -- 172.21.15.76:0/2065208348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5c8102f60 msgr2=0x7fc5c819eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:47.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.134+0000 7fc5c7fff700 1 --2- 172.21.15.76:0/2065208348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5c8102f60 0x7fc5c819eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:47.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.134+0000 7fc5c7fff700 1 -- 172.21.15.76:0/2065208348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5c8107660 msgr2=0x7fc5c81987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:47.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.134+0000 7fc5c7fff700 1 --2- 172.21.15.76:0/2065208348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5c8107660 0x7fc5c81987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:47.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.134+0000 7fc5c7fff700 1 -- 172.21.15.76:0/2065208348 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c8198fc0 con 0x7fc5c81025b0 2024-03-18T14:51:47.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.134+0000 7fc5c7fff700 1 --2- 172.21.15.76:0/2065208348 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5c81025b0 0x7fc5c819e660 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc00e9a0 tx=0x7fc5bc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:47.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.134+0000 7fc5c57fa700 1 -- 172.21.15.76:0/2065208348 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc00ca10 con 0x7fc5c81025b0 2024-03-18T14:51:47.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.134+0000 7fc5c57fa700 1 -- 172.21.15.76:0/2065208348 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5bc00cbb0 con 0x7fc5c81025b0 2024-03-18T14:51:47.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.134+0000 7fc5ce650700 1 -- 172.21.15.76:0/2065208348 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c81992b0 con 0x7fc5c81025b0 2024-03-18T14:51:47.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.134+0000 7fc5c57fa700 1 -- 172.21.15.76:0/2065208348 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc018680 con 0x7fc5c81025b0 2024-03-18T14:51:47.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.134+0000 7fc5ce650700 1 -- 172.21.15.76:0/2065208348 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c81a54c0 con 0x7fc5c81025b0 2024-03-18T14:51:47.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.135+0000 7fc5ce650700 1 -- 172.21.15.76:0/2065208348 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc5c8062a60 con 0x7fc5c81025b0 2024-03-18T14:51:47.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.136+0000 7fc5c57fa700 1 -- 172.21.15.76:0/2065208348 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc5bc010430 con 0x7fc5c81025b0 2024-03-18T14:51:47.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.137+0000 7fc5c57fa700 1 --2- 172.21.15.76:0/2065208348 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc5b00a29a0 0x7fc5b00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:47.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.137+0000 7fc5c77fe700 1 --2- 172.21.15.76:0/2065208348 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc5b00a29a0 0x7fc5b00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:47.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.137+0000 7fc5c57fa700 1 -- 172.21.15.76:0/2065208348 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fc5bc0cf4d0 con 0x7fc5c81025b0 2024-03-18T14:51:47.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.137+0000 7fc5c57fa700 1 --2- 172.21.15.76:0/2065208348 >> [v2:172.21.15.77:6800/3243872836,v1:172.21.15.77:6801/3243872836] conn(0x7fc5b00ac180 0x7fc5b00ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:47.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.137+0000 7fc5c57fa700 1 -- 172.21.15.76:0/2065208348 --> [v2:172.21.15.77:6800/3243872836,v1:172.21.15.77:6801/3243872836] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc5bc01bec0 con 0x7fc5b00ac180 2024-03-18T14:51:47.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.137+0000 7fc5c57fa700 1 -- 172.21.15.76:0/2065208348 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fc5bc0cf930 con 0x7fc5c81025b0 2024-03-18T14:51:47.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.138+0000 7fc5c77fe700 1 --2- 172.21.15.76:0/2065208348 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc5b00a29a0 0x7fc5b00a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8108a00 tx=0x7fc5b8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:47.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.138+0000 7fc5ccbed700 1 --2- 172.21.15.76:0/2065208348 >> [v2:172.21.15.77:6800/3243872836,v1:172.21.15.77:6801/3243872836] conn(0x7fc5b00ac180 0x7fc5b00ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:47.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.138+0000 7fc5ccbed700 1 --2- 172.21.15.76:0/2065208348 >> [v2:172.21.15.77:6800/3243872836,v1:172.21.15.77:6801/3243872836] conn(0x7fc5b00ac180 0x7fc5b00ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:47.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.139+0000 7fc5c57fa700 1 -- 172.21.15.76:0/2065208348 <== osd.1 v2:172.21.15.77:6800/3243872836 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc5bc01bec0 con 0x7fc5b00ac180 2024-03-18T14:51:47.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.154+0000 7fc5ce650700 1 -- 172.21.15.76:0/2065208348 --> [v2:172.21.15.77:6800/3243872836,v1:172.21.15.77:6801/3243872836] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc5c8062e60 con 0x7fc5b00ac180 2024-03-18T14:51:47.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.155+0000 7fc5c57fa700 1 -- 172.21.15.76:0/2065208348 <== osd.1 v2:172.21.15.77:6800/3243872836 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc5c8062e60 con 0x7fc5b00ac180 2024-03-18T14:51:47.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.155+0000 7fc5ce650700 1 -- 172.21.15.76:0/2065208348 >> [v2:172.21.15.77:6800/3243872836,v1:172.21.15.77:6801/3243872836] conn(0x7fc5b00ac180 msgr2=0x7fc5b00ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:47.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.155+0000 7fc5ce650700 1 --2- 172.21.15.76:0/2065208348 >> [v2:172.21.15.77:6800/3243872836,v1:172.21.15.77:6801/3243872836] conn(0x7fc5b00ac180 0x7fc5b00ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:47.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.155+0000 7fc5ce650700 1 -- 172.21.15.76:0/2065208348 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc5b00a29a0 msgr2=0x7fc5b00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:47.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.155+0000 7fc5ce650700 1 --2- 172.21.15.76:0/2065208348 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc5b00a29a0 0x7fc5b00a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8108a00 tx=0x7fc5b8009260 comp rx=0 tx=0).stop 2024-03-18T14:51:47.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.155+0000 7fc5ce650700 1 -- 172.21.15.76:0/2065208348 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5c81025b0 msgr2=0x7fc5c819e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:47.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.155+0000 7fc5ce650700 1 --2- 172.21.15.76:0/2065208348 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5c81025b0 0x7fc5c819e660 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc00e9a0 tx=0x7fc5bc00ecb0 comp rx=0 tx=0).stop 2024-03-18T14:51:47.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.155+0000 7fc5ccbed700 1 -- 172.21.15.76:0/2065208348 reap_dead start 2024-03-18T14:51:47.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.155+0000 7fc5ce650700 1 -- 172.21.15.76:0/2065208348 shutdown_connections 2024-03-18T14:51:47.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.155+0000 7fc5ce650700 1 -- 172.21.15.76:0/2065208348 >> 172.21.15.76:0/2065208348 conn(0x7fc5c80793f0 msgr2=0x7fc5c81003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:47.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.155+0000 7fc5ce650700 1 -- 172.21.15.76:0/2065208348 shutdown_connections 2024-03-18T14:51:47.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:47.155+0000 7fc5ce650700 1 -- 172.21.15.76:0/2065208348 wait complete. 2024-03-18T14:51:47.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:47 smithi076 ceph-mon[32205]: pgmap v306: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:51:47.701 INFO:teuthology.orchestra.run.smithi076.stdout:34359738445 2024-03-18T14:51:47.702 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-18T14:51:47.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:47 smithi174 ceph-mon[36787]: pgmap v306: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:51:47.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:47 smithi077 ceph-mon[35865]: pgmap v306: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:51:48.947 INFO:teuthology.orchestra.run.smithi076.stdout:167503724596 2024-03-18T14:51:48.948 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-18T14:51:49.381 INFO:teuthology.orchestra.run.smithi076.stdout:146028888120 2024-03-18T14:51:49.382 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-18T14:51:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:49 smithi076 ceph-mon[32205]: pgmap v307: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T14:51:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:49 smithi174 ceph-mon[36787]: pgmap v307: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T14:51:49.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:49 smithi077 ceph-mon[35865]: pgmap v307: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T14:51:49.835 INFO:teuthology.orchestra.run.smithi076.stdout:55834574920 2024-03-18T14:51:49.836 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-18T14:51:49.901 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:51.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.137+0000 7fe0a282c700 1 -- 172.21.15.76:0/3129272288 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe09c06b310 msgr2=0x7fe09c10e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:51.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.137+0000 7fe0a282c700 1 --2- 172.21.15.76:0/3129272288 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe09c06b310 0x7fe09c10e820 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe090009990 tx=0x7fe090009ca0 comp rx=0 tx=0).stop 2024-03-18T14:51:51.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.137+0000 7fe0a282c700 1 -- 172.21.15.76:0/3129272288 shutdown_connections 2024-03-18T14:51:51.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.137+0000 7fe0a282c700 1 --2- 172.21.15.76:0/3129272288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe09c10ed60 0x7fe09c111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:51.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.137+0000 7fe0a282c700 1 --2- 172.21.15.76:0/3129272288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe09c06a9f0 0x7fe09c06add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:51.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.137+0000 7fe0a282c700 1 --2- 172.21.15.76:0/3129272288 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe09c06b310 0x7fe09c10e820 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:51.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.137+0000 7fe0a282c700 1 -- 172.21.15.76:0/3129272288 >> 172.21.15.76:0/3129272288 conn(0x7fe09c0781e0 msgr2=0x7fe09c0785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:51.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.137+0000 7fe0a282c700 1 -- 172.21.15.76:0/3129272288 shutdown_connections 2024-03-18T14:51:51.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.137+0000 7fe0a282c700 1 -- 172.21.15.76:0/3129272288 wait complete. 2024-03-18T14:51:51.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.138+0000 7fe0a282c700 1 Processor -- start 2024-03-18T14:51:51.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.138+0000 7fe0a282c700 1 -- start start 2024-03-18T14:51:51.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.138+0000 7fe0a282c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe09c06a9f0 0x7fe09c1a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:51.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.138+0000 7fe0a282c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe09c06b310 0x7fe09c1a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.138+0000 7fe0a282c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe09c10ed60 0x7fe09c19aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.138+0000 7fe0a282c700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe09c1147b0 con 0x7fe09c10ed60 2024-03-18T14:51:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.138+0000 7fe0a282c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe09c114630 con 0x7fe09c06b310 2024-03-18T14:51:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.138+0000 7fe0a282c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe09c114930 con 0x7fe09c06a9f0 2024-03-18T14:51:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe0a0dc9700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe09c10ed60 0x7fe09c19aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe09bfff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe09c06a9f0 0x7fe09c1a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe09bfff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe09c06a9f0 0x7fe09c1a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:35972/0 (socket says 172.21.15.76:35972) 2024-03-18T14:51:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe0a0dc9700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe09c10ed60 0x7fe09c19aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51896/0 (socket says 172.21.15.76:51896) 2024-03-18T14:51:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe09b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe09c06b310 0x7fe09c1a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe09bfff700 1 -- 172.21.15.76:0/163528687 learned_addr learned my addr 172.21.15.76:0/163528687 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe09bfff700 1 -- 172.21.15.76:0/163528687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe09c06b310 msgr2=0x7fe09c1a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe09bfff700 1 --2- 172.21.15.76:0/163528687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe09c06b310 0x7fe09c1a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe09bfff700 1 -- 172.21.15.76:0/163528687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe09c10ed60 msgr2=0x7fe09c19aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe09bfff700 1 --2- 172.21.15.76:0/163528687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe09c10ed60 0x7fe09c19aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:51.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe09bfff700 1 -- 172.21.15.76:0/163528687 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe09c19b1d0 con 0x7fe09c06a9f0 2024-03-18T14:51:51.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe09b7fe700 1 --2- 172.21.15.76:0/163528687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe09c06b310 0x7fe09c1a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:51.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe0a0dc9700 1 --2- 172.21.15.76:0/163528687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe09c10ed60 0x7fe09c19aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:51:51.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe09bfff700 1 --2- 172.21.15.76:0/163528687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe09c06a9f0 0x7fe09c1a08b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00e9a0 tx=0x7fe08c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:51.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe0997fa700 1 -- 172.21.15.76:0/163528687 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08c00ca10 con 0x7fe09c06a9f0 2024-03-18T14:51:51.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe0997fa700 1 -- 172.21.15.76:0/163528687 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe08c00cbb0 con 0x7fe09c06a9f0 2024-03-18T14:51:51.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe0997fa700 1 -- 172.21.15.76:0/163528687 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08c018680 con 0x7fe09c06a9f0 2024-03-18T14:51:51.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.139+0000 7fe0a282c700 1 -- 172.21.15.76:0/163528687 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe09c19b4c0 con 0x7fe09c06a9f0 2024-03-18T14:51:51.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.140+0000 7fe0a282c700 1 -- 172.21.15.76:0/163528687 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe09c1a7780 con 0x7fe09c06a9f0 2024-03-18T14:51:51.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.140+0000 7fe0a282c700 1 -- 172.21.15.76:0/163528687 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe09c062a60 con 0x7fe09c06a9f0 2024-03-18T14:51:51.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.142+0000 7fe0997fa700 1 -- 172.21.15.76:0/163528687 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe08c010430 con 0x7fe09c06a9f0 2024-03-18T14:51:51.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.143+0000 7fe0997fa700 1 --2- 172.21.15.76:0/163528687 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe0840a29a0 0x7fe0840a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:51.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.143+0000 7fe09b7fe700 1 --2- 172.21.15.76:0/163528687 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe0840a29a0 0x7fe0840a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:51.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.143+0000 7fe0997fa700 1 -- 172.21.15.76:0/163528687 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fe08c0cf4f0 con 0x7fe09c06a9f0 2024-03-18T14:51:51.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.143+0000 7fe0997fa700 1 --2- 172.21.15.76:0/163528687 >> [v2:172.21.15.77:6824/4105457096,v1:172.21.15.77:6825/4105457096] conn(0x7fe0840ac180 0x7fe0840ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:51.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.143+0000 7fe0997fa700 1 -- 172.21.15.76:0/163528687 --> [v2:172.21.15.77:6824/4105457096,v1:172.21.15.77:6825/4105457096] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe08c01bec0 con 0x7fe0840ac180 2024-03-18T14:51:51.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.143+0000 7fe0997fa700 1 -- 172.21.15.76:0/163528687 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fe08c0cf8a0 con 0x7fe09c06a9f0 2024-03-18T14:51:51.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.143+0000 7fe0a0dc9700 1 --2- 172.21.15.76:0/163528687 >> [v2:172.21.15.77:6824/4105457096,v1:172.21.15.77:6825/4105457096] conn(0x7fe0840ac180 0x7fe0840ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:51.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.143+0000 7fe09b7fe700 1 --2- 172.21.15.76:0/163528687 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe0840a29a0 0x7fe0840a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe09000b2e0 tx=0x7fe09001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:51.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.144+0000 7fe0a0dc9700 1 --2- 172.21.15.76:0/163528687 >> [v2:172.21.15.77:6824/4105457096,v1:172.21.15.77:6825/4105457096] conn(0x7fe0840ac180 0x7fe0840ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:51.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.144+0000 7fe0997fa700 1 -- 172.21.15.76:0/163528687 <== osd.10 v2:172.21.15.77:6824/4105457096 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe08c01bec0 con 0x7fe0840ac180 2024-03-18T14:51:51.164 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:51.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.166+0000 7fe0a282c700 1 -- 172.21.15.76:0/163528687 --> [v2:172.21.15.77:6824/4105457096,v1:172.21.15.77:6825/4105457096] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe09c062ee0 con 0x7fe0840ac180 2024-03-18T14:51:51.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.166+0000 7fe0997fa700 1 -- 172.21.15.76:0/163528687 <== osd.10 v2:172.21.15.77:6824/4105457096 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe09c062ee0 con 0x7fe0840ac180 2024-03-18T14:51:51.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.167+0000 7fe0a282c700 1 -- 172.21.15.76:0/163528687 >> [v2:172.21.15.77:6824/4105457096,v1:172.21.15.77:6825/4105457096] conn(0x7fe0840ac180 msgr2=0x7fe0840ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:51.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.167+0000 7fe0a282c700 1 --2- 172.21.15.76:0/163528687 >> [v2:172.21.15.77:6824/4105457096,v1:172.21.15.77:6825/4105457096] conn(0x7fe0840ac180 0x7fe0840ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:51.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.167+0000 7fe0a282c700 1 -- 172.21.15.76:0/163528687 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe0840a29a0 msgr2=0x7fe0840a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:51.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.167+0000 7fe0a282c700 1 --2- 172.21.15.76:0/163528687 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe0840a29a0 0x7fe0840a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe09000b2e0 tx=0x7fe09001a040 comp rx=0 tx=0).stop 2024-03-18T14:51:51.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.167+0000 7fe0a282c700 1 -- 172.21.15.76:0/163528687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe09c06a9f0 msgr2=0x7fe09c1a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:51.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.167+0000 7fe0a282c700 1 --2- 172.21.15.76:0/163528687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe09c06a9f0 0x7fe09c1a08b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00e9a0 tx=0x7fe08c00ecb0 comp rx=0 tx=0).stop 2024-03-18T14:51:51.169 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:51.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.167+0000 7fe0a0dc9700 1 -- 172.21.15.76:0/163528687 reap_dead start 2024-03-18T14:51:51.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.167+0000 7fe0a282c700 1 -- 172.21.15.76:0/163528687 shutdown_connections 2024-03-18T14:51:51.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.167+0000 7fe0a282c700 1 -- 172.21.15.76:0/163528687 >> 172.21.15.76:0/163528687 conn(0x7fe09c0781e0 msgr2=0x7fe09c110270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:51.170 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:51.171 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:51.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.170+0000 7fe0a282c700 1 -- 172.21.15.76:0/163528687 shutdown_connections 2024-03-18T14:51:51.173 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:51.173 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:51.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:51.171+0000 7fe0a282c700 1 -- 172.21.15.76:0/163528687 wait complete. 2024-03-18T14:51:51.174 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:51.607 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:51 smithi076 ceph-mon[32205]: pgmap v308: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-18T14:51:51.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:51 smithi174 ceph-mon[36787]: pgmap v308: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-18T14:51:51.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:51 smithi077 ceph-mon[35865]: pgmap v308: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-18T14:51:52.436 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:53.650 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:53 smithi076 ceph-mon[32205]: pgmap v309: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-18T14:51:53.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:53 smithi174 ceph-mon[36787]: pgmap v309: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-18T14:51:53.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:53 smithi077 ceph-mon[35865]: pgmap v309: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-18T14:51:54.134 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:51:54.163 INFO:teuthology.orchestra.run.smithi076.stdout:313532612635 2024-03-18T14:51:54.164 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-18T14:51:55.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:55 smithi076 ceph-mon[32205]: pgmap v310: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-18T14:51:55.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:55 smithi174 ceph-mon[36787]: pgmap v310: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-18T14:51:55.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:55 smithi077 ceph-mon[35865]: pgmap v310: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-18T14:51:57.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:57 smithi076 ceph-mon[32205]: pgmap v311: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-18T14:51:57.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:57 smithi174 ceph-mon[36787]: pgmap v311: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-18T14:51:57.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:57 smithi077 ceph-mon[35865]: pgmap v311: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-18T14:51:59.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:51:59 smithi076 ceph-mon[32205]: pgmap v312: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T14:51:59.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.658+0000 7fe9e8a68700 1 -- 172.21.15.76:0/2130006921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9e4101c30 msgr2=0x7fe9e4102010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:59.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.658+0000 7fe9e8a68700 1 --2- 172.21.15.76:0/2130006921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9e4101c30 0x7fe9e4102010 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe9d40099c0 tx=0x7fe9d4009cd0 comp rx=0 tx=0).stop 2024-03-18T14:51:59.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.658+0000 7fe9e8a68700 1 -- 172.21.15.76:0/2130006921 shutdown_connections 2024-03-18T14:51:59.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.658+0000 7fe9e8a68700 1 --2- 172.21.15.76:0/2130006921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9e4106550 0x7fe9e410e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:59.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.658+0000 7fe9e8a68700 1 --2- 172.21.15.76:0/2130006921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9e4101c30 0x7fe9e4102010 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:59.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.658+0000 7fe9e8a68700 1 --2- 172.21.15.76:0/2130006921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9e41025e0 0x7fe9e4105e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:59.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.658+0000 7fe9e8a68700 1 -- 172.21.15.76:0/2130006921 >> 172.21.15.76:0/2130006921 conn(0x7fe9e40777d0 msgr2=0x7fe9e40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:59.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.658+0000 7fe9e8a68700 1 -- 172.21.15.76:0/2130006921 shutdown_connections 2024-03-18T14:51:59.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.658+0000 7fe9e8a68700 1 -- 172.21.15.76:0/2130006921 wait complete. 2024-03-18T14:51:59.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e8a68700 1 Processor -- start 2024-03-18T14:51:59.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e8a68700 1 -- start start 2024-03-18T14:51:59.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e8a68700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9e4101c30 0x7fe9e4197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:59.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e8a68700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9e41025e0 0x7fe9e4197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:59.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e8a68700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9e4106550 0x7fe9e41981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:59.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e8a68700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e4111550 con 0x7fe9e41025e0 2024-03-18T14:51:59.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e8a68700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e41113d0 con 0x7fe9e4106550 2024-03-18T14:51:59.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e8a68700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e41116d0 con 0x7fe9e4101c30 2024-03-18T14:51:59.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e1d9b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9e41025e0 0x7fe9e4197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:59.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e1d9b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9e41025e0 0x7fe9e4197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34354/0 (socket says 172.21.15.76:34354) 2024-03-18T14:51:59.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e1d9b700 1 -- 172.21.15.76:0/1139494404 learned_addr learned my addr 172.21.15.76:0/1139494404 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:51:59.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e259c700 1 --2- 172.21.15.76:0/1139494404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9e4101c30 0x7fe9e4197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:59.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e2d9d700 1 --2- 172.21.15.76:0/1139494404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9e4106550 0x7fe9e41981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:59.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e1d9b700 1 -- 172.21.15.76:0/1139494404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9e4101c30 msgr2=0x7fe9e4197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:59.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e1d9b700 1 --2- 172.21.15.76:0/1139494404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9e4101c30 0x7fe9e4197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:59.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e1d9b700 1 -- 172.21.15.76:0/1139494404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9e4106550 msgr2=0x7fe9e41981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:59.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e1d9b700 1 --2- 172.21.15.76:0/1139494404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9e4106550 0x7fe9e41981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:59.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e1d9b700 1 -- 172.21.15.76:0/1139494404 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e41a6fd0 con 0x7fe9e41025e0 2024-03-18T14:51:59.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.659+0000 7fe9e1d9b700 1 --2- 172.21.15.76:0/1139494404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9e41025e0 0x7fe9e4197ca0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc00e9f0 tx=0x7fe9cc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:59.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.660+0000 7fe9db7fe700 1 -- 172.21.15.76:0/1139494404 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc004a30 con 0x7fe9e41025e0 2024-03-18T14:51:59.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.660+0000 7fe9db7fe700 1 -- 172.21.15.76:0/1139494404 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9cc00cca0 con 0x7fe9e41025e0 2024-03-18T14:51:59.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.660+0000 7fe9e8a68700 1 -- 172.21.15.76:0/1139494404 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9e41a72c0 con 0x7fe9e41025e0 2024-03-18T14:51:59.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.660+0000 7fe9db7fe700 1 -- 172.21.15.76:0/1139494404 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc0056c0 con 0x7fe9e41025e0 2024-03-18T14:51:59.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.660+0000 7fe9e8a68700 1 -- 172.21.15.76:0/1139494404 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9e41a77b0 con 0x7fe9e41025e0 2024-03-18T14:51:59.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.661+0000 7fe9e8a68700 1 -- 172.21.15.76:0/1139494404 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9e4068f90 con 0x7fe9e41025e0 2024-03-18T14:51:59.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.662+0000 7fe9db7fe700 1 -- 172.21.15.76:0/1139494404 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe9cc00c7c0 con 0x7fe9e41025e0 2024-03-18T14:51:59.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.663+0000 7fe9db7fe700 1 --2- 172.21.15.76:0/1139494404 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe9d00a2920 0x7fe9d00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:51:59.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.663+0000 7fe9e259c700 1 --2- 172.21.15.76:0/1139494404 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe9d00a2920 0x7fe9d00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:51:59.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.663+0000 7fe9db7fe700 1 -- 172.21.15.76:0/1139494404 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fe9cc0cef40 con 0x7fe9e41025e0 2024-03-18T14:51:59.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.663+0000 7fe9e259c700 1 --2- 172.21.15.76:0/1139494404 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe9d00a2920 0x7fe9d00a4de0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4000c00 tx=0x7fe9d4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:51:59.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.666+0000 7fe9db7fe700 1 -- 172.21.15.76:0/1139494404 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe9cc053740 con 0x7fe9e41025e0 2024-03-18T14:51:59.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:51:59 smithi174 ceph-mon[36787]: pgmap v312: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T14:51:59.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:51:59 smithi077 ceph-mon[35865]: pgmap v312: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T14:51:59.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.869+0000 7fe9e8a68700 1 -- 172.21.15.76:0/1139494404 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fe9e402db70 con 0x7fe9e41025e0 2024-03-18T14:51:59.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.869+0000 7fe9db7fe700 1 -- 172.21.15.76:0/1139494404 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe9cc09bf30 con 0x7fe9e41025e0 2024-03-18T14:51:59.871 INFO:teuthology.orchestra.run.smithi076.stdout:257698037799 2024-03-18T14:51:59.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.872+0000 7fe9e8a68700 1 -- 172.21.15.76:0/1139494404 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe9d00a2920 msgr2=0x7fe9d00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:59.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.872+0000 7fe9e8a68700 1 --2- 172.21.15.76:0/1139494404 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe9d00a2920 0x7fe9d00a4de0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4000c00 tx=0x7fe9d4005fb0 comp rx=0 tx=0).stop 2024-03-18T14:51:59.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.872+0000 7fe9e8a68700 1 -- 172.21.15.76:0/1139494404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9e41025e0 msgr2=0x7fe9e4197ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:51:59.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.872+0000 7fe9e8a68700 1 --2- 172.21.15.76:0/1139494404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9e41025e0 0x7fe9e4197ca0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc00e9f0 tx=0x7fe9cc00ed00 comp rx=0 tx=0).stop 2024-03-18T14:51:59.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.872+0000 7fe9e8a68700 1 -- 172.21.15.76:0/1139494404 shutdown_connections 2024-03-18T14:51:59.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.872+0000 7fe9e8a68700 1 --2- 172.21.15.76:0/1139494404 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe9d00a2920 0x7fe9d00a4de0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:59.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.872+0000 7fe9e8a68700 1 --2- 172.21.15.76:0/1139494404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9e4106550 0x7fe9e41981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:59.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.872+0000 7fe9e8a68700 1 --2- 172.21.15.76:0/1139494404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9e4101c30 0x7fe9e4197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:59.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.872+0000 7fe9e8a68700 1 --2- 172.21.15.76:0/1139494404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9e41025e0 0x7fe9e4197ca0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:51:59.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.872+0000 7fe9e8a68700 1 -- 172.21.15.76:0/1139494404 >> 172.21.15.76:0/1139494404 conn(0x7fe9e40777d0 msgr2=0x7fe9e40fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:51:59.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.872+0000 7fe9e8a68700 1 -- 172.21.15.76:0/1139494404 shutdown_connections 2024-03-18T14:51:59.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:51:59.872+0000 7fe9e8a68700 1 -- 172.21.15.76:0/1139494404 wait complete. 2024-03-18T14:52:00.577 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:00 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1139494404' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T14:52:00.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:00 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1139494404' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T14:52:00.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:00 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1139494404' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T14:52:01.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:01 smithi076 ceph-mon[32205]: pgmap v313: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:01.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:01 smithi174 ceph-mon[36787]: pgmap v313: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:01.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:01 smithi077 ceph-mon[35865]: pgmap v313: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:01.920 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:52:02.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.571+0000 7f9100789700 1 -- 172.21.15.76:0/1018976301 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90f8107660 msgr2=0x7f90f810f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:02.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.571+0000 7f9100789700 1 --2- 172.21.15.76:0/1018976301 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90f8107660 0x7f90f810f1e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f90f000b0d0 tx=0x7f90f000b3e0 comp rx=0 tx=0).stop 2024-03-18T14:52:02.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.571+0000 7f9100789700 1 -- 172.21.15.76:0/1018976301 shutdown_connections 2024-03-18T14:52:02.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.571+0000 7f9100789700 1 --2- 172.21.15.76:0/1018976301 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90f8107660 0x7f90f810f1e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:02.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.571+0000 7f9100789700 1 --2- 172.21.15.76:0/1018976301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90f81025b0 0x7f90f8102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:02.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.571+0000 7f9100789700 1 --2- 172.21.15.76:0/1018976301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f8102f60 0x7f90f8106f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:02.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.571+0000 7f9100789700 1 -- 172.21.15.76:0/1018976301 >> 172.21.15.76:0/1018976301 conn(0x7f90f80793f0 msgr2=0x7f90f8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:02.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.572+0000 7f9100789700 1 -- 172.21.15.76:0/1018976301 shutdown_connections 2024-03-18T14:52:02.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.572+0000 7f9100789700 1 -- 172.21.15.76:0/1018976301 wait complete. 2024-03-18T14:52:02.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.572+0000 7f9100789700 1 Processor -- start 2024-03-18T14:52:02.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.572+0000 7f9100789700 1 -- start start 2024-03-18T14:52:02.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.572+0000 7f9100789700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90f81025b0 0x7f90f819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:02.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.572+0000 7f9100789700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90f8102f60 0x7f90f819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.572+0000 7f9100789700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f8107660 0x7f90f81987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.572+0000 7f9100789700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f90f8112600 con 0x7f90f8102f60 2024-03-18T14:52:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.572+0000 7f9100789700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f90f8112480 con 0x7f90f8107660 2024-03-18T14:52:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.572+0000 7f9100789700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f90f8112780 con 0x7f90f81025b0 2024-03-18T14:52:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.572+0000 7f90fdd24700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90f8102f60 0x7f90f819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.573+0000 7f90fe525700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90f81025b0 0x7f90f819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.573+0000 7f90fdd24700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90f8102f60 0x7f90f819eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34380/0 (socket says 172.21.15.76:34380) 2024-03-18T14:52:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.573+0000 7f90fdd24700 1 -- 172.21.15.76:0/866163856 learned_addr learned my addr 172.21.15.76:0/866163856 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.573+0000 7f90fed26700 1 --2- 172.21.15.76:0/866163856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f8107660 0x7f90f81987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.573+0000 7f90fdd24700 1 -- 172.21.15.76:0/866163856 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90f81025b0 msgr2=0x7f90f819e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.573+0000 7f90fdd24700 1 --2- 172.21.15.76:0/866163856 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90f81025b0 0x7f90f819e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.573+0000 7f90fdd24700 1 -- 172.21.15.76:0/866163856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f8107660 msgr2=0x7f90f81987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.573+0000 7f90fdd24700 1 --2- 172.21.15.76:0/866163856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f8107660 0x7f90f81987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.573+0000 7f90fdd24700 1 -- 172.21.15.76:0/866163856 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f8198fc0 con 0x7f90f8102f60 2024-03-18T14:52:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.573+0000 7f90fed26700 1 --2- 172.21.15.76:0/866163856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f8107660 0x7f90f81987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.573+0000 7f90fdd24700 1 --2- 172.21.15.76:0/866163856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90f8102f60 0x7f90f819eba0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f90e800e940 tx=0x7f90e800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.573+0000 7f90ef7fe700 1 -- 172.21.15.76:0/866163856 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e800c9e0 con 0x7f90f8102f60 2024-03-18T14:52:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.573+0000 7f90ef7fe700 1 -- 172.21.15.76:0/866163856 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90e800cb80 con 0x7f90f8102f60 2024-03-18T14:52:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.573+0000 7f9100789700 1 -- 172.21.15.76:0/866163856 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90f81992b0 con 0x7f90f8102f60 2024-03-18T14:52:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.573+0000 7f90ef7fe700 1 -- 172.21.15.76:0/866163856 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e8018660 con 0x7f90f8102f60 2024-03-18T14:52:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.573+0000 7f9100789700 1 -- 172.21.15.76:0/866163856 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90f81a54c0 con 0x7f90f8102f60 2024-03-18T14:52:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.574+0000 7f9100789700 1 -- 172.21.15.76:0/866163856 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90f8103a10 con 0x7f90f8102f60 2024-03-18T14:52:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.575+0000 7f90ef7fe700 1 -- 172.21.15.76:0/866163856 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f90e8010430 con 0x7f90f8102f60 2024-03-18T14:52:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.576+0000 7f90ef7fe700 1 --2- 172.21.15.76:0/866163856 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f90e40a2920 0x7f90e40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.576+0000 7f90fe525700 1 --2- 172.21.15.76:0/866163856 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f90e40a2920 0x7f90e40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.576+0000 7f90ef7fe700 1 -- 172.21.15.76:0/866163856 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f90e80ceda0 con 0x7f90f8102f60 2024-03-18T14:52:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.577+0000 7f90fe525700 1 --2- 172.21.15.76:0/866163856 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f90e40a2920 0x7f90e40a4de0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f90f4005950 tx=0x7f90f400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.580+0000 7f90ef7fe700 1 -- 172.21.15.76:0/866163856 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f90e809be10 con 0x7f90f8102f60 2024-03-18T14:52:02.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.790+0000 7f9100789700 1 -- 172.21.15.76:0/866163856 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f90f802db70 con 0x7f90f8102f60 2024-03-18T14:52:02.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.791+0000 7f90ef7fe700 1 -- 172.21.15.76:0/866163856 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f90e809b710 con 0x7f90f8102f60 2024-03-18T14:52:02.792 INFO:teuthology.orchestra.run.smithi076.stdout:77309411399 2024-03-18T14:52:02.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.793+0000 7f9100789700 1 -- 172.21.15.76:0/866163856 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f90e40a2920 msgr2=0x7f90e40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:02.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.793+0000 7f9100789700 1 --2- 172.21.15.76:0/866163856 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f90e40a2920 0x7f90e40a4de0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f90f4005950 tx=0x7f90f400b410 comp rx=0 tx=0).stop 2024-03-18T14:52:02.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.793+0000 7f9100789700 1 -- 172.21.15.76:0/866163856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90f8102f60 msgr2=0x7f90f819eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:02.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.793+0000 7f9100789700 1 --2- 172.21.15.76:0/866163856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90f8102f60 0x7f90f819eba0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f90e800e940 tx=0x7f90e800ec50 comp rx=0 tx=0).stop 2024-03-18T14:52:02.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.793+0000 7f9100789700 1 -- 172.21.15.76:0/866163856 shutdown_connections 2024-03-18T14:52:02.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.793+0000 7f9100789700 1 --2- 172.21.15.76:0/866163856 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f90e40a2920 0x7f90e40a4de0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:02.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.793+0000 7f9100789700 1 --2- 172.21.15.76:0/866163856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f8107660 0x7f90f81987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:02.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.793+0000 7f9100789700 1 --2- 172.21.15.76:0/866163856 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90f81025b0 0x7f90f819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:02.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.793+0000 7f9100789700 1 --2- 172.21.15.76:0/866163856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90f8102f60 0x7f90f819eba0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:02.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.793+0000 7f9100789700 1 -- 172.21.15.76:0/866163856 >> 172.21.15.76:0/866163856 conn(0x7f90f80793f0 msgr2=0x7f90f81003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:02.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.793+0000 7f9100789700 1 -- 172.21.15.76:0/866163856 shutdown_connections 2024-03-18T14:52:02.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:02.793+0000 7f9100789700 1 -- 172.21.15.76:0/866163856 wait complete. 2024-03-18T14:52:03.574 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411394 got 77309411399 for osd.2 2024-03-18T14:52:03.575 DEBUG:teuthology.parallel:result is None 2024-03-18T14:52:03.701 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:03 smithi076 ceph-mon[32205]: pgmap v314: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-18T14:52:03.701 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:03 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:52:03.701 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:03 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/866163856' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T14:52:03.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:03 smithi174 ceph-mon[36787]: pgmap v314: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-18T14:52:03.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:03 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:52:03.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:03 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/866163856' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T14:52:03.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:03 smithi077 ceph-mon[35865]: pgmap v314: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-18T14:52:03.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:03 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:52:03.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:03 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/866163856' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T14:52:04.593 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037794 got 257698037799 for osd.8 2024-03-18T14:52:04.594 DEBUG:teuthology.parallel:result is None 2024-03-18T14:52:05.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.093+0000 7f14a6231700 1 -- 172.21.15.76:0/1924975485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14a0102510 msgr2=0x7f14a01028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:05.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.093+0000 7f14a6231700 1 --2- 172.21.15.76:0/1924975485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14a0102510 0x7f14a01028f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f14900099c0 tx=0x7f1490009cd0 comp rx=0 tx=0).stop 2024-03-18T14:52:05.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.093+0000 7f14a6231700 1 -- 172.21.15.76:0/1924975485 shutdown_connections 2024-03-18T14:52:05.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.093+0000 7f14a6231700 1 --2- 172.21.15.76:0/1924975485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14a01075f0 0x7f14a010f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:05.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.093+0000 7f14a6231700 1 --2- 172.21.15.76:0/1924975485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14a0102510 0x7f14a01028f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:05.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.093+0000 7f14a6231700 1 --2- 172.21.15.76:0/1924975485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14a0102ec0 0x7f14a0106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:05.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.093+0000 7f14a6231700 1 -- 172.21.15.76:0/1924975485 >> 172.21.15.76:0/1924975485 conn(0x7f14a00fdd80 msgr2=0x7f14a01001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:05.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.093+0000 7f14a6231700 1 -- 172.21.15.76:0/1924975485 shutdown_connections 2024-03-18T14:52:05.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.093+0000 7f14a6231700 1 -- 172.21.15.76:0/1924975485 wait complete. 2024-03-18T14:52:05.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.094+0000 7f14a6231700 1 Processor -- start 2024-03-18T14:52:05.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.094+0000 7f14a6231700 1 -- start start 2024-03-18T14:52:05.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.094+0000 7f14a6231700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14a0102510 0x7f14a019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:05.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.094+0000 7f14a6231700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14a0102ec0 0x7f14a019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:05.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.094+0000 7f14a6231700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14a01075f0 0x7f14a0198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:05.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.094+0000 7f14a6231700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f14a01125d0 con 0x7f14a0102ec0 2024-03-18T14:52:05.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.094+0000 7f14a6231700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f14a0112450 con 0x7f14a0102510 2024-03-18T14:52:05.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.094+0000 7f14a6231700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f14a0112750 con 0x7f14a01075f0 2024-03-18T14:52:05.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.094+0000 7f1496ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14a0102ec0 0x7f14a019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:05.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.094+0000 7f1496ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14a0102ec0 0x7f14a019ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34398/0 (socket says 172.21.15.76:34398) 2024-03-18T14:52:05.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.094+0000 7f1496ffd700 1 -- 172.21.15.76:0/2779164647 learned_addr learned my addr 172.21.15.76:0/2779164647 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:05.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.094+0000 7f149f7fe700 1 --2- 172.21.15.76:0/2779164647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14a0102510 0x7f14a019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:05.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.094+0000 7f149ffff700 1 --2- 172.21.15.76:0/2779164647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14a01075f0 0x7f14a0198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:05.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.095+0000 7f1496ffd700 1 -- 172.21.15.76:0/2779164647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14a01075f0 msgr2=0x7f14a0198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:05.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.095+0000 7f1496ffd700 1 --2- 172.21.15.76:0/2779164647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14a01075f0 0x7f14a0198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:05.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.095+0000 7f1496ffd700 1 -- 172.21.15.76:0/2779164647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14a0102510 msgr2=0x7f14a019e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:05.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.095+0000 7f1496ffd700 1 --2- 172.21.15.76:0/2779164647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14a0102510 0x7f14a019e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:05.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.095+0000 7f1496ffd700 1 -- 172.21.15.76:0/2779164647 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14a0198fd0 con 0x7f14a0102ec0 2024-03-18T14:52:05.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.095+0000 7f149f7fe700 1 --2- 172.21.15.76:0/2779164647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14a0102510 0x7f14a019e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:05.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.095+0000 7f1496ffd700 1 --2- 172.21.15.76:0/2779164647 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14a0102ec0 0x7f14a019ebf0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f148800d950 tx=0x7f148800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:05.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.095+0000 7f149d7fa700 1 -- 172.21.15.76:0/2779164647 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488016070 con 0x7f14a0102ec0 2024-03-18T14:52:05.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.095+0000 7f14a6231700 1 -- 172.21.15.76:0/2779164647 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14a01992c0 con 0x7f14a0102ec0 2024-03-18T14:52:05.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.095+0000 7f149d7fa700 1 -- 172.21.15.76:0/2779164647 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1488004d10 con 0x7f14a0102ec0 2024-03-18T14:52:05.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.095+0000 7f149d7fa700 1 -- 172.21.15.76:0/2779164647 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488005650 con 0x7f14a0102ec0 2024-03-18T14:52:05.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.095+0000 7f14a6231700 1 -- 172.21.15.76:0/2779164647 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14a0074500 con 0x7f14a0102ec0 2024-03-18T14:52:05.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.096+0000 7f14a6231700 1 -- 172.21.15.76:0/2779164647 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14a0068f90 con 0x7f14a0102ec0 2024-03-18T14:52:05.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.097+0000 7f149d7fa700 1 -- 172.21.15.76:0/2779164647 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f1488004750 con 0x7f14a0102ec0 2024-03-18T14:52:05.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.098+0000 7f149d7fa700 1 --2- 172.21.15.76:0/2779164647 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f148c0a2920 0x7f148c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:05.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.098+0000 7f149f7fe700 1 --2- 172.21.15.76:0/2779164647 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f148c0a2920 0x7f148c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:05.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.098+0000 7f149d7fa700 1 -- 172.21.15.76:0/2779164647 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f14880cec00 con 0x7f14a0102ec0 2024-03-18T14:52:05.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.098+0000 7f149f7fe700 1 --2- 172.21.15.76:0/2779164647 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f148c0a2920 0x7f148c0a4de0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1490000c00 tx=0x7f1490005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:05.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.102+0000 7f149d7fa700 1 -- 172.21.15.76:0/2779164647 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f148801f020 con 0x7f14a0102ec0 2024-03-18T14:52:05.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.303+0000 7f14a6231700 1 -- 172.21.15.76:0/2779164647 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f14a0199da0 con 0x7f14a0102ec0 2024-03-18T14:52:05.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.304+0000 7f149d7fa700 1 -- 172.21.15.76:0/2779164647 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f148809bcf0 con 0x7f14a0102ec0 2024-03-18T14:52:05.305 INFO:teuthology.orchestra.run.smithi076.stdout:167503724599 2024-03-18T14:52:05.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.306+0000 7f14a6231700 1 -- 172.21.15.76:0/2779164647 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f148c0a2920 msgr2=0x7f148c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:05.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.306+0000 7f14a6231700 1 --2- 172.21.15.76:0/2779164647 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f148c0a2920 0x7f148c0a4de0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1490000c00 tx=0x7f1490005fb0 comp rx=0 tx=0).stop 2024-03-18T14:52:05.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.306+0000 7f14a6231700 1 -- 172.21.15.76:0/2779164647 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14a0102ec0 msgr2=0x7f14a019ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:05.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.306+0000 7f14a6231700 1 --2- 172.21.15.76:0/2779164647 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14a0102ec0 0x7f14a019ebf0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f148800d950 tx=0x7f148800dc60 comp rx=0 tx=0).stop 2024-03-18T14:52:05.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.306+0000 7f14a6231700 1 -- 172.21.15.76:0/2779164647 shutdown_connections 2024-03-18T14:52:05.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.306+0000 7f14a6231700 1 --2- 172.21.15.76:0/2779164647 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f148c0a2920 0x7f148c0a4de0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:05.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.306+0000 7f14a6231700 1 --2- 172.21.15.76:0/2779164647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14a01075f0 0x7f14a0198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:05.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.306+0000 7f14a6231700 1 --2- 172.21.15.76:0/2779164647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14a0102510 0x7f14a019e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:05.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.306+0000 7f14a6231700 1 --2- 172.21.15.76:0/2779164647 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14a0102ec0 0x7f14a019ebf0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:05.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.306+0000 7f14a6231700 1 -- 172.21.15.76:0/2779164647 >> 172.21.15.76:0/2779164647 conn(0x7f14a00fdd80 msgr2=0x7f14a00ffaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:05.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.306+0000 7f14a6231700 1 -- 172.21.15.76:0/2779164647 shutdown_connections 2024-03-18T14:52:05.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.306+0000 7f14a6231700 1 -- 172.21.15.76:0/2779164647 wait complete. 2024-03-18T14:52:05.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:05 smithi076 ceph-mon[32205]: pgmap v315: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:05.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:05 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:05.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:05 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:05.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:05 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:05.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:05 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:05.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:05 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2779164647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T14:52:05.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:05 smithi174 ceph-mon[36787]: pgmap v315: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:05.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:05 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:05.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:05 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:05.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:05 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:05.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:05 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:05.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:05 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2779164647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T14:52:05.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:05 smithi077 ceph-mon[35865]: pgmap v315: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:05.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:05 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:05.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:05 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:05.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:05 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:05.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:05 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:05.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:05 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2779164647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T14:52:05.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.794+0000 7efe39a39700 1 -- 172.21.15.76:0/737946244 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe34107210 msgr2=0x7efe341075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:05.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.794+0000 7efe39a39700 1 --2- 172.21.15.76:0/737946244 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe34107210 0x7efe341075f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7efe24009990 tx=0x7efe24009ca0 comp rx=0 tx=0).stop 2024-03-18T14:52:05.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.794+0000 7efe39a39700 1 -- 172.21.15.76:0/737946244 shutdown_connections 2024-03-18T14:52:05.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.794+0000 7efe39a39700 1 --2- 172.21.15.76:0/737946244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe34101b70 0x7efe3410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:05.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.794+0000 7efe39a39700 1 --2- 172.21.15.76:0/737946244 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe34107210 0x7efe341075f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:05.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.794+0000 7efe39a39700 1 --2- 172.21.15.76:0/737946244 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe341011d0 0x7efe34101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:05.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.794+0000 7efe39a39700 1 -- 172.21.15.76:0/737946244 >> 172.21.15.76:0/737946244 conn(0x7efe34078100 msgr2=0x7efe34078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:05.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.794+0000 7efe39a39700 1 -- 172.21.15.76:0/737946244 shutdown_connections 2024-03-18T14:52:05.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.794+0000 7efe39a39700 1 -- 172.21.15.76:0/737946244 wait complete. 2024-03-18T14:52:05.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.794+0000 7efe39a39700 1 Processor -- start 2024-03-18T14:52:05.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe39a39700 1 -- start start 2024-03-18T14:52:05.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe39a39700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe341011d0 0x7efe34102e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:05.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe39a39700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe34101b70 0x7efe341033c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:05.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe39a39700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe34107210 0x7efe34103900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:05.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe39a39700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efe341114c0 con 0x7efe341011d0 2024-03-18T14:52:05.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe39a39700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efe34111340 con 0x7efe34107210 2024-03-18T14:52:05.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe39a39700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efe34111640 con 0x7efe34101b70 2024-03-18T14:52:05.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe32ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe341011d0 0x7efe34102e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:05.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe327fc700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe34101b70 0x7efe341033c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:05.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe32ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe341011d0 0x7efe34102e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34414/0 (socket says 172.21.15.76:34414) 2024-03-18T14:52:05.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe32ffd700 1 -- 172.21.15.76:0/4137347745 learned_addr learned my addr 172.21.15.76:0/4137347745 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:05.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe337fe700 1 --2- 172.21.15.76:0/4137347745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe34107210 0x7efe34103900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:05.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe327fc700 1 -- 172.21.15.76:0/4137347745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe34107210 msgr2=0x7efe34103900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:05.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe327fc700 1 --2- 172.21.15.76:0/4137347745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe34107210 0x7efe34103900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:05.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe327fc700 1 -- 172.21.15.76:0/4137347745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe341011d0 msgr2=0x7efe34102e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:05.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe327fc700 1 --2- 172.21.15.76:0/4137347745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe341011d0 0x7efe34102e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:05.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe327fc700 1 -- 172.21.15.76:0/4137347745 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe341a7020 con 0x7efe34101b70 2024-03-18T14:52:05.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.795+0000 7efe32ffd700 1 --2- 172.21.15.76:0/4137347745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe341011d0 0x7efe34102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:05.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.796+0000 7efe327fc700 1 --2- 172.21.15.76:0/4137347745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe34101b70 0x7efe341033c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efe2800e9f0 tx=0x7efe2800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:05.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.796+0000 7efe38a37700 1 -- 172.21.15.76:0/4137347745 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe28004220 con 0x7efe34101b70 2024-03-18T14:52:05.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.796+0000 7efe38a37700 1 -- 172.21.15.76:0/4137347745 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe28004d10 con 0x7efe34101b70 2024-03-18T14:52:05.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.796+0000 7efe39a39700 1 -- 172.21.15.76:0/4137347745 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe341a7310 con 0x7efe34101b70 2024-03-18T14:52:05.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.796+0000 7efe38a37700 1 -- 172.21.15.76:0/4137347745 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe28005020 con 0x7efe34101b70 2024-03-18T14:52:05.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.796+0000 7efe39a39700 1 -- 172.21.15.76:0/4137347745 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe341a7800 con 0x7efe34101b70 2024-03-18T14:52:05.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.797+0000 7efe39a39700 1 -- 172.21.15.76:0/4137347745 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe34068f90 con 0x7efe34101b70 2024-03-18T14:52:05.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.798+0000 7efe38a37700 1 -- 172.21.15.76:0/4137347745 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7efe28004750 con 0x7efe34101b70 2024-03-18T14:52:05.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.798+0000 7efe38a37700 1 --2- 172.21.15.76:0/4137347745 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efe1c0a2920 0x7efe1c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:05.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.798+0000 7efe32ffd700 1 --2- 172.21.15.76:0/4137347745 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efe1c0a2920 0x7efe1c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:05.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.798+0000 7efe38a37700 1 -- 172.21.15.76:0/4137347745 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7efe280ce8d0 con 0x7efe34101b70 2024-03-18T14:52:05.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.799+0000 7efe32ffd700 1 --2- 172.21.15.76:0/4137347745 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efe1c0a2920 0x7efe1c0a4de0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efe24006010 tx=0x7efe2400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:05.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:05.802+0000 7efe38a37700 1 -- 172.21.15.76:0/4137347745 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efe2809b9c0 con 0x7efe34101b70 2024-03-18T14:52:06.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.018+0000 7efe39a39700 1 -- 172.21.15.76:0/4137347745 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7efe341a7ac0 con 0x7efe34101b70 2024-03-18T14:52:06.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.019+0000 7efe38a37700 1 -- 172.21.15.76:0/4137347745 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7efe2809b2c0 con 0x7efe34101b70 2024-03-18T14:52:06.020 INFO:teuthology.orchestra.run.smithi076.stdout:201863462962 2024-03-18T14:52:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.021+0000 7efe39a39700 1 -- 172.21.15.76:0/4137347745 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efe1c0a2920 msgr2=0x7efe1c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.021+0000 7efe39a39700 1 --2- 172.21.15.76:0/4137347745 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efe1c0a2920 0x7efe1c0a4de0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efe24006010 tx=0x7efe2400b260 comp rx=0 tx=0).stop 2024-03-18T14:52:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.021+0000 7efe39a39700 1 -- 172.21.15.76:0/4137347745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe34101b70 msgr2=0x7efe341033c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.021+0000 7efe39a39700 1 --2- 172.21.15.76:0/4137347745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe34101b70 0x7efe341033c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efe2800e9f0 tx=0x7efe2800ed00 comp rx=0 tx=0).stop 2024-03-18T14:52:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.021+0000 7efe39a39700 1 -- 172.21.15.76:0/4137347745 shutdown_connections 2024-03-18T14:52:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.021+0000 7efe39a39700 1 --2- 172.21.15.76:0/4137347745 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efe1c0a2920 0x7efe1c0a4de0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.021+0000 7efe39a39700 1 --2- 172.21.15.76:0/4137347745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe34107210 0x7efe34103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.021+0000 7efe39a39700 1 --2- 172.21.15.76:0/4137347745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe341011d0 0x7efe34102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.021+0000 7efe39a39700 1 --2- 172.21.15.76:0/4137347745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe34101b70 0x7efe341033c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.021+0000 7efe39a39700 1 -- 172.21.15.76:0/4137347745 >> 172.21.15.76:0/4137347745 conn(0x7efe34078100 msgr2=0x7efe34100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.021+0000 7efe39a39700 1 -- 172.21.15.76:0/4137347745 shutdown_connections 2024-03-18T14:52:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.021+0000 7efe39a39700 1 -- 172.21.15.76:0/4137347745 wait complete. 2024-03-18T14:52:06.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:06 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/4137347745' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T14:52:06.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-18T14:52:06.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-18T14:52:06.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-18T14:52:06.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-03-18T14:52:06.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-18T14:52:06.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-03-18T14:52:06.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:06 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-18T14:52:06.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:06 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/4137347745' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T14:52:06.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-18T14:52:06.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-18T14:52:06.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-18T14:52:06.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-03-18T14:52:06.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-18T14:52:06.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-03-18T14:52:06.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:06 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-18T14:52:06.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:06 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/4137347745' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T14:52:06.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-18T14:52:06.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-18T14:52:06.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-18T14:52:06.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-03-18T14:52:06.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-18T14:52:06.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-03-18T14:52:06.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:06 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-18T14:52:06.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.786+0000 7fb016481700 1 -- 172.21.15.76:0/2877184249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb010069f80 msgr2=0x7fb01006a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:06.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.786+0000 7fb016481700 1 --2- 172.21.15.76:0/2877184249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb010069f80 0x7fb01006a3e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb0040099c0 tx=0x7fb004009cd0 comp rx=0 tx=0).stop 2024-03-18T14:52:06.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.786+0000 7fb016481700 1 -- 172.21.15.76:0/2877184249 shutdown_connections 2024-03-18T14:52:06.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.786+0000 7fb016481700 1 --2- 172.21.15.76:0/2877184249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb01006a920 0x7fb01010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:06.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.786+0000 7fb016481700 1 --2- 172.21.15.76:0/2877184249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb010108320 0x7fb010108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:06.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.786+0000 7fb016481700 1 --2- 172.21.15.76:0/2877184249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb010069f80 0x7fb01006a3e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:06.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.786+0000 7fb016481700 1 -- 172.21.15.76:0/2877184249 >> 172.21.15.76:0/2877184249 conn(0x7fb010078100 msgr2=0x7fb010078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:06.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.786+0000 7fb016481700 1 -- 172.21.15.76:0/2877184249 shutdown_connections 2024-03-18T14:52:06.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.786+0000 7fb016481700 1 -- 172.21.15.76:0/2877184249 wait complete. 2024-03-18T14:52:06.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.787+0000 7fb016481700 1 Processor -- start 2024-03-18T14:52:06.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.787+0000 7fb016481700 1 -- start start 2024-03-18T14:52:06.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.787+0000 7fb016481700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb010069f80 0x7fb01019e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:06.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.787+0000 7fb016481700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb01006a920 0x7fb01019ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:06.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.787+0000 7fb016481700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb010108320 0x7fb0101988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:06.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.787+0000 7fb016481700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb010112580 con 0x7fb010069f80 2024-03-18T14:52:06.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.787+0000 7fb016481700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb010112400 con 0x7fb010108320 2024-03-18T14:52:06.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.787+0000 7fb016481700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb010112700 con 0x7fb01006a920 2024-03-18T14:52:06.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb00ffff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb010069f80 0x7fb01019e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:06.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb00f7fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb01006a920 0x7fb01019ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:06.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb00ffff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb010069f80 0x7fb01019e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34420/0 (socket says 172.21.15.76:34420) 2024-03-18T14:52:06.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb014a1e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb010108320 0x7fb0101988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:06.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb00f7fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb01006a920 0x7fb01019ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:46114/0 (socket says 172.21.15.76:46114) 2024-03-18T14:52:06.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb00ffff700 1 -- 172.21.15.76:0/886601170 learned_addr learned my addr 172.21.15.76:0/886601170 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:06.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb00f7fe700 1 -- 172.21.15.76:0/886601170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb010108320 msgr2=0x7fb0101988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:06.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb00f7fe700 1 --2- 172.21.15.76:0/886601170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb010108320 0x7fb0101988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:06.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb00f7fe700 1 -- 172.21.15.76:0/886601170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb010069f80 msgr2=0x7fb01019e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:06.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb00f7fe700 1 --2- 172.21.15.76:0/886601170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb010069f80 0x7fb01019e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:06.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb00f7fe700 1 -- 172.21.15.76:0/886601170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb010199040 con 0x7fb01006a920 2024-03-18T14:52:06.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb014a1e700 1 --2- 172.21.15.76:0/886601170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb010108320 0x7fb0101988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:06.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb00ffff700 1 --2- 172.21.15.76:0/886601170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb010069f80 0x7fb01019e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:06.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb00f7fe700 1 --2- 172.21.15.76:0/886601170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb01006a920 0x7fb01019ec60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb0040098e0 tx=0x7fb004005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:06.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb00d7fa700 1 -- 172.21.15.76:0/886601170 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb004027070 con 0x7fb01006a920 2024-03-18T14:52:06.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb00d7fa700 1 -- 172.21.15.76:0/886601170 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb004003680 con 0x7fb01006a920 2024-03-18T14:52:06.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb016481700 1 -- 172.21.15.76:0/886601170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0101992d0 con 0x7fb01006a920 2024-03-18T14:52:06.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb00d7fa700 1 -- 172.21.15.76:0/886601170 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb004022e10 con 0x7fb01006a920 2024-03-18T14:52:06.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.788+0000 7fb016481700 1 -- 172.21.15.76:0/886601170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0101a55f0 con 0x7fb01006a920 2024-03-18T14:52:06.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.789+0000 7fb016481700 1 -- 172.21.15.76:0/886601170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0101039d0 con 0x7fb01006a920 2024-03-18T14:52:06.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.790+0000 7fb00d7fa700 1 -- 172.21.15.76:0/886601170 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fb00402a070 con 0x7fb01006a920 2024-03-18T14:52:06.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.791+0000 7fb00d7fa700 1 --2- 172.21.15.76:0/886601170 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7faff80a2920 0x7faff80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:06.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.791+0000 7fb00ffff700 1 --2- 172.21.15.76:0/886601170 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7faff80a2920 0x7faff80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:06.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.791+0000 7fb00d7fa700 1 -- 172.21.15.76:0/886601170 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fb0040cf590 con 0x7fb01006a920 2024-03-18T14:52:06.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.791+0000 7fb00ffff700 1 --2- 172.21.15.76:0/886601170 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7faff80a2920 0x7faff80a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb010199a20 tx=0x7fb000009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:06.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.794+0000 7fb00d7fa700 1 -- 172.21.15.76:0/886601170 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb004013070 con 0x7fb01006a920 2024-03-18T14:52:06.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.994+0000 7fb016481700 1 -- 172.21.15.76:0/886601170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fb01002db70 con 0x7fb01006a920 2024-03-18T14:52:06.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.995+0000 7fb00d7fa700 1 -- 172.21.15.76:0/886601170 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb00409c530 con 0x7fb01006a920 2024-03-18T14:52:06.996 INFO:teuthology.orchestra.run.smithi076.stdout:34359738449 2024-03-18T14:52:06.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.997+0000 7fb016481700 1 -- 172.21.15.76:0/886601170 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7faff80a2920 msgr2=0x7faff80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:06.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.997+0000 7fb016481700 1 --2- 172.21.15.76:0/886601170 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7faff80a2920 0x7faff80a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb010199a20 tx=0x7fb000009260 comp rx=0 tx=0).stop 2024-03-18T14:52:06.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.997+0000 7fb016481700 1 -- 172.21.15.76:0/886601170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb01006a920 msgr2=0x7fb01019ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:06.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.997+0000 7fb016481700 1 --2- 172.21.15.76:0/886601170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb01006a920 0x7fb01019ec60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb0040098e0 tx=0x7fb004005f10 comp rx=0 tx=0).stop 2024-03-18T14:52:06.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.997+0000 7fb016481700 1 -- 172.21.15.76:0/886601170 shutdown_connections 2024-03-18T14:52:06.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.997+0000 7fb016481700 1 --2- 172.21.15.76:0/886601170 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7faff80a2920 0x7faff80a4de0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:06.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.997+0000 7fb016481700 1 --2- 172.21.15.76:0/886601170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb010108320 0x7fb0101988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:06.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.997+0000 7fb016481700 1 --2- 172.21.15.76:0/886601170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb010069f80 0x7fb01019e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:06.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.997+0000 7fb016481700 1 --2- 172.21.15.76:0/886601170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb01006a920 0x7fb01019ec60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:06.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.997+0000 7fb016481700 1 -- 172.21.15.76:0/886601170 >> 172.21.15.76:0/886601170 conn(0x7fb010078100 msgr2=0x7fb0100ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:06.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.997+0000 7fb016481700 1 -- 172.21.15.76:0/886601170 shutdown_connections 2024-03-18T14:52:06.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:06.997+0000 7fb016481700 1 -- 172.21.15.76:0/886601170 wait complete. 2024-03-18T14:52:07.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.024+0000 7f0b3ac5a700 1 -- 172.21.15.76:0/3780406500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b34107680 msgr2=0x7f0b3410f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:07.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.024+0000 7f0b3ac5a700 1 --2- 172.21.15.76:0/3780406500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b34107680 0x7f0b3410f250 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c009fc0 tx=0x7f0b2c00b330 comp rx=0 tx=0).stop 2024-03-18T14:52:07.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.024+0000 7f0b3ac5a700 1 -- 172.21.15.76:0/3780406500 shutdown_connections 2024-03-18T14:52:07.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.024+0000 7f0b3ac5a700 1 --2- 172.21.15.76:0/3780406500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b34107680 0x7f0b3410f250 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:07.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.024+0000 7f0b3ac5a700 1 --2- 172.21.15.76:0/3780406500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b341025a0 0x7f0b34102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:07.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.024+0000 7f0b3ac5a700 1 --2- 172.21.15.76:0/3780406500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b34102f50 0x7f0b34106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:07.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.024+0000 7f0b3ac5a700 1 -- 172.21.15.76:0/3780406500 >> 172.21.15.76:0/3780406500 conn(0x7f0b340fde10 msgr2=0x7f0b34100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:07.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.024+0000 7f0b3ac5a700 1 -- 172.21.15.76:0/3780406500 shutdown_connections 2024-03-18T14:52:07.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.024+0000 7f0b3ac5a700 1 -- 172.21.15.76:0/3780406500 wait complete. 2024-03-18T14:52:07.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b3ac5a700 1 Processor -- start 2024-03-18T14:52:07.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b3ac5a700 1 -- start start 2024-03-18T14:52:07.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b3ac5a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b341025a0 0x7f0b341a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:07.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b3ac5a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b34102f50 0x7f0b341a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:07.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b3ac5a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b34107680 0x7f0b3419aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:07.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b3ac5a700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0b34112670 con 0x7f0b34107680 2024-03-18T14:52:07.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b3ac5a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0b341124f0 con 0x7f0b341025a0 2024-03-18T14:52:07.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b3ac5a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0b341127f0 con 0x7f0b34102f50 2024-03-18T14:52:07.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b391f7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b34107680 0x7f0b3419aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:07.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b391f7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b34107680 0x7f0b3419aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34434/0 (socket says 172.21.15.76:34434) 2024-03-18T14:52:07.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b391f7700 1 -- 172.21.15.76:0/4094823658 learned_addr learned my addr 172.21.15.76:0/4094823658 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:07.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b389f6700 1 --2- 172.21.15.76:0/4094823658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b341025a0 0x7f0b341a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:07.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b33fff700 1 --2- 172.21.15.76:0/4094823658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b34102f50 0x7f0b341a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:07.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b391f7700 1 -- 172.21.15.76:0/4094823658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b34102f50 msgr2=0x7f0b341a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:07.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b391f7700 1 --2- 172.21.15.76:0/4094823658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b34102f50 0x7f0b341a0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:07.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b391f7700 1 -- 172.21.15.76:0/4094823658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b341025a0 msgr2=0x7f0b341a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:07.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b391f7700 1 --2- 172.21.15.76:0/4094823658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b341025a0 0x7f0b341a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:07.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b391f7700 1 -- 172.21.15.76:0/4094823658 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b3419b230 con 0x7f0b34107680 2024-03-18T14:52:07.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b33fff700 1 --2- 172.21.15.76:0/4094823658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b34102f50 0x7f0b341a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:52:07.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b389f6700 1 --2- 172.21.15.76:0/4094823658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b341025a0 0x7f0b341a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:07.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.025+0000 7f0b391f7700 1 --2- 172.21.15.76:0/4094823658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b34107680 0x7f0b3419aad0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c009780 tx=0x7f0b2c004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:07.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.026+0000 7f0b31ffb700 1 -- 172.21.15.76:0/4094823658 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c021070 con 0x7f0b34107680 2024-03-18T14:52:07.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.026+0000 7f0b3ac5a700 1 -- 172.21.15.76:0/4094823658 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b3419b4c0 con 0x7f0b34107680 2024-03-18T14:52:07.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.026+0000 7f0b31ffb700 1 -- 172.21.15.76:0/4094823658 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b2c007ac0 con 0x7f0b34107680 2024-03-18T14:52:07.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.026+0000 7f0b31ffb700 1 -- 172.21.15.76:0/4094823658 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c0052a0 con 0x7f0b34107680 2024-03-18T14:52:07.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.026+0000 7f0b3ac5a700 1 -- 172.21.15.76:0/4094823658 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b341a77e0 con 0x7f0b34107680 2024-03-18T14:52:07.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.027+0000 7f0b3ac5a700 1 -- 172.21.15.76:0/4094823658 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b34068f90 con 0x7f0b34107680 2024-03-18T14:52:07.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.028+0000 7f0b31ffb700 1 -- 172.21.15.76:0/4094823658 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f0b2c024070 con 0x7f0b34107680 2024-03-18T14:52:07.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.029+0000 7f0b31ffb700 1 --2- 172.21.15.76:0/4094823658 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0b240a2920 0x7f0b240a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:07.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.029+0000 7f0b389f6700 1 --2- 172.21.15.76:0/4094823658 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0b240a2920 0x7f0b240a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:07.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.029+0000 7f0b31ffb700 1 -- 172.21.15.76:0/4094823658 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f0b2c0cfde0 con 0x7f0b34107680 2024-03-18T14:52:07.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.029+0000 7f0b389f6700 1 --2- 172.21.15.76:0/4094823658 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0b240a2920 0x7f0b240a4de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0b28009680 tx=0x7f0b28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:07.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.032+0000 7f0b31ffb700 1 -- 172.21.15.76:0/4094823658 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0b2c09cd80 con 0x7f0b34107680 2024-03-18T14:52:07.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.232+0000 7f0b3ac5a700 1 -- 172.21.15.76:0/4094823658 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f0b341a79b0 con 0x7f0b34107680 2024-03-18T14:52:07.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.233+0000 7f0b31ffb700 1 -- 172.21.15.76:0/4094823658 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f0b2c09c680 con 0x7f0b34107680 2024-03-18T14:52:07.234 INFO:teuthology.orchestra.run.smithi076.stdout:347892351002 2024-03-18T14:52:07.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.235+0000 7f0b3ac5a700 1 -- 172.21.15.76:0/4094823658 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0b240a2920 msgr2=0x7f0b240a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:07.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.235+0000 7f0b3ac5a700 1 --2- 172.21.15.76:0/4094823658 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0b240a2920 0x7f0b240a4de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0b28009680 tx=0x7f0b28009260 comp rx=0 tx=0).stop 2024-03-18T14:52:07.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.235+0000 7f0b3ac5a700 1 -- 172.21.15.76:0/4094823658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b34107680 msgr2=0x7f0b3419aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:07.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.235+0000 7f0b3ac5a700 1 --2- 172.21.15.76:0/4094823658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b34107680 0x7f0b3419aad0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c009780 tx=0x7f0b2c004f20 comp rx=0 tx=0).stop 2024-03-18T14:52:07.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.235+0000 7f0b3ac5a700 1 -- 172.21.15.76:0/4094823658 shutdown_connections 2024-03-18T14:52:07.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.235+0000 7f0b3ac5a700 1 --2- 172.21.15.76:0/4094823658 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f0b240a2920 0x7f0b240a4de0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:07.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.235+0000 7f0b3ac5a700 1 --2- 172.21.15.76:0/4094823658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b34107680 0x7f0b3419aad0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:07.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.235+0000 7f0b3ac5a700 1 --2- 172.21.15.76:0/4094823658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b341025a0 0x7f0b341a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:07.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.235+0000 7f0b3ac5a700 1 --2- 172.21.15.76:0/4094823658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b34102f50 0x7f0b341a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:07.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.235+0000 7f0b3ac5a700 1 -- 172.21.15.76:0/4094823658 >> 172.21.15.76:0/4094823658 conn(0x7f0b340fde10 msgr2=0x7f0b3410d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:07.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.235+0000 7f0b3ac5a700 1 -- 172.21.15.76:0/4094823658 shutdown_connections 2024-03-18T14:52:07.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:07.235+0000 7f0b3ac5a700 1 -- 172.21.15.76:0/4094823658 wait complete. 2024-03-18T14:52:07.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:07 smithi076 ceph-mon[32205]: pgmap v316: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:52:07.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:07 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-18T14:52:07.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:07 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-18T14:52:07.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:07 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-18T14:52:07.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:07 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-03-18T14:52:07.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:07 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-18T14:52:07.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:07 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-03-18T14:52:07.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:07 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-18T14:52:07.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:07 smithi076 ceph-mon[32205]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T14:52:07.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:07 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/886601170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T14:52:07.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:07 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/4094823658' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T14:52:07.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:07 smithi174 ceph-mon[36787]: pgmap v316: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:52:07.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:07 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-18T14:52:07.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:07 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-18T14:52:07.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:07 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-18T14:52:07.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:07 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-03-18T14:52:07.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:07 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-18T14:52:07.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:07 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-03-18T14:52:07.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:07 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-18T14:52:07.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:07 smithi174 ceph-mon[36787]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T14:52:07.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:07 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/886601170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T14:52:07.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:07 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/4094823658' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T14:52:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:07 smithi077 ceph-mon[35865]: pgmap v316: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:52:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:07 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-18T14:52:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:07 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-18T14:52:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:07 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-18T14:52:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:07 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-03-18T14:52:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:07 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-18T14:52:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:07 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-03-18T14:52:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:07 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-18T14:52:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:07 smithi077 ceph-mon[35865]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T14:52:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:07 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/886601170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T14:52:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:07 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/4094823658' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T14:52:07.919 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738445 got 34359738449 for osd.0 2024-03-18T14:52:07.919 DEBUG:teuthology.parallel:result is None 2024-03-18T14:52:08.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.127+0000 7fab81c59700 1 -- 172.21.15.76:0/1963854869 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab7c102ce0 msgr2=0x7fab7c10f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:08.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.127+0000 7fab81c59700 1 --2- 172.21.15.76:0/1963854869 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab7c102ce0 0x7fab7c10f120 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fab7400b0d0 tx=0x7fab7400b3e0 comp rx=0 tx=0).stop 2024-03-18T14:52:08.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.127+0000 7fab81c59700 1 -- 172.21.15.76:0/1963854869 shutdown_connections 2024-03-18T14:52:08.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.127+0000 7fab81c59700 1 --2- 172.21.15.76:0/1963854869 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab7c102ce0 0x7fab7c10f120 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:08.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.127+0000 7fab81c59700 1 --2- 172.21.15.76:0/1963854869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab7c108300 0x7fab7c1086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:08.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.127+0000 7fab81c59700 1 --2- 172.21.15.76:0/1963854869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab7c102320 0x7fab7c1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:08.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.128+0000 7fab81c59700 1 -- 172.21.15.76:0/1963854869 >> 172.21.15.76:0/1963854869 conn(0x7fab7c079400 msgr2=0x7fab7c079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:08.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.128+0000 7fab81c59700 1 -- 172.21.15.76:0/1963854869 shutdown_connections 2024-03-18T14:52:08.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.128+0000 7fab81c59700 1 -- 172.21.15.76:0/1963854869 wait complete. 2024-03-18T14:52:08.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.128+0000 7fab81c59700 1 Processor -- start 2024-03-18T14:52:08.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.128+0000 7fab81c59700 1 -- start start 2024-03-18T14:52:08.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.128+0000 7fab81c59700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab7c102320 0x7fab7c0745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:08.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.128+0000 7fab81c59700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab7c102ce0 0x7fab7c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:08.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.128+0000 7fab81c59700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab7c108300 0x7fab7c075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:08.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.128+0000 7fab81c59700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c1125b0 con 0x7fab7c108300 2024-03-18T14:52:08.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.128+0000 7fab81c59700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c112430 con 0x7fab7c102320 2024-03-18T14:52:08.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.128+0000 7fab81c59700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c112730 con 0x7fab7c102ce0 2024-03-18T14:52:08.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.129+0000 7fab7bfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab7c108300 0x7fab7c075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:08.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.129+0000 7fab7b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab7c102320 0x7fab7c0745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:08.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.129+0000 7fab7b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab7c102320 0x7fab7c0745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:40376/0 (socket says 172.21.15.76:40376) 2024-03-18T14:52:08.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.129+0000 7fab7bfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab7c108300 0x7fab7c075050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34446/0 (socket says 172.21.15.76:34446) 2024-03-18T14:52:08.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.129+0000 7fab7b7fe700 1 -- 172.21.15.76:0/3652762192 learned_addr learned my addr 172.21.15.76:0/3652762192 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:08.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.129+0000 7fab7affd700 1 --2- 172.21.15.76:0/3652762192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab7c102ce0 0x7fab7c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:08.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.129+0000 7fab7bfff700 1 -- 172.21.15.76:0/3652762192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab7c102ce0 msgr2=0x7fab7c074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:08.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.129+0000 7fab7bfff700 1 --2- 172.21.15.76:0/3652762192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab7c102ce0 0x7fab7c074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:08.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.129+0000 7fab7bfff700 1 -- 172.21.15.76:0/3652762192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab7c102320 msgr2=0x7fab7c0745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:08.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.129+0000 7fab7bfff700 1 --2- 172.21.15.76:0/3652762192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab7c102320 0x7fab7c0745d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:08.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.129+0000 7fab7bfff700 1 -- 172.21.15.76:0/3652762192 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab7c1a6fc0 con 0x7fab7c108300 2024-03-18T14:52:08.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.129+0000 7fab7bfff700 1 --2- 172.21.15.76:0/3652762192 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab7c108300 0x7fab7c075050 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fab7400b770 tx=0x7fab74007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:08.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.129+0000 7fab78ff9700 1 -- 172.21.15.76:0/3652762192 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab74021020 con 0x7fab7c108300 2024-03-18T14:52:08.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.129+0000 7fab78ff9700 1 -- 172.21.15.76:0/3652762192 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab74004c50 con 0x7fab7c108300 2024-03-18T14:52:08.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.129+0000 7fab81c59700 1 -- 172.21.15.76:0/3652762192 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab7c1a7250 con 0x7fab7c108300 2024-03-18T14:52:08.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.129+0000 7fab78ff9700 1 -- 172.21.15.76:0/3652762192 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab740052b0 con 0x7fab7c108300 2024-03-18T14:52:08.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.129+0000 7fab81c59700 1 -- 172.21.15.76:0/3652762192 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab7c1a7700 con 0x7fab7c108300 2024-03-18T14:52:08.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.130+0000 7fab81c59700 1 -- 172.21.15.76:0/3652762192 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab7c068f90 con 0x7fab7c108300 2024-03-18T14:52:08.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.131+0000 7fab78ff9700 1 -- 172.21.15.76:0/3652762192 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fab74024020 con 0x7fab7c108300 2024-03-18T14:52:08.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.132+0000 7fab78ff9700 1 --2- 172.21.15.76:0/3652762192 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fab640a29a0 0x7fab640a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:08.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.132+0000 7fab7b7fe700 1 --2- 172.21.15.76:0/3652762192 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fab640a29a0 0x7fab640a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:08.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.132+0000 7fab78ff9700 1 -- 172.21.15.76:0/3652762192 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fab740cfb90 con 0x7fab7c108300 2024-03-18T14:52:08.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.132+0000 7fab7b7fe700 1 --2- 172.21.15.76:0/3652762192 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fab640a29a0 0x7fab640a4e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009ca0 tx=0x7fab6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:08.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.136+0000 7fab78ff9700 1 -- 172.21.15.76:0/3652762192 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fab7402b020 con 0x7fab7c108300 2024-03-18T14:52:08.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.345+0000 7fab81c59700 1 -- 172.21.15.76:0/3652762192 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fab7c02db70 con 0x7fab7c108300 2024-03-18T14:52:08.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.345+0000 7fab78ff9700 1 -- 172.21.15.76:0/3652762192 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fab7401de20 con 0x7fab7c108300 2024-03-18T14:52:08.346 INFO:teuthology.orchestra.run.smithi076.stdout:292057776164 2024-03-18T14:52:08.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.347+0000 7fab81c59700 1 -- 172.21.15.76:0/3652762192 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fab640a29a0 msgr2=0x7fab640a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:08.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.347+0000 7fab81c59700 1 --2- 172.21.15.76:0/3652762192 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fab640a29a0 0x7fab640a4e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009ca0 tx=0x7fab6c009260 comp rx=0 tx=0).stop 2024-03-18T14:52:08.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.347+0000 7fab81c59700 1 -- 172.21.15.76:0/3652762192 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab7c108300 msgr2=0x7fab7c075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:08.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.347+0000 7fab81c59700 1 --2- 172.21.15.76:0/3652762192 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab7c108300 0x7fab7c075050 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fab7400b770 tx=0x7fab74007a20 comp rx=0 tx=0).stop 2024-03-18T14:52:08.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.347+0000 7fab81c59700 1 -- 172.21.15.76:0/3652762192 shutdown_connections 2024-03-18T14:52:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.347+0000 7fab81c59700 1 --2- 172.21.15.76:0/3652762192 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fab640a29a0 0x7fab640a4e60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.347+0000 7fab81c59700 1 --2- 172.21.15.76:0/3652762192 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab7c108300 0x7fab7c075050 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.347+0000 7fab81c59700 1 --2- 172.21.15.76:0/3652762192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab7c102320 0x7fab7c0745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.347+0000 7fab81c59700 1 --2- 172.21.15.76:0/3652762192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab7c102ce0 0x7fab7c074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.347+0000 7fab81c59700 1 -- 172.21.15.76:0/3652762192 >> 172.21.15.76:0/3652762192 conn(0x7fab7c079400 msgr2=0x7fab7c100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.348+0000 7fab81c59700 1 -- 172.21.15.76:0/3652762192 shutdown_connections 2024-03-18T14:52:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:08.348+0000 7fab81c59700 1 -- 172.21.15.76:0/3652762192 wait complete. 2024-03-18T14:52:08.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:08 smithi076 ceph-mon[32205]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T14:52:08.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:08 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3652762192' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T14:52:08.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:08 smithi174 ceph-mon[36787]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T14:52:08.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:08 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3652762192' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T14:52:08.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:08 smithi077 ceph-mon[35865]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T14:52:08.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:08 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3652762192' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T14:52:09.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:09 smithi076 ceph-mon[32205]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-18T14:52:09.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:09 smithi174 ceph-mon[36787]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-18T14:52:09.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:09 smithi077 ceph-mon[35865]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-18T14:52:09.776 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724596 got 167503724599 for osd.5 2024-03-18T14:52:09.776 DEBUG:teuthology.parallel:result is None 2024-03-18T14:52:10.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.175+0000 7f368db59700 1 -- 172.21.15.76:0/2821931092 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3688107210 msgr2=0x7f36881075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:10.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.175+0000 7f368db59700 1 --2- 172.21.15.76:0/2821931092 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3688107210 0x7f36881075f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3678009990 tx=0x7f3678009ca0 comp rx=0 tx=0).stop 2024-03-18T14:52:10.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.175+0000 7f368db59700 1 -- 172.21.15.76:0/2821931092 shutdown_connections 2024-03-18T14:52:10.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.175+0000 7f368db59700 1 --2- 172.21.15.76:0/2821931092 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3688101b70 0x7f368810e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:10.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.175+0000 7f368db59700 1 --2- 172.21.15.76:0/2821931092 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3688107210 0x7f36881075f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:10.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.175+0000 7f368db59700 1 --2- 172.21.15.76:0/2821931092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36881011d0 0x7f3688101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:10.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.175+0000 7f368db59700 1 -- 172.21.15.76:0/2821931092 >> 172.21.15.76:0/2821931092 conn(0x7f3688078100 msgr2=0x7f3688078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:10.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.175+0000 7f368db59700 1 -- 172.21.15.76:0/2821931092 shutdown_connections 2024-03-18T14:52:10.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.175+0000 7f368db59700 1 -- 172.21.15.76:0/2821931092 wait complete. 2024-03-18T14:52:10.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.176+0000 7f368db59700 1 Processor -- start 2024-03-18T14:52:10.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.176+0000 7f368db59700 1 -- start start 2024-03-18T14:52:10.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.176+0000 7f368db59700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36881011d0 0x7f3688199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:10.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.176+0000 7f368db59700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3688101b70 0x7f3688199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:10.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.176+0000 7f368db59700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3688107210 0x7f36881a4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:10.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.176+0000 7f368db59700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f36881114a0 con 0x7f36881011d0 2024-03-18T14:52:10.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.176+0000 7f368db59700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3688111320 con 0x7f3688101b70 2024-03-18T14:52:10.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.176+0000 7f368db59700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3688111620 con 0x7f3688107210 2024-03-18T14:52:10.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.176+0000 7f36877fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36881011d0 0x7f3688199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:10.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.176+0000 7f36877fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36881011d0 0x7f3688199980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52900/0 (socket says 172.21.15.76:52900) 2024-03-18T14:52:10.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.176+0000 7f36877fe700 1 -- 172.21.15.76:0/3064005479 learned_addr learned my addr 172.21.15.76:0/3064005479 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:10.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.176+0000 7f367effd700 1 --2- 172.21.15.76:0/3064005479 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3688101b70 0x7f3688199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:10.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.177+0000 7f3687fff700 1 --2- 172.21.15.76:0/3064005479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3688107210 0x7f36881a4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:10.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.177+0000 7f36877fe700 1 -- 172.21.15.76:0/3064005479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3688107210 msgr2=0x7f36881a4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:10.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.177+0000 7f36877fe700 1 --2- 172.21.15.76:0/3064005479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3688107210 0x7f36881a4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:10.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.177+0000 7f36877fe700 1 -- 172.21.15.76:0/3064005479 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3688101b70 msgr2=0x7f3688199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:10.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.177+0000 7f36877fe700 1 --2- 172.21.15.76:0/3064005479 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3688101b70 0x7f3688199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:10.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.177+0000 7f36877fe700 1 -- 172.21.15.76:0/3064005479 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36881a4f40 con 0x7f36881011d0 2024-03-18T14:52:10.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.177+0000 7f36877fe700 1 --2- 172.21.15.76:0/3064005479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36881011d0 0x7f3688199980 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3678005910 tx=0x7f367800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:10.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.177+0000 7f36857fa700 1 -- 172.21.15.76:0/3064005479 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3678027070 con 0x7f36881011d0 2024-03-18T14:52:10.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.177+0000 7f36857fa700 1 -- 172.21.15.76:0/3064005479 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3678004650 con 0x7f36881011d0 2024-03-18T14:52:10.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.177+0000 7f36857fa700 1 -- 172.21.15.76:0/3064005479 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367800f460 con 0x7f36881011d0 2024-03-18T14:52:10.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.177+0000 7f368db59700 1 -- 172.21.15.76:0/3064005479 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36881a5110 con 0x7f36881011d0 2024-03-18T14:52:10.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.177+0000 7f368db59700 1 -- 172.21.15.76:0/3064005479 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36881a5560 con 0x7f36881011d0 2024-03-18T14:52:10.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.178+0000 7f368db59700 1 -- 172.21.15.76:0/3064005479 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3688068f90 con 0x7f36881011d0 2024-03-18T14:52:10.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.179+0000 7f36857fa700 1 -- 172.21.15.76:0/3064005479 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f367802a070 con 0x7f36881011d0 2024-03-18T14:52:10.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.180+0000 7f36857fa700 1 --2- 172.21.15.76:0/3064005479 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f36740a2920 0x7f36740a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:10.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.180+0000 7f367effd700 1 --2- 172.21.15.76:0/3064005479 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f36740a2920 0x7f36740a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:10.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.180+0000 7f36857fa700 1 -- 172.21.15.76:0/3064005479 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f36780cfe90 con 0x7f36881011d0 2024-03-18T14:52:10.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.180+0000 7f367effd700 1 --2- 172.21.15.76:0/3064005479 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f36740a2920 0x7f36740a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f36880787c0 tx=0x7f3670009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:10.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.183+0000 7f36857fa700 1 -- 172.21.15.76:0/3064005479 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3678031070 con 0x7f36881011d0 2024-03-18T14:52:10.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.390+0000 7f368db59700 1 -- 172.21.15.76:0/3064005479 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f368802dd30 con 0x7f36881011d0 2024-03-18T14:52:10.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.391+0000 7f36857fa700 1 -- 172.21.15.76:0/3064005479 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3678013070 con 0x7f36881011d0 2024-03-18T14:52:10.391 INFO:teuthology.orchestra.run.smithi076.stdout:55834574925 2024-03-18T14:52:10.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.393+0000 7f368db59700 1 -- 172.21.15.76:0/3064005479 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f36740a2920 msgr2=0x7f36740a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:10.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.393+0000 7f368db59700 1 --2- 172.21.15.76:0/3064005479 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f36740a2920 0x7f36740a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f36880787c0 tx=0x7f3670009260 comp rx=0 tx=0).stop 2024-03-18T14:52:10.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.393+0000 7f368db59700 1 -- 172.21.15.76:0/3064005479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36881011d0 msgr2=0x7f3688199980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:10.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.393+0000 7f368db59700 1 --2- 172.21.15.76:0/3064005479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36881011d0 0x7f3688199980 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3678005910 tx=0x7f367800bab0 comp rx=0 tx=0).stop 2024-03-18T14:52:10.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.393+0000 7f368db59700 1 -- 172.21.15.76:0/3064005479 shutdown_connections 2024-03-18T14:52:10.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.393+0000 7f368db59700 1 --2- 172.21.15.76:0/3064005479 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f36740a2920 0x7f36740a4de0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:10.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.393+0000 7f368db59700 1 --2- 172.21.15.76:0/3064005479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3688107210 0x7f36881a4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:10.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.393+0000 7f368db59700 1 --2- 172.21.15.76:0/3064005479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36881011d0 0x7f3688199980 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:10.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.393+0000 7f368db59700 1 --2- 172.21.15.76:0/3064005479 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3688101b70 0x7f3688199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:10.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.393+0000 7f368db59700 1 -- 172.21.15.76:0/3064005479 >> 172.21.15.76:0/3064005479 conn(0x7f3688078100 msgr2=0x7f36880fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:10.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.393+0000 7f368db59700 1 -- 172.21.15.76:0/3064005479 shutdown_connections 2024-03-18T14:52:10.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:10.393+0000 7f368db59700 1 -- 172.21.15.76:0/3064005479 wait complete. 2024-03-18T14:52:10.663 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462958 got 201863462962 for osd.6 2024-03-18T14:52:10.663 DEBUG:teuthology.parallel:result is None 2024-03-18T14:52:11.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.438+0000 7f490aed0700 1 -- 172.21.15.76:0/1345749320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49041011d0 msgr2=0x7f4904101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:11.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.438+0000 7f490aed0700 1 --2- 172.21.15.76:0/1345749320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49041011d0 0x7f4904101630 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f48f0009990 tx=0x7f48f0009ca0 comp rx=0 tx=0).stop 2024-03-18T14:52:11.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.439+0000 7f490aed0700 1 -- 172.21.15.76:0/1345749320 shutdown_connections 2024-03-18T14:52:11.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.439+0000 7f490aed0700 1 --2- 172.21.15.76:0/1345749320 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4904101b70 0x7f490410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:11.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.439+0000 7f490aed0700 1 --2- 172.21.15.76:0/1345749320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4904107210 0x7f49041075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:11.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.439+0000 7f490aed0700 1 --2- 172.21.15.76:0/1345749320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49041011d0 0x7f4904101630 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:11.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.439+0000 7f490aed0700 1 -- 172.21.15.76:0/1345749320 >> 172.21.15.76:0/1345749320 conn(0x7f4904078100 msgr2=0x7f4904078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:11.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.439+0000 7f490aed0700 1 -- 172.21.15.76:0/1345749320 shutdown_connections 2024-03-18T14:52:11.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.439+0000 7f490aed0700 1 -- 172.21.15.76:0/1345749320 wait complete. 2024-03-18T14:52:11.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.439+0000 7f490aed0700 1 Processor -- start 2024-03-18T14:52:11.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.439+0000 7f490aed0700 1 -- start start 2024-03-18T14:52:11.441 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:52:11.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.439+0000 7f490aed0700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49041011d0 0x7f49041998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:11.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.439+0000 7f490aed0700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4904101b70 0x7f4904199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:11.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f490aed0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4904107210 0x7f49041a4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:11.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f490aed0700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4904111470 con 0x7f49041011d0 2024-03-18T14:52:11.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f490aed0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f49041112f0 con 0x7f4904107210 2024-03-18T14:52:11.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f490aed0700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f49041115f0 con 0x7f4904101b70 2024-03-18T14:52:11.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f4903fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4904101b70 0x7f4904199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:11.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f4903fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4904101b70 0x7f4904199e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:38008/0 (socket says 172.21.15.76:38008) 2024-03-18T14:52:11.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f4903fff700 1 -- 172.21.15.76:0/2621408239 learned_addr learned my addr 172.21.15.76:0/2621408239 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:11.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f4908c6c700 1 --2- 172.21.15.76:0/2621408239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49041011d0 0x7f49041998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:11.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f490946d700 1 --2- 172.21.15.76:0/2621408239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4904107210 0x7f49041a4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:11.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f4903fff700 1 -- 172.21.15.76:0/2621408239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4904107210 msgr2=0x7f49041a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:11.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f4903fff700 1 --2- 172.21.15.76:0/2621408239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4904107210 0x7f49041a4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:11.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f4903fff700 1 -- 172.21.15.76:0/2621408239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49041011d0 msgr2=0x7f49041998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:11.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f4903fff700 1 --2- 172.21.15.76:0/2621408239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49041011d0 0x7f49041998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:11.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f4903fff700 1 -- 172.21.15.76:0/2621408239 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49041a4eb0 con 0x7f4904101b70 2024-03-18T14:52:11.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f4903fff700 1 --2- 172.21.15.76:0/2621408239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4904101b70 0x7f4904199e30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f48f0000c00 tx=0x7f48f000bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:11.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f4901ffb700 1 -- 172.21.15.76:0/2621408239 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f0027070 con 0x7f4904101b70 2024-03-18T14:52:11.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f490aed0700 1 -- 172.21.15.76:0/2621408239 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49041a5080 con 0x7f4904101b70 2024-03-18T14:52:11.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f4901ffb700 1 -- 172.21.15.76:0/2621408239 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48f0003d70 con 0x7f4904101b70 2024-03-18T14:52:11.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.440+0000 7f4901ffb700 1 -- 172.21.15.76:0/2621408239 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f000f460 con 0x7f4904101b70 2024-03-18T14:52:11.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.441+0000 7f490aed0700 1 -- 172.21.15.76:0/2621408239 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49041a5480 con 0x7f4904101b70 2024-03-18T14:52:11.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.442+0000 7f490aed0700 1 -- 172.21.15.76:0/2621408239 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4904068f90 con 0x7f4904101b70 2024-03-18T14:52:11.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.443+0000 7f4901ffb700 1 -- 172.21.15.76:0/2621408239 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f48f002a070 con 0x7f4904101b70 2024-03-18T14:52:11.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.443+0000 7f4901ffb700 1 --2- 172.21.15.76:0/2621408239 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f48f40a2920 0x7f48f40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:11.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.443+0000 7f4908c6c700 1 --2- 172.21.15.76:0/2621408239 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f48f40a2920 0x7f48f40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:11.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.443+0000 7f4901ffb700 1 -- 172.21.15.76:0/2621408239 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f48f00cf9a0 con 0x7f4904101b70 2024-03-18T14:52:11.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.444+0000 7f4908c6c700 1 --2- 172.21.15.76:0/2621408239 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f48f40a2920 0x7f48f40a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009be0 tx=0x7f48f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:11.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.448+0000 7f4901ffb700 1 -- 172.21.15.76:0/2621408239 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f48f0013070 con 0x7f4904101b70 2024-03-18T14:52:11.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.655+0000 7f490aed0700 1 -- 172.21.15.76:0/2621408239 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f490402db70 con 0x7f4904101b70 2024-03-18T14:52:11.656 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:11 smithi076 ceph-mon[32205]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T14:52:11.656 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:11 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3064005479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T14:52:11.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.655+0000 7f4901ffb700 1 -- 172.21.15.76:0/2621408239 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f48f009c940 con 0x7f4904101b70 2024-03-18T14:52:11.656 INFO:teuthology.orchestra.run.smithi076.stdout:111669149763 2024-03-18T14:52:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.657+0000 7f490aed0700 1 -- 172.21.15.76:0/2621408239 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f48f40a2920 msgr2=0x7f48f40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.657+0000 7f490aed0700 1 --2- 172.21.15.76:0/2621408239 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f48f40a2920 0x7f48f40a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009be0 tx=0x7f48f8009260 comp rx=0 tx=0).stop 2024-03-18T14:52:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.657+0000 7f490aed0700 1 -- 172.21.15.76:0/2621408239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4904101b70 msgr2=0x7f4904199e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.657+0000 7f490aed0700 1 --2- 172.21.15.76:0/2621408239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4904101b70 0x7f4904199e30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f48f0000c00 tx=0x7f48f000bab0 comp rx=0 tx=0).stop 2024-03-18T14:52:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.657+0000 7f490aed0700 1 -- 172.21.15.76:0/2621408239 shutdown_connections 2024-03-18T14:52:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.657+0000 7f490aed0700 1 --2- 172.21.15.76:0/2621408239 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f48f40a2920 0x7f48f40a4de0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.657+0000 7f490aed0700 1 --2- 172.21.15.76:0/2621408239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4904107210 0x7f49041a4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.657+0000 7f490aed0700 1 --2- 172.21.15.76:0/2621408239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49041011d0 0x7f49041998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.657+0000 7f490aed0700 1 --2- 172.21.15.76:0/2621408239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4904101b70 0x7f4904199e30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.657+0000 7f490aed0700 1 -- 172.21.15.76:0/2621408239 >> 172.21.15.76:0/2621408239 conn(0x7f4904078100 msgr2=0x7f49040ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.657+0000 7f490aed0700 1 -- 172.21.15.76:0/2621408239 shutdown_connections 2024-03-18T14:52:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:11.657+0000 7f490aed0700 1 -- 172.21.15.76:0/2621408239 wait complete. 2024-03-18T14:52:11.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:11 smithi174 ceph-mon[36787]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T14:52:11.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:11 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3064005479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T14:52:11.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:11 smithi077 ceph-mon[35865]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T14:52:11.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:11 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3064005479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T14:52:12.652 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:12 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2621408239' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T14:52:12.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:12 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2621408239' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T14:52:12.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:12 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2621408239' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T14:52:13.283 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574920 got 55834574925 for osd.1 2024-03-18T14:52:13.283 DEBUG:teuthology.parallel:result is None 2024-03-18T14:52:13.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.485+0000 7fd0edfff700 1 -- 172.21.15.76:0/2137665890 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0e8101ec0 msgr2=0x7fd0e810dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:13.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.485+0000 7fd0edfff700 1 --2- 172.21.15.76:0/2137665890 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0e8101ec0 0x7fd0e810dfc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd0e000b0a0 tx=0x7fd0e000b3b0 comp rx=0 tx=0).stop 2024-03-18T14:52:13.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.485+0000 7fd0edfff700 1 -- 172.21.15.76:0/2137665890 shutdown_connections 2024-03-18T14:52:13.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.485+0000 7fd0edfff700 1 --2- 172.21.15.76:0/2137665890 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0e8101ec0 0x7fd0e810dfc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:13.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.485+0000 7fd0edfff700 1 --2- 172.21.15.76:0/2137665890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0e8106140 0x7fd0e8106520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:13.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.485+0000 7fd0edfff700 1 --2- 172.21.15.76:0/2137665890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0e8101500 0x7fd0e8101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:13.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.485+0000 7fd0edfff700 1 -- 172.21.15.76:0/2137665890 >> 172.21.15.76:0/2137665890 conn(0x7fd0e8077f80 msgr2=0x7fd0e8078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:13.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.485+0000 7fd0edfff700 1 -- 172.21.15.76:0/2137665890 shutdown_connections 2024-03-18T14:52:13.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.485+0000 7fd0edfff700 1 -- 172.21.15.76:0/2137665890 wait complete. 2024-03-18T14:52:13.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.485+0000 7fd0edfff700 1 Processor -- start 2024-03-18T14:52:13.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0edfff700 1 -- start start 2024-03-18T14:52:13.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0edfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0e8101500 0x7fd0e8199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:13.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0edfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0e8101ec0 0x7fd0e8199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:13.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0edfff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0e8106140 0x7fd0e81a48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:13.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0edfff700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e8111560 con 0x7fd0e8101ec0 2024-03-18T14:52:13.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0edfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e81113e0 con 0x7fd0e8101500 2024-03-18T14:52:13.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0edfff700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e81116e0 con 0x7fd0e8106140 2024-03-18T14:52:13.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0e6ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0e8101ec0 0x7fd0e8199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:13.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0e77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0e8101500 0x7fd0e8199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:13.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0e6ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0e8101ec0 0x7fd0e8199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52940/0 (socket says 172.21.15.76:52940) 2024-03-18T14:52:13.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0e6ffd700 1 -- 172.21.15.76:0/3001867844 learned_addr learned my addr 172.21.15.76:0/3001867844 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:13.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0e7fff700 1 --2- 172.21.15.76:0/3001867844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0e8106140 0x7fd0e81a48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:13.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0e77fe700 1 -- 172.21.15.76:0/3001867844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0e8106140 msgr2=0x7fd0e81a48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:13.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0e77fe700 1 --2- 172.21.15.76:0/3001867844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0e8106140 0x7fd0e81a48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:13.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0e77fe700 1 -- 172.21.15.76:0/3001867844 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0e8101ec0 msgr2=0x7fd0e8199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:13.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0e77fe700 1 --2- 172.21.15.76:0/3001867844 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0e8101ec0 0x7fd0e8199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:13.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0e77fe700 1 -- 172.21.15.76:0/3001867844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e81a4e20 con 0x7fd0e8101500 2024-03-18T14:52:13.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0e6ffd700 1 --2- 172.21.15.76:0/3001867844 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0e8101ec0 0x7fd0e8199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:13.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.486+0000 7fd0e77fe700 1 --2- 172.21.15.76:0/3001867844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0e8101500 0x7fd0e8199860 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800e9f0 tx=0x7fd0d800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:13.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.487+0000 7fd0e4ff9700 1 -- 172.21.15.76:0/3001867844 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d800c9c0 con 0x7fd0e8101500 2024-03-18T14:52:13.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.487+0000 7fd0e4ff9700 1 -- 172.21.15.76:0/3001867844 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0d800cb60 con 0x7fd0e8101500 2024-03-18T14:52:13.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.487+0000 7fd0edfff700 1 -- 172.21.15.76:0/3001867844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0e81a4ff0 con 0x7fd0e8101500 2024-03-18T14:52:13.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.487+0000 7fd0e4ff9700 1 -- 172.21.15.76:0/3001867844 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8018680 con 0x7fd0e8101500 2024-03-18T14:52:13.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.487+0000 7fd0edfff700 1 -- 172.21.15.76:0/3001867844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0e81a5500 con 0x7fd0e8101500 2024-03-18T14:52:13.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.488+0000 7fd0edfff700 1 -- 172.21.15.76:0/3001867844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0e81028a0 con 0x7fd0e8101500 2024-03-18T14:52:13.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.489+0000 7fd0e4ff9700 1 -- 172.21.15.76:0/3001867844 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd0d8010430 con 0x7fd0e8101500 2024-03-18T14:52:13.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.489+0000 7fd0e4ff9700 1 --2- 172.21.15.76:0/3001867844 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd0d00a2920 0x7fd0d00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:13.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.489+0000 7fd0e6ffd700 1 --2- 172.21.15.76:0/3001867844 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd0d00a2920 0x7fd0d00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:13.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.489+0000 7fd0e4ff9700 1 -- 172.21.15.76:0/3001867844 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fd0d80cec70 con 0x7fd0e8101500 2024-03-18T14:52:13.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.490+0000 7fd0e6ffd700 1 --2- 172.21.15.76:0/3001867844 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd0d00a2920 0x7fd0d00a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8105e00 tx=0x7fd0dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:13.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.494+0000 7fd0e4ff9700 1 -- 172.21.15.76:0/3001867844 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd0d809bb80 con 0x7fd0e8101500 2024-03-18T14:52:13.600 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:13 smithi076 ceph-mon[32205]: pgmap v321: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s; 7.2 KiB/s, 1 keys/s, 0 objects/s recovering 2024-03-18T14:52:13.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.694+0000 7fd0edfff700 1 -- 172.21.15.76:0/3001867844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fd0e81a5800 con 0x7fd0e8101500 2024-03-18T14:52:13.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.694+0000 7fd0e4ff9700 1 -- 172.21.15.76:0/3001867844 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd0d809b480 con 0x7fd0e8101500 2024-03-18T14:52:13.695 INFO:teuthology.orchestra.run.smithi076.stdout:223338299439 2024-03-18T14:52:13.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.696+0000 7fd0edfff700 1 -- 172.21.15.76:0/3001867844 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd0d00a2920 msgr2=0x7fd0d00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:13.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.696+0000 7fd0edfff700 1 --2- 172.21.15.76:0/3001867844 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd0d00a2920 0x7fd0d00a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8105e00 tx=0x7fd0dc009260 comp rx=0 tx=0).stop 2024-03-18T14:52:13.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.696+0000 7fd0edfff700 1 -- 172.21.15.76:0/3001867844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0e8101500 msgr2=0x7fd0e8199860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:13.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.696+0000 7fd0edfff700 1 --2- 172.21.15.76:0/3001867844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0e8101500 0x7fd0e8199860 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800e9f0 tx=0x7fd0d800ed00 comp rx=0 tx=0).stop 2024-03-18T14:52:13.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.696+0000 7fd0edfff700 1 -- 172.21.15.76:0/3001867844 shutdown_connections 2024-03-18T14:52:13.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.696+0000 7fd0edfff700 1 --2- 172.21.15.76:0/3001867844 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd0d00a2920 0x7fd0d00a4de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:13.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.696+0000 7fd0edfff700 1 --2- 172.21.15.76:0/3001867844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0e8106140 0x7fd0e81a48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:13.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.696+0000 7fd0edfff700 1 --2- 172.21.15.76:0/3001867844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0e8101500 0x7fd0e8199860 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:13.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.696+0000 7fd0edfff700 1 --2- 172.21.15.76:0/3001867844 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0e8101ec0 0x7fd0e8199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:13.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.696+0000 7fd0edfff700 1 -- 172.21.15.76:0/3001867844 >> 172.21.15.76:0/3001867844 conn(0x7fd0e8077f80 msgr2=0x7fd0e80ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:13.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.696+0000 7fd0edfff700 1 -- 172.21.15.76:0/3001867844 shutdown_connections 2024-03-18T14:52:13.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:13.696+0000 7fd0edfff700 1 -- 172.21.15.76:0/3001867844 wait complete. 2024-03-18T14:52:13.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:13 smithi174 ceph-mon[36787]: pgmap v321: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s; 7.2 KiB/s, 1 keys/s, 0 objects/s recovering 2024-03-18T14:52:13.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:13 smithi077 ceph-mon[35865]: pgmap v321: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s; 7.2 KiB/s, 1 keys/s, 0 objects/s recovering 2024-03-18T14:52:14.148 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350997 got 347892351002 for osd.11 2024-03-18T14:52:14.149 DEBUG:teuthology.parallel:result is None 2024-03-18T14:52:14.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:14 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3001867844' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T14:52:14.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:14 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3001867844' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T14:52:14.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:14 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3001867844' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T14:52:14.802 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776160 got 292057776164 for osd.9 2024-03-18T14:52:14.802 DEBUG:teuthology.parallel:result is None 2024-03-18T14:52:15.266 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149757 got 111669149763 for osd.3 2024-03-18T14:52:15.267 DEBUG:teuthology.parallel:result is None 2024-03-18T14:52:15.430 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:15 smithi076 ceph-mon[32205]: pgmap v322: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s; 7.2 KiB/s, 1 keys/s, 0 objects/s recovering 2024-03-18T14:52:15.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:15 smithi174 ceph-mon[36787]: pgmap v322: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s; 7.2 KiB/s, 1 keys/s, 0 objects/s recovering 2024-03-18T14:52:15.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:15 smithi077 ceph-mon[35865]: pgmap v322: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s; 7.2 KiB/s, 1 keys/s, 0 objects/s recovering 2024-03-18T14:52:16.690 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299432 got 223338299439 for osd.7 2024-03-18T14:52:16.690 DEBUG:teuthology.parallel:result is None 2024-03-18T14:52:17.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.389+0000 7f4a03557700 1 -- 172.21.15.76:0/1034865074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49fc102320 msgr2=0x7f49fc1027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:17.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.389+0000 7f4a03557700 1 --2- 172.21.15.76:0/1034865074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49fc102320 0x7f49fc1027a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f49ec0099c0 tx=0x7f49ec009cd0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.389+0000 7f4a03557700 1 -- 172.21.15.76:0/1034865074 shutdown_connections 2024-03-18T14:52:17.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.389+0000 7f4a03557700 1 --2- 172.21.15.76:0/1034865074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49fc102ce0 0x7f49fc10f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.389+0000 7f4a03557700 1 --2- 172.21.15.76:0/1034865074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49fc108300 0x7f49fc1086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.389+0000 7f4a03557700 1 --2- 172.21.15.76:0/1034865074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49fc102320 0x7f49fc1027a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.389+0000 7f4a03557700 1 -- 172.21.15.76:0/1034865074 >> 172.21.15.76:0/1034865074 conn(0x7f49fc079400 msgr2=0x7f49fc079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:17.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.389+0000 7f4a03557700 1 -- 172.21.15.76:0/1034865074 shutdown_connections 2024-03-18T14:52:17.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.389+0000 7f4a03557700 1 -- 172.21.15.76:0/1034865074 wait complete. 2024-03-18T14:52:17.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.389+0000 7f4a03557700 1 Processor -- start 2024-03-18T14:52:17.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a03557700 1 -- start start 2024-03-18T14:52:17.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a03557700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49fc102320 0x7f49fc0745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:17.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a03557700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49fc102ce0 0x7f49fc074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:17.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a03557700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49fc108300 0x7f49fc075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:17.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a03557700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc1125a0 con 0x7f49fc102ce0 2024-03-18T14:52:17.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a03557700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc112420 con 0x7f49fc102320 2024-03-18T14:52:17.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a03557700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc112720 con 0x7f49fc108300 2024-03-18T14:52:17.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a00af2700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49fc102ce0 0x7f49fc074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:17.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a00af2700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49fc102ce0 0x7f49fc074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52944/0 (socket says 172.21.15.76:52944) 2024-03-18T14:52:17.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a00af2700 1 -- 172.21.15.76:0/1863390906 learned_addr learned my addr 172.21.15.76:0/1863390906 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:17.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a012f3700 1 --2- 172.21.15.76:0/1863390906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49fc102320 0x7f49fc0745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:17.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a01af4700 1 --2- 172.21.15.76:0/1863390906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49fc108300 0x7f49fc075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:17.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a00af2700 1 -- 172.21.15.76:0/1863390906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49fc108300 msgr2=0x7f49fc075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:17.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a00af2700 1 --2- 172.21.15.76:0/1863390906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49fc108300 0x7f49fc075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a00af2700 1 -- 172.21.15.76:0/1863390906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49fc102320 msgr2=0x7f49fc0745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:17.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a00af2700 1 --2- 172.21.15.76:0/1863390906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49fc102320 0x7f49fc0745d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a00af2700 1 -- 172.21.15.76:0/1863390906 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49fc1a6f90 con 0x7f49fc102ce0 2024-03-18T14:52:17.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f4a00af2700 1 --2- 172.21.15.76:0/1863390906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49fc102ce0 0x7f49fc074b10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f49ec0059d0 tx=0x7f49ec00fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:17.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.390+0000 7f49f27fc700 1 -- 172.21.15.76:0/1863390906 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec027070 con 0x7f49fc102ce0 2024-03-18T14:52:17.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.391+0000 7f4a03557700 1 -- 172.21.15.76:0/1863390906 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49fc1a7220 con 0x7f49fc102ce0 2024-03-18T14:52:17.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.391+0000 7f49f27fc700 1 -- 172.21.15.76:0/1863390906 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49ec003680 con 0x7f49fc102ce0 2024-03-18T14:52:17.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.391+0000 7f49f27fc700 1 -- 172.21.15.76:0/1863390906 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec0173f0 con 0x7f49fc102ce0 2024-03-18T14:52:17.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.391+0000 7f4a03557700 1 -- 172.21.15.76:0/1863390906 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49fc1a7670 con 0x7f49fc102ce0 2024-03-18T14:52:17.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.391+0000 7f4a03557700 1 -- 172.21.15.76:0/1863390906 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49fc068f90 con 0x7f49fc102ce0 2024-03-18T14:52:17.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.393+0000 7f49f27fc700 1 -- 172.21.15.76:0/1863390906 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f49ec02a070 con 0x7f49fc102ce0 2024-03-18T14:52:17.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.393+0000 7f49f27fc700 1 --2- 172.21.15.76:0/1863390906 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f49e80a2920 0x7f49e80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:17.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.393+0000 7f4a012f3700 1 --2- 172.21.15.76:0/1863390906 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f49e80a2920 0x7f49e80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:17.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.394+0000 7f49f27fc700 1 -- 172.21.15.76:0/1863390906 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f49ec0cf730 con 0x7f49fc102ce0 2024-03-18T14:52:17.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.394+0000 7f4a012f3700 1 --2- 172.21.15.76:0/1863390906 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f49e80a2920 0x7f49e80a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f49f8009be0 tx=0x7f49f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:17.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.397+0000 7f49f27fc700 1 -- 172.21.15.76:0/1863390906 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f49ec013070 con 0x7f49fc102ce0 2024-03-18T14:52:17.512 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:17 smithi076 ceph-mon[32205]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 5.8 KiB/s, 1 keys/s, 0 objects/s recovering 2024-03-18T14:52:17.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.600+0000 7f4a03557700 1 -- 172.21.15.76:0/1863390906 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f49fc02db70 con 0x7f49fc102ce0 2024-03-18T14:52:17.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.600+0000 7f49f27fc700 1 -- 172.21.15.76:0/1863390906 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f49ec09c6d0 con 0x7f49fc102ce0 2024-03-18T14:52:17.601 INFO:teuthology.orchestra.run.smithi076.stdout:146028888128 2024-03-18T14:52:17.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.602+0000 7f4a03557700 1 -- 172.21.15.76:0/1863390906 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f49e80a2920 msgr2=0x7f49e80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:17.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.602+0000 7f4a03557700 1 --2- 172.21.15.76:0/1863390906 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f49e80a2920 0x7f49e80a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f49f8009be0 tx=0x7f49f8009260 comp rx=0 tx=0).stop 2024-03-18T14:52:17.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.602+0000 7f4a03557700 1 -- 172.21.15.76:0/1863390906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49fc102ce0 msgr2=0x7f49fc074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:17.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.602+0000 7f4a03557700 1 --2- 172.21.15.76:0/1863390906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49fc102ce0 0x7f49fc074b10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f49ec0059d0 tx=0x7f49ec00fa00 comp rx=0 tx=0).stop 2024-03-18T14:52:17.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.602+0000 7f4a03557700 1 -- 172.21.15.76:0/1863390906 shutdown_connections 2024-03-18T14:52:17.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.602+0000 7f4a03557700 1 --2- 172.21.15.76:0/1863390906 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f49e80a2920 0x7f49e80a4de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.602+0000 7f4a03557700 1 --2- 172.21.15.76:0/1863390906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49fc108300 0x7f49fc075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.602+0000 7f4a03557700 1 --2- 172.21.15.76:0/1863390906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49fc102320 0x7f49fc0745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.602+0000 7f4a03557700 1 --2- 172.21.15.76:0/1863390906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49fc102ce0 0x7f49fc074b10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.602+0000 7f4a03557700 1 -- 172.21.15.76:0/1863390906 >> 172.21.15.76:0/1863390906 conn(0x7f49fc079400 msgr2=0x7f49fc100ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:17.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.602+0000 7f4a03557700 1 -- 172.21.15.76:0/1863390906 shutdown_connections 2024-03-18T14:52:17.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.602+0000 7f4a03557700 1 -- 172.21.15.76:0/1863390906 wait complete. 2024-03-18T14:52:17.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.613+0000 7f4bf2fd0700 1 -- 172.21.15.76:0/1860300825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bec100990 msgr2=0x7f4bec100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:17.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.613+0000 7f4bf2fd0700 1 --2- 172.21.15.76:0/1860300825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bec100990 0x7f4bec100e10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc009990 tx=0x7f4bdc009ca0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.613+0000 7f4bf2fd0700 1 -- 172.21.15.76:0/1860300825 shutdown_connections 2024-03-18T14:52:17.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.613+0000 7f4bf2fd0700 1 --2- 172.21.15.76:0/1860300825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bec101350 0x7f4bec10d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.613+0000 7f4bf2fd0700 1 --2- 172.21.15.76:0/1860300825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bec106990 0x7f4bec106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.613+0000 7f4bf2fd0700 1 --2- 172.21.15.76:0/1860300825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bec100990 0x7f4bec100e10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.613+0000 7f4bf2fd0700 1 -- 172.21.15.76:0/1860300825 >> 172.21.15.76:0/1860300825 conn(0x7f4bec0777d0 msgr2=0x7f4bec0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:17.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.613+0000 7f4bf2fd0700 1 -- 172.21.15.76:0/1860300825 shutdown_connections 2024-03-18T14:52:17.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.613+0000 7f4bf2fd0700 1 -- 172.21.15.76:0/1860300825 wait complete. 2024-03-18T14:52:17.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.614+0000 7f4bf2fd0700 1 Processor -- start 2024-03-18T14:52:17.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.614+0000 7f4bf2fd0700 1 -- start start 2024-03-18T14:52:17.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.614+0000 7f4bf2fd0700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bec100990 0x7f4bec19ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:17.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.614+0000 7f4bf2fd0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bec101350 0x7f4bec19b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:17.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.614+0000 7f4bf2fd0700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bec106990 0x7f4bec19f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:17.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.614+0000 7f4bf2fd0700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec110cf0 con 0x7f4bec106990 2024-03-18T14:52:17.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.614+0000 7f4bf2fd0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec110b70 con 0x7f4bec101350 2024-03-18T14:52:17.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.614+0000 7f4bf2fd0700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec110e70 con 0x7f4bec100990 2024-03-18T14:52:17.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.614+0000 7f4bf156d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bec106990 0x7f4bec19f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:17.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.614+0000 7f4bf156d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bec106990 0x7f4bec19f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52972/0 (socket says 172.21.15.76:52972) 2024-03-18T14:52:17.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.614+0000 7f4bf156d700 1 -- 172.21.15.76:0/4048373375 learned_addr learned my addr 172.21.15.76:0/4048373375 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:17.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.614+0000 7f4bebfff700 1 --2- 172.21.15.76:0/4048373375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bec101350 0x7f4bec19b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:17.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.614+0000 7f4bf0d6c700 1 --2- 172.21.15.76:0/4048373375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bec100990 0x7f4bec19ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:17.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.615+0000 7f4bf156d700 1 -- 172.21.15.76:0/4048373375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bec100990 msgr2=0x7f4bec19ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.615+0000 7f4bf156d700 1 --2- 172.21.15.76:0/4048373375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bec100990 0x7f4bec19ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.615+0000 7f4bf156d700 1 -- 172.21.15.76:0/4048373375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bec101350 msgr2=0x7f4bec19b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.615+0000 7f4bf156d700 1 --2- 172.21.15.76:0/4048373375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bec101350 0x7f4bec19b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.615+0000 7f4bf156d700 1 -- 172.21.15.76:0/4048373375 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bec19fd60 con 0x7f4bec106990 2024-03-18T14:52:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.615+0000 7f4bf0d6c700 1 --2- 172.21.15.76:0/4048373375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bec100990 0x7f4bec19ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.615+0000 7f4bebfff700 1 --2- 172.21.15.76:0/4048373375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bec101350 0x7f4bec19b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.615+0000 7f4bf156d700 1 --2- 172.21.15.76:0/4048373375 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bec106990 0x7f4bec19f620 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400e940 tx=0x7f4bd400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.615+0000 7f4be9ffb700 1 -- 172.21.15.76:0/4048373375 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4004a30 con 0x7f4bec106990 2024-03-18T14:52:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.615+0000 7f4bf2fd0700 1 -- 172.21.15.76:0/4048373375 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bec1a0050 con 0x7f4bec106990 2024-03-18T14:52:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.615+0000 7f4be9ffb700 1 -- 172.21.15.76:0/4048373375 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bd4005500 con 0x7f4bec106990 2024-03-18T14:52:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.615+0000 7f4be9ffb700 1 -- 172.21.15.76:0/4048373375 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd400ce20 con 0x7f4bec106990 2024-03-18T14:52:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.615+0000 7f4bf2fd0700 1 -- 172.21.15.76:0/4048373375 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bec1a78d0 con 0x7f4bec106990 2024-03-18T14:52:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.617+0000 7f4bf2fd0700 1 -- 172.21.15.76:0/4048373375 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bec1020d0 con 0x7f4bec106990 2024-03-18T14:52:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.617+0000 7f4be9ffb700 1 -- 172.21.15.76:0/4048373375 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4bd4005020 con 0x7f4bec106990 2024-03-18T14:52:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.618+0000 7f4be9ffb700 1 --2- 172.21.15.76:0/4048373375 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f4bd80a2920 0x7f4bd80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.618+0000 7f4bf0d6c700 1 --2- 172.21.15.76:0/4048373375 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f4bd80a2920 0x7f4bd80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:17.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.618+0000 7f4be9ffb700 1 -- 172.21.15.76:0/4048373375 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f4bd40ced40 con 0x7f4bec106990 2024-03-18T14:52:17.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.619+0000 7f4bf0d6c700 1 --2- 172.21.15.76:0/4048373375 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f4bd80a2920 0x7f4bd80a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4be40060b0 tx=0x7f4be400c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:17.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.625+0000 7f4be9ffb700 1 -- 172.21.15.76:0/4048373375 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4bd409bcb0 con 0x7f4bec106990 2024-03-18T14:52:17.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:17 smithi174 ceph-mon[36787]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 5.8 KiB/s, 1 keys/s, 0 objects/s recovering 2024-03-18T14:52:17.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:17 smithi077 ceph-mon[35865]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 5.8 KiB/s, 1 keys/s, 0 objects/s recovering 2024-03-18T14:52:17.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.826+0000 7f4bf2fd0700 1 -- 172.21.15.76:0/4048373375 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f4bec19c0b0 con 0x7f4bec106990 2024-03-18T14:52:17.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.827+0000 7f4be9ffb700 1 -- 172.21.15.76:0/4048373375 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f4bd409b5b0 con 0x7f4bec106990 2024-03-18T14:52:17.828 INFO:teuthology.orchestra.run.smithi076.stdout:313532612640 2024-03-18T14:52:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.829+0000 7f4bf2fd0700 1 -- 172.21.15.76:0/4048373375 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f4bd80a2920 msgr2=0x7f4bd80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.829+0000 7f4bf2fd0700 1 --2- 172.21.15.76:0/4048373375 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f4bd80a2920 0x7f4bd80a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4be40060b0 tx=0x7f4be400c040 comp rx=0 tx=0).stop 2024-03-18T14:52:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.829+0000 7f4bf2fd0700 1 -- 172.21.15.76:0/4048373375 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bec106990 msgr2=0x7f4bec19f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.829+0000 7f4bf2fd0700 1 --2- 172.21.15.76:0/4048373375 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bec106990 0x7f4bec19f620 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400e940 tx=0x7f4bd400ec50 comp rx=0 tx=0).stop 2024-03-18T14:52:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.829+0000 7f4bf2fd0700 1 -- 172.21.15.76:0/4048373375 shutdown_connections 2024-03-18T14:52:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.829+0000 7f4bf2fd0700 1 --2- 172.21.15.76:0/4048373375 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f4bd80a2920 0x7f4bd80a4de0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.829+0000 7f4bf2fd0700 1 --2- 172.21.15.76:0/4048373375 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bec106990 0x7f4bec19f620 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.829+0000 7f4bf2fd0700 1 --2- 172.21.15.76:0/4048373375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bec100990 0x7f4bec19ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.829+0000 7f4bf2fd0700 1 --2- 172.21.15.76:0/4048373375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bec101350 0x7f4bec19b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.829+0000 7f4bf2fd0700 1 -- 172.21.15.76:0/4048373375 >> 172.21.15.76:0/4048373375 conn(0x7f4bec0777d0 msgr2=0x7f4bec104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.829+0000 7f4bf2fd0700 1 -- 172.21.15.76:0/4048373375 shutdown_connections 2024-03-18T14:52:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:17.829+0000 7f4bf2fd0700 1 -- 172.21.15.76:0/4048373375 wait complete. 2024-03-18T14:52:18.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:18 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1863390906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T14:52:18.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:18 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/4048373375' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T14:52:18.502 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888120 got 146028888128 for osd.4 2024-03-18T14:52:18.502 DEBUG:teuthology.parallel:result is None 2024-03-18T14:52:18.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:18 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1863390906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T14:52:18.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:18 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/4048373375' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T14:52:18.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:18 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1863390906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T14:52:18.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:18 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/4048373375' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T14:52:18.991 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612635 got 313532612640 for osd.10 2024-03-18T14:52:18.991 DEBUG:teuthology.parallel:result is None 2024-03-18T14:52:18.991 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-18T14:52:18.991 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T14:52:19.649 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:52:19.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:19 smithi076 ceph-mon[32205]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 5.2 KiB/s, 0 keys/s, 0 objects/s recovering 2024-03-18T14:52:19.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:19 smithi174 ceph-mon[36787]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 5.2 KiB/s, 0 keys/s, 0 objects/s recovering 2024-03-18T14:52:19.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:19 smithi077 ceph-mon[35865]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 5.2 KiB/s, 0 keys/s, 0 objects/s recovering 2024-03-18T14:52:21.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.355+0000 7f3e6af6a700 1 -- 172.21.15.76:0/3146161967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e64101b70 msgr2=0x7f3e6410e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:21.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.355+0000 7f3e6af6a700 1 --2- 172.21.15.76:0/3146161967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e64101b70 0x7f3e6410e010 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c009f90 tx=0x7f3e5c00b330 comp rx=0 tx=0).stop 2024-03-18T14:52:21.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.355+0000 7f3e6af6a700 1 -- 172.21.15.76:0/3146161967 shutdown_connections 2024-03-18T14:52:21.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.355+0000 7f3e6af6a700 1 --2- 172.21.15.76:0/3146161967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e64101b70 0x7f3e6410e010 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:21.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.355+0000 7f3e6af6a700 1 --2- 172.21.15.76:0/3146161967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e64107260 0x7f3e64107640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:21.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.355+0000 7f3e6af6a700 1 --2- 172.21.15.76:0/3146161967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e641011b0 0x7f3e64101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:21.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.355+0000 7f3e6af6a700 1 -- 172.21.15.76:0/3146161967 >> 172.21.15.76:0/3146161967 conn(0x7f3e640781e0 msgr2=0x7f3e640785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:21.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.355+0000 7f3e6af6a700 1 -- 172.21.15.76:0/3146161967 shutdown_connections 2024-03-18T14:52:21.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.355+0000 7f3e6af6a700 1 -- 172.21.15.76:0/3146161967 wait complete. 2024-03-18T14:52:21.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.356+0000 7f3e6af6a700 1 Processor -- start 2024-03-18T14:52:21.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.356+0000 7f3e6af6a700 1 -- start start 2024-03-18T14:52:21.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.356+0000 7f3e6af6a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e641011b0 0x7f3e641a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:21.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.356+0000 7f3e6af6a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e64101b70 0x7f3e641a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:21.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.356+0000 7f3e6af6a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e64107260 0x7f3e6419a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:21.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e6af6a700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3e641125f0 con 0x7f3e64107260 2024-03-18T14:52:21.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e6af6a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3e64112470 con 0x7f3e64101b70 2024-03-18T14:52:21.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e6af6a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3e64112770 con 0x7f3e641011b0 2024-03-18T14:52:21.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e63fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e64101b70 0x7f3e641a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:21.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e63fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e64101b70 0x7f3e641a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:37282/0 (socket says 172.21.15.76:37282) 2024-03-18T14:52:21.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e63fff700 1 -- 172.21.15.76:0/3394008838 learned_addr learned my addr 172.21.15.76:0/3394008838 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:21.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e69507700 1 --2- 172.21.15.76:0/3394008838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e64107260 0x7f3e6419a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:21.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e68d06700 1 --2- 172.21.15.76:0/3394008838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e641011b0 0x7f3e641a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:21.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e69507700 1 -- 172.21.15.76:0/3394008838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e641011b0 msgr2=0x7f3e641a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:21.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e69507700 1 --2- 172.21.15.76:0/3394008838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e641011b0 0x7f3e641a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:21.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e69507700 1 -- 172.21.15.76:0/3394008838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e64101b70 msgr2=0x7f3e641a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:21.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e69507700 1 --2- 172.21.15.76:0/3394008838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e64101b70 0x7f3e641a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:21.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e69507700 1 -- 172.21.15.76:0/3394008838 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e6419b0f0 con 0x7f3e64107260 2024-03-18T14:52:21.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e63fff700 1 --2- 172.21.15.76:0/3394008838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e64101b70 0x7f3e641a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:21.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e68d06700 1 --2- 172.21.15.76:0/3394008838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e641011b0 0x7f3e641a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:52:21.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e69507700 1 --2- 172.21.15.76:0/3394008838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e64107260 0x7f3e6419a990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c015000 tx=0x7f3e5c004f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:21.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e61ffb700 1 -- 172.21.15.76:0/3394008838 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c021070 con 0x7f3e64107260 2024-03-18T14:52:21.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.357+0000 7f3e6af6a700 1 -- 172.21.15.76:0/3394008838 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e6419b380 con 0x7f3e64107260 2024-03-18T14:52:21.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.358+0000 7f3e61ffb700 1 -- 172.21.15.76:0/3394008838 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e5c007ac0 con 0x7f3e64107260 2024-03-18T14:52:21.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.358+0000 7f3e61ffb700 1 -- 172.21.15.76:0/3394008838 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c01de60 con 0x7f3e64107260 2024-03-18T14:52:21.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.358+0000 7f3e6af6a700 1 -- 172.21.15.76:0/3394008838 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e641a76a0 con 0x7f3e64107260 2024-03-18T14:52:21.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.359+0000 7f3e6af6a700 1 -- 172.21.15.76:0/3394008838 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e64068f90 con 0x7f3e64107260 2024-03-18T14:52:21.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.360+0000 7f3e61ffb700 1 -- 172.21.15.76:0/3394008838 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f3e5c024070 con 0x7f3e64107260 2024-03-18T14:52:21.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.361+0000 7f3e61ffb700 1 --2- 172.21.15.76:0/3394008838 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3e540a2920 0x7f3e540a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:21.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.361+0000 7f3e68d06700 1 --2- 172.21.15.76:0/3394008838 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3e540a2920 0x7f3e540a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:21.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.362+0000 7f3e61ffb700 1 -- 172.21.15.76:0/3394008838 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f3e5c00f020 con 0x7f3e64107260 2024-03-18T14:52:21.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.362+0000 7f3e68d06700 1 --2- 172.21.15.76:0/3394008838 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3e540a2920 0x7f3e540a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3e6419bad0 tx=0x7f3e58009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:21.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.366+0000 7f3e61ffb700 1 -- 172.21.15.76:0/3394008838 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3e5c02b020 con 0x7f3e64107260 2024-03-18T14:52:21.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.580+0000 7f3e6af6a700 1 -- 172.21.15.76:0/3394008838 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3e64100370 con 0x7f3e540a2920 2024-03-18T14:52:21.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.588+0000 7f3e61ffb700 1 -- 172.21.15.76:0/3394008838 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258950 (secure 0 0 0) 0x7f3e64100370 con 0x7f3e540a2920 2024-03-18T14:52:21.593 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:21.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.591+0000 7f3e6af6a700 1 -- 172.21.15.76:0/3394008838 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3e540a2920 msgr2=0x7f3e540a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:21.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.591+0000 7f3e6af6a700 1 --2- 172.21.15.76:0/3394008838 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3e540a2920 0x7f3e540a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3e6419bad0 tx=0x7f3e58009260 comp rx=0 tx=0).stop 2024-03-18T14:52:21.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.591+0000 7f3e6af6a700 1 -- 172.21.15.76:0/3394008838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e64107260 msgr2=0x7f3e6419a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:21.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.591+0000 7f3e6af6a700 1 --2- 172.21.15.76:0/3394008838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e64107260 0x7f3e6419a990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c015000 tx=0x7f3e5c004f00 comp rx=0 tx=0).stop 2024-03-18T14:52:21.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.591+0000 7f3e6af6a700 1 -- 172.21.15.76:0/3394008838 shutdown_connections 2024-03-18T14:52:21.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.591+0000 7f3e6af6a700 1 --2- 172.21.15.76:0/3394008838 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3e540a2920 0x7f3e540a4de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:21.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.591+0000 7f3e6af6a700 1 --2- 172.21.15.76:0/3394008838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e64107260 0x7f3e6419a990 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:21.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.591+0000 7f3e6af6a700 1 --2- 172.21.15.76:0/3394008838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e641011b0 0x7f3e641a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:21.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.591+0000 7f3e6af6a700 1 --2- 172.21.15.76:0/3394008838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e64101b70 0x7f3e641a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:21.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.591+0000 7f3e6af6a700 1 -- 172.21.15.76:0/3394008838 >> 172.21.15.76:0/3394008838 conn(0x7f3e640781e0 msgr2=0x7f3e640ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:21.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.591+0000 7f3e6af6a700 1 -- 172.21.15.76:0/3394008838 shutdown_connections 2024-03-18T14:52:21.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:21.591+0000 7f3e6af6a700 1 -- 172.21.15.76:0/3394008838 wait complete. 2024-03-18T14:52:21.595 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-03-18T14:52:21.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:21 smithi076 ceph-mon[32205]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 4.8 KiB/s, 0 keys/s, 0 objects/s recovering 2024-03-18T14:52:21.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:21 smithi174 ceph-mon[36787]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 4.8 KiB/s, 0 keys/s, 0 objects/s recovering 2024-03-18T14:52:21.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:21 smithi077 ceph-mon[35865]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 4.8 KiB/s, 0 keys/s, 0 objects/s recovering 2024-03-18T14:52:22.303 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-03-18T14:52:20.348591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653515,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":575,"num_read_kb":406,"num_write":337,"num_write_kb":3045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1239648,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":272,"ondisk_log_size":272,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369168,"kb_used_data":11612,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572656,"statfs":{"total":1151940427776,"available":1151562399744,"internally_reserved":0,"allocated":11890688,"data_stored":7463493,"data_compressed":120345,"data_compressed_allocated":155648,"data_compressed_original":1843200,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":58912,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002830"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.367143+0000","last_change":"2024-03-18T14:51:01.365948+0000","last_active":"2024-03-18T14:52:07.367143+0000","last_peered":"2024-03-18T14:52:07.367143+0000","last_clean":"2024-03-18T14:52:07.367143+0000","last_became_active":"2024-03-18T14:51:01.364600+0000","last_became_peered":"2024-03-18T14:51:01.364600+0000","last_unstale":"2024-03-18T14:52:07.367143+0000","last_undegraded":"2024-03-18T14:52:07.367143+0000","last_fullsized":"2024-03-18T14:52:07.367143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:14:45.198618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366393+0000","last_change":"2024-03-18T14:51:01.360186+0000","last_active":"2024-03-18T14:52:07.366393+0000","last_peered":"2024-03-18T14:52:07.366393+0000","last_clean":"2024-03-18T14:52:07.366393+0000","last_became_active":"2024-03-18T14:51:01.359862+0000","last_became_peered":"2024-03-18T14:51:01.359862+0000","last_unstale":"2024-03-18T14:52:07.366393+0000","last_undegraded":"2024-03-18T14:52:07.366393+0000","last_fullsized":"2024-03-18T14:52:07.366393+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:46.206823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366284+0000","last_change":"2024-03-18T14:51:01.364635+0000","last_active":"2024-03-18T14:52:07.366284+0000","last_peered":"2024-03-18T14:52:07.366284+0000","last_clean":"2024-03-18T14:52:07.366284+0000","last_became_active":"2024-03-18T14:51:01.364320+0000","last_became_peered":"2024-03-18T14:51:01.364320+0000","last_unstale":"2024-03-18T14:52:07.366284+0000","last_undegraded":"2024-03-18T14:52:07.366284+0000","last_fullsized":"2024-03-18T14:52:07.366284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:08:20.227639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.372442+0000","last_change":"2024-03-18T14:52:07.372090+0000","last_active":"2024-03-18T14:52:07.372442+0000","last_peered":"2024-03-18T14:52:07.372442+0000","last_clean":"2024-03-18T14:52:07.372442+0000","last_became_active":"2024-03-18T14:52:07.371775+0000","last_became_peered":"2024-03-18T14:52:07.371775+0000","last_unstale":"2024-03-18T14:52:07.372442+0000","last_undegraded":"2024-03-18T14:52:07.372442+0000","last_fullsized":"2024-03-18T14:52:07.372442+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:48.128827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370622+0000","last_change":"2024-03-18T14:51:01.356839+0000","last_active":"2024-03-18T14:52:07.370622+0000","last_peered":"2024-03-18T14:52:07.370622+0000","last_clean":"2024-03-18T14:52:07.370622+0000","last_became_active":"2024-03-18T14:51:01.356531+0000","last_became_peered":"2024-03-18T14:51:01.356531+0000","last_unstale":"2024-03-18T14:52:07.370622+0000","last_undegraded":"2024-03-18T14:52:07.370622+0000","last_fullsized":"2024-03-18T14:52:07.370622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:28.499573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369212+0000","last_change":"2024-03-18T14:51:01.362912+0000","last_active":"2024-03-18T14:52:07.369212+0000","last_peered":"2024-03-18T14:52:07.369212+0000","last_clean":"2024-03-18T14:52:07.369212+0000","last_became_active":"2024-03-18T14:51:01.362590+0000","last_became_peered":"2024-03-18T14:51:01.362590+0000","last_unstale":"2024-03-18T14:52:07.369212+0000","last_undegraded":"2024-03-18T14:52:07.369212+0000","last_fullsized":"2024-03-18T14:52:07.369212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:49.199616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366574+0000","last_change":"2024-03-18T14:51:01.358652+0000","last_active":"2024-03-18T14:52:07.366574+0000","last_peered":"2024-03-18T14:52:07.366574+0000","last_clean":"2024-03-18T14:52:07.366574+0000","last_became_active":"2024-03-18T14:51:01.358312+0000","last_became_peered":"2024-03-18T14:51:01.358312+0000","last_unstale":"2024-03-18T14:52:07.366574+0000","last_undegraded":"2024-03-18T14:52:07.366574+0000","last_fullsized":"2024-03-18T14:52:07.366574+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:40.150240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368483+0000","last_change":"2024-03-18T14:51:01.360082+0000","last_active":"2024-03-18T14:52:07.368483+0000","last_peered":"2024-03-18T14:52:07.368483+0000","last_clean":"2024-03-18T14:52:07.368483+0000","last_became_active":"2024-03-18T14:51:01.359769+0000","last_became_peered":"2024-03-18T14:51:01.359769+0000","last_unstale":"2024-03-18T14:52:07.368483+0000","last_undegraded":"2024-03-18T14:52:07.368483+0000","last_fullsized":"2024-03-18T14:52:07.368483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:53:27.081546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.371122+0000","last_change":"2024-03-18T14:51:01.363947+0000","last_active":"2024-03-18T14:52:07.371122+0000","last_peered":"2024-03-18T14:52:07.371122+0000","last_clean":"2024-03-18T14:52:07.371122+0000","last_became_active":"2024-03-18T14:51:01.363106+0000","last_became_peered":"2024-03-18T14:51:01.363106+0000","last_unstale":"2024-03-18T14:52:07.371122+0000","last_undegraded":"2024-03-18T14:52:07.371122+0000","last_fullsized":"2024-03-18T14:52:07.371122+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:24:50.478426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369578+0000","last_change":"2024-03-18T14:51:01.357950+0000","last_active":"2024-03-18T14:52:07.369578+0000","last_peered":"2024-03-18T14:52:07.369578+0000","last_clean":"2024-03-18T14:52:07.369578+0000","last_became_active":"2024-03-18T14:51:01.357651+0000","last_became_peered":"2024-03-18T14:51:01.357651+0000","last_unstale":"2024-03-18T14:52:07.369578+0000","last_undegraded":"2024-03-18T14:52:07.369578+0000","last_fullsized":"2024-03-18T14:52:07.369578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:49:01.376008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366870+0000","last_change":"2024-03-18T14:51:01.360069+0000","last_active":"2024-03-18T14:52:07.366870+0000","last_peered":"2024-03-18T14:52:07.366870+0000","last_clean":"2024-03-18T14:52:07.366870+0000","last_became_active":"2024-03-18T14:51:01.358960+0000","last_became_peered":"2024-03-18T14:51:01.358960+0000","last_unstale":"2024-03-18T14:52:07.366870+0000","last_undegraded":"2024-03-18T14:52:07.366870+0000","last_fullsized":"2024-03-18T14:52:07.366870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:48:42.197329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.420023+0000","last_change":"2024-03-18T14:51:01.361200+0000","last_active":"2024-03-18T14:52:07.420023+0000","last_peered":"2024-03-18T14:52:07.420023+0000","last_clean":"2024-03-18T14:52:07.420023+0000","last_became_active":"2024-03-18T14:51:01.360879+0000","last_became_peered":"2024-03-18T14:51:01.360879+0000","last_unstale":"2024-03-18T14:52:07.420023+0000","last_undegraded":"2024-03-18T14:52:07.420023+0000","last_fullsized":"2024-03-18T14:52:07.420023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:05.335627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481365+0000","last_change":"2024-03-18T14:51:01.352441+0000","last_active":"2024-03-18T14:52:07.481365+0000","last_peered":"2024-03-18T14:52:07.481365+0000","last_clean":"2024-03-18T14:52:07.481365+0000","last_became_active":"2024-03-18T14:51:01.352118+0000","last_became_peered":"2024-03-18T14:51:01.352118+0000","last_unstale":"2024-03-18T14:52:07.481365+0000","last_undegraded":"2024-03-18T14:52:07.481365+0000","last_fullsized":"2024-03-18T14:52:07.481365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:45:42.781439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366198+0000","last_change":"2024-03-18T14:51:01.361257+0000","last_active":"2024-03-18T14:52:07.366198+0000","last_peered":"2024-03-18T14:52:07.366198+0000","last_clean":"2024-03-18T14:52:07.366198+0000","last_became_active":"2024-03-18T14:51:01.360898+0000","last_became_peered":"2024-03-18T14:51:01.360898+0000","last_unstale":"2024-03-18T14:52:07.366198+0000","last_undegraded":"2024-03-18T14:52:07.366198+0000","last_fullsized":"2024-03-18T14:52:07.366198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:38:26.200678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.374822+0000","last_change":"2024-03-18T14:52:07.374822+0000","last_active":"2024-03-18T14:52:07.374822+0000","last_peered":"2024-03-18T14:52:07.374822+0000","last_clean":"2024-03-18T14:52:07.374822+0000","last_became_active":"2024-03-18T14:52:07.374537+0000","last_became_peered":"2024-03-18T14:52:07.374537+0000","last_unstale":"2024-03-18T14:52:07.374822+0000","last_undegraded":"2024-03-18T14:52:07.374822+0000","last_fullsized":"2024-03-18T14:52:07.374822+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:57:29.971366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370713+0000","last_change":"2024-03-18T14:51:01.357223+0000","last_active":"2024-03-18T14:52:07.370713+0000","last_peered":"2024-03-18T14:52:07.370713+0000","last_clean":"2024-03-18T14:52:07.370713+0000","last_became_active":"2024-03-18T14:51:01.356934+0000","last_became_peered":"2024-03-18T14:51:01.356934+0000","last_unstale":"2024-03-18T14:52:07.370713+0000","last_undegraded":"2024-03-18T14:52:07.370713+0000","last_fullsized":"2024-03-18T14:52:07.370713+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:28.683605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.367323+0000","last_change":"2024-03-18T14:51:01.366414+0000","last_active":"2024-03-18T14:52:07.367323+0000","last_peered":"2024-03-18T14:52:07.367323+0000","last_clean":"2024-03-18T14:52:07.367323+0000","last_became_active":"2024-03-18T14:51:01.365534+0000","last_became_peered":"2024-03-18T14:51:01.365534+0000","last_unstale":"2024-03-18T14:52:07.367323+0000","last_undegraded":"2024-03-18T14:52:07.367323+0000","last_fullsized":"2024-03-18T14:52:07.367323+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:28:51.568662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.363976+0000","last_change":"2024-03-18T14:51:01.354084+0000","last_active":"2024-03-18T14:52:07.363976+0000","last_peered":"2024-03-18T14:52:07.363976+0000","last_clean":"2024-03-18T14:52:07.363976+0000","last_became_active":"2024-03-18T14:51:01.353599+0000","last_became_peered":"2024-03-18T14:51:01.353599+0000","last_unstale":"2024-03-18T14:52:07.363976+0000","last_undegraded":"2024-03-18T14:52:07.363976+0000","last_fullsized":"2024-03-18T14:52:07.363976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:40.502783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368760+0000","last_change":"2024-03-18T14:51:01.353391+0000","last_active":"2024-03-18T14:52:07.368760+0000","last_peered":"2024-03-18T14:52:07.368760+0000","last_clean":"2024-03-18T14:52:07.368760+0000","last_became_active":"2024-03-18T14:51:01.353084+0000","last_became_peered":"2024-03-18T14:51:01.353084+0000","last_unstale":"2024-03-18T14:52:07.368760+0000","last_undegraded":"2024-03-18T14:52:07.368760+0000","last_fullsized":"2024-03-18T14:52:07.368760+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:18:02.774589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.363884+0000","last_change":"2024-03-18T14:51:00.942617+0000","last_active":"2024-03-18T14:52:07.363884+0000","last_peered":"2024-03-18T14:52:07.363884+0000","last_clean":"2024-03-18T14:52:07.363884+0000","last_became_active":"2024-03-18T14:51:00.942276+0000","last_became_peered":"2024-03-18T14:51:00.942276+0000","last_unstale":"2024-03-18T14:52:07.363884+0000","last_undegraded":"2024-03-18T14:52:07.363884+0000","last_fullsized":"2024-03-18T14:52:07.363884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:03:05.984066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368670+0000","last_change":"2024-03-18T14:51:01.363887+0000","last_active":"2024-03-18T14:52:07.368670+0000","last_peered":"2024-03-18T14:52:07.368670+0000","last_clean":"2024-03-18T14:52:07.368670+0000","last_became_active":"2024-03-18T14:51:01.361359+0000","last_became_peered":"2024-03-18T14:51:01.361359+0000","last_unstale":"2024-03-18T14:52:07.368670+0000","last_undegraded":"2024-03-18T14:52:07.368670+0000","last_fullsized":"2024-03-18T14:52:07.368670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:22.996930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368879+0000","last_change":"2024-03-18T14:51:00.940392+0000","last_active":"2024-03-18T14:52:07.368879+0000","last_peered":"2024-03-18T14:52:07.368879+0000","last_clean":"2024-03-18T14:52:07.368879+0000","last_became_active":"2024-03-18T14:51:00.940055+0000","last_became_peered":"2024-03-18T14:51:00.940055+0000","last_unstale":"2024-03-18T14:52:07.368879+0000","last_undegraded":"2024-03-18T14:52:07.368879+0000","last_fullsized":"2024-03-18T14:52:07.368879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:57:28.523549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366493+0000","last_change":"2024-03-18T14:51:01.361458+0000","last_active":"2024-03-18T14:52:07.366493+0000","last_peered":"2024-03-18T14:52:07.366493+0000","last_clean":"2024-03-18T14:52:07.366493+0000","last_became_active":"2024-03-18T14:51:01.361105+0000","last_became_peered":"2024-03-18T14:51:01.361105+0000","last_unstale":"2024-03-18T14:52:07.366493+0000","last_undegraded":"2024-03-18T14:52:07.366493+0000","last_fullsized":"2024-03-18T14:52:07.366493+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:22:59.410080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.379071+0000","last_change":"2024-03-18T14:52:07.378642+0000","last_active":"2024-03-18T14:52:07.379071+0000","last_peered":"2024-03-18T14:52:07.379071+0000","last_clean":"2024-03-18T14:52:07.379071+0000","last_became_active":"2024-03-18T14:52:07.371947+0000","last_became_peered":"2024-03-18T14:52:07.371947+0000","last_unstale":"2024-03-18T14:52:07.379071+0000","last_undegraded":"2024-03-18T14:52:07.379071+0000","last_fullsized":"2024-03-18T14:52:07.379071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:46:00.898462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.371816+0000","last_change":"2024-03-18T14:51:01.366305+0000","last_active":"2024-03-18T14:52:07.371816+0000","last_peered":"2024-03-18T14:52:07.371816+0000","last_clean":"2024-03-18T14:52:07.371816+0000","last_became_active":"2024-03-18T14:51:01.364626+0000","last_became_peered":"2024-03-18T14:51:01.364626+0000","last_unstale":"2024-03-18T14:52:07.371816+0000","last_undegraded":"2024-03-18T14:52:07.371816+0000","last_fullsized":"2024-03-18T14:52:07.371816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:06.212030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370835+0000","last_change":"2024-03-18T14:51:00.932514+0000","last_active":"2024-03-18T14:52:07.370835+0000","last_peered":"2024-03-18T14:52:07.370835+0000","last_clean":"2024-03-18T14:52:07.370835+0000","last_became_active":"2024-03-18T14:51:00.932183+0000","last_became_peered":"2024-03-18T14:51:00.932183+0000","last_unstale":"2024-03-18T14:52:07.370835+0000","last_undegraded":"2024-03-18T14:52:07.370835+0000","last_fullsized":"2024-03-18T14:52:07.370835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:36:37.230185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.424080+0000","last_change":"2024-03-18T14:51:01.358939+0000","last_active":"2024-03-18T14:52:07.424080+0000","last_peered":"2024-03-18T14:52:07.424080+0000","last_clean":"2024-03-18T14:52:07.424080+0000","last_became_active":"2024-03-18T14:51:01.358601+0000","last_became_peered":"2024-03-18T14:51:01.358601+0000","last_unstale":"2024-03-18T14:52:07.424080+0000","last_undegraded":"2024-03-18T14:52:07.424080+0000","last_fullsized":"2024-03-18T14:52:07.424080+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:40:14.876143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":95,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:14.107620+0000","last_change":"2024-03-18T14:51:00.945116+0000","last_active":"2024-03-18T14:52:14.107620+0000","last_peered":"2024-03-18T14:52:14.107620+0000","last_clean":"2024-03-18T14:52:14.107620+0000","last_became_active":"2024-03-18T14:51:00.944791+0000","last_became_peered":"2024-03-18T14:51:00.944791+0000","last_unstale":"2024-03-18T14:52:14.107620+0000","last_undegraded":"2024-03-18T14:52:14.107620+0000","last_fullsized":"2024-03-18T14:52:14.107620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:06.637940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370715+0000","last_change":"2024-03-18T14:51:01.358682+0000","last_active":"2024-03-18T14:52:07.370715+0000","last_peered":"2024-03-18T14:52:07.370715+0000","last_clean":"2024-03-18T14:52:07.370715+0000","last_became_active":"2024-03-18T14:51:01.358378+0000","last_became_peered":"2024-03-18T14:51:01.358378+0000","last_unstale":"2024-03-18T14:52:07.370715+0000","last_undegraded":"2024-03-18T14:52:07.370715+0000","last_fullsized":"2024-03-18T14:52:07.370715+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:42.697944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369158+0000","last_change":"2024-03-18T14:51:00.943976+0000","last_active":"2024-03-18T14:52:07.369158+0000","last_peered":"2024-03-18T14:52:07.369158+0000","last_clean":"2024-03-18T14:52:07.369158+0000","last_became_active":"2024-03-18T14:51:00.943614+0000","last_became_peered":"2024-03-18T14:51:00.943614+0000","last_unstale":"2024-03-18T14:52:07.369158+0000","last_undegraded":"2024-03-18T14:52:07.369158+0000","last_fullsized":"2024-03-18T14:52:07.369158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:07:00.744708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423803+0000","last_change":"2024-03-18T14:51:01.365316+0000","last_active":"2024-03-18T14:52:07.423803+0000","last_peered":"2024-03-18T14:52:07.423803+0000","last_clean":"2024-03-18T14:52:07.423803+0000","last_became_active":"2024-03-18T14:51:01.365019+0000","last_became_peered":"2024-03-18T14:51:01.365019+0000","last_unstale":"2024-03-18T14:52:07.423803+0000","last_undegraded":"2024-03-18T14:52:07.423803+0000","last_fullsized":"2024-03-18T14:52:07.423803+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:39:59.040907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481239+0000","last_change":"2024-03-18T14:51:00.936923+0000","last_active":"2024-03-18T14:52:07.481239+0000","last_peered":"2024-03-18T14:52:07.481239+0000","last_clean":"2024-03-18T14:52:07.481239+0000","last_became_active":"2024-03-18T14:51:00.936591+0000","last_became_peered":"2024-03-18T14:51:00.936591+0000","last_unstale":"2024-03-18T14:52:07.481239+0000","last_undegraded":"2024-03-18T14:52:07.481239+0000","last_fullsized":"2024-03-18T14:52:07.481239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:48:46.423033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368536+0000","last_change":"2024-03-18T14:51:01.356827+0000","last_active":"2024-03-18T14:52:07.368536+0000","last_peered":"2024-03-18T14:52:07.368536+0000","last_clean":"2024-03-18T14:52:07.368536+0000","last_became_active":"2024-03-18T14:51:01.356148+0000","last_became_peered":"2024-03-18T14:51:01.356148+0000","last_unstale":"2024-03-18T14:52:07.368536+0000","last_undegraded":"2024-03-18T14:52:07.368536+0000","last_fullsized":"2024-03-18T14:52:07.368536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:18.477890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423674+0000","last_change":"2024-03-18T14:51:00.934024+0000","last_active":"2024-03-18T14:52:07.423674+0000","last_peered":"2024-03-18T14:52:07.423674+0000","last_clean":"2024-03-18T14:52:07.423674+0000","last_became_active":"2024-03-18T14:51:00.933739+0000","last_became_peered":"2024-03-18T14:51:00.933739+0000","last_unstale":"2024-03-18T14:52:07.423674+0000","last_undegraded":"2024-03-18T14:52:07.423674+0000","last_fullsized":"2024-03-18T14:52:07.423674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:03.967260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.363735+0000","last_change":"2024-03-18T14:51:01.356519+0000","last_active":"2024-03-18T14:52:07.363735+0000","last_peered":"2024-03-18T14:52:07.363735+0000","last_clean":"2024-03-18T14:52:07.363735+0000","last_became_active":"2024-03-18T14:51:01.356193+0000","last_became_peered":"2024-03-18T14:51:01.356193+0000","last_unstale":"2024-03-18T14:52:07.363735+0000","last_undegraded":"2024-03-18T14:52:07.363735+0000","last_fullsized":"2024-03-18T14:52:07.363735+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:34:59.633241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":93,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:16.109029+0000","last_change":"2024-03-18T14:51:00.940189+0000","last_active":"2024-03-18T14:52:16.109029+0000","last_peered":"2024-03-18T14:52:16.109029+0000","last_clean":"2024-03-18T14:52:16.109029+0000","last_became_active":"2024-03-18T14:51:00.939820+0000","last_became_peered":"2024-03-18T14:51:00.939820+0000","last_unstale":"2024-03-18T14:52:16.109029+0000","last_undegraded":"2024-03-18T14:52:16.109029+0000","last_fullsized":"2024-03-18T14:52:16.109029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:44:09.749747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.371040+0000","last_change":"2024-03-18T14:51:01.355756+0000","last_active":"2024-03-18T14:52:07.371040+0000","last_peered":"2024-03-18T14:52:07.371040+0000","last_clean":"2024-03-18T14:52:07.371040+0000","last_became_active":"2024-03-18T14:51:01.352871+0000","last_became_peered":"2024-03-18T14:51:01.352871+0000","last_unstale":"2024-03-18T14:52:07.371040+0000","last_undegraded":"2024-03-18T14:52:07.371040+0000","last_fullsized":"2024-03-18T14:52:07.371040+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:20:26.533066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366850+0000","last_change":"2024-03-18T14:51:00.941908+0000","last_active":"2024-03-18T14:52:07.366850+0000","last_peered":"2024-03-18T14:52:07.366850+0000","last_clean":"2024-03-18T14:52:07.366850+0000","last_became_active":"2024-03-18T14:51:00.940978+0000","last_became_peered":"2024-03-18T14:51:00.940978+0000","last_unstale":"2024-03-18T14:52:07.366850+0000","last_undegraded":"2024-03-18T14:52:07.366850+0000","last_fullsized":"2024-03-18T14:52:07.366850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:52.705243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366489+0000","last_change":"2024-03-18T14:51:01.363462+0000","last_active":"2024-03-18T14:52:07.366489+0000","last_peered":"2024-03-18T14:52:07.366489+0000","last_clean":"2024-03-18T14:52:07.366489+0000","last_became_active":"2024-03-18T14:51:01.362262+0000","last_became_peered":"2024-03-18T14:51:01.362262+0000","last_unstale":"2024-03-18T14:52:07.366489+0000","last_undegraded":"2024-03-18T14:52:07.366489+0000","last_fullsized":"2024-03-18T14:52:07.366489+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:07:18.952876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.424170+0000","last_change":"2024-03-18T14:51:00.936050+0000","last_active":"2024-03-18T14:52:07.424170+0000","last_peered":"2024-03-18T14:52:07.424170+0000","last_clean":"2024-03-18T14:52:07.424170+0000","last_became_active":"2024-03-18T14:51:00.935676+0000","last_became_peered":"2024-03-18T14:51:00.935676+0000","last_unstale":"2024-03-18T14:52:07.424170+0000","last_undegraded":"2024-03-18T14:52:07.424170+0000","last_fullsized":"2024-03-18T14:52:07.424170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:29.423110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.371929+0000","last_change":"2024-03-18T14:51:01.364295+0000","last_active":"2024-03-18T14:52:07.371929+0000","last_peered":"2024-03-18T14:52:07.371929+0000","last_clean":"2024-03-18T14:52:07.371929+0000","last_became_active":"2024-03-18T14:51:01.363916+0000","last_became_peered":"2024-03-18T14:51:01.363916+0000","last_unstale":"2024-03-18T14:52:07.371929+0000","last_undegraded":"2024-03-18T14:52:07.371929+0000","last_fullsized":"2024-03-18T14:52:07.371929+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:21:17.905819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'26","reported_seq":209,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:16.432827+0000","last_change":"2024-03-18T14:52:07.427262+0000","last_active":"2024-03-18T14:52:16.432827+0000","last_peered":"2024-03-18T14:52:16.432827+0000","last_clean":"2024-03-18T14:52:16.432827+0000","last_became_active":"2024-03-18T14:52:07.421388+0000","last_became_peered":"2024-03-18T14:52:07.421388+0000","last_unstale":"2024-03-18T14:52:16.432827+0000","last_undegraded":"2024-03-18T14:52:16.432827+0000","last_fullsized":"2024-03-18T14:52:16.432827+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:28.159787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58912,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":142,"num_read_kb":128,"num_write":27,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":58912,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370722+0000","last_change":"2024-03-18T14:51:01.362426+0000","last_active":"2024-03-18T14:52:07.370722+0000","last_peered":"2024-03-18T14:52:07.370722+0000","last_clean":"2024-03-18T14:52:07.370722+0000","last_became_active":"2024-03-18T14:51:01.362130+0000","last_became_peered":"2024-03-18T14:51:01.362130+0000","last_unstale":"2024-03-18T14:52:07.370722+0000","last_undegraded":"2024-03-18T14:52:07.370722+0000","last_fullsized":"2024-03-18T14:52:07.370722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:14.545997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368774+0000","last_change":"2024-03-18T14:51:00.934028+0000","last_active":"2024-03-18T14:52:07.368774+0000","last_peered":"2024-03-18T14:52:07.368774+0000","last_clean":"2024-03-18T14:52:07.368774+0000","last_became_active":"2024-03-18T14:51:00.933594+0000","last_became_peered":"2024-03-18T14:51:00.933594+0000","last_unstale":"2024-03-18T14:52:07.368774+0000","last_undegraded":"2024-03-18T14:52:07.368774+0000","last_fullsized":"2024-03-18T14:52:07.368774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:55:08.675533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481326+0000","last_change":"2024-03-18T14:51:01.353544+0000","last_active":"2024-03-18T14:52:07.481326+0000","last_peered":"2024-03-18T14:52:07.481326+0000","last_clean":"2024-03-18T14:52:07.481326+0000","last_became_active":"2024-03-18T14:51:01.353241+0000","last_became_peered":"2024-03-18T14:51:01.353241+0000","last_unstale":"2024-03-18T14:52:07.481326+0000","last_undegraded":"2024-03-18T14:52:07.481326+0000","last_fullsized":"2024-03-18T14:52:07.481326+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:45:27.617681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423792+0000","last_change":"2024-03-18T14:51:00.944492+0000","last_active":"2024-03-18T14:52:07.423792+0000","last_peered":"2024-03-18T14:52:07.423792+0000","last_clean":"2024-03-18T14:52:07.423792+0000","last_became_active":"2024-03-18T14:51:00.944165+0000","last_became_peered":"2024-03-18T14:51:00.944165+0000","last_unstale":"2024-03-18T14:52:07.423792+0000","last_undegraded":"2024-03-18T14:52:07.423792+0000","last_fullsized":"2024-03-18T14:52:07.423792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:47:05.968966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366823+0000","last_change":"2024-03-18T14:51:01.356782+0000","last_active":"2024-03-18T14:52:07.366823+0000","last_peered":"2024-03-18T14:52:07.366823+0000","last_clean":"2024-03-18T14:52:07.366823+0000","last_became_active":"2024-03-18T14:51:01.356242+0000","last_became_peered":"2024-03-18T14:51:01.356242+0000","last_unstale":"2024-03-18T14:52:07.366823+0000","last_undegraded":"2024-03-18T14:52:07.366823+0000","last_fullsized":"2024-03-18T14:52:07.366823+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:56:07.662545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":95,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:14.112951+0000","last_change":"2024-03-18T14:51:00.937873+0000","last_active":"2024-03-18T14:52:14.112951+0000","last_peered":"2024-03-18T14:52:14.112951+0000","last_clean":"2024-03-18T14:52:14.112951+0000","last_became_active":"2024-03-18T14:51:00.937562+0000","last_became_peered":"2024-03-18T14:51:00.937562+0000","last_unstale":"2024-03-18T14:52:14.112951+0000","last_undegraded":"2024-03-18T14:52:14.112951+0000","last_fullsized":"2024-03-18T14:52:14.112951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:36:44.342414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366508+0000","last_change":"2024-03-18T14:51:01.358681+0000","last_active":"2024-03-18T14:52:07.366508+0000","last_peered":"2024-03-18T14:52:07.366508+0000","last_clean":"2024-03-18T14:52:07.366508+0000","last_became_active":"2024-03-18T14:51:01.358374+0000","last_became_peered":"2024-03-18T14:51:01.358374+0000","last_unstale":"2024-03-18T14:52:07.366508+0000","last_undegraded":"2024-03-18T14:52:07.366508+0000","last_fullsized":"2024-03-18T14:52:07.366508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:43:47.804271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370451+0000","last_change":"2024-03-18T14:51:00.935639+0000","last_active":"2024-03-18T14:52:07.370451+0000","last_peered":"2024-03-18T14:52:07.370451+0000","last_clean":"2024-03-18T14:52:07.370451+0000","last_became_active":"2024-03-18T14:51:00.935334+0000","last_became_peered":"2024-03-18T14:51:00.935334+0000","last_unstale":"2024-03-18T14:52:07.370451+0000","last_undegraded":"2024-03-18T14:52:07.370451+0000","last_fullsized":"2024-03-18T14:52:07.370451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:46.966812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423741+0000","last_change":"2024-03-18T14:51:01.364755+0000","last_active":"2024-03-18T14:52:07.423741+0000","last_peered":"2024-03-18T14:52:07.423741+0000","last_clean":"2024-03-18T14:52:07.423741+0000","last_became_active":"2024-03-18T14:51:01.364448+0000","last_became_peered":"2024-03-18T14:51:01.364448+0000","last_unstale":"2024-03-18T14:52:07.423741+0000","last_undegraded":"2024-03-18T14:52:07.423741+0000","last_fullsized":"2024-03-18T14:52:07.423741+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:37:49.480787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370995+0000","last_change":"2024-03-18T14:51:00.933271+0000","last_active":"2024-03-18T14:52:07.370995+0000","last_peered":"2024-03-18T14:52:07.370995+0000","last_clean":"2024-03-18T14:52:07.370995+0000","last_became_active":"2024-03-18T14:51:00.932928+0000","last_became_peered":"2024-03-18T14:51:00.932928+0000","last_unstale":"2024-03-18T14:52:07.370995+0000","last_undegraded":"2024-03-18T14:52:07.370995+0000","last_fullsized":"2024-03-18T14:52:07.370995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:11:55.281763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.367160+0000","last_change":"2024-03-18T14:51:01.362911+0000","last_active":"2024-03-18T14:52:07.367160+0000","last_peered":"2024-03-18T14:52:07.367160+0000","last_clean":"2024-03-18T14:52:07.367160+0000","last_became_active":"2024-03-18T14:51:01.362598+0000","last_became_peered":"2024-03-18T14:51:01.362598+0000","last_unstale":"2024-03-18T14:52:07.367160+0000","last_undegraded":"2024-03-18T14:52:07.367160+0000","last_fullsized":"2024-03-18T14:52:07.367160+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:22:58.627753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.363811+0000","last_change":"2024-03-18T14:51:00.935554+0000","last_active":"2024-03-18T14:52:07.363811+0000","last_peered":"2024-03-18T14:52:07.363811+0000","last_clean":"2024-03-18T14:52:07.363811+0000","last_became_active":"2024-03-18T14:51:00.935184+0000","last_became_peered":"2024-03-18T14:51:00.935184+0000","last_unstale":"2024-03-18T14:52:07.363811+0000","last_undegraded":"2024-03-18T14:52:07.363811+0000","last_fullsized":"2024-03-18T14:52:07.363811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:21:13.973570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369358+0000","last_change":"2024-03-18T14:51:01.355990+0000","last_active":"2024-03-18T14:52:07.369358+0000","last_peered":"2024-03-18T14:52:07.369358+0000","last_clean":"2024-03-18T14:52:07.369358+0000","last_became_active":"2024-03-18T14:51:01.355692+0000","last_became_peered":"2024-03-18T14:51:01.355692+0000","last_unstale":"2024-03-18T14:52:07.369358+0000","last_undegraded":"2024-03-18T14:52:07.369358+0000","last_fullsized":"2024-03-18T14:52:07.369358+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:27:19.742684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366869+0000","last_change":"2024-03-18T14:51:00.943947+0000","last_active":"2024-03-18T14:52:07.366869+0000","last_peered":"2024-03-18T14:52:07.366869+0000","last_clean":"2024-03-18T14:52:07.366869+0000","last_became_active":"2024-03-18T14:51:00.943565+0000","last_became_peered":"2024-03-18T14:51:00.943565+0000","last_unstale":"2024-03-18T14:52:07.366869+0000","last_undegraded":"2024-03-18T14:52:07.366869+0000","last_fullsized":"2024-03-18T14:52:07.366869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:33:06.103148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369281+0000","last_change":"2024-03-18T14:51:01.363221+0000","last_active":"2024-03-18T14:52:07.369281+0000","last_peered":"2024-03-18T14:52:07.369281+0000","last_clean":"2024-03-18T14:52:07.369281+0000","last_became_active":"2024-03-18T14:51:01.362848+0000","last_became_peered":"2024-03-18T14:51:01.362848+0000","last_unstale":"2024-03-18T14:52:07.369281+0000","last_undegraded":"2024-03-18T14:52:07.369281+0000","last_fullsized":"2024-03-18T14:52:07.369281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:04.139365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423779+0000","last_change":"2024-03-18T14:51:01.356237+0000","last_active":"2024-03-18T14:52:07.423779+0000","last_peered":"2024-03-18T14:52:07.423779+0000","last_clean":"2024-03-18T14:52:07.423779+0000","last_became_active":"2024-03-18T14:51:01.355922+0000","last_became_peered":"2024-03-18T14:51:01.355922+0000","last_unstale":"2024-03-18T14:52:07.423779+0000","last_undegraded":"2024-03-18T14:52:07.423779+0000","last_fullsized":"2024-03-18T14:52:07.423779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:04:50.491752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":92,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:14.112616+0000","last_change":"2024-03-18T14:51:00.936058+0000","last_active":"2024-03-18T14:52:14.112616+0000","last_peered":"2024-03-18T14:52:14.112616+0000","last_clean":"2024-03-18T14:52:14.112616+0000","last_became_active":"2024-03-18T14:51:00.935731+0000","last_became_peered":"2024-03-18T14:51:00.935731+0000","last_unstale":"2024-03-18T14:52:14.112616+0000","last_undegraded":"2024-03-18T14:52:14.112616+0000","last_fullsized":"2024-03-18T14:52:14.112616+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:47:49.023943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368957+0000","last_change":"2024-03-18T14:51:01.359742+0000","last_active":"2024-03-18T14:52:07.368957+0000","last_peered":"2024-03-18T14:52:07.368957+0000","last_clean":"2024-03-18T14:52:07.368957+0000","last_became_active":"2024-03-18T14:51:01.358507+0000","last_became_peered":"2024-03-18T14:51:01.358507+0000","last_unstale":"2024-03-18T14:52:07.368957+0000","last_undegraded":"2024-03-18T14:52:07.368957+0000","last_fullsized":"2024-03-18T14:52:07.368957+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:17.129695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.419644+0000","last_change":"2024-03-18T14:51:00.934593+0000","last_active":"2024-03-18T14:52:07.419644+0000","last_peered":"2024-03-18T14:52:07.419644+0000","last_clean":"2024-03-18T14:52:07.419644+0000","last_became_active":"2024-03-18T14:51:00.934253+0000","last_became_peered":"2024-03-18T14:51:00.934253+0000","last_unstale":"2024-03-18T14:52:07.419644+0000","last_undegraded":"2024-03-18T14:52:07.419644+0000","last_fullsized":"2024-03-18T14:52:07.419644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:43:26.323162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":351,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366618+0000","last_change":"2024-03-18T14:49:51.158965+0000","last_active":"2024-03-18T14:52:07.366618+0000","last_peered":"2024-03-18T14:52:07.366618+0000","last_clean":"2024-03-18T14:52:07.366618+0000","last_became_active":"2024-03-18T14:49:51.158580+0000","last_became_peered":"2024-03-18T14:49:51.158580+0000","last_unstale":"2024-03-18T14:52:07.366618+0000","last_undegraded":"2024-03-18T14:52:07.366618+0000","last_fullsized":"2024-03-18T14:52:07.366618+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:46:27.210244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:46:27.210244+0000","last_clean_scrub_stamp":"2024-03-18T14:46:27.210244+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:37:08.104617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481115+0000","last_change":"2024-03-18T14:51:01.359972+0000","last_active":"2024-03-18T14:52:07.481115+0000","last_peered":"2024-03-18T14:52:07.481115+0000","last_clean":"2024-03-18T14:52:07.481115+0000","last_became_active":"2024-03-18T14:51:01.359430+0000","last_became_peered":"2024-03-18T14:51:01.359430+0000","last_unstale":"2024-03-18T14:52:07.481115+0000","last_undegraded":"2024-03-18T14:52:07.481115+0000","last_fullsized":"2024-03-18T14:52:07.481115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:00:14.099499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366722+0000","last_change":"2024-03-18T14:51:00.934196+0000","last_active":"2024-03-18T14:52:07.366722+0000","last_peered":"2024-03-18T14:52:07.366722+0000","last_clean":"2024-03-18T14:52:07.366722+0000","last_became_active":"2024-03-18T14:51:00.933875+0000","last_became_peered":"2024-03-18T14:51:00.933875+0000","last_unstale":"2024-03-18T14:52:07.366722+0000","last_undegraded":"2024-03-18T14:52:07.366722+0000","last_fullsized":"2024-03-18T14:52:07.366722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:56.797227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370796+0000","last_change":"2024-03-18T14:51:00.941862+0000","last_active":"2024-03-18T14:52:07.370796+0000","last_peered":"2024-03-18T14:52:07.370796+0000","last_clean":"2024-03-18T14:52:07.370796+0000","last_became_active":"2024-03-18T14:51:00.941536+0000","last_became_peered":"2024-03-18T14:51:00.941536+0000","last_unstale":"2024-03-18T14:52:07.370796+0000","last_undegraded":"2024-03-18T14:52:07.370796+0000","last_fullsized":"2024-03-18T14:52:07.370796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:56.364552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481330+0000","last_change":"2024-03-18T14:51:01.352404+0000","last_active":"2024-03-18T14:52:07.481330+0000","last_peered":"2024-03-18T14:52:07.481330+0000","last_clean":"2024-03-18T14:52:07.481330+0000","last_became_active":"2024-03-18T14:51:01.352077+0000","last_became_peered":"2024-03-18T14:51:01.352077+0000","last_unstale":"2024-03-18T14:52:07.481330+0000","last_undegraded":"2024-03-18T14:52:07.481330+0000","last_fullsized":"2024-03-18T14:52:07.481330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:30:38.050712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.364070+0000","last_change":"2024-03-18T14:51:00.935564+0000","last_active":"2024-03-18T14:52:07.364070+0000","last_peered":"2024-03-18T14:52:07.364070+0000","last_clean":"2024-03-18T14:52:07.364070+0000","last_became_active":"2024-03-18T14:51:00.935268+0000","last_became_peered":"2024-03-18T14:51:00.935268+0000","last_unstale":"2024-03-18T14:52:07.364070+0000","last_undegraded":"2024-03-18T14:52:07.364070+0000","last_fullsized":"2024-03-18T14:52:07.364070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:22:33.504184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368810+0000","last_change":"2024-03-18T14:51:01.356096+0000","last_active":"2024-03-18T14:52:07.368810+0000","last_peered":"2024-03-18T14:52:07.368810+0000","last_clean":"2024-03-18T14:52:07.368810+0000","last_became_active":"2024-03-18T14:51:01.355688+0000","last_became_peered":"2024-03-18T14:51:01.355688+0000","last_unstale":"2024-03-18T14:52:07.368810+0000","last_undegraded":"2024-03-18T14:52:07.368810+0000","last_fullsized":"2024-03-18T14:52:07.368810+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:04:38.678482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.419726+0000","last_change":"2024-03-18T14:51:00.942610+0000","last_active":"2024-03-18T14:52:07.419726+0000","last_peered":"2024-03-18T14:52:07.419726+0000","last_clean":"2024-03-18T14:52:07.419726+0000","last_became_active":"2024-03-18T14:51:00.942203+0000","last_became_peered":"2024-03-18T14:51:00.942203+0000","last_unstale":"2024-03-18T14:52:07.419726+0000","last_undegraded":"2024-03-18T14:52:07.419726+0000","last_fullsized":"2024-03-18T14:52:07.419726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:17:26.324282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366425+0000","last_change":"2024-03-18T14:51:01.362931+0000","last_active":"2024-03-18T14:52:07.366425+0000","last_peered":"2024-03-18T14:52:07.366425+0000","last_clean":"2024-03-18T14:52:07.366425+0000","last_became_active":"2024-03-18T14:51:01.362624+0000","last_became_peered":"2024-03-18T14:51:01.362624+0000","last_unstale":"2024-03-18T14:52:07.366425+0000","last_undegraded":"2024-03-18T14:52:07.366425+0000","last_fullsized":"2024-03-18T14:52:07.366425+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:40:52.512871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370722+0000","last_change":"2024-03-18T14:51:00.938971+0000","last_active":"2024-03-18T14:52:07.370722+0000","last_peered":"2024-03-18T14:52:07.370722+0000","last_clean":"2024-03-18T14:52:07.370722+0000","last_became_active":"2024-03-18T14:51:00.938645+0000","last_became_peered":"2024-03-18T14:51:00.938645+0000","last_unstale":"2024-03-18T14:52:07.370722+0000","last_undegraded":"2024-03-18T14:52:07.370722+0000","last_fullsized":"2024-03-18T14:52:07.370722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:18:54.999927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370470+0000","last_change":"2024-03-18T14:51:01.359510+0000","last_active":"2024-03-18T14:52:07.370470+0000","last_peered":"2024-03-18T14:52:07.370470+0000","last_clean":"2024-03-18T14:52:07.370470+0000","last_became_active":"2024-03-18T14:51:01.359192+0000","last_became_peered":"2024-03-18T14:51:01.359192+0000","last_unstale":"2024-03-18T14:52:07.370470+0000","last_undegraded":"2024-03-18T14:52:07.370470+0000","last_fullsized":"2024-03-18T14:52:07.370470+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:11:52.942825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368576+0000","last_change":"2024-03-18T14:51:00.940194+0000","last_active":"2024-03-18T14:52:07.368576+0000","last_peered":"2024-03-18T14:52:07.368576+0000","last_clean":"2024-03-18T14:52:07.368576+0000","last_became_active":"2024-03-18T14:51:00.939861+0000","last_became_peered":"2024-03-18T14:51:00.939861+0000","last_unstale":"2024-03-18T14:52:07.368576+0000","last_undegraded":"2024-03-18T14:52:07.368576+0000","last_fullsized":"2024-03-18T14:52:07.368576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:06:30.314102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366604+0000","last_change":"2024-03-18T14:51:01.355098+0000","last_active":"2024-03-18T14:52:07.366604+0000","last_peered":"2024-03-18T14:52:07.366604+0000","last_clean":"2024-03-18T14:52:07.366604+0000","last_became_active":"2024-03-18T14:51:01.354452+0000","last_became_peered":"2024-03-18T14:51:01.354452+0000","last_unstale":"2024-03-18T14:52:07.366604+0000","last_undegraded":"2024-03-18T14:52:07.366604+0000","last_fullsized":"2024-03-18T14:52:07.366604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:40:54.650294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.381529+0000","last_change":"2024-03-18T14:52:07.381159+0000","last_active":"2024-03-18T14:52:07.381529+0000","last_peered":"2024-03-18T14:52:07.381529+0000","last_clean":"2024-03-18T14:52:07.381529+0000","last_became_active":"2024-03-18T14:52:07.371943+0000","last_became_peered":"2024-03-18T14:52:07.371943+0000","last_unstale":"2024-03-18T14:52:07.381529+0000","last_undegraded":"2024-03-18T14:52:07.381529+0000","last_fullsized":"2024-03-18T14:52:07.381529+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:11.329239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368784+0000","last_change":"2024-03-18T14:51:01.357920+0000","last_active":"2024-03-18T14:52:07.368784+0000","last_peered":"2024-03-18T14:52:07.368784+0000","last_clean":"2024-03-18T14:52:07.368784+0000","last_became_active":"2024-03-18T14:51:01.357565+0000","last_became_peered":"2024-03-18T14:51:01.357565+0000","last_unstale":"2024-03-18T14:52:07.368784+0000","last_undegraded":"2024-03-18T14:52:07.368784+0000","last_fullsized":"2024-03-18T14:52:07.368784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:39:57.880178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370714+0000","last_change":"2024-03-18T14:51:00.937887+0000","last_active":"2024-03-18T14:52:07.370714+0000","last_peered":"2024-03-18T14:52:07.370714+0000","last_clean":"2024-03-18T14:52:07.370714+0000","last_became_active":"2024-03-18T14:51:00.937588+0000","last_became_peered":"2024-03-18T14:51:00.937588+0000","last_unstale":"2024-03-18T14:52:07.370714+0000","last_undegraded":"2024-03-18T14:52:07.370714+0000","last_fullsized":"2024-03-18T14:52:07.370714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:43:37.558412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366705+0000","last_change":"2024-03-18T14:51:01.354175+0000","last_active":"2024-03-18T14:52:07.366705+0000","last_peered":"2024-03-18T14:52:07.366705+0000","last_clean":"2024-03-18T14:52:07.366705+0000","last_became_active":"2024-03-18T14:51:01.353839+0000","last_became_peered":"2024-03-18T14:51:01.353839+0000","last_unstale":"2024-03-18T14:52:07.366705+0000","last_undegraded":"2024-03-18T14:52:07.366705+0000","last_fullsized":"2024-03-18T14:52:07.366705+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:21:52.372938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369167+0000","last_change":"2024-03-18T14:51:00.941600+0000","last_active":"2024-03-18T14:52:07.369167+0000","last_peered":"2024-03-18T14:52:07.369167+0000","last_clean":"2024-03-18T14:52:07.369167+0000","last_became_active":"2024-03-18T14:51:00.941291+0000","last_became_peered":"2024-03-18T14:51:00.941291+0000","last_unstale":"2024-03-18T14:52:07.369167+0000","last_undegraded":"2024-03-18T14:52:07.369167+0000","last_fullsized":"2024-03-18T14:52:07.369167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:04:11.142029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366985+0000","last_change":"2024-03-18T14:51:01.359058+0000","last_active":"2024-03-18T14:52:07.366985+0000","last_peered":"2024-03-18T14:52:07.366985+0000","last_clean":"2024-03-18T14:52:07.366985+0000","last_became_active":"2024-03-18T14:51:01.358437+0000","last_became_peered":"2024-03-18T14:51:01.358437+0000","last_unstale":"2024-03-18T14:52:07.366985+0000","last_undegraded":"2024-03-18T14:52:07.366985+0000","last_fullsized":"2024-03-18T14:52:07.366985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:39:58.945080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.424033+0000","last_change":"2024-03-18T14:51:00.933647+0000","last_active":"2024-03-18T14:52:07.424033+0000","last_peered":"2024-03-18T14:52:07.424033+0000","last_clean":"2024-03-18T14:52:07.424033+0000","last_became_active":"2024-03-18T14:51:00.933320+0000","last_became_peered":"2024-03-18T14:51:00.933320+0000","last_unstale":"2024-03-18T14:52:07.424033+0000","last_undegraded":"2024-03-18T14:52:07.424033+0000","last_fullsized":"2024-03-18T14:52:07.424033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:16:46.498845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366728+0000","last_change":"2024-03-18T14:51:01.359874+0000","last_active":"2024-03-18T14:52:07.366728+0000","last_peered":"2024-03-18T14:52:07.366728+0000","last_clean":"2024-03-18T14:52:07.366728+0000","last_became_active":"2024-03-18T14:51:01.359335+0000","last_became_peered":"2024-03-18T14:51:01.359335+0000","last_unstale":"2024-03-18T14:52:07.366728+0000","last_undegraded":"2024-03-18T14:52:07.366728+0000","last_fullsized":"2024-03-18T14:52:07.366728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:03.583402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481395+0000","last_change":"2024-03-18T14:51:00.935755+0000","last_active":"2024-03-18T14:52:07.481395+0000","last_peered":"2024-03-18T14:52:07.481395+0000","last_clean":"2024-03-18T14:52:07.481395+0000","last_became_active":"2024-03-18T14:51:00.935421+0000","last_became_peered":"2024-03-18T14:51:00.935421+0000","last_unstale":"2024-03-18T14:52:07.481395+0000","last_undegraded":"2024-03-18T14:52:07.481395+0000","last_fullsized":"2024-03-18T14:52:07.481395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:46:11.680972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.367029+0000","last_change":"2024-03-18T14:51:01.355605+0000","last_active":"2024-03-18T14:52:07.367029+0000","last_peered":"2024-03-18T14:52:07.367029+0000","last_clean":"2024-03-18T14:52:07.367029+0000","last_became_active":"2024-03-18T14:51:01.355295+0000","last_became_peered":"2024-03-18T14:51:01.355295+0000","last_unstale":"2024-03-18T14:52:07.367029+0000","last_undegraded":"2024-03-18T14:52:07.367029+0000","last_fullsized":"2024-03-18T14:52:07.367029+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:15:20.466314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366671+0000","last_change":"2024-03-18T14:51:01.359704+0000","last_active":"2024-03-18T14:52:07.366671+0000","last_peered":"2024-03-18T14:52:07.366671+0000","last_clean":"2024-03-18T14:52:07.366671+0000","last_became_active":"2024-03-18T14:51:01.359367+0000","last_became_peered":"2024-03-18T14:51:01.359367+0000","last_unstale":"2024-03-18T14:52:07.366671+0000","last_undegraded":"2024-03-18T14:52:07.366671+0000","last_fullsized":"2024-03-18T14:52:07.366671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:26.933993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.420506+0000","last_change":"2024-03-18T14:52:07.420506+0000","last_active":"2024-03-18T14:52:07.420506+0000","last_peered":"2024-03-18T14:52:07.420506+0000","last_clean":"2024-03-18T14:52:07.420506+0000","last_became_active":"2024-03-18T14:52:07.420254+0000","last_became_peered":"2024-03-18T14:52:07.420254+0000","last_unstale":"2024-03-18T14:52:07.420506+0000","last_undegraded":"2024-03-18T14:52:07.420506+0000","last_fullsized":"2024-03-18T14:52:07.420506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:32:08.702462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368828+0000","last_change":"2024-03-18T14:51:01.355678+0000","last_active":"2024-03-18T14:52:07.368828+0000","last_peered":"2024-03-18T14:52:07.368828+0000","last_clean":"2024-03-18T14:52:07.368828+0000","last_became_active":"2024-03-18T14:51:01.355197+0000","last_became_peered":"2024-03-18T14:51:01.355197+0000","last_unstale":"2024-03-18T14:52:07.368828+0000","last_undegraded":"2024-03-18T14:52:07.368828+0000","last_fullsized":"2024-03-18T14:52:07.368828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:34.194834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366340+0000","last_change":"2024-03-18T14:51:01.363964+0000","last_active":"2024-03-18T14:52:07.366340+0000","last_peered":"2024-03-18T14:52:07.366340+0000","last_clean":"2024-03-18T14:52:07.366340+0000","last_became_active":"2024-03-18T14:51:01.363590+0000","last_became_peered":"2024-03-18T14:51:01.363590+0000","last_unstale":"2024-03-18T14:52:07.366340+0000","last_undegraded":"2024-03-18T14:52:07.366340+0000","last_fullsized":"2024-03-18T14:52:07.366340+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:59.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481350+0000","last_change":"2024-03-18T14:51:01.361599+0000","last_active":"2024-03-18T14:52:07.481350+0000","last_peered":"2024-03-18T14:52:07.481350+0000","last_clean":"2024-03-18T14:52:07.481350+0000","last_became_active":"2024-03-18T14:51:01.361280+0000","last_became_peered":"2024-03-18T14:51:01.361280+0000","last_unstale":"2024-03-18T14:52:07.481350+0000","last_undegraded":"2024-03-18T14:52:07.481350+0000","last_fullsized":"2024-03-18T14:52:07.481350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:52:50.572257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.367005+0000","last_change":"2024-03-18T14:51:01.360495+0000","last_active":"2024-03-18T14:52:07.367005+0000","last_peered":"2024-03-18T14:52:07.367005+0000","last_clean":"2024-03-18T14:52:07.367005+0000","last_became_active":"2024-03-18T14:51:01.360146+0000","last_became_peered":"2024-03-18T14:51:01.360146+0000","last_unstale":"2024-03-18T14:52:07.367005+0000","last_undegraded":"2024-03-18T14:52:07.367005+0000","last_fullsized":"2024-03-18T14:52:07.367005+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:52.048137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368892+0000","last_change":"2024-03-18T14:51:01.357650+0000","last_active":"2024-03-18T14:52:07.368892+0000","last_peered":"2024-03-18T14:52:07.368892+0000","last_clean":"2024-03-18T14:52:07.368892+0000","last_became_active":"2024-03-18T14:51:01.356265+0000","last_became_peered":"2024-03-18T14:51:01.356265+0000","last_unstale":"2024-03-18T14:52:07.368892+0000","last_undegraded":"2024-03-18T14:52:07.368892+0000","last_fullsized":"2024-03-18T14:52:07.368892+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:49.328404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368727+0000","last_change":"2024-03-18T14:51:01.363753+0000","last_active":"2024-03-18T14:52:07.368727+0000","last_peered":"2024-03-18T14:52:07.368727+0000","last_clean":"2024-03-18T14:52:07.368727+0000","last_became_active":"2024-03-18T14:51:01.363411+0000","last_became_peered":"2024-03-18T14:51:01.363411+0000","last_unstale":"2024-03-18T14:52:07.368727+0000","last_undegraded":"2024-03-18T14:52:07.368727+0000","last_fullsized":"2024-03-18T14:52:07.368727+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:55.452209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423670+0000","last_change":"2024-03-18T14:51:01.360723+0000","last_active":"2024-03-18T14:52:07.423670+0000","last_peered":"2024-03-18T14:52:07.423670+0000","last_clean":"2024-03-18T14:52:07.423670+0000","last_became_active":"2024-03-18T14:51:01.360130+0000","last_became_peered":"2024-03-18T14:51:01.360130+0000","last_unstale":"2024-03-18T14:52:07.423670+0000","last_undegraded":"2024-03-18T14:52:07.423670+0000","last_fullsized":"2024-03-18T14:52:07.423670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:51.634011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368533+0000","last_change":"2024-03-18T14:51:01.356462+0000","last_active":"2024-03-18T14:52:07.368533+0000","last_peered":"2024-03-18T14:52:07.368533+0000","last_clean":"2024-03-18T14:52:07.368533+0000","last_became_active":"2024-03-18T14:51:01.355395+0000","last_became_peered":"2024-03-18T14:51:01.355395+0000","last_unstale":"2024-03-18T14:52:07.368533+0000","last_undegraded":"2024-03-18T14:52:07.368533+0000","last_fullsized":"2024-03-18T14:52:07.368533+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:37:15.326354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370868+0000","last_change":"2024-03-18T14:51:01.358626+0000","last_active":"2024-03-18T14:52:07.370868+0000","last_peered":"2024-03-18T14:52:07.370868+0000","last_clean":"2024-03-18T14:52:07.370868+0000","last_became_active":"2024-03-18T14:51:01.358293+0000","last_became_peered":"2024-03-18T14:51:01.358293+0000","last_unstale":"2024-03-18T14:52:07.370868+0000","last_undegraded":"2024-03-18T14:52:07.370868+0000","last_fullsized":"2024-03-18T14:52:07.370868+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:52:41.139885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481309+0000","last_change":"2024-03-18T14:51:01.355276+0000","last_active":"2024-03-18T14:52:07.481309+0000","last_peered":"2024-03-18T14:52:07.481309+0000","last_clean":"2024-03-18T14:52:07.481309+0000","last_became_active":"2024-03-18T14:51:01.354661+0000","last_became_peered":"2024-03-18T14:51:01.354661+0000","last_unstale":"2024-03-18T14:52:07.481309+0000","last_undegraded":"2024-03-18T14:52:07.481309+0000","last_fullsized":"2024-03-18T14:52:07.481309+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:28.569055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.372297+0000","last_change":"2024-03-18T14:52:07.372297+0000","last_active":"2024-03-18T14:52:07.372297+0000","last_peered":"2024-03-18T14:52:07.372297+0000","last_clean":"2024-03-18T14:52:07.372297+0000","last_became_active":"2024-03-18T14:52:07.371918+0000","last_became_peered":"2024-03-18T14:52:07.371918+0000","last_unstale":"2024-03-18T14:52:07.372297+0000","last_undegraded":"2024-03-18T14:52:07.372297+0000","last_fullsized":"2024-03-18T14:52:07.372297+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:28:23.431373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280632,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62913,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":439,"num_read_kb":287,"num_write":154,"num_write_kb":105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":58912,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":188739,"data_compressed":13263,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":0,"internal_metadata":0},"log_size":133,"ondisk_log_size":133,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351005,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":950272,"data_stored":459161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612641,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712896,"statfs":{"total":95995035648,"available":95962005504,"internally_reserved":0,"allocated":1097728,"data_stored":1109945,"data_compressed":37553,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776167,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":459397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037804,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":459317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299440,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":1051247,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462965,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":458786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":517491,"data_compressed":1859,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":974848,"data_stored":460057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":459929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":970752,"data_stored":459097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888128,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":518526,"data_compressed":9545,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724602,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":1050540,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":58912,"data_compressed":1859,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":59947,"data_compressed":9545,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":60881,"data_compressed":1859,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:52:22.307 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T14:52:22.558 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:52:22.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:22.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:22 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:22.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:22 smithi076 ceph-mon[32205]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:52:22.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:22 smithi076 ceph-mon[32205]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 4.8 KiB/s, 0 objects/s recovering 2024-03-18T14:52:22.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:22 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:22.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:22 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:22.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:22 smithi174 ceph-mon[36787]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:52:22.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:22 smithi174 ceph-mon[36787]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 4.8 KiB/s, 0 objects/s recovering 2024-03-18T14:52:22.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:22 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:22.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:22 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:22.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:22 smithi077 ceph-mon[35865]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:52:22.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:22 smithi077 ceph-mon[35865]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 4.8 KiB/s, 0 objects/s recovering 2024-03-18T14:52:23.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:23 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:52:23.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:23 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:52:23.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:23 smithi076 ceph-mon[32205]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:23.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:23 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:52:23.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:23 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:52:23.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:23 smithi174 ceph-mon[36787]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:23 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:52:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:23 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:52:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:23 smithi077 ceph-mon[35865]: from='mgr.14152 172.21.15.76:0/1689095628' entity='mgr.x' 2024-03-18T14:52:24.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.290+0000 7f8c92d74700 1 -- 172.21.15.76:0/3455186000 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c8c107210 msgr2=0x7f8c8c1075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:24.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.290+0000 7f8c92d74700 1 --2- 172.21.15.76:0/3455186000 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c8c107210 0x7f8c8c1075f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8c8400b0a0 tx=0x7f8c8400b3b0 comp rx=0 tx=0).stop 2024-03-18T14:52:24.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.290+0000 7f8c92d74700 1 -- 172.21.15.76:0/3455186000 shutdown_connections 2024-03-18T14:52:24.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.290+0000 7f8c92d74700 1 --2- 172.21.15.76:0/3455186000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c8c101b70 0x7f8c8c10e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:24.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.290+0000 7f8c92d74700 1 --2- 172.21.15.76:0/3455186000 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c8c107210 0x7f8c8c1075f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:24.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.290+0000 7f8c92d74700 1 --2- 172.21.15.76:0/3455186000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c8c1011d0 0x7f8c8c101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:24.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.290+0000 7f8c92d74700 1 -- 172.21.15.76:0/3455186000 >> 172.21.15.76:0/3455186000 conn(0x7f8c8c078100 msgr2=0x7f8c8c078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:24.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.290+0000 7f8c92d74700 1 -- 172.21.15.76:0/3455186000 shutdown_connections 2024-03-18T14:52:24.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.290+0000 7f8c92d74700 1 -- 172.21.15.76:0/3455186000 wait complete. 2024-03-18T14:52:24.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.290+0000 7f8c92d74700 1 Processor -- start 2024-03-18T14:52:24.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c92d74700 1 -- start start 2024-03-18T14:52:24.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c92d74700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c8c1011d0 0x7f8c8c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:24.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c92d74700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c8c101b70 0x7f8c8c19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:24.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c92d74700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c8c107210 0x7f8c8c198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:24.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c92d74700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8c111470 con 0x7f8c8c101b70 2024-03-18T14:52:24.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c92d74700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8c1112f0 con 0x7f8c8c107210 2024-03-18T14:52:24.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c92d74700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8c1115f0 con 0x7f8c8c1011d0 2024-03-18T14:52:24.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c8bfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c8c101b70 0x7f8c8c19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:24.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c8bfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c8c101b70 0x7f8c8c19ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45622/0 (socket says 172.21.15.76:45622) 2024-03-18T14:52:24.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c8bfff700 1 -- 172.21.15.76:0/2935398854 learned_addr learned my addr 172.21.15.76:0/2935398854 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:24.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c91311700 1 --2- 172.21.15.76:0/2935398854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c8c107210 0x7f8c8c198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:24.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c90b10700 1 --2- 172.21.15.76:0/2935398854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c8c1011d0 0x7f8c8c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:24.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c8bfff700 1 -- 172.21.15.76:0/2935398854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c8c1011d0 msgr2=0x7f8c8c19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:24.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c8bfff700 1 --2- 172.21.15.76:0/2935398854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c8c1011d0 0x7f8c8c19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:24.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c8bfff700 1 -- 172.21.15.76:0/2935398854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c8c107210 msgr2=0x7f8c8c198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:24.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c8bfff700 1 --2- 172.21.15.76:0/2935398854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c8c107210 0x7f8c8c198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:24.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c8bfff700 1 -- 172.21.15.76:0/2935398854 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c8c199030 con 0x7f8c8c101b70 2024-03-18T14:52:24.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c91311700 1 --2- 172.21.15.76:0/2935398854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c8c107210 0x7f8c8c198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:24.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.291+0000 7f8c8bfff700 1 --2- 172.21.15.76:0/2935398854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c8c101b70 0x7f8c8c19ec10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000e9c0 tx=0x7f8c8000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:24.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.292+0000 7f8c89ffb700 1 -- 172.21.15.76:0/2935398854 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8000cbd0 con 0x7f8c8c101b70 2024-03-18T14:52:24.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.292+0000 7f8c89ffb700 1 -- 172.21.15.76:0/2935398854 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c8000ee60 con 0x7f8c8c101b70 2024-03-18T14:52:24.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.292+0000 7f8c92d74700 1 -- 172.21.15.76:0/2935398854 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c8c199320 con 0x7f8c8c101b70 2024-03-18T14:52:24.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.292+0000 7f8c89ffb700 1 -- 172.21.15.76:0/2935398854 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c80005630 con 0x7f8c8c101b70 2024-03-18T14:52:24.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.292+0000 7f8c92d74700 1 -- 172.21.15.76:0/2935398854 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c8c1a5530 con 0x7f8c8c101b70 2024-03-18T14:52:24.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.293+0000 7f8c92d74700 1 -- 172.21.15.76:0/2935398854 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c8c1028c0 con 0x7f8c8c101b70 2024-03-18T14:52:24.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.294+0000 7f8c89ffb700 1 -- 172.21.15.76:0/2935398854 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8c80004750 con 0x7f8c8c101b70 2024-03-18T14:52:24.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.294+0000 7f8c89ffb700 1 --2- 172.21.15.76:0/2935398854 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8c740a2920 0x7f8c740a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:24.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.294+0000 7f8c90b10700 1 --2- 172.21.15.76:0/2935398854 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8c740a2920 0x7f8c740a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:24.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.295+0000 7f8c89ffb700 1 -- 172.21.15.76:0/2935398854 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f8c800cee40 con 0x7f8c8c101b70 2024-03-18T14:52:24.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.295+0000 7f8c90b10700 1 --2- 172.21.15.76:0/2935398854 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8c740a2920 0x7f8c740a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8c84009e60 tx=0x7f8c8400b700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:24.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.299+0000 7f8c89ffb700 1 -- 172.21.15.76:0/2935398854 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8c8009bdb0 con 0x7f8c8c101b70 2024-03-18T14:52:24.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.490+0000 7f8c92d74700 1 -- 172.21.15.76:0/2935398854 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8c8c104fc0 con 0x7f8c740a2920 2024-03-18T14:52:24.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.497+0000 7f8c89ffb700 1 -- 172.21.15.76:0/2935398854 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258947 (secure 0 0 0) 0x7f8c8c104fc0 con 0x7f8c740a2920 2024-03-18T14:52:24.501 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:24.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.500+0000 7f8c92d74700 1 -- 172.21.15.76:0/2935398854 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8c740a2920 msgr2=0x7f8c740a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:24.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.500+0000 7f8c92d74700 1 --2- 172.21.15.76:0/2935398854 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8c740a2920 0x7f8c740a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8c84009e60 tx=0x7f8c8400b700 comp rx=0 tx=0).stop 2024-03-18T14:52:24.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.500+0000 7f8c92d74700 1 -- 172.21.15.76:0/2935398854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c8c101b70 msgr2=0x7f8c8c19ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:24.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.500+0000 7f8c92d74700 1 --2- 172.21.15.76:0/2935398854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c8c101b70 0x7f8c8c19ec10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000e9c0 tx=0x7f8c8000ecd0 comp rx=0 tx=0).stop 2024-03-18T14:52:24.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.500+0000 7f8c92d74700 1 -- 172.21.15.76:0/2935398854 shutdown_connections 2024-03-18T14:52:24.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.500+0000 7f8c92d74700 1 --2- 172.21.15.76:0/2935398854 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8c740a2920 0x7f8c740a4de0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:24.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.500+0000 7f8c92d74700 1 --2- 172.21.15.76:0/2935398854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c8c107210 0x7f8c8c198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:24.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.500+0000 7f8c92d74700 1 --2- 172.21.15.76:0/2935398854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c8c1011d0 0x7f8c8c19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:24.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.500+0000 7f8c92d74700 1 --2- 172.21.15.76:0/2935398854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c8c101b70 0x7f8c8c19ec10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:24.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.500+0000 7f8c92d74700 1 -- 172.21.15.76:0/2935398854 >> 172.21.15.76:0/2935398854 conn(0x7f8c8c078100 msgr2=0x7f8c8c1004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:24.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.500+0000 7f8c92d74700 1 -- 172.21.15.76:0/2935398854 shutdown_connections 2024-03-18T14:52:24.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:24.500+0000 7f8c92d74700 1 -- 172.21.15.76:0/2935398854 wait complete. 2024-03-18T14:52:24.503 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-03-18T14:52:24.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:24 smithi076 ceph-mon[32205]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:24.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:24 smithi174 ceph-mon[36787]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:24.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:24 smithi077 ceph-mon[35865]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:25.147 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-03-18T14:52:24.349446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653515,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":605,"num_read_kb":421,"num_write":337,"num_write_kb":3045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1239648,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":272,"ondisk_log_size":272,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369168,"kb_used_data":11612,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572656,"statfs":{"total":1151940427776,"available":1151562399744,"internally_reserved":0,"allocated":11890688,"data_stored":7463493,"data_compressed":120345,"data_compressed_allocated":155648,"data_compressed_original":1843200,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002894"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.367143+0000","last_change":"2024-03-18T14:51:01.365948+0000","last_active":"2024-03-18T14:52:07.367143+0000","last_peered":"2024-03-18T14:52:07.367143+0000","last_clean":"2024-03-18T14:52:07.367143+0000","last_became_active":"2024-03-18T14:51:01.364600+0000","last_became_peered":"2024-03-18T14:51:01.364600+0000","last_unstale":"2024-03-18T14:52:07.367143+0000","last_undegraded":"2024-03-18T14:52:07.367143+0000","last_fullsized":"2024-03-18T14:52:07.367143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:14:45.198618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366393+0000","last_change":"2024-03-18T14:51:01.360186+0000","last_active":"2024-03-18T14:52:07.366393+0000","last_peered":"2024-03-18T14:52:07.366393+0000","last_clean":"2024-03-18T14:52:07.366393+0000","last_became_active":"2024-03-18T14:51:01.359862+0000","last_became_peered":"2024-03-18T14:51:01.359862+0000","last_unstale":"2024-03-18T14:52:07.366393+0000","last_undegraded":"2024-03-18T14:52:07.366393+0000","last_fullsized":"2024-03-18T14:52:07.366393+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:46.206823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366284+0000","last_change":"2024-03-18T14:51:01.364635+0000","last_active":"2024-03-18T14:52:07.366284+0000","last_peered":"2024-03-18T14:52:07.366284+0000","last_clean":"2024-03-18T14:52:07.366284+0000","last_became_active":"2024-03-18T14:51:01.364320+0000","last_became_peered":"2024-03-18T14:51:01.364320+0000","last_unstale":"2024-03-18T14:52:07.366284+0000","last_undegraded":"2024-03-18T14:52:07.366284+0000","last_fullsized":"2024-03-18T14:52:07.366284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:08:20.227639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.372442+0000","last_change":"2024-03-18T14:52:07.372090+0000","last_active":"2024-03-18T14:52:07.372442+0000","last_peered":"2024-03-18T14:52:07.372442+0000","last_clean":"2024-03-18T14:52:07.372442+0000","last_became_active":"2024-03-18T14:52:07.371775+0000","last_became_peered":"2024-03-18T14:52:07.371775+0000","last_unstale":"2024-03-18T14:52:07.372442+0000","last_undegraded":"2024-03-18T14:52:07.372442+0000","last_fullsized":"2024-03-18T14:52:07.372442+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:48.128827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370622+0000","last_change":"2024-03-18T14:51:01.356839+0000","last_active":"2024-03-18T14:52:07.370622+0000","last_peered":"2024-03-18T14:52:07.370622+0000","last_clean":"2024-03-18T14:52:07.370622+0000","last_became_active":"2024-03-18T14:51:01.356531+0000","last_became_peered":"2024-03-18T14:51:01.356531+0000","last_unstale":"2024-03-18T14:52:07.370622+0000","last_undegraded":"2024-03-18T14:52:07.370622+0000","last_fullsized":"2024-03-18T14:52:07.370622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:28.499573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369212+0000","last_change":"2024-03-18T14:51:01.362912+0000","last_active":"2024-03-18T14:52:07.369212+0000","last_peered":"2024-03-18T14:52:07.369212+0000","last_clean":"2024-03-18T14:52:07.369212+0000","last_became_active":"2024-03-18T14:51:01.362590+0000","last_became_peered":"2024-03-18T14:51:01.362590+0000","last_unstale":"2024-03-18T14:52:07.369212+0000","last_undegraded":"2024-03-18T14:52:07.369212+0000","last_fullsized":"2024-03-18T14:52:07.369212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:49.199616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366574+0000","last_change":"2024-03-18T14:51:01.358652+0000","last_active":"2024-03-18T14:52:07.366574+0000","last_peered":"2024-03-18T14:52:07.366574+0000","last_clean":"2024-03-18T14:52:07.366574+0000","last_became_active":"2024-03-18T14:51:01.358312+0000","last_became_peered":"2024-03-18T14:51:01.358312+0000","last_unstale":"2024-03-18T14:52:07.366574+0000","last_undegraded":"2024-03-18T14:52:07.366574+0000","last_fullsized":"2024-03-18T14:52:07.366574+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:40.150240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368483+0000","last_change":"2024-03-18T14:51:01.360082+0000","last_active":"2024-03-18T14:52:07.368483+0000","last_peered":"2024-03-18T14:52:07.368483+0000","last_clean":"2024-03-18T14:52:07.368483+0000","last_became_active":"2024-03-18T14:51:01.359769+0000","last_became_peered":"2024-03-18T14:51:01.359769+0000","last_unstale":"2024-03-18T14:52:07.368483+0000","last_undegraded":"2024-03-18T14:52:07.368483+0000","last_fullsized":"2024-03-18T14:52:07.368483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:53:27.081546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.371122+0000","last_change":"2024-03-18T14:51:01.363947+0000","last_active":"2024-03-18T14:52:07.371122+0000","last_peered":"2024-03-18T14:52:07.371122+0000","last_clean":"2024-03-18T14:52:07.371122+0000","last_became_active":"2024-03-18T14:51:01.363106+0000","last_became_peered":"2024-03-18T14:51:01.363106+0000","last_unstale":"2024-03-18T14:52:07.371122+0000","last_undegraded":"2024-03-18T14:52:07.371122+0000","last_fullsized":"2024-03-18T14:52:07.371122+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:24:50.478426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369578+0000","last_change":"2024-03-18T14:51:01.357950+0000","last_active":"2024-03-18T14:52:07.369578+0000","last_peered":"2024-03-18T14:52:07.369578+0000","last_clean":"2024-03-18T14:52:07.369578+0000","last_became_active":"2024-03-18T14:51:01.357651+0000","last_became_peered":"2024-03-18T14:51:01.357651+0000","last_unstale":"2024-03-18T14:52:07.369578+0000","last_undegraded":"2024-03-18T14:52:07.369578+0000","last_fullsized":"2024-03-18T14:52:07.369578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:49:01.376008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366870+0000","last_change":"2024-03-18T14:51:01.360069+0000","last_active":"2024-03-18T14:52:07.366870+0000","last_peered":"2024-03-18T14:52:07.366870+0000","last_clean":"2024-03-18T14:52:07.366870+0000","last_became_active":"2024-03-18T14:51:01.358960+0000","last_became_peered":"2024-03-18T14:51:01.358960+0000","last_unstale":"2024-03-18T14:52:07.366870+0000","last_undegraded":"2024-03-18T14:52:07.366870+0000","last_fullsized":"2024-03-18T14:52:07.366870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:48:42.197329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.420023+0000","last_change":"2024-03-18T14:51:01.361200+0000","last_active":"2024-03-18T14:52:07.420023+0000","last_peered":"2024-03-18T14:52:07.420023+0000","last_clean":"2024-03-18T14:52:07.420023+0000","last_became_active":"2024-03-18T14:51:01.360879+0000","last_became_peered":"2024-03-18T14:51:01.360879+0000","last_unstale":"2024-03-18T14:52:07.420023+0000","last_undegraded":"2024-03-18T14:52:07.420023+0000","last_fullsized":"2024-03-18T14:52:07.420023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:05.335627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481365+0000","last_change":"2024-03-18T14:51:01.352441+0000","last_active":"2024-03-18T14:52:07.481365+0000","last_peered":"2024-03-18T14:52:07.481365+0000","last_clean":"2024-03-18T14:52:07.481365+0000","last_became_active":"2024-03-18T14:51:01.352118+0000","last_became_peered":"2024-03-18T14:51:01.352118+0000","last_unstale":"2024-03-18T14:52:07.481365+0000","last_undegraded":"2024-03-18T14:52:07.481365+0000","last_fullsized":"2024-03-18T14:52:07.481365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:45:42.781439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366198+0000","last_change":"2024-03-18T14:51:01.361257+0000","last_active":"2024-03-18T14:52:07.366198+0000","last_peered":"2024-03-18T14:52:07.366198+0000","last_clean":"2024-03-18T14:52:07.366198+0000","last_became_active":"2024-03-18T14:51:01.360898+0000","last_became_peered":"2024-03-18T14:51:01.360898+0000","last_unstale":"2024-03-18T14:52:07.366198+0000","last_undegraded":"2024-03-18T14:52:07.366198+0000","last_fullsized":"2024-03-18T14:52:07.366198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:38:26.200678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.374822+0000","last_change":"2024-03-18T14:52:07.374822+0000","last_active":"2024-03-18T14:52:07.374822+0000","last_peered":"2024-03-18T14:52:07.374822+0000","last_clean":"2024-03-18T14:52:07.374822+0000","last_became_active":"2024-03-18T14:52:07.374537+0000","last_became_peered":"2024-03-18T14:52:07.374537+0000","last_unstale":"2024-03-18T14:52:07.374822+0000","last_undegraded":"2024-03-18T14:52:07.374822+0000","last_fullsized":"2024-03-18T14:52:07.374822+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:57:29.971366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370713+0000","last_change":"2024-03-18T14:51:01.357223+0000","last_active":"2024-03-18T14:52:07.370713+0000","last_peered":"2024-03-18T14:52:07.370713+0000","last_clean":"2024-03-18T14:52:07.370713+0000","last_became_active":"2024-03-18T14:51:01.356934+0000","last_became_peered":"2024-03-18T14:51:01.356934+0000","last_unstale":"2024-03-18T14:52:07.370713+0000","last_undegraded":"2024-03-18T14:52:07.370713+0000","last_fullsized":"2024-03-18T14:52:07.370713+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:28.683605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.367323+0000","last_change":"2024-03-18T14:51:01.366414+0000","last_active":"2024-03-18T14:52:07.367323+0000","last_peered":"2024-03-18T14:52:07.367323+0000","last_clean":"2024-03-18T14:52:07.367323+0000","last_became_active":"2024-03-18T14:51:01.365534+0000","last_became_peered":"2024-03-18T14:51:01.365534+0000","last_unstale":"2024-03-18T14:52:07.367323+0000","last_undegraded":"2024-03-18T14:52:07.367323+0000","last_fullsized":"2024-03-18T14:52:07.367323+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:28:51.568662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.363976+0000","last_change":"2024-03-18T14:51:01.354084+0000","last_active":"2024-03-18T14:52:07.363976+0000","last_peered":"2024-03-18T14:52:07.363976+0000","last_clean":"2024-03-18T14:52:07.363976+0000","last_became_active":"2024-03-18T14:51:01.353599+0000","last_became_peered":"2024-03-18T14:51:01.353599+0000","last_unstale":"2024-03-18T14:52:07.363976+0000","last_undegraded":"2024-03-18T14:52:07.363976+0000","last_fullsized":"2024-03-18T14:52:07.363976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:40.502783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368760+0000","last_change":"2024-03-18T14:51:01.353391+0000","last_active":"2024-03-18T14:52:07.368760+0000","last_peered":"2024-03-18T14:52:07.368760+0000","last_clean":"2024-03-18T14:52:07.368760+0000","last_became_active":"2024-03-18T14:51:01.353084+0000","last_became_peered":"2024-03-18T14:51:01.353084+0000","last_unstale":"2024-03-18T14:52:07.368760+0000","last_undegraded":"2024-03-18T14:52:07.368760+0000","last_fullsized":"2024-03-18T14:52:07.368760+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:18:02.774589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.363884+0000","last_change":"2024-03-18T14:51:00.942617+0000","last_active":"2024-03-18T14:52:07.363884+0000","last_peered":"2024-03-18T14:52:07.363884+0000","last_clean":"2024-03-18T14:52:07.363884+0000","last_became_active":"2024-03-18T14:51:00.942276+0000","last_became_peered":"2024-03-18T14:51:00.942276+0000","last_unstale":"2024-03-18T14:52:07.363884+0000","last_undegraded":"2024-03-18T14:52:07.363884+0000","last_fullsized":"2024-03-18T14:52:07.363884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:03:05.984066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368670+0000","last_change":"2024-03-18T14:51:01.363887+0000","last_active":"2024-03-18T14:52:07.368670+0000","last_peered":"2024-03-18T14:52:07.368670+0000","last_clean":"2024-03-18T14:52:07.368670+0000","last_became_active":"2024-03-18T14:51:01.361359+0000","last_became_peered":"2024-03-18T14:51:01.361359+0000","last_unstale":"2024-03-18T14:52:07.368670+0000","last_undegraded":"2024-03-18T14:52:07.368670+0000","last_fullsized":"2024-03-18T14:52:07.368670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:22.996930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368879+0000","last_change":"2024-03-18T14:51:00.940392+0000","last_active":"2024-03-18T14:52:07.368879+0000","last_peered":"2024-03-18T14:52:07.368879+0000","last_clean":"2024-03-18T14:52:07.368879+0000","last_became_active":"2024-03-18T14:51:00.940055+0000","last_became_peered":"2024-03-18T14:51:00.940055+0000","last_unstale":"2024-03-18T14:52:07.368879+0000","last_undegraded":"2024-03-18T14:52:07.368879+0000","last_fullsized":"2024-03-18T14:52:07.368879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:57:28.523549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366493+0000","last_change":"2024-03-18T14:51:01.361458+0000","last_active":"2024-03-18T14:52:07.366493+0000","last_peered":"2024-03-18T14:52:07.366493+0000","last_clean":"2024-03-18T14:52:07.366493+0000","last_became_active":"2024-03-18T14:51:01.361105+0000","last_became_peered":"2024-03-18T14:51:01.361105+0000","last_unstale":"2024-03-18T14:52:07.366493+0000","last_undegraded":"2024-03-18T14:52:07.366493+0000","last_fullsized":"2024-03-18T14:52:07.366493+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:22:59.410080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.379071+0000","last_change":"2024-03-18T14:52:07.378642+0000","last_active":"2024-03-18T14:52:07.379071+0000","last_peered":"2024-03-18T14:52:07.379071+0000","last_clean":"2024-03-18T14:52:07.379071+0000","last_became_active":"2024-03-18T14:52:07.371947+0000","last_became_peered":"2024-03-18T14:52:07.371947+0000","last_unstale":"2024-03-18T14:52:07.379071+0000","last_undegraded":"2024-03-18T14:52:07.379071+0000","last_fullsized":"2024-03-18T14:52:07.379071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:46:00.898462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.371816+0000","last_change":"2024-03-18T14:51:01.366305+0000","last_active":"2024-03-18T14:52:07.371816+0000","last_peered":"2024-03-18T14:52:07.371816+0000","last_clean":"2024-03-18T14:52:07.371816+0000","last_became_active":"2024-03-18T14:51:01.364626+0000","last_became_peered":"2024-03-18T14:51:01.364626+0000","last_unstale":"2024-03-18T14:52:07.371816+0000","last_undegraded":"2024-03-18T14:52:07.371816+0000","last_fullsized":"2024-03-18T14:52:07.371816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:06.212030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370835+0000","last_change":"2024-03-18T14:51:00.932514+0000","last_active":"2024-03-18T14:52:07.370835+0000","last_peered":"2024-03-18T14:52:07.370835+0000","last_clean":"2024-03-18T14:52:07.370835+0000","last_became_active":"2024-03-18T14:51:00.932183+0000","last_became_peered":"2024-03-18T14:51:00.932183+0000","last_unstale":"2024-03-18T14:52:07.370835+0000","last_undegraded":"2024-03-18T14:52:07.370835+0000","last_fullsized":"2024-03-18T14:52:07.370835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:36:37.230185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.424080+0000","last_change":"2024-03-18T14:51:01.358939+0000","last_active":"2024-03-18T14:52:07.424080+0000","last_peered":"2024-03-18T14:52:07.424080+0000","last_clean":"2024-03-18T14:52:07.424080+0000","last_became_active":"2024-03-18T14:51:01.358601+0000","last_became_peered":"2024-03-18T14:51:01.358601+0000","last_unstale":"2024-03-18T14:52:07.424080+0000","last_undegraded":"2024-03-18T14:52:07.424080+0000","last_fullsized":"2024-03-18T14:52:07.424080+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:40:14.876143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":100,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:19.112410+0000","last_change":"2024-03-18T14:51:00.945116+0000","last_active":"2024-03-18T14:52:19.112410+0000","last_peered":"2024-03-18T14:52:19.112410+0000","last_clean":"2024-03-18T14:52:19.112410+0000","last_became_active":"2024-03-18T14:51:00.944791+0000","last_became_peered":"2024-03-18T14:51:00.944791+0000","last_unstale":"2024-03-18T14:52:19.112410+0000","last_undegraded":"2024-03-18T14:52:19.112410+0000","last_fullsized":"2024-03-18T14:52:19.112410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:06.637940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370715+0000","last_change":"2024-03-18T14:51:01.358682+0000","last_active":"2024-03-18T14:52:07.370715+0000","last_peered":"2024-03-18T14:52:07.370715+0000","last_clean":"2024-03-18T14:52:07.370715+0000","last_became_active":"2024-03-18T14:51:01.358378+0000","last_became_peered":"2024-03-18T14:51:01.358378+0000","last_unstale":"2024-03-18T14:52:07.370715+0000","last_undegraded":"2024-03-18T14:52:07.370715+0000","last_fullsized":"2024-03-18T14:52:07.370715+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:42.697944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369158+0000","last_change":"2024-03-18T14:51:00.943976+0000","last_active":"2024-03-18T14:52:07.369158+0000","last_peered":"2024-03-18T14:52:07.369158+0000","last_clean":"2024-03-18T14:52:07.369158+0000","last_became_active":"2024-03-18T14:51:00.943614+0000","last_became_peered":"2024-03-18T14:51:00.943614+0000","last_unstale":"2024-03-18T14:52:07.369158+0000","last_undegraded":"2024-03-18T14:52:07.369158+0000","last_fullsized":"2024-03-18T14:52:07.369158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:07:00.744708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423803+0000","last_change":"2024-03-18T14:51:01.365316+0000","last_active":"2024-03-18T14:52:07.423803+0000","last_peered":"2024-03-18T14:52:07.423803+0000","last_clean":"2024-03-18T14:52:07.423803+0000","last_became_active":"2024-03-18T14:51:01.365019+0000","last_became_peered":"2024-03-18T14:51:01.365019+0000","last_unstale":"2024-03-18T14:52:07.423803+0000","last_undegraded":"2024-03-18T14:52:07.423803+0000","last_fullsized":"2024-03-18T14:52:07.423803+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:39:59.040907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481239+0000","last_change":"2024-03-18T14:51:00.936923+0000","last_active":"2024-03-18T14:52:07.481239+0000","last_peered":"2024-03-18T14:52:07.481239+0000","last_clean":"2024-03-18T14:52:07.481239+0000","last_became_active":"2024-03-18T14:51:00.936591+0000","last_became_peered":"2024-03-18T14:51:00.936591+0000","last_unstale":"2024-03-18T14:52:07.481239+0000","last_undegraded":"2024-03-18T14:52:07.481239+0000","last_fullsized":"2024-03-18T14:52:07.481239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:48:46.423033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368536+0000","last_change":"2024-03-18T14:51:01.356827+0000","last_active":"2024-03-18T14:52:07.368536+0000","last_peered":"2024-03-18T14:52:07.368536+0000","last_clean":"2024-03-18T14:52:07.368536+0000","last_became_active":"2024-03-18T14:51:01.356148+0000","last_became_peered":"2024-03-18T14:51:01.356148+0000","last_unstale":"2024-03-18T14:52:07.368536+0000","last_undegraded":"2024-03-18T14:52:07.368536+0000","last_fullsized":"2024-03-18T14:52:07.368536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:18.477890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423674+0000","last_change":"2024-03-18T14:51:00.934024+0000","last_active":"2024-03-18T14:52:07.423674+0000","last_peered":"2024-03-18T14:52:07.423674+0000","last_clean":"2024-03-18T14:52:07.423674+0000","last_became_active":"2024-03-18T14:51:00.933739+0000","last_became_peered":"2024-03-18T14:51:00.933739+0000","last_unstale":"2024-03-18T14:52:07.423674+0000","last_undegraded":"2024-03-18T14:52:07.423674+0000","last_fullsized":"2024-03-18T14:52:07.423674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:03.967260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.363735+0000","last_change":"2024-03-18T14:51:01.356519+0000","last_active":"2024-03-18T14:52:07.363735+0000","last_peered":"2024-03-18T14:52:07.363735+0000","last_clean":"2024-03-18T14:52:07.363735+0000","last_became_active":"2024-03-18T14:51:01.356193+0000","last_became_peered":"2024-03-18T14:51:01.356193+0000","last_unstale":"2024-03-18T14:52:07.363735+0000","last_undegraded":"2024-03-18T14:52:07.363735+0000","last_fullsized":"2024-03-18T14:52:07.363735+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:34:59.633241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":98,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:21.114024+0000","last_change":"2024-03-18T14:51:00.940189+0000","last_active":"2024-03-18T14:52:21.114024+0000","last_peered":"2024-03-18T14:52:21.114024+0000","last_clean":"2024-03-18T14:52:21.114024+0000","last_became_active":"2024-03-18T14:51:00.939820+0000","last_became_peered":"2024-03-18T14:51:00.939820+0000","last_unstale":"2024-03-18T14:52:21.114024+0000","last_undegraded":"2024-03-18T14:52:21.114024+0000","last_fullsized":"2024-03-18T14:52:21.114024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:44:09.749747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":76,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.371040+0000","last_change":"2024-03-18T14:51:01.355756+0000","last_active":"2024-03-18T14:52:07.371040+0000","last_peered":"2024-03-18T14:52:07.371040+0000","last_clean":"2024-03-18T14:52:07.371040+0000","last_became_active":"2024-03-18T14:51:01.352871+0000","last_became_peered":"2024-03-18T14:51:01.352871+0000","last_unstale":"2024-03-18T14:52:07.371040+0000","last_undegraded":"2024-03-18T14:52:07.371040+0000","last_fullsized":"2024-03-18T14:52:07.371040+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:20:26.533066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366850+0000","last_change":"2024-03-18T14:51:00.941908+0000","last_active":"2024-03-18T14:52:07.366850+0000","last_peered":"2024-03-18T14:52:07.366850+0000","last_clean":"2024-03-18T14:52:07.366850+0000","last_became_active":"2024-03-18T14:51:00.940978+0000","last_became_peered":"2024-03-18T14:51:00.940978+0000","last_unstale":"2024-03-18T14:52:07.366850+0000","last_undegraded":"2024-03-18T14:52:07.366850+0000","last_fullsized":"2024-03-18T14:52:07.366850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:52.705243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366489+0000","last_change":"2024-03-18T14:51:01.363462+0000","last_active":"2024-03-18T14:52:07.366489+0000","last_peered":"2024-03-18T14:52:07.366489+0000","last_clean":"2024-03-18T14:52:07.366489+0000","last_became_active":"2024-03-18T14:51:01.362262+0000","last_became_peered":"2024-03-18T14:51:01.362262+0000","last_unstale":"2024-03-18T14:52:07.366489+0000","last_undegraded":"2024-03-18T14:52:07.366489+0000","last_fullsized":"2024-03-18T14:52:07.366489+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:07:18.952876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.424170+0000","last_change":"2024-03-18T14:51:00.936050+0000","last_active":"2024-03-18T14:52:07.424170+0000","last_peered":"2024-03-18T14:52:07.424170+0000","last_clean":"2024-03-18T14:52:07.424170+0000","last_became_active":"2024-03-18T14:51:00.935676+0000","last_became_peered":"2024-03-18T14:51:00.935676+0000","last_unstale":"2024-03-18T14:52:07.424170+0000","last_undegraded":"2024-03-18T14:52:07.424170+0000","last_fullsized":"2024-03-18T14:52:07.424170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:29.423110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.371929+0000","last_change":"2024-03-18T14:51:01.364295+0000","last_active":"2024-03-18T14:52:07.371929+0000","last_peered":"2024-03-18T14:52:07.371929+0000","last_clean":"2024-03-18T14:52:07.371929+0000","last_became_active":"2024-03-18T14:51:01.363916+0000","last_became_peered":"2024-03-18T14:51:01.363916+0000","last_unstale":"2024-03-18T14:52:07.371929+0000","last_undegraded":"2024-03-18T14:52:07.371929+0000","last_fullsized":"2024-03-18T14:52:07.371929+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:21:17.905819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'26","reported_seq":219,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:21.437984+0000","last_change":"2024-03-18T14:52:07.427262+0000","last_active":"2024-03-18T14:52:21.437984+0000","last_peered":"2024-03-18T14:52:21.437984+0000","last_clean":"2024-03-18T14:52:21.437984+0000","last_became_active":"2024-03-18T14:52:07.421388+0000","last_became_peered":"2024-03-18T14:52:07.421388+0000","last_unstale":"2024-03-18T14:52:21.437984+0000","last_undegraded":"2024-03-18T14:52:21.437984+0000","last_fullsized":"2024-03-18T14:52:21.437984+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:28.159787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58912,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":152,"num_read_kb":133,"num_write":27,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":58912,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370722+0000","last_change":"2024-03-18T14:51:01.362426+0000","last_active":"2024-03-18T14:52:07.370722+0000","last_peered":"2024-03-18T14:52:07.370722+0000","last_clean":"2024-03-18T14:52:07.370722+0000","last_became_active":"2024-03-18T14:51:01.362130+0000","last_became_peered":"2024-03-18T14:51:01.362130+0000","last_unstale":"2024-03-18T14:52:07.370722+0000","last_undegraded":"2024-03-18T14:52:07.370722+0000","last_fullsized":"2024-03-18T14:52:07.370722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:14.545997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368774+0000","last_change":"2024-03-18T14:51:00.934028+0000","last_active":"2024-03-18T14:52:07.368774+0000","last_peered":"2024-03-18T14:52:07.368774+0000","last_clean":"2024-03-18T14:52:07.368774+0000","last_became_active":"2024-03-18T14:51:00.933594+0000","last_became_peered":"2024-03-18T14:51:00.933594+0000","last_unstale":"2024-03-18T14:52:07.368774+0000","last_undegraded":"2024-03-18T14:52:07.368774+0000","last_fullsized":"2024-03-18T14:52:07.368774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:55:08.675533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481326+0000","last_change":"2024-03-18T14:51:01.353544+0000","last_active":"2024-03-18T14:52:07.481326+0000","last_peered":"2024-03-18T14:52:07.481326+0000","last_clean":"2024-03-18T14:52:07.481326+0000","last_became_active":"2024-03-18T14:51:01.353241+0000","last_became_peered":"2024-03-18T14:51:01.353241+0000","last_unstale":"2024-03-18T14:52:07.481326+0000","last_undegraded":"2024-03-18T14:52:07.481326+0000","last_fullsized":"2024-03-18T14:52:07.481326+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:45:27.617681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423792+0000","last_change":"2024-03-18T14:51:00.944492+0000","last_active":"2024-03-18T14:52:07.423792+0000","last_peered":"2024-03-18T14:52:07.423792+0000","last_clean":"2024-03-18T14:52:07.423792+0000","last_became_active":"2024-03-18T14:51:00.944165+0000","last_became_peered":"2024-03-18T14:51:00.944165+0000","last_unstale":"2024-03-18T14:52:07.423792+0000","last_undegraded":"2024-03-18T14:52:07.423792+0000","last_fullsized":"2024-03-18T14:52:07.423792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:47:05.968966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366823+0000","last_change":"2024-03-18T14:51:01.356782+0000","last_active":"2024-03-18T14:52:07.366823+0000","last_peered":"2024-03-18T14:52:07.366823+0000","last_clean":"2024-03-18T14:52:07.366823+0000","last_became_active":"2024-03-18T14:51:01.356242+0000","last_became_peered":"2024-03-18T14:51:01.356242+0000","last_unstale":"2024-03-18T14:52:07.366823+0000","last_undegraded":"2024-03-18T14:52:07.366823+0000","last_fullsized":"2024-03-18T14:52:07.366823+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:56:07.662545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":100,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:19.118152+0000","last_change":"2024-03-18T14:51:00.937873+0000","last_active":"2024-03-18T14:52:19.118152+0000","last_peered":"2024-03-18T14:52:19.118152+0000","last_clean":"2024-03-18T14:52:19.118152+0000","last_became_active":"2024-03-18T14:51:00.937562+0000","last_became_peered":"2024-03-18T14:51:00.937562+0000","last_unstale":"2024-03-18T14:52:19.118152+0000","last_undegraded":"2024-03-18T14:52:19.118152+0000","last_fullsized":"2024-03-18T14:52:19.118152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:36:44.342414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366508+0000","last_change":"2024-03-18T14:51:01.358681+0000","last_active":"2024-03-18T14:52:07.366508+0000","last_peered":"2024-03-18T14:52:07.366508+0000","last_clean":"2024-03-18T14:52:07.366508+0000","last_became_active":"2024-03-18T14:51:01.358374+0000","last_became_peered":"2024-03-18T14:51:01.358374+0000","last_unstale":"2024-03-18T14:52:07.366508+0000","last_undegraded":"2024-03-18T14:52:07.366508+0000","last_fullsized":"2024-03-18T14:52:07.366508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:43:47.804271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370451+0000","last_change":"2024-03-18T14:51:00.935639+0000","last_active":"2024-03-18T14:52:07.370451+0000","last_peered":"2024-03-18T14:52:07.370451+0000","last_clean":"2024-03-18T14:52:07.370451+0000","last_became_active":"2024-03-18T14:51:00.935334+0000","last_became_peered":"2024-03-18T14:51:00.935334+0000","last_unstale":"2024-03-18T14:52:07.370451+0000","last_undegraded":"2024-03-18T14:52:07.370451+0000","last_fullsized":"2024-03-18T14:52:07.370451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:46.966812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423741+0000","last_change":"2024-03-18T14:51:01.364755+0000","last_active":"2024-03-18T14:52:07.423741+0000","last_peered":"2024-03-18T14:52:07.423741+0000","last_clean":"2024-03-18T14:52:07.423741+0000","last_became_active":"2024-03-18T14:51:01.364448+0000","last_became_peered":"2024-03-18T14:51:01.364448+0000","last_unstale":"2024-03-18T14:52:07.423741+0000","last_undegraded":"2024-03-18T14:52:07.423741+0000","last_fullsized":"2024-03-18T14:52:07.423741+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:37:49.480787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370995+0000","last_change":"2024-03-18T14:51:00.933271+0000","last_active":"2024-03-18T14:52:07.370995+0000","last_peered":"2024-03-18T14:52:07.370995+0000","last_clean":"2024-03-18T14:52:07.370995+0000","last_became_active":"2024-03-18T14:51:00.932928+0000","last_became_peered":"2024-03-18T14:51:00.932928+0000","last_unstale":"2024-03-18T14:52:07.370995+0000","last_undegraded":"2024-03-18T14:52:07.370995+0000","last_fullsized":"2024-03-18T14:52:07.370995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:11:55.281763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.367160+0000","last_change":"2024-03-18T14:51:01.362911+0000","last_active":"2024-03-18T14:52:07.367160+0000","last_peered":"2024-03-18T14:52:07.367160+0000","last_clean":"2024-03-18T14:52:07.367160+0000","last_became_active":"2024-03-18T14:51:01.362598+0000","last_became_peered":"2024-03-18T14:51:01.362598+0000","last_unstale":"2024-03-18T14:52:07.367160+0000","last_undegraded":"2024-03-18T14:52:07.367160+0000","last_fullsized":"2024-03-18T14:52:07.367160+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:22:58.627753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.363811+0000","last_change":"2024-03-18T14:51:00.935554+0000","last_active":"2024-03-18T14:52:07.363811+0000","last_peered":"2024-03-18T14:52:07.363811+0000","last_clean":"2024-03-18T14:52:07.363811+0000","last_became_active":"2024-03-18T14:51:00.935184+0000","last_became_peered":"2024-03-18T14:51:00.935184+0000","last_unstale":"2024-03-18T14:52:07.363811+0000","last_undegraded":"2024-03-18T14:52:07.363811+0000","last_fullsized":"2024-03-18T14:52:07.363811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:21:13.973570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369358+0000","last_change":"2024-03-18T14:51:01.355990+0000","last_active":"2024-03-18T14:52:07.369358+0000","last_peered":"2024-03-18T14:52:07.369358+0000","last_clean":"2024-03-18T14:52:07.369358+0000","last_became_active":"2024-03-18T14:51:01.355692+0000","last_became_peered":"2024-03-18T14:51:01.355692+0000","last_unstale":"2024-03-18T14:52:07.369358+0000","last_undegraded":"2024-03-18T14:52:07.369358+0000","last_fullsized":"2024-03-18T14:52:07.369358+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:27:19.742684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366869+0000","last_change":"2024-03-18T14:51:00.943947+0000","last_active":"2024-03-18T14:52:07.366869+0000","last_peered":"2024-03-18T14:52:07.366869+0000","last_clean":"2024-03-18T14:52:07.366869+0000","last_became_active":"2024-03-18T14:51:00.943565+0000","last_became_peered":"2024-03-18T14:51:00.943565+0000","last_unstale":"2024-03-18T14:52:07.366869+0000","last_undegraded":"2024-03-18T14:52:07.366869+0000","last_fullsized":"2024-03-18T14:52:07.366869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:33:06.103148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369281+0000","last_change":"2024-03-18T14:51:01.363221+0000","last_active":"2024-03-18T14:52:07.369281+0000","last_peered":"2024-03-18T14:52:07.369281+0000","last_clean":"2024-03-18T14:52:07.369281+0000","last_became_active":"2024-03-18T14:51:01.362848+0000","last_became_peered":"2024-03-18T14:51:01.362848+0000","last_unstale":"2024-03-18T14:52:07.369281+0000","last_undegraded":"2024-03-18T14:52:07.369281+0000","last_fullsized":"2024-03-18T14:52:07.369281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:04.139365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423779+0000","last_change":"2024-03-18T14:51:01.356237+0000","last_active":"2024-03-18T14:52:07.423779+0000","last_peered":"2024-03-18T14:52:07.423779+0000","last_clean":"2024-03-18T14:52:07.423779+0000","last_became_active":"2024-03-18T14:51:01.355922+0000","last_became_peered":"2024-03-18T14:51:01.355922+0000","last_unstale":"2024-03-18T14:52:07.423779+0000","last_undegraded":"2024-03-18T14:52:07.423779+0000","last_fullsized":"2024-03-18T14:52:07.423779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:04:50.491752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":97,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:19.117698+0000","last_change":"2024-03-18T14:51:00.936058+0000","last_active":"2024-03-18T14:52:19.117698+0000","last_peered":"2024-03-18T14:52:19.117698+0000","last_clean":"2024-03-18T14:52:19.117698+0000","last_became_active":"2024-03-18T14:51:00.935731+0000","last_became_peered":"2024-03-18T14:51:00.935731+0000","last_unstale":"2024-03-18T14:52:19.117698+0000","last_undegraded":"2024-03-18T14:52:19.117698+0000","last_fullsized":"2024-03-18T14:52:19.117698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:47:49.023943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368957+0000","last_change":"2024-03-18T14:51:01.359742+0000","last_active":"2024-03-18T14:52:07.368957+0000","last_peered":"2024-03-18T14:52:07.368957+0000","last_clean":"2024-03-18T14:52:07.368957+0000","last_became_active":"2024-03-18T14:51:01.358507+0000","last_became_peered":"2024-03-18T14:51:01.358507+0000","last_unstale":"2024-03-18T14:52:07.368957+0000","last_undegraded":"2024-03-18T14:52:07.368957+0000","last_fullsized":"2024-03-18T14:52:07.368957+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:17.129695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.419644+0000","last_change":"2024-03-18T14:51:00.934593+0000","last_active":"2024-03-18T14:52:07.419644+0000","last_peered":"2024-03-18T14:52:07.419644+0000","last_clean":"2024-03-18T14:52:07.419644+0000","last_became_active":"2024-03-18T14:51:00.934253+0000","last_became_peered":"2024-03-18T14:51:00.934253+0000","last_unstale":"2024-03-18T14:52:07.419644+0000","last_undegraded":"2024-03-18T14:52:07.419644+0000","last_fullsized":"2024-03-18T14:52:07.419644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:43:26.323162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":351,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366618+0000","last_change":"2024-03-18T14:49:51.158965+0000","last_active":"2024-03-18T14:52:07.366618+0000","last_peered":"2024-03-18T14:52:07.366618+0000","last_clean":"2024-03-18T14:52:07.366618+0000","last_became_active":"2024-03-18T14:49:51.158580+0000","last_became_peered":"2024-03-18T14:49:51.158580+0000","last_unstale":"2024-03-18T14:52:07.366618+0000","last_undegraded":"2024-03-18T14:52:07.366618+0000","last_fullsized":"2024-03-18T14:52:07.366618+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:46:27.210244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:46:27.210244+0000","last_clean_scrub_stamp":"2024-03-18T14:46:27.210244+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:37:08.104617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481115+0000","last_change":"2024-03-18T14:51:01.359972+0000","last_active":"2024-03-18T14:52:07.481115+0000","last_peered":"2024-03-18T14:52:07.481115+0000","last_clean":"2024-03-18T14:52:07.481115+0000","last_became_active":"2024-03-18T14:51:01.359430+0000","last_became_peered":"2024-03-18T14:51:01.359430+0000","last_unstale":"2024-03-18T14:52:07.481115+0000","last_undegraded":"2024-03-18T14:52:07.481115+0000","last_fullsized":"2024-03-18T14:52:07.481115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:00:14.099499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366722+0000","last_change":"2024-03-18T14:51:00.934196+0000","last_active":"2024-03-18T14:52:07.366722+0000","last_peered":"2024-03-18T14:52:07.366722+0000","last_clean":"2024-03-18T14:52:07.366722+0000","last_became_active":"2024-03-18T14:51:00.933875+0000","last_became_peered":"2024-03-18T14:51:00.933875+0000","last_unstale":"2024-03-18T14:52:07.366722+0000","last_undegraded":"2024-03-18T14:52:07.366722+0000","last_fullsized":"2024-03-18T14:52:07.366722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:56.797227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370796+0000","last_change":"2024-03-18T14:51:00.941862+0000","last_active":"2024-03-18T14:52:07.370796+0000","last_peered":"2024-03-18T14:52:07.370796+0000","last_clean":"2024-03-18T14:52:07.370796+0000","last_became_active":"2024-03-18T14:51:00.941536+0000","last_became_peered":"2024-03-18T14:51:00.941536+0000","last_unstale":"2024-03-18T14:52:07.370796+0000","last_undegraded":"2024-03-18T14:52:07.370796+0000","last_fullsized":"2024-03-18T14:52:07.370796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:56.364552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481330+0000","last_change":"2024-03-18T14:51:01.352404+0000","last_active":"2024-03-18T14:52:07.481330+0000","last_peered":"2024-03-18T14:52:07.481330+0000","last_clean":"2024-03-18T14:52:07.481330+0000","last_became_active":"2024-03-18T14:51:01.352077+0000","last_became_peered":"2024-03-18T14:51:01.352077+0000","last_unstale":"2024-03-18T14:52:07.481330+0000","last_undegraded":"2024-03-18T14:52:07.481330+0000","last_fullsized":"2024-03-18T14:52:07.481330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:30:38.050712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.364070+0000","last_change":"2024-03-18T14:51:00.935564+0000","last_active":"2024-03-18T14:52:07.364070+0000","last_peered":"2024-03-18T14:52:07.364070+0000","last_clean":"2024-03-18T14:52:07.364070+0000","last_became_active":"2024-03-18T14:51:00.935268+0000","last_became_peered":"2024-03-18T14:51:00.935268+0000","last_unstale":"2024-03-18T14:52:07.364070+0000","last_undegraded":"2024-03-18T14:52:07.364070+0000","last_fullsized":"2024-03-18T14:52:07.364070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:22:33.504184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368810+0000","last_change":"2024-03-18T14:51:01.356096+0000","last_active":"2024-03-18T14:52:07.368810+0000","last_peered":"2024-03-18T14:52:07.368810+0000","last_clean":"2024-03-18T14:52:07.368810+0000","last_became_active":"2024-03-18T14:51:01.355688+0000","last_became_peered":"2024-03-18T14:51:01.355688+0000","last_unstale":"2024-03-18T14:52:07.368810+0000","last_undegraded":"2024-03-18T14:52:07.368810+0000","last_fullsized":"2024-03-18T14:52:07.368810+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:04:38.678482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.419726+0000","last_change":"2024-03-18T14:51:00.942610+0000","last_active":"2024-03-18T14:52:07.419726+0000","last_peered":"2024-03-18T14:52:07.419726+0000","last_clean":"2024-03-18T14:52:07.419726+0000","last_became_active":"2024-03-18T14:51:00.942203+0000","last_became_peered":"2024-03-18T14:51:00.942203+0000","last_unstale":"2024-03-18T14:52:07.419726+0000","last_undegraded":"2024-03-18T14:52:07.419726+0000","last_fullsized":"2024-03-18T14:52:07.419726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:17:26.324282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366425+0000","last_change":"2024-03-18T14:51:01.362931+0000","last_active":"2024-03-18T14:52:07.366425+0000","last_peered":"2024-03-18T14:52:07.366425+0000","last_clean":"2024-03-18T14:52:07.366425+0000","last_became_active":"2024-03-18T14:51:01.362624+0000","last_became_peered":"2024-03-18T14:51:01.362624+0000","last_unstale":"2024-03-18T14:52:07.366425+0000","last_undegraded":"2024-03-18T14:52:07.366425+0000","last_fullsized":"2024-03-18T14:52:07.366425+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:40:52.512871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370722+0000","last_change":"2024-03-18T14:51:00.938971+0000","last_active":"2024-03-18T14:52:07.370722+0000","last_peered":"2024-03-18T14:52:07.370722+0000","last_clean":"2024-03-18T14:52:07.370722+0000","last_became_active":"2024-03-18T14:51:00.938645+0000","last_became_peered":"2024-03-18T14:51:00.938645+0000","last_unstale":"2024-03-18T14:52:07.370722+0000","last_undegraded":"2024-03-18T14:52:07.370722+0000","last_fullsized":"2024-03-18T14:52:07.370722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:18:54.999927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370470+0000","last_change":"2024-03-18T14:51:01.359510+0000","last_active":"2024-03-18T14:52:07.370470+0000","last_peered":"2024-03-18T14:52:07.370470+0000","last_clean":"2024-03-18T14:52:07.370470+0000","last_became_active":"2024-03-18T14:51:01.359192+0000","last_became_peered":"2024-03-18T14:51:01.359192+0000","last_unstale":"2024-03-18T14:52:07.370470+0000","last_undegraded":"2024-03-18T14:52:07.370470+0000","last_fullsized":"2024-03-18T14:52:07.370470+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:11:52.942825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368576+0000","last_change":"2024-03-18T14:51:00.940194+0000","last_active":"2024-03-18T14:52:07.368576+0000","last_peered":"2024-03-18T14:52:07.368576+0000","last_clean":"2024-03-18T14:52:07.368576+0000","last_became_active":"2024-03-18T14:51:00.939861+0000","last_became_peered":"2024-03-18T14:51:00.939861+0000","last_unstale":"2024-03-18T14:52:07.368576+0000","last_undegraded":"2024-03-18T14:52:07.368576+0000","last_fullsized":"2024-03-18T14:52:07.368576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:06:30.314102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366604+0000","last_change":"2024-03-18T14:51:01.355098+0000","last_active":"2024-03-18T14:52:07.366604+0000","last_peered":"2024-03-18T14:52:07.366604+0000","last_clean":"2024-03-18T14:52:07.366604+0000","last_became_active":"2024-03-18T14:51:01.354452+0000","last_became_peered":"2024-03-18T14:51:01.354452+0000","last_unstale":"2024-03-18T14:52:07.366604+0000","last_undegraded":"2024-03-18T14:52:07.366604+0000","last_fullsized":"2024-03-18T14:52:07.366604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:40:54.650294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.381529+0000","last_change":"2024-03-18T14:52:07.381159+0000","last_active":"2024-03-18T14:52:07.381529+0000","last_peered":"2024-03-18T14:52:07.381529+0000","last_clean":"2024-03-18T14:52:07.381529+0000","last_became_active":"2024-03-18T14:52:07.371943+0000","last_became_peered":"2024-03-18T14:52:07.371943+0000","last_unstale":"2024-03-18T14:52:07.381529+0000","last_undegraded":"2024-03-18T14:52:07.381529+0000","last_fullsized":"2024-03-18T14:52:07.381529+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:11.329239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368784+0000","last_change":"2024-03-18T14:51:01.357920+0000","last_active":"2024-03-18T14:52:07.368784+0000","last_peered":"2024-03-18T14:52:07.368784+0000","last_clean":"2024-03-18T14:52:07.368784+0000","last_became_active":"2024-03-18T14:51:01.357565+0000","last_became_peered":"2024-03-18T14:51:01.357565+0000","last_unstale":"2024-03-18T14:52:07.368784+0000","last_undegraded":"2024-03-18T14:52:07.368784+0000","last_fullsized":"2024-03-18T14:52:07.368784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:39:57.880178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370714+0000","last_change":"2024-03-18T14:51:00.937887+0000","last_active":"2024-03-18T14:52:07.370714+0000","last_peered":"2024-03-18T14:52:07.370714+0000","last_clean":"2024-03-18T14:52:07.370714+0000","last_became_active":"2024-03-18T14:51:00.937588+0000","last_became_peered":"2024-03-18T14:51:00.937588+0000","last_unstale":"2024-03-18T14:52:07.370714+0000","last_undegraded":"2024-03-18T14:52:07.370714+0000","last_fullsized":"2024-03-18T14:52:07.370714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:43:37.558412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366705+0000","last_change":"2024-03-18T14:51:01.354175+0000","last_active":"2024-03-18T14:52:07.366705+0000","last_peered":"2024-03-18T14:52:07.366705+0000","last_clean":"2024-03-18T14:52:07.366705+0000","last_became_active":"2024-03-18T14:51:01.353839+0000","last_became_peered":"2024-03-18T14:51:01.353839+0000","last_unstale":"2024-03-18T14:52:07.366705+0000","last_undegraded":"2024-03-18T14:52:07.366705+0000","last_fullsized":"2024-03-18T14:52:07.366705+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:21:52.372938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369167+0000","last_change":"2024-03-18T14:51:00.941600+0000","last_active":"2024-03-18T14:52:07.369167+0000","last_peered":"2024-03-18T14:52:07.369167+0000","last_clean":"2024-03-18T14:52:07.369167+0000","last_became_active":"2024-03-18T14:51:00.941291+0000","last_became_peered":"2024-03-18T14:51:00.941291+0000","last_unstale":"2024-03-18T14:52:07.369167+0000","last_undegraded":"2024-03-18T14:52:07.369167+0000","last_fullsized":"2024-03-18T14:52:07.369167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:04:11.142029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366985+0000","last_change":"2024-03-18T14:51:01.359058+0000","last_active":"2024-03-18T14:52:07.366985+0000","last_peered":"2024-03-18T14:52:07.366985+0000","last_clean":"2024-03-18T14:52:07.366985+0000","last_became_active":"2024-03-18T14:51:01.358437+0000","last_became_peered":"2024-03-18T14:51:01.358437+0000","last_unstale":"2024-03-18T14:52:07.366985+0000","last_undegraded":"2024-03-18T14:52:07.366985+0000","last_fullsized":"2024-03-18T14:52:07.366985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:39:58.945080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.424033+0000","last_change":"2024-03-18T14:51:00.933647+0000","last_active":"2024-03-18T14:52:07.424033+0000","last_peered":"2024-03-18T14:52:07.424033+0000","last_clean":"2024-03-18T14:52:07.424033+0000","last_became_active":"2024-03-18T14:51:00.933320+0000","last_became_peered":"2024-03-18T14:51:00.933320+0000","last_unstale":"2024-03-18T14:52:07.424033+0000","last_undegraded":"2024-03-18T14:52:07.424033+0000","last_fullsized":"2024-03-18T14:52:07.424033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:16:46.498845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366728+0000","last_change":"2024-03-18T14:51:01.359874+0000","last_active":"2024-03-18T14:52:07.366728+0000","last_peered":"2024-03-18T14:52:07.366728+0000","last_clean":"2024-03-18T14:52:07.366728+0000","last_became_active":"2024-03-18T14:51:01.359335+0000","last_became_peered":"2024-03-18T14:51:01.359335+0000","last_unstale":"2024-03-18T14:52:07.366728+0000","last_undegraded":"2024-03-18T14:52:07.366728+0000","last_fullsized":"2024-03-18T14:52:07.366728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:03.583402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481395+0000","last_change":"2024-03-18T14:51:00.935755+0000","last_active":"2024-03-18T14:52:07.481395+0000","last_peered":"2024-03-18T14:52:07.481395+0000","last_clean":"2024-03-18T14:52:07.481395+0000","last_became_active":"2024-03-18T14:51:00.935421+0000","last_became_peered":"2024-03-18T14:51:00.935421+0000","last_unstale":"2024-03-18T14:52:07.481395+0000","last_undegraded":"2024-03-18T14:52:07.481395+0000","last_fullsized":"2024-03-18T14:52:07.481395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:46:11.680972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.367029+0000","last_change":"2024-03-18T14:51:01.355605+0000","last_active":"2024-03-18T14:52:07.367029+0000","last_peered":"2024-03-18T14:52:07.367029+0000","last_clean":"2024-03-18T14:52:07.367029+0000","last_became_active":"2024-03-18T14:51:01.355295+0000","last_became_peered":"2024-03-18T14:51:01.355295+0000","last_unstale":"2024-03-18T14:52:07.367029+0000","last_undegraded":"2024-03-18T14:52:07.367029+0000","last_fullsized":"2024-03-18T14:52:07.367029+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:15:20.466314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366671+0000","last_change":"2024-03-18T14:51:01.359704+0000","last_active":"2024-03-18T14:52:07.366671+0000","last_peered":"2024-03-18T14:52:07.366671+0000","last_clean":"2024-03-18T14:52:07.366671+0000","last_became_active":"2024-03-18T14:51:01.359367+0000","last_became_peered":"2024-03-18T14:51:01.359367+0000","last_unstale":"2024-03-18T14:52:07.366671+0000","last_undegraded":"2024-03-18T14:52:07.366671+0000","last_fullsized":"2024-03-18T14:52:07.366671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:26.933993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.420506+0000","last_change":"2024-03-18T14:52:07.420506+0000","last_active":"2024-03-18T14:52:07.420506+0000","last_peered":"2024-03-18T14:52:07.420506+0000","last_clean":"2024-03-18T14:52:07.420506+0000","last_became_active":"2024-03-18T14:52:07.420254+0000","last_became_peered":"2024-03-18T14:52:07.420254+0000","last_unstale":"2024-03-18T14:52:07.420506+0000","last_undegraded":"2024-03-18T14:52:07.420506+0000","last_fullsized":"2024-03-18T14:52:07.420506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:32:08.702462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368828+0000","last_change":"2024-03-18T14:51:01.355678+0000","last_active":"2024-03-18T14:52:07.368828+0000","last_peered":"2024-03-18T14:52:07.368828+0000","last_clean":"2024-03-18T14:52:07.368828+0000","last_became_active":"2024-03-18T14:51:01.355197+0000","last_became_peered":"2024-03-18T14:51:01.355197+0000","last_unstale":"2024-03-18T14:52:07.368828+0000","last_undegraded":"2024-03-18T14:52:07.368828+0000","last_fullsized":"2024-03-18T14:52:07.368828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:34.194834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366340+0000","last_change":"2024-03-18T14:51:01.363964+0000","last_active":"2024-03-18T14:52:07.366340+0000","last_peered":"2024-03-18T14:52:07.366340+0000","last_clean":"2024-03-18T14:52:07.366340+0000","last_became_active":"2024-03-18T14:51:01.363590+0000","last_became_peered":"2024-03-18T14:51:01.363590+0000","last_unstale":"2024-03-18T14:52:07.366340+0000","last_undegraded":"2024-03-18T14:52:07.366340+0000","last_fullsized":"2024-03-18T14:52:07.366340+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:59.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481350+0000","last_change":"2024-03-18T14:51:01.361599+0000","last_active":"2024-03-18T14:52:07.481350+0000","last_peered":"2024-03-18T14:52:07.481350+0000","last_clean":"2024-03-18T14:52:07.481350+0000","last_became_active":"2024-03-18T14:51:01.361280+0000","last_became_peered":"2024-03-18T14:51:01.361280+0000","last_unstale":"2024-03-18T14:52:07.481350+0000","last_undegraded":"2024-03-18T14:52:07.481350+0000","last_fullsized":"2024-03-18T14:52:07.481350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:52:50.572257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.367005+0000","last_change":"2024-03-18T14:51:01.360495+0000","last_active":"2024-03-18T14:52:07.367005+0000","last_peered":"2024-03-18T14:52:07.367005+0000","last_clean":"2024-03-18T14:52:07.367005+0000","last_became_active":"2024-03-18T14:51:01.360146+0000","last_became_peered":"2024-03-18T14:51:01.360146+0000","last_unstale":"2024-03-18T14:52:07.367005+0000","last_undegraded":"2024-03-18T14:52:07.367005+0000","last_fullsized":"2024-03-18T14:52:07.367005+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:52.048137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368892+0000","last_change":"2024-03-18T14:51:01.357650+0000","last_active":"2024-03-18T14:52:07.368892+0000","last_peered":"2024-03-18T14:52:07.368892+0000","last_clean":"2024-03-18T14:52:07.368892+0000","last_became_active":"2024-03-18T14:51:01.356265+0000","last_became_peered":"2024-03-18T14:51:01.356265+0000","last_unstale":"2024-03-18T14:52:07.368892+0000","last_undegraded":"2024-03-18T14:52:07.368892+0000","last_fullsized":"2024-03-18T14:52:07.368892+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:49.328404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368727+0000","last_change":"2024-03-18T14:51:01.363753+0000","last_active":"2024-03-18T14:52:07.368727+0000","last_peered":"2024-03-18T14:52:07.368727+0000","last_clean":"2024-03-18T14:52:07.368727+0000","last_became_active":"2024-03-18T14:51:01.363411+0000","last_became_peered":"2024-03-18T14:51:01.363411+0000","last_unstale":"2024-03-18T14:52:07.368727+0000","last_undegraded":"2024-03-18T14:52:07.368727+0000","last_fullsized":"2024-03-18T14:52:07.368727+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:55.452209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423670+0000","last_change":"2024-03-18T14:51:01.360723+0000","last_active":"2024-03-18T14:52:07.423670+0000","last_peered":"2024-03-18T14:52:07.423670+0000","last_clean":"2024-03-18T14:52:07.423670+0000","last_became_active":"2024-03-18T14:51:01.360130+0000","last_became_peered":"2024-03-18T14:51:01.360130+0000","last_unstale":"2024-03-18T14:52:07.423670+0000","last_undegraded":"2024-03-18T14:52:07.423670+0000","last_fullsized":"2024-03-18T14:52:07.423670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:51.634011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368533+0000","last_change":"2024-03-18T14:51:01.356462+0000","last_active":"2024-03-18T14:52:07.368533+0000","last_peered":"2024-03-18T14:52:07.368533+0000","last_clean":"2024-03-18T14:52:07.368533+0000","last_became_active":"2024-03-18T14:51:01.355395+0000","last_became_peered":"2024-03-18T14:51:01.355395+0000","last_unstale":"2024-03-18T14:52:07.368533+0000","last_undegraded":"2024-03-18T14:52:07.368533+0000","last_fullsized":"2024-03-18T14:52:07.368533+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:37:15.326354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370868+0000","last_change":"2024-03-18T14:51:01.358626+0000","last_active":"2024-03-18T14:52:07.370868+0000","last_peered":"2024-03-18T14:52:07.370868+0000","last_clean":"2024-03-18T14:52:07.370868+0000","last_became_active":"2024-03-18T14:51:01.358293+0000","last_became_peered":"2024-03-18T14:51:01.358293+0000","last_unstale":"2024-03-18T14:52:07.370868+0000","last_undegraded":"2024-03-18T14:52:07.370868+0000","last_fullsized":"2024-03-18T14:52:07.370868+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:52:41.139885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481309+0000","last_change":"2024-03-18T14:51:01.355276+0000","last_active":"2024-03-18T14:52:07.481309+0000","last_peered":"2024-03-18T14:52:07.481309+0000","last_clean":"2024-03-18T14:52:07.481309+0000","last_became_active":"2024-03-18T14:51:01.354661+0000","last_became_peered":"2024-03-18T14:51:01.354661+0000","last_unstale":"2024-03-18T14:52:07.481309+0000","last_undegraded":"2024-03-18T14:52:07.481309+0000","last_fullsized":"2024-03-18T14:52:07.481309+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:28.569055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.372297+0000","last_change":"2024-03-18T14:52:07.372297+0000","last_active":"2024-03-18T14:52:07.372297+0000","last_peered":"2024-03-18T14:52:07.372297+0000","last_clean":"2024-03-18T14:52:07.372297+0000","last_became_active":"2024-03-18T14:52:07.371918+0000","last_became_peered":"2024-03-18T14:52:07.371918+0000","last_unstale":"2024-03-18T14:52:07.372297+0000","last_undegraded":"2024-03-18T14:52:07.372297+0000","last_fullsized":"2024-03-18T14:52:07.372297+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:28:23.431373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280632,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62913,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":469,"num_read_kb":302,"num_write":154,"num_write_kb":105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":58912,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":188739,"data_compressed":13263,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":0,"internal_metadata":0},"log_size":133,"ondisk_log_size":133,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351006,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":950272,"data_stored":459161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612642,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712896,"statfs":{"total":95995035648,"available":95962005504,"internally_reserved":0,"allocated":1097728,"data_stored":1109945,"data_compressed":37553,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776168,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":459397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037804,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":459317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299441,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":1051247,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462966,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":458786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":517491,"data_compressed":1859,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":974848,"data_stored":460057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":459929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":970752,"data_stored":459097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888129,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":518526,"data_compressed":9545,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724603,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":1050540,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":58912,"data_compressed":1859,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":59947,"data_compressed":9545,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":60881,"data_compressed":1859,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:52:25.151 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-18T14:52:25.151 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-18T14:52:25.151 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-18T14:52:25.151 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-18T14:52:25.395 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:52:25.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:25 smithi076 ceph-mon[32205]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:52:25.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:25 smithi174 ceph-mon[36787]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:52:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:25 smithi077 ceph-mon[35865]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:52:26.749 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:26 smithi076 ceph-mon[32205]: pgmap v328: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:52:26.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:26 smithi174 ceph-mon[36787]: pgmap v328: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:52:26.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:26 smithi077 ceph-mon[35865]: pgmap v328: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:52:27.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.127+0000 7f317b2d8700 1 -- 172.21.15.76:0/2727063179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f317406aa20 msgr2=0x7f317406aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:27.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.127+0000 7f317b2d8700 1 --2- 172.21.15.76:0/2727063179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f317406aa20 0x7f317406aea0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f31640099c0 tx=0x7f3164009cd0 comp rx=0 tx=0).stop 2024-03-18T14:52:27.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.127+0000 7f317b2d8700 1 -- 172.21.15.76:0/2727063179 shutdown_connections 2024-03-18T14:52:27.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.127+0000 7f317b2d8700 1 --2- 172.21.15.76:0/2727063179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f317406b3e0 0x7f317410f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:27.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.127+0000 7f317b2d8700 1 --2- 172.21.15.76:0/2727063179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3174108300 0x7f31741086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:27.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.127+0000 7f317b2d8700 1 --2- 172.21.15.76:0/2727063179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f317406aa20 0x7f317406aea0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:27.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.127+0000 7f317b2d8700 1 -- 172.21.15.76:0/2727063179 >> 172.21.15.76:0/2727063179 conn(0x7f31740781e0 msgr2=0x7f31740785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:27.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.127+0000 7f317b2d8700 1 -- 172.21.15.76:0/2727063179 shutdown_connections 2024-03-18T14:52:27.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.127+0000 7f317b2d8700 1 -- 172.21.15.76:0/2727063179 wait complete. 2024-03-18T14:52:27.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.127+0000 7f317b2d8700 1 Processor -- start 2024-03-18T14:52:27.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f317b2d8700 1 -- start start 2024-03-18T14:52:27.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f317b2d8700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f317406aa20 0x7f317419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:27.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f317b2d8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f317406b3e0 0x7f317419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:27.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f317b2d8700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3174108300 0x7f31741987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:27.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f317b2d8700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f31741125b0 con 0x7f317406aa20 2024-03-18T14:52:27.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f317b2d8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3174112430 con 0x7f317406b3e0 2024-03-18T14:52:27.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f317b2d8700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3174112730 con 0x7f3174108300 2024-03-18T14:52:27.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f3179074700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f317406aa20 0x7f317419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:27.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f3179074700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f317406aa20 0x7f317419e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45630/0 (socket says 172.21.15.76:45630) 2024-03-18T14:52:27.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f3179074700 1 -- 172.21.15.76:0/984521084 learned_addr learned my addr 172.21.15.76:0/984521084 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:27.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f3178873700 1 --2- 172.21.15.76:0/984521084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f317406b3e0 0x7f317419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:27.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f3179875700 1 --2- 172.21.15.76:0/984521084 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3174108300 0x7f31741987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:27.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f3179074700 1 -- 172.21.15.76:0/984521084 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3174108300 msgr2=0x7f31741987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:27.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f3179074700 1 --2- 172.21.15.76:0/984521084 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3174108300 0x7f31741987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:27.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f3179074700 1 -- 172.21.15.76:0/984521084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f317406b3e0 msgr2=0x7f317419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:27.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f3179074700 1 --2- 172.21.15.76:0/984521084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f317406b3e0 0x7f317419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:27.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f3179074700 1 -- 172.21.15.76:0/984521084 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3174198fd0 con 0x7f317406aa20 2024-03-18T14:52:27.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f3178873700 1 --2- 172.21.15.76:0/984521084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f317406b3e0 0x7f317419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:27.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f3179074700 1 --2- 172.21.15.76:0/984521084 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f317406aa20 0x7f317419e620 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f317000e9a0 tx=0x7f317000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:27.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f316a7fc700 1 -- 172.21.15.76:0/984521084 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317000ca10 con 0x7f317406aa20 2024-03-18T14:52:27.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f317b2d8700 1 -- 172.21.15.76:0/984521084 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31741992c0 con 0x7f317406aa20 2024-03-18T14:52:27.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f316a7fc700 1 -- 172.21.15.76:0/984521084 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f317000cbb0 con 0x7f317406aa20 2024-03-18T14:52:27.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f316a7fc700 1 -- 172.21.15.76:0/984521084 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3170018680 con 0x7f317406aa20 2024-03-18T14:52:27.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.128+0000 7f317b2d8700 1 -- 172.21.15.76:0/984521084 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31741a5480 con 0x7f317406aa20 2024-03-18T14:52:27.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.129+0000 7f317b2d8700 1 -- 172.21.15.76:0/984521084 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31741039b0 con 0x7f317406aa20 2024-03-18T14:52:27.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.130+0000 7f316a7fc700 1 -- 172.21.15.76:0/984521084 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f3170010430 con 0x7f317406aa20 2024-03-18T14:52:27.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.131+0000 7f316a7fc700 1 --2- 172.21.15.76:0/984521084 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f31600a2920 0x7f31600a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:27.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.131+0000 7f3178873700 1 --2- 172.21.15.76:0/984521084 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f31600a2920 0x7f31600a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:27.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.131+0000 7f316a7fc700 1 -- 172.21.15.76:0/984521084 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f31700cee00 con 0x7f317406aa20 2024-03-18T14:52:27.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.131+0000 7f3178873700 1 --2- 172.21.15.76:0/984521084 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f31600a2920 0x7f31600a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3174078750 tx=0x7f3164005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:27.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.135+0000 7f316a7fc700 1 -- 172.21.15.76:0/984521084 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f317009bd70 con 0x7f317406aa20 2024-03-18T14:52:27.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.376+0000 7f317b2d8700 1 -- 172.21.15.76:0/984521084 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f317402db70 con 0x7f317406aa20 2024-03-18T14:52:27.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.377+0000 7f316a7fc700 1 -- 172.21.15.76:0/984521084 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f317009b670 con 0x7f317406aa20 2024-03-18T14:52:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:27.378 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-18T14:52:27.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.379+0000 7f317b2d8700 1 -- 172.21.15.76:0/984521084 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f31600a2920 msgr2=0x7f31600a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:27.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.379+0000 7f317b2d8700 1 --2- 172.21.15.76:0/984521084 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f31600a2920 0x7f31600a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3174078750 tx=0x7f3164005fb0 comp rx=0 tx=0).stop 2024-03-18T14:52:27.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.379+0000 7f317b2d8700 1 -- 172.21.15.76:0/984521084 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f317406aa20 msgr2=0x7f317419e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:27.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.379+0000 7f317b2d8700 1 --2- 172.21.15.76:0/984521084 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f317406aa20 0x7f317419e620 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f317000e9a0 tx=0x7f317000ecb0 comp rx=0 tx=0).stop 2024-03-18T14:52:27.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.379+0000 7f317b2d8700 1 -- 172.21.15.76:0/984521084 shutdown_connections 2024-03-18T14:52:27.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.379+0000 7f317b2d8700 1 --2- 172.21.15.76:0/984521084 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f31600a2920 0x7f31600a4de0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:27.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.379+0000 7f317b2d8700 1 --2- 172.21.15.76:0/984521084 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3174108300 0x7f31741987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:27.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.379+0000 7f317b2d8700 1 --2- 172.21.15.76:0/984521084 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f317406aa20 0x7f317419e620 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:27.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.379+0000 7f317b2d8700 1 --2- 172.21.15.76:0/984521084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f317406b3e0 0x7f317419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:27.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.379+0000 7f317b2d8700 1 -- 172.21.15.76:0/984521084 >> 172.21.15.76:0/984521084 conn(0x7f31740781e0 msgr2=0x7f31740ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:27.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.379+0000 7f317b2d8700 1 -- 172.21.15.76:0/984521084 shutdown_connections 2024-03-18T14:52:27.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:27.379+0000 7f317b2d8700 1 -- 172.21.15.76:0/984521084 wait complete. 2024-03-18T14:52:27.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:27 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/984521084' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T14:52:27.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:27 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/984521084' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T14:52:27.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:27 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/984521084' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T14:52:28.094 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-18T14:52:28.094 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:52:28.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.199+0000 7f925c932640 1 -- 172.21.15.76:0/2517911382 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92581022b0 msgr2=0x7f9258108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:28.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.199+0000 7f925c932640 1 --2- 172.21.15.76:0/2517911382 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92581022b0 0x7f9258108b40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9250009ca0 tx=0x7f925002c0f0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.199+0000 7f925c932640 1 -- 172.21.15.76:0/2517911382 shutdown_connections 2024-03-18T14:52:28.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.199+0000 7f925c932640 1 --2- 172.21.15.76:0/2517911382 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92581022b0 0x7f9258108b40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.199+0000 7f925c932640 1 --2- 172.21.15.76:0/2517911382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92581018f0 0x7f9258101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.199+0000 7f925c932640 1 --2- 172.21.15.76:0/2517911382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9258100700 0x7f9258100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.199+0000 7f925c932640 1 -- 172.21.15.76:0/2517911382 >> 172.21.15.76:0/2517911382 conn(0x7f92580fbc70 msgr2=0x7f92580fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:28.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.199+0000 7f925c932640 1 -- 172.21.15.76:0/2517911382 shutdown_connections 2024-03-18T14:52:28.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.199+0000 7f925c932640 1 -- 172.21.15.76:0/2517911382 wait complete. 2024-03-18T14:52:28.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.200+0000 7f925c932640 1 Processor -- start 2024-03-18T14:52:28.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.200+0000 7f925c932640 1 -- start start 2024-03-18T14:52:28.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.200+0000 7f925c932640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9258100700 0x7f92581969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:28.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.200+0000 7f925c932640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92581018f0 0x7f9258196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:28.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.200+0000 7f925c932640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92581022b0 0x7f925819df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:28.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.200+0000 7f925c932640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f925810c380 con 0x7f92581018f0 2024-03-18T14:52:28.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.200+0000 7f925c932640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f925810c200 con 0x7f9258100700 2024-03-18T14:52:28.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.200+0000 7f925c932640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f925810c500 con 0x7f92581022b0 2024-03-18T14:52:28.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.200+0000 7f924ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92581018f0 0x7f9258196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:28.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.200+0000 7f924ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92581018f0 0x7f9258196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45648/0 (socket says 172.21.15.76:45648) 2024-03-18T14:52:28.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.200+0000 7f925659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9258100700 0x7f92581969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:28.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.200+0000 7f924ffff640 1 -- 172.21.15.76:0/3791197992 learned_addr learned my addr 172.21.15.76:0/3791197992 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:28.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.200+0000 7f9256d9d640 1 --2- 172.21.15.76:0/3791197992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92581022b0 0x7f925819df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:28.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.201+0000 7f924ffff640 1 -- 172.21.15.76:0/3791197992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92581022b0 msgr2=0x7f925819df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:28.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.201+0000 7f924ffff640 1 --2- 172.21.15.76:0/3791197992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92581022b0 0x7f925819df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.201+0000 7f924ffff640 1 -- 172.21.15.76:0/3791197992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9258100700 msgr2=0x7f92581969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:28.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.201+0000 7f924ffff640 1 --2- 172.21.15.76:0/3791197992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9258100700 0x7f92581969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.201+0000 7f924ffff640 1 -- 172.21.15.76:0/3791197992 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925819e5e0 con 0x7f92581018f0 2024-03-18T14:52:28.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.201+0000 7f925659c640 1 --2- 172.21.15.76:0/3791197992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9258100700 0x7f92581969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:28.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.201+0000 7f924ffff640 1 --2- 172.21.15.76:0/3791197992 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92581018f0 0x7f9258196f00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f924000ea60 tx=0x7f924000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:28.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.201+0000 7f924f7fe640 1 -- 172.21.15.76:0/3791197992 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924000cde0 con 0x7f92581018f0 2024-03-18T14:52:28.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.201+0000 7f925c932640 1 -- 172.21.15.76:0/3791197992 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f925819e8d0 con 0x7f92581018f0 2024-03-18T14:52:28.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.201+0000 7f924f7fe640 1 -- 172.21.15.76:0/3791197992 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92400035f0 con 0x7f92581018f0 2024-03-18T14:52:28.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.201+0000 7f924f7fe640 1 -- 172.21.15.76:0/3791197992 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9240010670 con 0x7f92581018f0 2024-03-18T14:52:28.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.201+0000 7f925c932640 1 -- 172.21.15.76:0/3791197992 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f925819ee70 con 0x7f92581018f0 2024-03-18T14:52:28.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.202+0000 7f925c932640 1 -- 172.21.15.76:0/3791197992 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9258107a20 con 0x7f92581018f0 2024-03-18T14:52:28.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.203+0000 7f924f7fe640 1 -- 172.21.15.76:0/3791197992 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f92400037a0 con 0x7f92581018f0 2024-03-18T14:52:28.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.204+0000 7f924f7fe640 1 --2- 172.21.15.76:0/3791197992 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f92440a2890 0x7f92440a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:28.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.204+0000 7f925659c640 1 --2- 172.21.15.76:0/3791197992 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f92440a2890 0x7f92440a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:28.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.204+0000 7f924f7fe640 1 -- 172.21.15.76:0/3791197992 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f92400cef70 con 0x7f92581018f0 2024-03-18T14:52:28.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.204+0000 7f925659c640 1 --2- 172.21.15.76:0/3791197992 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f92440a2890 0x7f92440a4d50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9258101750 tx=0x7f9248005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:28.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.207+0000 7f924f7fe640 1 -- 172.21.15.76:0/3791197992 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f924009bf70 con 0x7f92581018f0 2024-03-18T14:52:28.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.383+0000 7f925c932640 1 -- 172.21.15.76:0/3791197992 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f925802d1b0 con 0x7f92581018f0 2024-03-18T14:52:28.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.385+0000 7f924f7fe640 1 -- 172.21.15.76:0/3791197992 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22580 (secure 0 0 0) 0x7f924009b870 con 0x7f92581018f0 2024-03-18T14:52:28.386 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:28.386 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:52:07.361301+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:52:28.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.387+0000 7f925c932640 1 -- 172.21.15.76:0/3791197992 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f92440a2890 msgr2=0x7f92440a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:28.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.387+0000 7f925c932640 1 --2- 172.21.15.76:0/3791197992 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f92440a2890 0x7f92440a4d50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9258101750 tx=0x7f9248005f90 comp rx=0 tx=0).stop 2024-03-18T14:52:28.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.387+0000 7f925c932640 1 -- 172.21.15.76:0/3791197992 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92581018f0 msgr2=0x7f9258196f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:28.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.387+0000 7f925c932640 1 --2- 172.21.15.76:0/3791197992 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92581018f0 0x7f9258196f00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f924000ea60 tx=0x7f924000ef20 comp rx=0 tx=0).stop 2024-03-18T14:52:28.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.387+0000 7f925c932640 1 -- 172.21.15.76:0/3791197992 shutdown_connections 2024-03-18T14:52:28.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.387+0000 7f925c932640 1 --2- 172.21.15.76:0/3791197992 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f92440a2890 0x7f92440a4d50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.387+0000 7f925c932640 1 --2- 172.21.15.76:0/3791197992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92581022b0 0x7f925819df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.387+0000 7f925c932640 1 --2- 172.21.15.76:0/3791197992 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92581018f0 0x7f9258196f00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.387+0000 7f925c932640 1 --2- 172.21.15.76:0/3791197992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9258100700 0x7f92581969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.387+0000 7f925c932640 1 -- 172.21.15.76:0/3791197992 >> 172.21.15.76:0/3791197992 conn(0x7f92580fbc70 msgr2=0x7f9258104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:28.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.387+0000 7f925c932640 1 -- 172.21.15.76:0/3791197992 shutdown_connections 2024-03-18T14:52:28.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.387+0000 7f925c932640 1 -- 172.21.15.76:0/3791197992 wait complete. 2024-03-18T14:52:28.399 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:46:26.588384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:50:59.606852+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:51:00.465325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:52:28.400 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:52:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.504+0000 7f5dab33b640 1 -- 172.21.15.76:0/1794220214 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5da4101800 msgr2=0x7f5da4101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.504+0000 7f5dab33b640 1 --2- 172.21.15.76:0/1794220214 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5da4101800 0x7f5da4101c20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5da0009900 tx=0x7f5da002bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.504+0000 7f5dab33b640 1 -- 172.21.15.76:0/1794220214 shutdown_connections 2024-03-18T14:52:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.504+0000 7f5dab33b640 1 --2- 172.21.15.76:0/1794220214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5da41033b0 0x7f5da4109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.504+0000 7f5dab33b640 1 --2- 172.21.15.76:0/1794220214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5da41029f0 0x7f5da4102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.504+0000 7f5dab33b640 1 --2- 172.21.15.76:0/1794220214 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5da4101800 0x7f5da4101c20 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.504+0000 7f5dab33b640 1 -- 172.21.15.76:0/1794220214 >> 172.21.15.76:0/1794220214 conn(0x7f5da40fcd70 msgr2=0x7f5da40ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.504+0000 7f5dab33b640 1 -- 172.21.15.76:0/1794220214 shutdown_connections 2024-03-18T14:52:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.504+0000 7f5dab33b640 1 -- 172.21.15.76:0/1794220214 wait complete. 2024-03-18T14:52:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5dab33b640 1 Processor -- start 2024-03-18T14:52:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5dab33b640 1 -- start start 2024-03-18T14:52:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5dab33b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5da4101800 0x7f5da4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5dab33b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5da41029f0 0x7f5da4198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5dab33b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5da41033b0 0x7f5da419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5dab33b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5da410d480 con 0x7f5da41033b0 2024-03-18T14:52:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5dab33b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5da410d300 con 0x7f5da41029f0 2024-03-18T14:52:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5dab33b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5da410d600 con 0x7f5da4101800 2024-03-18T14:52:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5da98d8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5da41033b0 0x7f5da419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5da98d8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5da41033b0 0x7f5da419f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59902/0 (socket says 172.21.15.76:59902) 2024-03-18T14:52:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5da98d8640 1 -- 172.21.15.76:0/2812853454 learned_addr learned my addr 172.21.15.76:0/2812853454 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5da90d7640 1 --2- 172.21.15.76:0/2812853454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5da4101800 0x7f5da4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5da88d6640 1 --2- 172.21.15.76:0/2812853454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5da41029f0 0x7f5da4198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5da98d8640 1 -- 172.21.15.76:0/2812853454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5da4101800 msgr2=0x7f5da4197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5da98d8640 1 --2- 172.21.15.76:0/2812853454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5da4101800 0x7f5da4197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5da98d8640 1 -- 172.21.15.76:0/2812853454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5da41029f0 msgr2=0x7f5da4198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5da98d8640 1 --2- 172.21.15.76:0/2812853454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5da41029f0 0x7f5da4198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5da98d8640 1 -- 172.21.15.76:0/2812853454 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5da419f710 con 0x7f5da41033b0 2024-03-18T14:52:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5da98d8640 1 --2- 172.21.15.76:0/2812853454 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5da41033b0 0x7f5da419f0a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c00eb10 tx=0x7f5d9c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5d9a7fc640 1 -- 172.21.15.76:0/2812853454 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c004b20 con 0x7f5da41033b0 2024-03-18T14:52:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5d9a7fc640 1 -- 172.21.15.76:0/2812853454 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d9c004cc0 con 0x7f5da41033b0 2024-03-18T14:52:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5dab33b640 1 -- 172.21.15.76:0/2812853454 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5da419fa00 con 0x7f5da41033b0 2024-03-18T14:52:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.505+0000 7f5d9a7fc640 1 -- 172.21.15.76:0/2812853454 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c010760 con 0x7f5da41033b0 2024-03-18T14:52:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.506+0000 7f5dab33b640 1 -- 172.21.15.76:0/2812853454 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5da419ff50 con 0x7f5da41033b0 2024-03-18T14:52:28.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.506+0000 7f5dab33b640 1 -- 172.21.15.76:0/2812853454 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5da4108ba0 con 0x7f5da41033b0 2024-03-18T14:52:28.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.507+0000 7f5d9a7fc640 1 -- 172.21.15.76:0/2812853454 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5d9c005020 con 0x7f5da41033b0 2024-03-18T14:52:28.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.508+0000 7f5d9a7fc640 1 --2- 172.21.15.76:0/2812853454 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5d900a2890 0x7f5d900a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:28.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.508+0000 7f5da90d7640 1 --2- 172.21.15.76:0/2812853454 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5d900a2890 0x7f5d900a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:28.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.508+0000 7f5d9a7fc640 1 -- 172.21.15.76:0/2812853454 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f5d9c0cf2a0 con 0x7f5da41033b0 2024-03-18T14:52:28.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.509+0000 7f5da90d7640 1 --2- 172.21.15.76:0/2812853454 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5d900a2890 0x7f5d900a4d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5da0000c00 tx=0x7f5da0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:28.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.513+0000 7f5d9a7fc640 1 -- 172.21.15.76:0/2812853454 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5d9c09c210 con 0x7f5da41033b0 2024-03-18T14:52:28.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:28 smithi076 ceph-mon[32205]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T14:52:28.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:28 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3791197992' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:28.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.692+0000 7f5dab33b640 1 -- 172.21.15.76:0/2812853454 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5da40623e0 con 0x7f5da41033b0 2024-03-18T14:52:28.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.693+0000 7f5d9a7fc640 1 -- 172.21.15.76:0/2812853454 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f5d9c09bb10 con 0x7f5da41033b0 2024-03-18T14:52:28.693 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-18T14:52:28.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.694+0000 7f5dab33b640 1 -- 172.21.15.76:0/2812853454 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5d900a2890 msgr2=0x7f5d900a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:28.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.695+0000 7f5dab33b640 1 --2- 172.21.15.76:0/2812853454 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5d900a2890 0x7f5d900a4d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5da0000c00 tx=0x7f5da0005fd0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.695+0000 7f5dab33b640 1 -- 172.21.15.76:0/2812853454 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5da41033b0 msgr2=0x7f5da419f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:28.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.695+0000 7f5dab33b640 1 --2- 172.21.15.76:0/2812853454 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5da41033b0 0x7f5da419f0a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c00eb10 tx=0x7f5d9c00c2d0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.695+0000 7f5dab33b640 1 -- 172.21.15.76:0/2812853454 shutdown_connections 2024-03-18T14:52:28.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.695+0000 7f5dab33b640 1 --2- 172.21.15.76:0/2812853454 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5d900a2890 0x7f5d900a4d50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.695+0000 7f5dab33b640 1 --2- 172.21.15.76:0/2812853454 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5da41033b0 0x7f5da419f0a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.695+0000 7f5dab33b640 1 --2- 172.21.15.76:0/2812853454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5da41029f0 0x7f5da4198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.695+0000 7f5dab33b640 1 --2- 172.21.15.76:0/2812853454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5da4101800 0x7f5da4197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.695+0000 7f5dab33b640 1 -- 172.21.15.76:0/2812853454 >> 172.21.15.76:0/2812853454 conn(0x7f5da40fcd70 msgr2=0x7f5da4105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:28.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.695+0000 7f5dab33b640 1 -- 172.21.15.76:0/2812853454 shutdown_connections 2024-03-18T14:52:28.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.695+0000 7f5dab33b640 1 -- 172.21.15.76:0/2812853454 wait complete. 2024-03-18T14:52:28.707 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:52:28.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:28 smithi174 ceph-mon[36787]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T14:52:28.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:28 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3791197992' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:28.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.810+0000 7f8d972cc640 1 -- 172.21.15.76:0/1699686774 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d901074c0 msgr2=0x7f8d90109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:28.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.810+0000 7f8d972cc640 1 --2- 172.21.15.76:0/1699686774 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d901074c0 0x7f8d90109910 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009ca0 tx=0x7f8d8802c0f0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.810+0000 7f8d972cc640 1 -- 172.21.15.76:0/1699686774 shutdown_connections 2024-03-18T14:52:28.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.810+0000 7f8d972cc640 1 --2- 172.21.15.76:0/1699686774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d90104b90 0x7f8d90106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.810+0000 7f8d972cc640 1 --2- 172.21.15.76:0/1699686774 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d901074c0 0x7f8d90109910 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.810+0000 7f8d972cc640 1 --2- 172.21.15.76:0/1699686774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d90102260 0x7f8d90104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.810+0000 7f8d972cc640 1 -- 172.21.15.76:0/1699686774 >> 172.21.15.76:0/1699686774 conn(0x7f8d900fbc70 msgr2=0x7f8d900fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:28.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.811+0000 7f8d972cc640 1 -- 172.21.15.76:0/1699686774 shutdown_connections 2024-03-18T14:52:28.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.811+0000 7f8d972cc640 1 -- 172.21.15.76:0/1699686774 wait complete. 2024-03-18T14:52:28.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.811+0000 7f8d972cc640 1 Processor -- start 2024-03-18T14:52:28.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.811+0000 7f8d972cc640 1 -- start start 2024-03-18T14:52:28.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.811+0000 7f8d972cc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d90102260 0x7f8d90197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:28.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.811+0000 7f8d972cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d90104b90 0x7f8d90198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:28.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.811+0000 7f8d972cc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d901074c0 0x7f8d9019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:28.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.811+0000 7f8d972cc640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9010d480 con 0x7f8d90102260 2024-03-18T14:52:28.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.811+0000 7f8d972cc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9010d300 con 0x7f8d90104b90 2024-03-18T14:52:28.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.811+0000 7f8d972cc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9010d600 con 0x7f8d901074c0 2024-03-18T14:52:28.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.811+0000 7f8d95068640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d90102260 0x7f8d90197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:28.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.811+0000 7f8d95068640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d90102260 0x7f8d90197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59924/0 (socket says 172.21.15.76:59924) 2024-03-18T14:52:28.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.811+0000 7f8d95068640 1 -- 172.21.15.76:0/4268544620 learned_addr learned my addr 172.21.15.76:0/4268544620 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:28.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.811+0000 7f8d94867640 1 --2- 172.21.15.76:0/4268544620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d90104b90 0x7f8d90198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:28.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.811+0000 7f8d95869640 1 --2- 172.21.15.76:0/4268544620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d901074c0 0x7f8d9019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:28.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.812+0000 7f8d95068640 1 -- 172.21.15.76:0/4268544620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d901074c0 msgr2=0x7f8d9019f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:28.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.812+0000 7f8d95068640 1 --2- 172.21.15.76:0/4268544620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d901074c0 0x7f8d9019f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.812+0000 7f8d95068640 1 -- 172.21.15.76:0/4268544620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d90104b90 msgr2=0x7f8d90198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:28.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.812+0000 7f8d95068640 1 --2- 172.21.15.76:0/4268544620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d90104b90 0x7f8d90198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:28.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.812+0000 7f8d95068640 1 -- 172.21.15.76:0/4268544620 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9019f750 con 0x7f8d90102260 2024-03-18T14:52:28.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.812+0000 7f8d95068640 1 --2- 172.21.15.76:0/4268544620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d90102260 0x7f8d90197b30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00e9f0 tx=0x7f8d8c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:28.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.812+0000 7f8d867fc640 1 -- 172.21.15.76:0/4268544620 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c00cd80 con 0x7f8d90102260 2024-03-18T14:52:28.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.812+0000 7f8d972cc640 1 -- 172.21.15.76:0/4268544620 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d9019fa40 con 0x7f8d90102260 2024-03-18T14:52:28.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.812+0000 7f8d867fc640 1 -- 172.21.15.76:0/4268544620 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d8c010910 con 0x7f8d90102260 2024-03-18T14:52:28.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.812+0000 7f8d867fc640 1 -- 172.21.15.76:0/4268544620 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c018990 con 0x7f8d90102260 2024-03-18T14:52:28.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.812+0000 7f8d972cc640 1 -- 172.21.15.76:0/4268544620 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d9019ff90 con 0x7f8d90102260 2024-03-18T14:52:28.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.813+0000 7f8d972cc640 1 -- 172.21.15.76:0/4268544620 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d900fd760 con 0x7f8d90102260 2024-03-18T14:52:28.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.814+0000 7f8d867fc640 1 -- 172.21.15.76:0/4268544620 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8d8c010ac0 con 0x7f8d90102260 2024-03-18T14:52:28.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.815+0000 7f8d867fc640 1 --2- 172.21.15.76:0/4268544620 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8d7c0a2890 0x7f8d7c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:28.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.815+0000 7f8d94867640 1 --2- 172.21.15.76:0/4268544620 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8d7c0a2890 0x7f8d7c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:28.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.815+0000 7f8d867fc640 1 -- 172.21.15.76:0/4268544620 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f8d8c0cef10 con 0x7f8d90102260 2024-03-18T14:52:28.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.815+0000 7f8d94867640 1 --2- 172.21.15.76:0/4268544620 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8d7c0a2890 0x7f8d7c0a4d50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8d90199040 tx=0x7f8d80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:28.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.819+0000 7f8d867fc640 1 -- 172.21.15.76:0/4268544620 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8d8c09b830 con 0x7f8d90102260 2024-03-18T14:52:28.870 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:28 smithi077 ceph-mon[35865]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T14:52:28.870 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:28 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3791197992' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:28.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.996+0000 7f8d972cc640 1 -- 172.21.15.76:0/4268544620 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8d900623e0 con 0x7f8d90102260 2024-03-18T14:52:28.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.996+0000 7f8d867fc640 1 -- 172.21.15.76:0/4268544620 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f8d8c09f8d0 con 0x7f8d90102260 2024-03-18T14:52:28.997 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-18T14:52:28.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.999+0000 7f8d972cc640 1 -- 172.21.15.76:0/4268544620 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8d7c0a2890 msgr2=0x7f8d7c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:28.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.999+0000 7f8d972cc640 1 --2- 172.21.15.76:0/4268544620 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8d7c0a2890 0x7f8d7c0a4d50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8d90199040 tx=0x7f8d80005f90 comp rx=0 tx=0).stop 2024-03-18T14:52:28.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.999+0000 7f8d972cc640 1 -- 172.21.15.76:0/4268544620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d90102260 msgr2=0x7f8d90197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:29.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.999+0000 7f8d972cc640 1 --2- 172.21.15.76:0/4268544620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d90102260 0x7f8d90197b30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00e9f0 tx=0x7f8d8c00eeb0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.999+0000 7f8d972cc640 1 -- 172.21.15.76:0/4268544620 shutdown_connections 2024-03-18T14:52:29.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.999+0000 7f8d972cc640 1 --2- 172.21.15.76:0/4268544620 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8d7c0a2890 0x7f8d7c0a4d50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.999+0000 7f8d972cc640 1 --2- 172.21.15.76:0/4268544620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d901074c0 0x7f8d9019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.999+0000 7f8d972cc640 1 --2- 172.21.15.76:0/4268544620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d90104b90 0x7f8d90198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.999+0000 7f8d972cc640 1 --2- 172.21.15.76:0/4268544620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d90102260 0x7f8d90197b30 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.999+0000 7f8d972cc640 1 -- 172.21.15.76:0/4268544620 >> 172.21.15.76:0/4268544620 conn(0x7f8d900fbc70 msgr2=0x7f8d90102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:29.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.999+0000 7f8d972cc640 1 -- 172.21.15.76:0/4268544620 shutdown_connections 2024-03-18T14:52:29.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:28.999+0000 7f8d972cc640 1 -- 172.21.15.76:0/4268544620 wait complete. 2024-03-18T14:52:29.010 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:52:29.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.114+0000 7fc7f8bf4640 1 -- 172.21.15.76:0/695436354 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7f41022a0 msgr2=0x7fc7f4108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:29.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.114+0000 7fc7f8bf4640 1 --2- 172.21.15.76:0/695436354 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7f41022a0 0x7fc7f4108b30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4009900 tx=0x7fc7e402bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:29.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.114+0000 7fc7f8bf4640 1 -- 172.21.15.76:0/695436354 shutdown_connections 2024-03-18T14:52:29.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.114+0000 7fc7f8bf4640 1 --2- 172.21.15.76:0/695436354 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7f41022a0 0x7fc7f4108b30 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.114+0000 7fc7f8bf4640 1 --2- 172.21.15.76:0/695436354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7f41018e0 0x7fc7f4101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.114+0000 7fc7f8bf4640 1 --2- 172.21.15.76:0/695436354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f41006f0 0x7fc7f4100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.114+0000 7fc7f8bf4640 1 -- 172.21.15.76:0/695436354 >> 172.21.15.76:0/695436354 conn(0x7fc7f40fbc80 msgr2=0x7fc7f40fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:29.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.114+0000 7fc7f8bf4640 1 -- 172.21.15.76:0/695436354 shutdown_connections 2024-03-18T14:52:29.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.114+0000 7fc7f8bf4640 1 -- 172.21.15.76:0/695436354 wait complete. 2024-03-18T14:52:29.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.114+0000 7fc7f8bf4640 1 Processor -- start 2024-03-18T14:52:29.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.114+0000 7fc7f8bf4640 1 -- start start 2024-03-18T14:52:29.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f8bf4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7f41006f0 0x7fc7f4104750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:29.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f8bf4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7f41018e0 0x7fc7f4102d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:29.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f8bf4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f41022a0 0x7fc7f41032d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:29.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f8bf4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f410c350 con 0x7fc7f41018e0 2024-03-18T14:52:29.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f8bf4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f410c1d0 con 0x7fc7f41006f0 2024-03-18T14:52:29.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f8bf4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f410c4d0 con 0x7fc7f41022a0 2024-03-18T14:52:29.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f1d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7f41018e0 0x7fc7f4102d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:29.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7f41006f0 0x7fc7f4104750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:29.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f1d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7f41018e0 0x7fc7f4102d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59928/0 (socket says 172.21.15.76:59928) 2024-03-18T14:52:29.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f1d9b640 1 -- 172.21.15.76:0/3810185669 learned_addr learned my addr 172.21.15.76:0/3810185669 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:29.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f2d9d640 1 --2- 172.21.15.76:0/3810185669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f41022a0 0x7fc7f41032d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:29.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f1d9b640 1 -- 172.21.15.76:0/3810185669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f41022a0 msgr2=0x7fc7f41032d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:29.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f1d9b640 1 --2- 172.21.15.76:0/3810185669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f41022a0 0x7fc7f41032d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f1d9b640 1 -- 172.21.15.76:0/3810185669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7f41006f0 msgr2=0x7fc7f4104750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:29.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f1d9b640 1 --2- 172.21.15.76:0/3810185669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7f41006f0 0x7fc7f4104750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f1d9b640 1 -- 172.21.15.76:0/3810185669 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f4103b60 con 0x7fc7f41018e0 2024-03-18T14:52:29.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f259c640 1 --2- 172.21.15.76:0/3810185669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7f41006f0 0x7fc7f4104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:29.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f1d9b640 1 --2- 172.21.15.76:0/3810185669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7f41018e0 0x7fc7f4102d90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc00ebc0 tx=0x7fc7dc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:29.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7eb7fe640 1 -- 172.21.15.76:0/3810185669 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc010630 con 0x7fc7f41018e0 2024-03-18T14:52:29.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7eb7fe640 1 -- 172.21.15.76:0/3810185669 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7dc010cb0 con 0x7fc7f41018e0 2024-03-18T14:52:29.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f8bf4640 1 -- 172.21.15.76:0/3810185669 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7f41a1be0 con 0x7fc7f41018e0 2024-03-18T14:52:29.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7eb7fe640 1 -- 172.21.15.76:0/3810185669 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc018bd0 con 0x7fc7f41018e0 2024-03-18T14:52:29.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.115+0000 7fc7f8bf4640 1 -- 172.21.15.76:0/3810185669 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7f41a2130 con 0x7fc7f41018e0 2024-03-18T14:52:29.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.116+0000 7fc7f8bf4640 1 -- 172.21.15.76:0/3810185669 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7f41a2300 con 0x7fc7f41018e0 2024-03-18T14:52:29.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.117+0000 7fc7eb7fe640 1 -- 172.21.15.76:0/3810185669 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc7dc0107d0 con 0x7fc7f41018e0 2024-03-18T14:52:29.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.118+0000 7fc7eb7fe640 1 --2- 172.21.15.76:0/3810185669 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc7e00a2890 0x7fc7e00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:29.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.118+0000 7fc7f259c640 1 --2- 172.21.15.76:0/3810185669 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc7e00a2890 0x7fc7e00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:29.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.118+0000 7fc7eb7fe640 1 -- 172.21.15.76:0/3810185669 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc7dc0cf1a0 con 0x7fc7f41018e0 2024-03-18T14:52:29.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.119+0000 7fc7f259c640 1 --2- 172.21.15.76:0/3810185669 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc7e00a2890 0x7fc7e00a4d50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4101740 tx=0x7fc7ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:29.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.123+0000 7fc7eb7fe640 1 -- 172.21.15.76:0/3810185669 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc7dc09bac0 con 0x7fc7f41018e0 2024-03-18T14:52:29.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.298+0000 7fc7f8bf4640 1 -- 172.21.15.76:0/3810185669 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc7f41a2690 con 0x7fc7f41018e0 2024-03-18T14:52:29.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.298+0000 7fc7eb7fe640 1 -- 172.21.15.76:0/3810185669 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fc7dc09fb60 con 0x7fc7f41018e0 2024-03-18T14:52:29.299 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-18T14:52:29.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.300+0000 7fc7f8bf4640 1 -- 172.21.15.76:0/3810185669 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc7e00a2890 msgr2=0x7fc7e00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:29.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.300+0000 7fc7f8bf4640 1 --2- 172.21.15.76:0/3810185669 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc7e00a2890 0x7fc7e00a4d50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4101740 tx=0x7fc7ec005f90 comp rx=0 tx=0).stop 2024-03-18T14:52:29.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.300+0000 7fc7f8bf4640 1 -- 172.21.15.76:0/3810185669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7f41018e0 msgr2=0x7fc7f4102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:29.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.300+0000 7fc7f8bf4640 1 --2- 172.21.15.76:0/3810185669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7f41018e0 0x7fc7f4102d90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc00ebc0 tx=0x7fc7dc00c6a0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.300+0000 7fc7f8bf4640 1 -- 172.21.15.76:0/3810185669 shutdown_connections 2024-03-18T14:52:29.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.300+0000 7fc7f8bf4640 1 --2- 172.21.15.76:0/3810185669 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc7e00a2890 0x7fc7e00a4d50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.300+0000 7fc7f8bf4640 1 --2- 172.21.15.76:0/3810185669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f41022a0 0x7fc7f41032d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.300+0000 7fc7f8bf4640 1 --2- 172.21.15.76:0/3810185669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7f41018e0 0x7fc7f4102d90 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.300+0000 7fc7f8bf4640 1 --2- 172.21.15.76:0/3810185669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7f41006f0 0x7fc7f4104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.300+0000 7fc7f8bf4640 1 -- 172.21.15.76:0/3810185669 >> 172.21.15.76:0/3810185669 conn(0x7fc7f40fbc80 msgr2=0x7fc7f4069db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:29.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.300+0000 7fc7f8bf4640 1 -- 172.21.15.76:0/3810185669 shutdown_connections 2024-03-18T14:52:29.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.300+0000 7fc7f8bf4640 1 -- 172.21.15.76:0/3810185669 wait complete. 2024-03-18T14:52:29.312 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:52:29.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.415+0000 7f8a6f6dc640 1 -- 172.21.15.76:0/997956651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a68102280 msgr2=0x7f8a68104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:29.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.415+0000 7f8a6f6dc640 1 --2- 172.21.15.76:0/997956651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a68102280 0x7f8a68104670 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8a64009900 tx=0x7f8a6402bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:29.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.415+0000 7f8a6f6dc640 1 -- 172.21.15.76:0/997956651 shutdown_connections 2024-03-18T14:52:29.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.415+0000 7f8a6f6dc640 1 --2- 172.21.15.76:0/997956651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a681074e0 0x7f8a68109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6f6dc640 1 --2- 172.21.15.76:0/997956651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a68104bb0 0x7f8a68106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6f6dc640 1 --2- 172.21.15.76:0/997956651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a68102280 0x7f8a68104670 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6f6dc640 1 -- 172.21.15.76:0/997956651 >> 172.21.15.76:0/997956651 conn(0x7f8a680fbc70 msgr2=0x7f8a680fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:29.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6f6dc640 1 -- 172.21.15.76:0/997956651 shutdown_connections 2024-03-18T14:52:29.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6f6dc640 1 -- 172.21.15.76:0/997956651 wait complete. 2024-03-18T14:52:29.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6f6dc640 1 Processor -- start 2024-03-18T14:52:29.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6f6dc640 1 -- start start 2024-03-18T14:52:29.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6f6dc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a68102280 0x7f8a68197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:29.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6f6dc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a68104bb0 0x7f8a68198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:29.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6f6dc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a681074e0 0x7f8a6819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:29.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6f6dc640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6810d4a0 con 0x7f8a681074e0 2024-03-18T14:52:29.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6f6dc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6810d320 con 0x7f8a68104bb0 2024-03-18T14:52:29.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6f6dc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6810d620 con 0x7f8a68102280 2024-03-18T14:52:29.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6dc79640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a681074e0 0x7f8a6819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:29.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6dc79640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a681074e0 0x7f8a6819f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59944/0 (socket says 172.21.15.76:59944) 2024-03-18T14:52:29.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6dc79640 1 -- 172.21.15.76:0/2801007342 learned_addr learned my addr 172.21.15.76:0/2801007342 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:29.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6d478640 1 --2- 172.21.15.76:0/2801007342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a68102280 0x7f8a68197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:29.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.416+0000 7f8a6cc77640 1 --2- 172.21.15.76:0/2801007342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a68104bb0 0x7f8a68198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:29.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.417+0000 7f8a6dc79640 1 -- 172.21.15.76:0/2801007342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a68102280 msgr2=0x7f8a68197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:29.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.417+0000 7f8a6dc79640 1 --2- 172.21.15.76:0/2801007342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a68102280 0x7f8a68197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.417+0000 7f8a6dc79640 1 -- 172.21.15.76:0/2801007342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a68104bb0 msgr2=0x7f8a68198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:29.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.417+0000 7f8a6dc79640 1 --2- 172.21.15.76:0/2801007342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a68104bb0 0x7f8a68198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.417+0000 7f8a6dc79640 1 -- 172.21.15.76:0/2801007342 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6819f740 con 0x7f8a681074e0 2024-03-18T14:52:29.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.417+0000 7f8a6dc79640 1 --2- 172.21.15.76:0/2801007342 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a681074e0 0x7f8a6819f0d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8a6000eb10 tx=0x7f8a6000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:29.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.417+0000 7f8a5e7fc640 1 -- 172.21.15.76:0/2801007342 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60004b20 con 0x7f8a681074e0 2024-03-18T14:52:29.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.417+0000 7f8a5e7fc640 1 -- 172.21.15.76:0/2801007342 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a60004cc0 con 0x7f8a681074e0 2024-03-18T14:52:29.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.417+0000 7f8a6f6dc640 1 -- 172.21.15.76:0/2801007342 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a6819fa30 con 0x7f8a681074e0 2024-03-18T14:52:29.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.417+0000 7f8a5e7fc640 1 -- 172.21.15.76:0/2801007342 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60010760 con 0x7f8a681074e0 2024-03-18T14:52:29.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.417+0000 7f8a6f6dc640 1 -- 172.21.15.76:0/2801007342 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a6819ffd0 con 0x7f8a681074e0 2024-03-18T14:52:29.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.418+0000 7f8a6f6dc640 1 -- 172.21.15.76:0/2801007342 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a680fd760 con 0x7f8a681074e0 2024-03-18T14:52:29.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.419+0000 7f8a5e7fc640 1 -- 172.21.15.76:0/2801007342 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8a60005000 con 0x7f8a681074e0 2024-03-18T14:52:29.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.419+0000 7f8a5e7fc640 1 --2- 172.21.15.76:0/2801007342 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8a540a2910 0x7f8a540a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:29.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.420+0000 7f8a6d478640 1 --2- 172.21.15.76:0/2801007342 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8a540a2910 0x7f8a540a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:29.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.420+0000 7f8a5e7fc640 1 -- 172.21.15.76:0/2801007342 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f8a600cf590 con 0x7f8a681074e0 2024-03-18T14:52:29.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.420+0000 7f8a6d478640 1 --2- 172.21.15.76:0/2801007342 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8a540a2910 0x7f8a540a4dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8a6402c7b0 tx=0x7f8a64005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:29.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.423+0000 7f8a5e7fc640 1 -- 172.21.15.76:0/2801007342 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8a6009c500 con 0x7f8a681074e0 2024-03-18T14:52:29.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.639+0000 7f8a6f6dc640 1 -- 172.21.15.76:0/2801007342 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a6802d1b0 con 0x7f8a681074e0 2024-03-18T14:52:29.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.640+0000 7f8a5e7fc640 1 -- 172.21.15.76:0/2801007342 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11863 (secure 0 0 0) 0x7f8a6009be00 con 0x7f8a681074e0 2024-03-18T14:52:29.641 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:29.641 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/2801069369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":2801069369},{"type":"v1","addr":"172.21.15.76:6839","nonce":2801069369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2645229232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2645229232},{"type":"v1","addr":"172.21.15.76:6841","nonce":2645229232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T14:51:01.293009+0000","modified":"2024-03-18T14:51:07.032425+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24490,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6839/399810652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":399810652},{"type":"v1","addr":"172.21.15.174:6839","nonce":399810652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1685041985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1685041985},{"type":"v1","addr":"172.21.15.77:6839","nonce":1685041985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24463,24490,24524]},"id":1}]} 2024-03-18T14:52:29.641 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 23 2024-03-18T14:52:29.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.642+0000 7f8a6f6dc640 1 -- 172.21.15.76:0/2801007342 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8a540a2910 msgr2=0x7f8a540a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:29.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.642+0000 7f8a6f6dc640 1 --2- 172.21.15.76:0/2801007342 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8a540a2910 0x7f8a540a4dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8a6402c7b0 tx=0x7f8a64005fd0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.642+0000 7f8a6f6dc640 1 -- 172.21.15.76:0/2801007342 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a681074e0 msgr2=0x7f8a6819f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:29.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.642+0000 7f8a6f6dc640 1 --2- 172.21.15.76:0/2801007342 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a681074e0 0x7f8a6819f0d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8a6000eb10 tx=0x7f8a6000c2d0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.642+0000 7f8a6f6dc640 1 -- 172.21.15.76:0/2801007342 shutdown_connections 2024-03-18T14:52:29.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.642+0000 7f8a6f6dc640 1 --2- 172.21.15.76:0/2801007342 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8a540a2910 0x7f8a540a4dd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.642+0000 7f8a6f6dc640 1 --2- 172.21.15.76:0/2801007342 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a681074e0 0x7f8a6819f0d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.642+0000 7f8a6f6dc640 1 --2- 172.21.15.76:0/2801007342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a68104bb0 0x7f8a68198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.642+0000 7f8a6f6dc640 1 --2- 172.21.15.76:0/2801007342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a68102280 0x7f8a68197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.642+0000 7f8a6f6dc640 1 -- 172.21.15.76:0/2801007342 >> 172.21.15.76:0/2801007342 conn(0x7f8a680fbc70 msgr2=0x7f8a68102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:29.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.642+0000 7f8a6f6dc640 1 -- 172.21.15.76:0/2801007342 shutdown_connections 2024-03-18T14:52:29.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.642+0000 7f8a6f6dc640 1 -- 172.21.15.76:0/2801007342 wait complete. 2024-03-18T14:52:29.652 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:29 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2812853454' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:52:29.653 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:29 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/4268544620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:52:29.653 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:29 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3810185669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:52:29.654 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:52:29.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.754+0000 7f1b2bd10640 1 -- 172.21.15.76:0/1914355945 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b241022b0 msgr2=0x7f1b24108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:29.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.754+0000 7f1b2bd10640 1 --2- 172.21.15.76:0/1914355945 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b241022b0 0x7f1b24108b40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c009ca0 tx=0x7f1b1c02c0f0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.754+0000 7f1b2bd10640 1 -- 172.21.15.76:0/1914355945 shutdown_connections 2024-03-18T14:52:29.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.754+0000 7f1b2bd10640 1 --2- 172.21.15.76:0/1914355945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b241018f0 0x7f1b24101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.754+0000 7f1b2bd10640 1 --2- 172.21.15.76:0/1914355945 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b241022b0 0x7f1b24108b40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.754+0000 7f1b2bd10640 1 --2- 172.21.15.76:0/1914355945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b24100700 0x7f1b24100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.754+0000 7f1b2bd10640 1 -- 172.21.15.76:0/1914355945 >> 172.21.15.76:0/1914355945 conn(0x7f1b240fbc70 msgr2=0x7f1b240fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:29.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.754+0000 7f1b2bd10640 1 -- 172.21.15.76:0/1914355945 shutdown_connections 2024-03-18T14:52:29.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.754+0000 7f1b2bd10640 1 -- 172.21.15.76:0/1914355945 wait complete. 2024-03-18T14:52:29.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.755+0000 7f1b2bd10640 1 Processor -- start 2024-03-18T14:52:29.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.755+0000 7f1b2bd10640 1 -- start start 2024-03-18T14:52:29.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.755+0000 7f1b2bd10640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b24100700 0x7f1b240694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:29.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.755+0000 7f1b2bd10640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b241018f0 0x7f1b2419c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:29.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.755+0000 7f1b2bd10640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b241022b0 0x7f1b2419edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:29.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.755+0000 7f1b2bd10640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1b2410c360 con 0x7f1b241018f0 2024-03-18T14:52:29.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.755+0000 7f1b2bd10640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1b2410c1e0 con 0x7f1b24100700 2024-03-18T14:52:29.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.755+0000 7f1b2bd10640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1b2410c4e0 con 0x7f1b241022b0 2024-03-18T14:52:29.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.755+0000 7f1b292ab640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b241018f0 0x7f1b2419c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:29.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.755+0000 7f1b29aac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b24100700 0x7f1b240694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:29.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.755+0000 7f1b292ab640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b241018f0 0x7f1b2419c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59960/0 (socket says 172.21.15.76:59960) 2024-03-18T14:52:29.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.755+0000 7f1b29aac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b24100700 0x7f1b240694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:34576/0 (socket says 172.21.15.76:34576) 2024-03-18T14:52:29.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.755+0000 7f1b292ab640 1 -- 172.21.15.76:0/2279874053 learned_addr learned my addr 172.21.15.76:0/2279874053 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:29.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.755+0000 7f1b2a2ad640 1 --2- 172.21.15.76:0/2279874053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b241022b0 0x7f1b2419edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:29.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.756+0000 7f1b292ab640 1 -- 172.21.15.76:0/2279874053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b241022b0 msgr2=0x7f1b2419edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:29.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.756+0000 7f1b292ab640 1 --2- 172.21.15.76:0/2279874053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b241022b0 0x7f1b2419edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.756+0000 7f1b292ab640 1 -- 172.21.15.76:0/2279874053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b24100700 msgr2=0x7f1b240694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:29.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.756+0000 7f1b292ab640 1 --2- 172.21.15.76:0/2279874053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b24100700 0x7f1b240694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.756+0000 7f1b292ab640 1 -- 172.21.15.76:0/2279874053 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b2419f6b0 con 0x7f1b241018f0 2024-03-18T14:52:29.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.756+0000 7f1b292ab640 1 --2- 172.21.15.76:0/2279874053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b241018f0 0x7f1b2419c7b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400ea60 tx=0x7f1b1400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:29.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.756+0000 7f1b1affd640 1 -- 172.21.15.76:0/2279874053 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1400cde0 con 0x7f1b241018f0 2024-03-18T14:52:29.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.756+0000 7f1b1affd640 1 -- 172.21.15.76:0/2279874053 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b140035f0 con 0x7f1b241018f0 2024-03-18T14:52:29.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.756+0000 7f1b2bd10640 1 -- 172.21.15.76:0/2279874053 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b2419f9a0 con 0x7f1b241018f0 2024-03-18T14:52:29.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.756+0000 7f1b1affd640 1 -- 172.21.15.76:0/2279874053 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b14010670 con 0x7f1b241018f0 2024-03-18T14:52:29.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.756+0000 7f1b2bd10640 1 -- 172.21.15.76:0/2279874053 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b2419ff40 con 0x7f1b241018f0 2024-03-18T14:52:29.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.757+0000 7f1b2bd10640 1 -- 172.21.15.76:0/2279874053 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b241079c0 con 0x7f1b241018f0 2024-03-18T14:52:29.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.758+0000 7f1b1affd640 1 -- 172.21.15.76:0/2279874053 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f1b140037a0 con 0x7f1b241018f0 2024-03-18T14:52:29.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.758+0000 7f1b1affd640 1 --2- 172.21.15.76:0/2279874053 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1b100a2910 0x7f1b100a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:29.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.758+0000 7f1b29aac640 1 --2- 172.21.15.76:0/2279874053 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1b100a2910 0x7f1b100a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:29.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.759+0000 7f1b1affd640 1 -- 172.21.15.76:0/2279874053 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f1b140ced90 con 0x7f1b241018f0 2024-03-18T14:52:29.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.759+0000 7f1b29aac640 1 --2- 172.21.15.76:0/2279874053 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1b100a2910 0x7f1b100a4dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1b24101750 tx=0x7f1b20005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:29.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.763+0000 7f1b1affd640 1 -- 172.21.15.76:0/2279874053 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1b140d2030 con 0x7f1b241018f0 2024-03-18T14:52:29.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.942+0000 7f1b2bd10640 1 -- 172.21.15.76:0/2279874053 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b2402d3b0 con 0x7f1b241018f0 2024-03-18T14:52:29.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.943+0000 7f1b1affd640 1 -- 172.21.15.76:0/2279874053 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22580 (secure 0 0 0) 0x7f1b1409bd00 con 0x7f1b241018f0 2024-03-18T14:52:29.944 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:29.945 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:52:07.361301+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:52:29.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.946+0000 7f1b2bd10640 1 -- 172.21.15.76:0/2279874053 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1b100a2910 msgr2=0x7f1b100a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:29.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.946+0000 7f1b2bd10640 1 --2- 172.21.15.76:0/2279874053 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1b100a2910 0x7f1b100a4dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1b24101750 tx=0x7f1b20005f90 comp rx=0 tx=0).stop 2024-03-18T14:52:29.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.946+0000 7f1b2bd10640 1 -- 172.21.15.76:0/2279874053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b241018f0 msgr2=0x7f1b2419c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:29.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.946+0000 7f1b2bd10640 1 --2- 172.21.15.76:0/2279874053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b241018f0 0x7f1b2419c7b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400ea60 tx=0x7f1b1400ef20 comp rx=0 tx=0).stop 2024-03-18T14:52:29.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.946+0000 7f1b2bd10640 1 -- 172.21.15.76:0/2279874053 shutdown_connections 2024-03-18T14:52:29.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.946+0000 7f1b2bd10640 1 --2- 172.21.15.76:0/2279874053 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1b100a2910 0x7f1b100a4dd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.946+0000 7f1b2bd10640 1 --2- 172.21.15.76:0/2279874053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b241022b0 0x7f1b2419edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.946+0000 7f1b2bd10640 1 --2- 172.21.15.76:0/2279874053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b241018f0 0x7f1b2419c7b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.946+0000 7f1b2bd10640 1 --2- 172.21.15.76:0/2279874053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b24100700 0x7f1b240694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:29.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.946+0000 7f1b2bd10640 1 -- 172.21.15.76:0/2279874053 >> 172.21.15.76:0/2279874053 conn(0x7f1b240fbc70 msgr2=0x7f1b24104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:29.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.946+0000 7f1b2bd10640 1 -- 172.21.15.76:0/2279874053 shutdown_connections 2024-03-18T14:52:29.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:29.946+0000 7f1b2bd10640 1 -- 172.21.15.76:0/2279874053 wait complete. 2024-03-18T14:52:29.958 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:46:26.588384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:50:59.606852+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:51:00.465325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:52:29.958 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:52:29.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:29 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2812853454' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:52:29.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:29 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/4268544620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:52:29.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:29 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3810185669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:52:29.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:29 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2812853454' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:52:29.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:29 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/4268544620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:52:29.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:29 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3810185669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:52:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.062+0000 7f43b22ea640 1 -- 172.21.15.76:0/3977490838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43ac104bd0 msgr2=0x7f43ac106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.062+0000 7f43b22ea640 1 --2- 172.21.15.76:0/3977490838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43ac104bd0 0x7f43ac106fc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009900 tx=0x7f43a002bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.062+0000 7f43b22ea640 1 -- 172.21.15.76:0/3977490838 shutdown_connections 2024-03-18T14:52:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.062+0000 7f43b22ea640 1 --2- 172.21.15.76:0/3977490838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43ac107500 0x7f43ac109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.062+0000 7f43b22ea640 1 --2- 172.21.15.76:0/3977490838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43ac104bd0 0x7f43ac106fc0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.062+0000 7f43b22ea640 1 --2- 172.21.15.76:0/3977490838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43ac1022a0 0x7f43ac104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.062+0000 7f43b22ea640 1 -- 172.21.15.76:0/3977490838 >> 172.21.15.76:0/3977490838 conn(0x7f43ac0fbc70 msgr2=0x7f43ac0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.062+0000 7f43b22ea640 1 -- 172.21.15.76:0/3977490838 shutdown_connections 2024-03-18T14:52:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.062+0000 7f43b22ea640 1 -- 172.21.15.76:0/3977490838 wait complete. 2024-03-18T14:52:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.063+0000 7f43b22ea640 1 Processor -- start 2024-03-18T14:52:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.063+0000 7f43b22ea640 1 -- start start 2024-03-18T14:52:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.063+0000 7f43b22ea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43ac1022a0 0x7f43ac197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.063+0000 7f43b22ea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43ac104bd0 0x7f43ac198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.063+0000 7f43b22ea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43ac107500 0x7f43ac19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.063+0000 7f43b22ea640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac10d440 con 0x7f43ac104bd0 2024-03-18T14:52:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.063+0000 7f43b22ea640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac10d2c0 con 0x7f43ac1022a0 2024-03-18T14:52:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.063+0000 7f43b22ea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac10d5c0 con 0x7f43ac107500 2024-03-18T14:52:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.063+0000 7f43ab7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43ac104bd0 0x7f43ac198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.063+0000 7f43ab7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43ac104bd0 0x7f43ac198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59984/0 (socket says 172.21.15.76:59984) 2024-03-18T14:52:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.063+0000 7f43ab7fe640 1 -- 172.21.15.76:0/2577553876 learned_addr learned my addr 172.21.15.76:0/2577553876 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.063+0000 7f43abfff640 1 --2- 172.21.15.76:0/2577553876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43ac1022a0 0x7f43ac197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.063+0000 7f43b0887640 1 --2- 172.21.15.76:0/2577553876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43ac107500 0x7f43ac19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.064+0000 7f43ab7fe640 1 -- 172.21.15.76:0/2577553876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43ac107500 msgr2=0x7f43ac19f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.064+0000 7f43ab7fe640 1 --2- 172.21.15.76:0/2577553876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43ac107500 0x7f43ac19f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.064+0000 7f43ab7fe640 1 -- 172.21.15.76:0/2577553876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43ac1022a0 msgr2=0x7f43ac197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.064+0000 7f43ab7fe640 1 --2- 172.21.15.76:0/2577553876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43ac1022a0 0x7f43ac197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.064+0000 7f43ab7fe640 1 -- 172.21.15.76:0/2577553876 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43ac19f790 con 0x7f43ac104bd0 2024-03-18T14:52:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.064+0000 7f43abfff640 1 --2- 172.21.15.76:0/2577553876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43ac1022a0 0x7f43ac197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.064+0000 7f43ab7fe640 1 --2- 172.21.15.76:0/2577553876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43ac104bd0 0x7f43ac198100 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f43a0000c00 tx=0x7f43a002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.064+0000 7f43a97fa640 1 -- 172.21.15.76:0/2577553876 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a0041070 con 0x7f43ac104bd0 2024-03-18T14:52:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.064+0000 7f43a97fa640 1 -- 172.21.15.76:0/2577553876 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43a0004450 con 0x7f43ac104bd0 2024-03-18T14:52:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.064+0000 7f43b22ea640 1 -- 172.21.15.76:0/2577553876 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43ac100090 con 0x7f43ac104bd0 2024-03-18T14:52:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.064+0000 7f43a97fa640 1 -- 172.21.15.76:0/2577553876 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a0003df0 con 0x7f43ac104bd0 2024-03-18T14:52:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.064+0000 7f43b22ea640 1 -- 172.21.15.76:0/2577553876 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43ac100580 con 0x7f43ac104bd0 2024-03-18T14:52:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.065+0000 7f43b22ea640 1 -- 172.21.15.76:0/2577553876 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43ac0fd760 con 0x7f43ac104bd0 2024-03-18T14:52:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.066+0000 7f43a97fa640 1 -- 172.21.15.76:0/2577553876 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f43a0044070 con 0x7f43ac104bd0 2024-03-18T14:52:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.067+0000 7f43a97fa640 1 --2- 172.21.15.76:0/2577553876 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f43940a2890 0x7f43940a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.067+0000 7f43abfff640 1 --2- 172.21.15.76:0/2577553876 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f43940a2890 0x7f43940a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.067+0000 7f43a97fa640 1 -- 172.21.15.76:0/2577553876 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f43a00f05a0 con 0x7f43ac104bd0 2024-03-18T14:52:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.067+0000 7f43abfff640 1 --2- 172.21.15.76:0/2577553876 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f43940a2890 0x7f43940a4d50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f439c0095f0 tx=0x7f439c012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:30.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.071+0000 7f43a97fa640 1 -- 172.21.15.76:0/2577553876 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f43a00bd6b0 con 0x7f43ac104bd0 2024-03-18T14:52:30.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.246+0000 7f43b22ea640 1 -- 172.21.15.76:0/2577553876 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f43ac02d400 con 0x7f43ac104bd0 2024-03-18T14:52:30.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.246+0000 7f43a97fa640 1 -- 172.21.15.76:0/2577553876 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f43a00bcfb0 con 0x7f43ac104bd0 2024-03-18T14:52:30.247 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-18T14:52:30.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.248+0000 7f43b22ea640 1 -- 172.21.15.76:0/2577553876 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f43940a2890 msgr2=0x7f43940a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.248+0000 7f43b22ea640 1 --2- 172.21.15.76:0/2577553876 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f43940a2890 0x7f43940a4d50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f439c0095f0 tx=0x7f439c012040 comp rx=0 tx=0).stop 2024-03-18T14:52:30.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.248+0000 7f43b22ea640 1 -- 172.21.15.76:0/2577553876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43ac104bd0 msgr2=0x7f43ac198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.248+0000 7f43b22ea640 1 --2- 172.21.15.76:0/2577553876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43ac104bd0 0x7f43ac198100 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f43a0000c00 tx=0x7f43a002c640 comp rx=0 tx=0).stop 2024-03-18T14:52:30.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.248+0000 7f43b22ea640 1 -- 172.21.15.76:0/2577553876 shutdown_connections 2024-03-18T14:52:30.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.248+0000 7f43b22ea640 1 --2- 172.21.15.76:0/2577553876 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f43940a2890 0x7f43940a4d50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.248+0000 7f43b22ea640 1 --2- 172.21.15.76:0/2577553876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43ac107500 0x7f43ac19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.248+0000 7f43b22ea640 1 --2- 172.21.15.76:0/2577553876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43ac104bd0 0x7f43ac198100 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.248+0000 7f43b22ea640 1 --2- 172.21.15.76:0/2577553876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43ac1022a0 0x7f43ac197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.249+0000 7f43b22ea640 1 -- 172.21.15.76:0/2577553876 >> 172.21.15.76:0/2577553876 conn(0x7f43ac0fbc70 msgr2=0x7f43ac102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:30.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.249+0000 7f43b22ea640 1 -- 172.21.15.76:0/2577553876 shutdown_connections 2024-03-18T14:52:30.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.249+0000 7f43b22ea640 1 -- 172.21.15.76:0/2577553876 wait complete. 2024-03-18T14:52:30.261 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:52:30.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.363+0000 7f8a7851b640 1 -- 172.21.15.76:0/3000512944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a701018e0 msgr2=0x7f8a70101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.363+0000 7f8a7851b640 1 --2- 172.21.15.76:0/3000512944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a701018e0 0x7f8a70101d60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009900 tx=0x7f8a6002bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.363+0000 7f8a7851b640 1 -- 172.21.15.76:0/3000512944 shutdown_connections 2024-03-18T14:52:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.363+0000 7f8a7851b640 1 --2- 172.21.15.76:0/3000512944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a701022a0 0x7f8a70108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.363+0000 7f8a7851b640 1 --2- 172.21.15.76:0/3000512944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a701018e0 0x7f8a70101d60 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.363+0000 7f8a7851b640 1 --2- 172.21.15.76:0/3000512944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a701006f0 0x7f8a70100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.363+0000 7f8a7851b640 1 -- 172.21.15.76:0/3000512944 >> 172.21.15.76:0/3000512944 conn(0x7f8a700fbc80 msgr2=0x7f8a700fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.363+0000 7f8a7851b640 1 -- 172.21.15.76:0/3000512944 shutdown_connections 2024-03-18T14:52:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.363+0000 7f8a7851b640 1 -- 172.21.15.76:0/3000512944 wait complete. 2024-03-18T14:52:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a7851b640 1 Processor -- start 2024-03-18T14:52:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a7851b640 1 -- start start 2024-03-18T14:52:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a7851b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a701006f0 0x7f8a7006a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a7851b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a701018e0 0x7f8a7019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a7851b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a701022a0 0x7f8a7019ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a7851b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7010c370 con 0x7f8a701022a0 2024-03-18T14:52:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a7851b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7010c1f0 con 0x7f8a701006f0 2024-03-18T14:52:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a7851b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7010c4f0 con 0x7f8a701018e0 2024-03-18T14:52:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a76ab8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a701022a0 0x7f8a7019ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a76ab8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a701022a0 0x7f8a7019ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60008/0 (socket says 172.21.15.76:60008) 2024-03-18T14:52:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a76ab8640 1 -- 172.21.15.76:0/3724217307 learned_addr learned my addr 172.21.15.76:0/3724217307 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a762b7640 1 --2- 172.21.15.76:0/3724217307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a701006f0 0x7f8a7006a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a75ab6640 1 --2- 172.21.15.76:0/3724217307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a701018e0 0x7f8a7019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a76ab8640 1 -- 172.21.15.76:0/3724217307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a701018e0 msgr2=0x7f8a7019c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a76ab8640 1 --2- 172.21.15.76:0/3724217307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a701018e0 0x7f8a7019c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a76ab8640 1 -- 172.21.15.76:0/3724217307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a701006f0 msgr2=0x7f8a7006a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a76ab8640 1 --2- 172.21.15.76:0/3724217307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a701006f0 0x7f8a7006a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a76ab8640 1 -- 172.21.15.76:0/3724217307 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a7019f740 con 0x7f8a701022a0 2024-03-18T14:52:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a762b7640 1 --2- 172.21.15.76:0/3724217307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a701006f0 0x7f8a7006a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a76ab8640 1 --2- 172.21.15.76:0/3724217307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a701022a0 0x7f8a7019ee80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800eb10 tx=0x7f8a6800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.364+0000 7f8a677fe640 1 -- 172.21.15.76:0/3724217307 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68004b20 con 0x7f8a701022a0 2024-03-18T14:52:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.365+0000 7f8a677fe640 1 -- 172.21.15.76:0/3724217307 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a68004cc0 con 0x7f8a701022a0 2024-03-18T14:52:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.365+0000 7f8a7851b640 1 -- 172.21.15.76:0/3724217307 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a7019fa30 con 0x7f8a701022a0 2024-03-18T14:52:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.365+0000 7f8a677fe640 1 -- 172.21.15.76:0/3724217307 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68010760 con 0x7f8a701022a0 2024-03-18T14:52:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.365+0000 7f8a7851b640 1 -- 172.21.15.76:0/3724217307 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a7019ff00 con 0x7f8a701022a0 2024-03-18T14:52:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.366+0000 7f8a7851b640 1 -- 172.21.15.76:0/3724217307 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a70107a90 con 0x7f8a701022a0 2024-03-18T14:52:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.367+0000 7f8a677fe640 1 -- 172.21.15.76:0/3724217307 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8a68005020 con 0x7f8a701022a0 2024-03-18T14:52:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.367+0000 7f8a677fe640 1 --2- 172.21.15.76:0/3724217307 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8a5c0a2890 0x7f8a5c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.367+0000 7f8a762b7640 1 --2- 172.21.15.76:0/3724217307 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8a5c0a2890 0x7f8a5c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.368+0000 7f8a677fe640 1 -- 172.21.15.76:0/3724217307 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f8a680cf130 con 0x7f8a701022a0 2024-03-18T14:52:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.368+0000 7f8a762b7640 1 --2- 172.21.15.76:0/3724217307 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8a5c0a2890 0x7f8a5c0a4d50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8a70101740 tx=0x7f8a6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:30.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.372+0000 7f8a677fe640 1 -- 172.21.15.76:0/3724217307 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8a6809c0a0 con 0x7f8a701022a0 2024-03-18T14:52:30.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.548+0000 7f8a7851b640 1 -- 172.21.15.76:0/3724217307 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8a7002d390 con 0x7f8a701022a0 2024-03-18T14:52:30.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.548+0000 7f8a677fe640 1 -- 172.21.15.76:0/3724217307 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f8a6809b9a0 con 0x7f8a701022a0 2024-03-18T14:52:30.550 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-18T14:52:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.550+0000 7f8a7851b640 1 -- 172.21.15.76:0/3724217307 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8a5c0a2890 msgr2=0x7f8a5c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.550+0000 7f8a7851b640 1 --2- 172.21.15.76:0/3724217307 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8a5c0a2890 0x7f8a5c0a4d50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8a70101740 tx=0x7f8a6c005f90 comp rx=0 tx=0).stop 2024-03-18T14:52:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.550+0000 7f8a7851b640 1 -- 172.21.15.76:0/3724217307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a701022a0 msgr2=0x7f8a7019ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.550+0000 7f8a7851b640 1 --2- 172.21.15.76:0/3724217307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a701022a0 0x7f8a7019ee80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800eb10 tx=0x7f8a6800c2d0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.550+0000 7f8a7851b640 1 -- 172.21.15.76:0/3724217307 shutdown_connections 2024-03-18T14:52:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.551+0000 7f8a7851b640 1 --2- 172.21.15.76:0/3724217307 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f8a5c0a2890 0x7f8a5c0a4d50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.551+0000 7f8a7851b640 1 --2- 172.21.15.76:0/3724217307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a701018e0 0x7f8a7019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.551+0000 7f8a7851b640 1 --2- 172.21.15.76:0/3724217307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8a701022a0 0x7f8a7019ee80 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.551+0000 7f8a7851b640 1 --2- 172.21.15.76:0/3724217307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a701006f0 0x7f8a7006a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.551+0000 7f8a7851b640 1 -- 172.21.15.76:0/3724217307 >> 172.21.15.76:0/3724217307 conn(0x7f8a700fbc80 msgr2=0x7f8a70104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.551+0000 7f8a7851b640 1 -- 172.21.15.76:0/3724217307 shutdown_connections 2024-03-18T14:52:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.551+0000 7f8a7851b640 1 -- 172.21.15.76:0/3724217307 wait complete. 2024-03-18T14:52:30.564 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:52:30.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.666+0000 7f01cbf97640 1 -- 172.21.15.76:0/3977616018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01c41029f0 msgr2=0x7f01c4102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.666+0000 7f01cbf97640 1 --2- 172.21.15.76:0/3977616018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01c41029f0 0x7f01c4102e70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009900 tx=0x7f01b402bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:30.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.666+0000 7f01cbf97640 1 -- 172.21.15.76:0/3977616018 shutdown_connections 2024-03-18T14:52:30.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.666+0000 7f01cbf97640 1 --2- 172.21.15.76:0/3977616018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01c41033b0 0x7f01c4109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.666+0000 7f01cbf97640 1 --2- 172.21.15.76:0/3977616018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01c41029f0 0x7f01c4102e70 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.666+0000 7f01cbf97640 1 --2- 172.21.15.76:0/3977616018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01c4101800 0x7f01c4101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.666+0000 7f01cbf97640 1 -- 172.21.15.76:0/3977616018 >> 172.21.15.76:0/3977616018 conn(0x7f01c40fcd70 msgr2=0x7f01c40ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.667+0000 7f01cbf97640 1 -- 172.21.15.76:0/3977616018 shutdown_connections 2024-03-18T14:52:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.667+0000 7f01cbf97640 1 -- 172.21.15.76:0/3977616018 wait complete. 2024-03-18T14:52:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.667+0000 7f01cbf97640 1 Processor -- start 2024-03-18T14:52:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.667+0000 7f01cbf97640 1 -- start start 2024-03-18T14:52:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.667+0000 7f01cbf97640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01c4101800 0x7f01c4197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.667+0000 7f01cbf97640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01c41029f0 0x7f01c4198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.667+0000 7f01cbf97640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01c41033b0 0x7f01c419f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.667+0000 7f01cbf97640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f01c410d4b0 con 0x7f01c41033b0 2024-03-18T14:52:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.667+0000 7f01cbf97640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f01c410d330 con 0x7f01c4101800 2024-03-18T14:52:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.667+0000 7f01cbf97640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f01c410d630 con 0x7f01c41029f0 2024-03-18T14:52:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.667+0000 7f01ca534640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01c41033b0 0x7f01c419f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:30.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.667+0000 7f01ca534640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01c41033b0 0x7f01c419f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60028/0 (socket says 172.21.15.76:60028) 2024-03-18T14:52:30.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.667+0000 7f01ca534640 1 -- 172.21.15.76:0/1726947323 learned_addr learned my addr 172.21.15.76:0/1726947323 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:30.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.667+0000 7f01c9532640 1 --2- 172.21.15.76:0/1726947323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01c41029f0 0x7f01c4198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:30.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.667+0000 7f01c9d33640 1 --2- 172.21.15.76:0/1726947323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01c4101800 0x7f01c4197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:30.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.668+0000 7f01ca534640 1 -- 172.21.15.76:0/1726947323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01c41029f0 msgr2=0x7f01c4198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.668+0000 7f01ca534640 1 --2- 172.21.15.76:0/1726947323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01c41029f0 0x7f01c4198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.668+0000 7f01ca534640 1 -- 172.21.15.76:0/1726947323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01c4101800 msgr2=0x7f01c4197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.668+0000 7f01ca534640 1 --2- 172.21.15.76:0/1726947323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01c4101800 0x7f01c4197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.668+0000 7f01ca534640 1 -- 172.21.15.76:0/1726947323 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c419f750 con 0x7f01c41033b0 2024-03-18T14:52:30.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.668+0000 7f01ca534640 1 --2- 172.21.15.76:0/1726947323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01c41033b0 0x7f01c419f0e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00efd0 tx=0x7f01bc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:30.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.668+0000 7f01baffd640 1 -- 172.21.15.76:0/1726947323 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01bc003b70 con 0x7f01c41033b0 2024-03-18T14:52:30.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.668+0000 7f01baffd640 1 -- 172.21.15.76:0/1726947323 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01bc003d10 con 0x7f01c41033b0 2024-03-18T14:52:30.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.668+0000 7f01cbf97640 1 -- 172.21.15.76:0/1726947323 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01c419fa40 con 0x7f01c41033b0 2024-03-18T14:52:30.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.668+0000 7f01baffd640 1 -- 172.21.15.76:0/1726947323 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01bc0109f0 con 0x7f01c41033b0 2024-03-18T14:52:30.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.668+0000 7f01cbf97640 1 -- 172.21.15.76:0/1726947323 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01c419ff10 con 0x7f01c41033b0 2024-03-18T14:52:30.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.669+0000 7f01cbf97640 1 -- 172.21.15.76:0/1726947323 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01c4108b40 con 0x7f01c41033b0 2024-03-18T14:52:30.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.670+0000 7f01baffd640 1 -- 172.21.15.76:0/1726947323 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f01bc010b90 con 0x7f01c41033b0 2024-03-18T14:52:30.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.670+0000 7f01baffd640 1 --2- 172.21.15.76:0/1726947323 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f01b00a2890 0x7f01b00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:30.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.671+0000 7f01c9d33640 1 --2- 172.21.15.76:0/1726947323 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f01b00a2890 0x7f01b00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:30.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.671+0000 7f01baffd640 1 -- 172.21.15.76:0/1726947323 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f01bc0cf3c0 con 0x7f01c41033b0 2024-03-18T14:52:30.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.671+0000 7f01c9d33640 1 --2- 172.21.15.76:0/1726947323 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f01b00a2890 0x7f01b00a4d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f01c4102850 tx=0x7f01c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:30.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.674+0000 7f01baffd640 1 -- 172.21.15.76:0/1726947323 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f01bc09c330 con 0x7f01c41033b0 2024-03-18T14:52:30.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.851+0000 7f01cbf97640 1 -- 172.21.15.76:0/1726947323 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f01c402d1b0 con 0x7f01c41033b0 2024-03-18T14:52:30.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.851+0000 7f01baffd640 1 -- 172.21.15.76:0/1726947323 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f01bc09bc30 con 0x7f01c41033b0 2024-03-18T14:52:30.853 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-18T14:52:30.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.853+0000 7f01cbf97640 1 -- 172.21.15.76:0/1726947323 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f01b00a2890 msgr2=0x7f01b00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.853+0000 7f01cbf97640 1 --2- 172.21.15.76:0/1726947323 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f01b00a2890 0x7f01b00a4d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f01c4102850 tx=0x7f01c0005f90 comp rx=0 tx=0).stop 2024-03-18T14:52:30.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.853+0000 7f01cbf97640 1 -- 172.21.15.76:0/1726947323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01c41033b0 msgr2=0x7f01c419f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.853+0000 7f01cbf97640 1 --2- 172.21.15.76:0/1726947323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01c41033b0 0x7f01c419f0e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00efd0 tx=0x7f01bc00c6a0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.853+0000 7f01cbf97640 1 -- 172.21.15.76:0/1726947323 shutdown_connections 2024-03-18T14:52:30.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.853+0000 7f01cbf97640 1 --2- 172.21.15.76:0/1726947323 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f01b00a2890 0x7f01b00a4d50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.853+0000 7f01cbf97640 1 --2- 172.21.15.76:0/1726947323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01c41029f0 0x7f01c4198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.853+0000 7f01cbf97640 1 --2- 172.21.15.76:0/1726947323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01c41033b0 0x7f01c419f0e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.853+0000 7f01cbf97640 1 --2- 172.21.15.76:0/1726947323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01c4101800 0x7f01c4197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.853+0000 7f01cbf97640 1 -- 172.21.15.76:0/1726947323 >> 172.21.15.76:0/1726947323 conn(0x7f01c40fcd70 msgr2=0x7f01c4105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:30.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.853+0000 7f01cbf97640 1 -- 172.21.15.76:0/1726947323 shutdown_connections 2024-03-18T14:52:30.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.853+0000 7f01cbf97640 1 -- 172.21.15.76:0/1726947323 wait complete. 2024-03-18T14:52:30.865 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:30 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2801007342' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:52:30.865 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:30 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2279874053' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:30.865 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:30 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2577553876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:52:30.865 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:30 smithi076 ceph-mon[32205]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:30.866 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:52:30.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:30 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2801007342' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:52:30.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:30 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2279874053' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:30.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:30 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2577553876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:52:30.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:30 smithi174 ceph-mon[36787]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:30.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.965+0000 7ffaa39bc640 1 -- 172.21.15.76:0/3938799849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa9c1013d0 msgr2=0x7ffa9c101850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.965+0000 7ffaa39bc640 1 --2- 172.21.15.76:0/3938799849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa9c1013d0 0x7ffa9c101850 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ffa94009900 tx=0x7ffa9402bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:30.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.965+0000 7ffaa39bc640 1 -- 172.21.15.76:0/3938799849 shutdown_connections 2024-03-18T14:52:30.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.965+0000 7ffaa39bc640 1 --2- 172.21.15.76:0/3938799849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa9c1013d0 0x7ffa9c101850 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.965+0000 7ffaa39bc640 1 --2- 172.21.15.76:0/3938799849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa9c100a30 0x7ffa9c100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.965+0000 7ffaa39bc640 1 --2- 172.21.15.76:0/3938799849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa9c1000d0 0x7ffa9c1004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.965+0000 7ffaa39bc640 1 -- 172.21.15.76:0/3938799849 >> 172.21.15.76:0/3938799849 conn(0x7ffa9c0fbc70 msgr2=0x7ffa9c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:30.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.966+0000 7ffaa39bc640 1 -- 172.21.15.76:0/3938799849 shutdown_connections 2024-03-18T14:52:30.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.966+0000 7ffaa39bc640 1 -- 172.21.15.76:0/3938799849 wait complete. 2024-03-18T14:52:30.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.966+0000 7ffaa39bc640 1 Processor -- start 2024-03-18T14:52:30.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.966+0000 7ffaa39bc640 1 -- start start 2024-03-18T14:52:30.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.966+0000 7ffaa39bc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa9c1000d0 0x7ffa9c199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:30.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.966+0000 7ffaa39bc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa9c100a30 0x7ffa9c19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:30.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.966+0000 7ffaa39bc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa9c1013d0 0x7ffa9c1a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:30.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.966+0000 7ffaa39bc640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9c10f670 con 0x7ffa9c1000d0 2024-03-18T14:52:30.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.966+0000 7ffaa39bc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9c10f4f0 con 0x7ffa9c100a30 2024-03-18T14:52:30.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.966+0000 7ffaa39bc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9c10f7f0 con 0x7ffa9c1013d0 2024-03-18T14:52:30.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.966+0000 7ffaa1758640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa9c1000d0 0x7ffa9c199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:30.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.966+0000 7ffaa1758640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa9c1000d0 0x7ffa9c199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60046/0 (socket says 172.21.15.76:60046) 2024-03-18T14:52:30.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.966+0000 7ffaa1758640 1 -- 172.21.15.76:0/1880568768 learned_addr learned my addr 172.21.15.76:0/1880568768 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:30.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.966+0000 7ffaa0f57640 1 --2- 172.21.15.76:0/1880568768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa9c100a30 0x7ffa9c19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:30.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.967+0000 7ffaa1f59640 1 --2- 172.21.15.76:0/1880568768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa9c1013d0 0x7ffa9c1a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:30.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.967+0000 7ffaa1758640 1 -- 172.21.15.76:0/1880568768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa9c1013d0 msgr2=0x7ffa9c1a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.967+0000 7ffaa1758640 1 --2- 172.21.15.76:0/1880568768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa9c1013d0 0x7ffa9c1a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.967+0000 7ffaa1758640 1 -- 172.21.15.76:0/1880568768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa9c100a30 msgr2=0x7ffa9c19a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:30.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.967+0000 7ffaa1758640 1 --2- 172.21.15.76:0/1880568768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa9c100a30 0x7ffa9c19a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:30.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.967+0000 7ffaa1758640 1 -- 172.21.15.76:0/1880568768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa9c1a18c0 con 0x7ffa9c1000d0 2024-03-18T14:52:30.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.967+0000 7ffaa0f57640 1 --2- 172.21.15.76:0/1880568768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa9c100a30 0x7ffa9c19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:30.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.967+0000 7ffaa1758640 1 --2- 172.21.15.76:0/1880568768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa9c1000d0 0x7ffa9c199c80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800eb90 tx=0x7ffa9800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:30.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.967+0000 7ffa927fc640 1 -- 172.21.15.76:0/1880568768 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa98004d20 con 0x7ffa9c1000d0 2024-03-18T14:52:30.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.967+0000 7ffa927fc640 1 -- 172.21.15.76:0/1880568768 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa9800ee90 con 0x7ffa9c1000d0 2024-03-18T14:52:30.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.967+0000 7ffa927fc640 1 -- 172.21.15.76:0/1880568768 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa98010830 con 0x7ffa9c1000d0 2024-03-18T14:52:30.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.967+0000 7ffaa39bc640 1 -- 172.21.15.76:0/1880568768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa9c1a1bb0 con 0x7ffa9c1000d0 2024-03-18T14:52:30.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.967+0000 7ffaa39bc640 1 -- 172.21.15.76:0/1880568768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa9c1a2100 con 0x7ffa9c1000d0 2024-03-18T14:52:30.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.968+0000 7ffaa39bc640 1 -- 172.21.15.76:0/1880568768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa9c10af60 con 0x7ffa9c1000d0 2024-03-18T14:52:30.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.969+0000 7ffa927fc640 1 -- 172.21.15.76:0/1880568768 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ffa98005020 con 0x7ffa9c1000d0 2024-03-18T14:52:30.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.970+0000 7ffa927fc640 1 --2- 172.21.15.76:0/1880568768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffa880a2910 0x7ffa880a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:30.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.970+0000 7ffaa0f57640 1 --2- 172.21.15.76:0/1880568768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffa880a2910 0x7ffa880a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:30.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.970+0000 7ffa927fc640 1 -- 172.21.15.76:0/1880568768 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7ffa980cf170 con 0x7ffa9c1000d0 2024-03-18T14:52:30.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.970+0000 7ffaa0f57640 1 --2- 172.21.15.76:0/1880568768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffa880a2910 0x7ffa880a4dd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c19b180 tx=0x7ffa8c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:30.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:30.974+0000 7ffa927fc640 1 -- 172.21.15.76:0/1880568768 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffa9809c0e0 con 0x7ffa9c1000d0 2024-03-18T14:52:30.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:30 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2801007342' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:52:30.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:30 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2279874053' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:30.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:30 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2577553876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:52:30.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:30 smithi077 ceph-mon[35865]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:31.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.188+0000 7ffaa39bc640 1 -- 172.21.15.76:0/1880568768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa9c02d1b0 con 0x7ffa9c1000d0 2024-03-18T14:52:31.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.189+0000 7ffa927fc640 1 -- 172.21.15.76:0/1880568768 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11863 (secure 0 0 0) 0x7ffa9809b9e0 con 0x7ffa9c1000d0 2024-03-18T14:52:31.191 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:31.191 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/2801069369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":2801069369},{"type":"v1","addr":"172.21.15.76:6839","nonce":2801069369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2645229232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2645229232},{"type":"v1","addr":"172.21.15.76:6841","nonce":2645229232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T14:51:01.293009+0000","modified":"2024-03-18T14:51:07.032425+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24490,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6839/399810652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":399810652},{"type":"v1","addr":"172.21.15.174:6839","nonce":399810652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1685041985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1685041985},{"type":"v1","addr":"172.21.15.77:6839","nonce":1685041985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24463,24490,24524]},"id":1}]} 2024-03-18T14:52:31.191 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 23 2024-03-18T14:52:31.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.191+0000 7ffaa39bc640 1 -- 172.21.15.76:0/1880568768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffa880a2910 msgr2=0x7ffa880a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.191+0000 7ffaa39bc640 1 --2- 172.21.15.76:0/1880568768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffa880a2910 0x7ffa880a4dd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c19b180 tx=0x7ffa8c005f90 comp rx=0 tx=0).stop 2024-03-18T14:52:31.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.191+0000 7ffaa39bc640 1 -- 172.21.15.76:0/1880568768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa9c1000d0 msgr2=0x7ffa9c199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.191+0000 7ffaa39bc640 1 --2- 172.21.15.76:0/1880568768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa9c1000d0 0x7ffa9c199c80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800eb90 tx=0x7ffa9800c6a0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.191+0000 7ffaa39bc640 1 -- 172.21.15.76:0/1880568768 shutdown_connections 2024-03-18T14:52:31.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.191+0000 7ffaa39bc640 1 --2- 172.21.15.76:0/1880568768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffa880a2910 0x7ffa880a4dd0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.191+0000 7ffaa39bc640 1 --2- 172.21.15.76:0/1880568768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa9c1013d0 0x7ffa9c1a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.191+0000 7ffaa39bc640 1 --2- 172.21.15.76:0/1880568768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa9c100a30 0x7ffa9c19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.191+0000 7ffaa39bc640 1 --2- 172.21.15.76:0/1880568768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa9c1000d0 0x7ffa9c199c80 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.191+0000 7ffaa39bc640 1 -- 172.21.15.76:0/1880568768 >> 172.21.15.76:0/1880568768 conn(0x7ffa9c0fbc70 msgr2=0x7ffa9c0fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:31.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.191+0000 7ffaa39bc640 1 -- 172.21.15.76:0/1880568768 shutdown_connections 2024-03-18T14:52:31.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.191+0000 7ffaa39bc640 1 -- 172.21.15.76:0/1880568768 wait complete. 2024-03-18T14:52:31.204 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:52:31.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.305+0000 7feeac99f640 1 -- 172.21.15.76:0/3324301994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feea8102c40 msgr2=0x7feea8103060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.305+0000 7feeac99f640 1 --2- 172.21.15.76:0/3324301994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feea8102c40 0x7feea8103060 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fee98009900 tx=0x7fee9802bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:31.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.305+0000 7feeac99f640 1 -- 172.21.15.76:0/3324301994 shutdown_connections 2024-03-18T14:52:31.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.305+0000 7feeac99f640 1 --2- 172.21.15.76:0/3324301994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feea80ff6b0 0x7feea80ffb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.305+0000 7feeac99f640 1 --2- 172.21.15.76:0/3324301994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feea8100070 0x7feea8109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.305+0000 7feeac99f640 1 --2- 172.21.15.76:0/3324301994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feea8102c40 0x7feea8103060 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.305+0000 7feeac99f640 1 -- 172.21.15.76:0/3324301994 >> 172.21.15.76:0/3324301994 conn(0x7feea80fb230 msgr2=0x7feea80fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:31.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.305+0000 7feeac99f640 1 -- 172.21.15.76:0/3324301994 shutdown_connections 2024-03-18T14:52:31.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.305+0000 7feeac99f640 1 -- 172.21.15.76:0/3324301994 wait complete. 2024-03-18T14:52:31.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.305+0000 7feeac99f640 1 Processor -- start 2024-03-18T14:52:31.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feeac99f640 1 -- start start 2024-03-18T14:52:31.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feeac99f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feea80ff6b0 0x7feea8102340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feeac99f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feea8100070 0x7feea8100980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feeac99f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feea8102c40 0x7feea8100ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feeac99f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7feea810d2d0 con 0x7feea8100070 2024-03-18T14:52:31.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feeac99f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7feea810d150 con 0x7feea8102c40 2024-03-18T14:52:31.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feeac99f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feea810d450 con 0x7feea80ff6b0 2024-03-18T14:52:31.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feea6ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feea8100070 0x7feea8100980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feea6ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feea8100070 0x7feea8100980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60070/0 (socket says 172.21.15.76:60070) 2024-03-18T14:52:31.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feea6ffd640 1 -- 172.21.15.76:0/1732387796 learned_addr learned my addr 172.21.15.76:0/1732387796 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:31.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feea77fe640 1 --2- 172.21.15.76:0/1732387796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feea80ff6b0 0x7feea8102340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feea7fff640 1 --2- 172.21.15.76:0/1732387796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feea8102c40 0x7feea8100ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feea6ffd640 1 -- 172.21.15.76:0/1732387796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feea80ff6b0 msgr2=0x7feea8102340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feea6ffd640 1 --2- 172.21.15.76:0/1732387796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feea80ff6b0 0x7feea8102340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feea6ffd640 1 -- 172.21.15.76:0/1732387796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feea8102c40 msgr2=0x7feea8100ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feea6ffd640 1 --2- 172.21.15.76:0/1732387796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feea8102c40 0x7feea8100ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feea6ffd640 1 -- 172.21.15.76:0/1732387796 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea8101750 con 0x7feea8100070 2024-03-18T14:52:31.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.306+0000 7feea6ffd640 1 --2- 172.21.15.76:0/1732387796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feea8100070 0x7feea8100980 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fee9c00e9f0 tx=0x7fee9c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:31.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.307+0000 7feea4ff9640 1 -- 172.21.15.76:0/1732387796 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9c00cd80 con 0x7feea8100070 2024-03-18T14:52:31.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.307+0000 7feeac99f640 1 -- 172.21.15.76:0/1732387796 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feea81a1a40 con 0x7feea8100070 2024-03-18T14:52:31.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.307+0000 7feea4ff9640 1 -- 172.21.15.76:0/1732387796 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee9c005500 con 0x7feea8100070 2024-03-18T14:52:31.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.307+0000 7feea4ff9640 1 -- 172.21.15.76:0/1732387796 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9c010690 con 0x7feea8100070 2024-03-18T14:52:31.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.307+0000 7feeac99f640 1 -- 172.21.15.76:0/1732387796 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feea81a1f90 con 0x7feea8100070 2024-03-18T14:52:31.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.308+0000 7feeac99f640 1 -- 172.21.15.76:0/1732387796 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feea8108b00 con 0x7feea8100070 2024-03-18T14:52:31.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.309+0000 7feea4ff9640 1 -- 172.21.15.76:0/1732387796 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fee9c0046c0 con 0x7feea8100070 2024-03-18T14:52:31.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.310+0000 7feea4ff9640 1 --2- 172.21.15.76:0/1732387796 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fee900a2890 0x7fee900a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.310+0000 7feea77fe640 1 --2- 172.21.15.76:0/1732387796 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fee900a2890 0x7fee900a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.310+0000 7feea4ff9640 1 -- 172.21.15.76:0/1732387796 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fee9c0cf220 con 0x7feea8100070 2024-03-18T14:52:31.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.310+0000 7feea77fe640 1 --2- 172.21.15.76:0/1732387796 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fee900a2890 0x7fee900a4d50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fee98000c00 tx=0x7fee98005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:31.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.313+0000 7feea4ff9640 1 -- 172.21.15.76:0/1732387796 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fee9c053740 con 0x7feea8100070 2024-03-18T14:52:31.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.492+0000 7feeac99f640 1 -- 172.21.15.76:0/1732387796 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feea802d3b0 con 0x7feea8100070 2024-03-18T14:52:31.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.494+0000 7feea4ff9640 1 -- 172.21.15.76:0/1732387796 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22580 (secure 0 0 0) 0x7fee9c09c190 con 0x7feea8100070 2024-03-18T14:52:31.496 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:31.496 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:52:07.361301+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:52:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.496+0000 7feeac99f640 1 -- 172.21.15.76:0/1732387796 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fee900a2890 msgr2=0x7fee900a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.496+0000 7feeac99f640 1 --2- 172.21.15.76:0/1732387796 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fee900a2890 0x7fee900a4d50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fee98000c00 tx=0x7fee98005fd0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.496+0000 7feeac99f640 1 -- 172.21.15.76:0/1732387796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feea8100070 msgr2=0x7feea8100980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.496+0000 7feeac99f640 1 --2- 172.21.15.76:0/1732387796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feea8100070 0x7feea8100980 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fee9c00e9f0 tx=0x7fee9c00edf0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.496+0000 7feeac99f640 1 -- 172.21.15.76:0/1732387796 shutdown_connections 2024-03-18T14:52:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.496+0000 7feeac99f640 1 --2- 172.21.15.76:0/1732387796 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fee900a2890 0x7fee900a4d50 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.496+0000 7feeac99f640 1 --2- 172.21.15.76:0/1732387796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feea8102c40 0x7feea8100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.496+0000 7feeac99f640 1 --2- 172.21.15.76:0/1732387796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feea8100070 0x7feea8100980 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.496+0000 7feeac99f640 1 --2- 172.21.15.76:0/1732387796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feea80ff6b0 0x7feea8102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.496+0000 7feeac99f640 1 -- 172.21.15.76:0/1732387796 >> 172.21.15.76:0/1732387796 conn(0x7feea80fb230 msgr2=0x7feea8105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:31.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.496+0000 7feeac99f640 1 -- 172.21.15.76:0/1732387796 shutdown_connections 2024-03-18T14:52:31.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.496+0000 7feeac99f640 1 -- 172.21.15.76:0/1732387796 wait complete. 2024-03-18T14:52:31.509 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:52:31.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.610+0000 7ffab00f7640 1 -- 172.21.15.76:0/1383854927 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaa81033b0 msgr2=0x7ffaa8109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.610+0000 7ffab00f7640 1 --2- 172.21.15.76:0/1383854927 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaa81033b0 0x7ffaa8109c40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0009ca0 tx=0x7ffaa002c0f0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.610+0000 7ffab00f7640 1 -- 172.21.15.76:0/1383854927 shutdown_connections 2024-03-18T14:52:31.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.610+0000 7ffab00f7640 1 --2- 172.21.15.76:0/1383854927 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaa81033b0 0x7ffaa8109c40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.610+0000 7ffab00f7640 1 --2- 172.21.15.76:0/1383854927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa81029f0 0x7ffaa8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.610+0000 7ffab00f7640 1 --2- 172.21.15.76:0/1383854927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaa8101800 0x7ffaa8101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.610+0000 7ffab00f7640 1 -- 172.21.15.76:0/1383854927 >> 172.21.15.76:0/1383854927 conn(0x7ffaa80fcd90 msgr2=0x7ffaa80ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:31.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.610+0000 7ffab00f7640 1 -- 172.21.15.76:0/1383854927 shutdown_connections 2024-03-18T14:52:31.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.610+0000 7ffab00f7640 1 -- 172.21.15.76:0/1383854927 wait complete. 2024-03-18T14:52:31.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.610+0000 7ffab00f7640 1 Processor -- start 2024-03-18T14:52:31.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.610+0000 7ffab00f7640 1 -- start start 2024-03-18T14:52:31.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.610+0000 7ffab00f7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaa8101800 0x7ffaa8197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.610+0000 7ffab00f7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaa81029f0 0x7ffaa8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.610+0000 7ffab00f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa81033b0 0x7ffaa819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.610+0000 7ffab00f7640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa810d480 con 0x7ffaa8101800 2024-03-18T14:52:31.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.610+0000 7ffab00f7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa810d300 con 0x7ffaa81033b0 2024-03-18T14:52:31.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.610+0000 7ffab00f7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa810d600 con 0x7ffaa81029f0 2024-03-18T14:52:31.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.611+0000 7ffaade93640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaa8101800 0x7ffaa8197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.611+0000 7ffaade93640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaa8101800 0x7ffaa8197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60096/0 (socket says 172.21.15.76:60096) 2024-03-18T14:52:31.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.611+0000 7ffaade93640 1 -- 172.21.15.76:0/2644441138 learned_addr learned my addr 172.21.15.76:0/2644441138 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:31.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.611+0000 7ffaad692640 1 --2- 172.21.15.76:0/2644441138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaa81029f0 0x7ffaa8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.611+0000 7ffaae694640 1 --2- 172.21.15.76:0/2644441138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa81033b0 0x7ffaa819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.611+0000 7ffaade93640 1 -- 172.21.15.76:0/2644441138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaa81029f0 msgr2=0x7ffaa8197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.611+0000 7ffaade93640 1 --2- 172.21.15.76:0/2644441138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaa81029f0 0x7ffaa8197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.611+0000 7ffaade93640 1 -- 172.21.15.76:0/2644441138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa81033b0 msgr2=0x7ffaa819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.611+0000 7ffaade93640 1 --2- 172.21.15.76:0/2644441138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa81033b0 0x7ffaa819f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.611+0000 7ffaade93640 1 -- 172.21.15.76:0/2644441138 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa819f6b0 con 0x7ffaa8101800 2024-03-18T14:52:31.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.611+0000 7ffaade93640 1 --2- 172.21.15.76:0/2644441138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaa8101800 0x7ffaa8197a90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800e9c0 tx=0x7ffa9800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:31.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.611+0000 7ffa9effd640 1 -- 172.21.15.76:0/2644441138 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9800cd80 con 0x7ffaa8101800 2024-03-18T14:52:31.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.611+0000 7ffab00f7640 1 -- 172.21.15.76:0/2644441138 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaa819f9a0 con 0x7ffaa8101800 2024-03-18T14:52:31.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.611+0000 7ffa9effd640 1 -- 172.21.15.76:0/2644441138 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa980035f0 con 0x7ffaa8101800 2024-03-18T14:52:31.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.611+0000 7ffa9effd640 1 -- 172.21.15.76:0/2644441138 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa98010690 con 0x7ffaa8101800 2024-03-18T14:52:31.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.611+0000 7ffab00f7640 1 -- 172.21.15.76:0/2644441138 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaa819fea0 con 0x7ffaa8101800 2024-03-18T14:52:31.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.612+0000 7ffab00f7640 1 -- 172.21.15.76:0/2644441138 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaa8108b20 con 0x7ffaa8101800 2024-03-18T14:52:31.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.613+0000 7ffa9effd640 1 -- 172.21.15.76:0/2644441138 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ffa980037a0 con 0x7ffaa8101800 2024-03-18T14:52:31.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.614+0000 7ffa9effd640 1 --2- 172.21.15.76:0/2644441138 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffa940a2890 0x7ffa940a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.614+0000 7ffaad692640 1 --2- 172.21.15.76:0/2644441138 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffa940a2890 0x7ffa940a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.614+0000 7ffa9effd640 1 -- 172.21.15.76:0/2644441138 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7ffa980cef20 con 0x7ffaa8101800 2024-03-18T14:52:31.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.614+0000 7ffaad692640 1 --2- 172.21.15.76:0/2644441138 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffa940a2890 0x7ffa940a4d50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8198fa0 tx=0x7ffaa4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:31.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.619+0000 7ffa9effd640 1 -- 172.21.15.76:0/2644441138 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffa9809be90 con 0x7ffaa8101800 2024-03-18T14:52:31.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.833+0000 7ffab00f7640 1 -- 172.21.15.76:0/2644441138 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffaa802d430 con 0x7ffaa8101800 2024-03-18T14:52:31.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.833+0000 7ffa9effd640 1 -- 172.21.15.76:0/2644441138 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11863 (secure 0 0 0) 0x7ffa9809b790 con 0x7ffaa8101800 2024-03-18T14:52:31.835 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:31.836 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/2801069369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":2801069369},{"type":"v1","addr":"172.21.15.76:6839","nonce":2801069369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2645229232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2645229232},{"type":"v1","addr":"172.21.15.76:6841","nonce":2645229232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T14:51:01.293009+0000","modified":"2024-03-18T14:51:07.032425+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24490,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6839/399810652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":399810652},{"type":"v1","addr":"172.21.15.174:6839","nonce":399810652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1685041985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1685041985},{"type":"v1","addr":"172.21.15.77:6839","nonce":1685041985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24463,24490,24524]},"id":1}]} 2024-03-18T14:52:31.836 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 23 2024-03-18T14:52:31.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.835+0000 7ffab00f7640 1 -- 172.21.15.76:0/2644441138 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffa940a2890 msgr2=0x7ffa940a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.835+0000 7ffab00f7640 1 --2- 172.21.15.76:0/2644441138 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffa940a2890 0x7ffa940a4d50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8198fa0 tx=0x7ffaa4005f90 comp rx=0 tx=0).stop 2024-03-18T14:52:31.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.836+0000 7ffab00f7640 1 -- 172.21.15.76:0/2644441138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaa8101800 msgr2=0x7ffaa8197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.836+0000 7ffab00f7640 1 --2- 172.21.15.76:0/2644441138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaa8101800 0x7ffaa8197a90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800e9c0 tx=0x7ffa9800ee80 comp rx=0 tx=0).stop 2024-03-18T14:52:31.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.836+0000 7ffab00f7640 1 -- 172.21.15.76:0/2644441138 shutdown_connections 2024-03-18T14:52:31.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.836+0000 7ffab00f7640 1 --2- 172.21.15.76:0/2644441138 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ffa940a2890 0x7ffa940a4d50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.836+0000 7ffab00f7640 1 --2- 172.21.15.76:0/2644441138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaa81029f0 0x7ffaa8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.836+0000 7ffab00f7640 1 --2- 172.21.15.76:0/2644441138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa81033b0 0x7ffaa819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.836+0000 7ffab00f7640 1 --2- 172.21.15.76:0/2644441138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaa8101800 0x7ffaa8197a90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.836+0000 7ffab00f7640 1 -- 172.21.15.76:0/2644441138 >> 172.21.15.76:0/2644441138 conn(0x7ffaa80fcd90 msgr2=0x7ffaa8105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:31.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.836+0000 7ffab00f7640 1 -- 172.21.15.76:0/2644441138 shutdown_connections 2024-03-18T14:52:31.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.836+0000 7ffab00f7640 1 -- 172.21.15.76:0/2644441138 wait complete. 2024-03-18T14:52:31.847 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:31 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3724217307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:52:31.847 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:31 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1726947323' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:52:31.847 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:31 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1880568768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:52:31.848 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:31 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1732387796' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:31.849 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-18T14:51:01.293009+0000', 'modified': '2024-03-18T14:51:07.032425+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24524, 'mds_1': 24490, 'mds_2': 24463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24422': {'gid': 24422, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.77:6833/2726911960', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6832', 'nonce': 2726911960}, {'type': 'v1', 'addr': '172.21.15.77:6833', 'nonce': 2726911960}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24451': {'gid': 24451, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6835/603893267', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6834', 'nonce': 603893267}, {'type': 'v1', 'addr': '172.21.15.174:6835', 'nonce': 603893267}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.77:6835/3551464902', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 3551464902}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 3551464902}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.174:6837/239386414', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6836', 'nonce': 239386414}, {'type': 'v1', 'addr': '172.21.15.174:6837', 'nonce': 239386414}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24490': {'gid': 24490, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.174:6839/399810652', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6838', 'nonce': 399810652}, {'type': 'v1', 'addr': '172.21.15.174:6839', 'nonce': 399810652}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24524': {'gid': 24524, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6839/1685041985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6838', 'nonce': 1685041985}, {'type': 'v1', 'addr': '172.21.15.77:6839', 'nonce': 1685041985}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24524, 'qdb_cluster': [24463, 24490, 24524]} 2024-03-18T14:52:31.849 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-18T14:52:31.849 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T14:52:31.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.950+0000 7fc758e61640 1 -- 172.21.15.76:0/227171025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7541006e0 msgr2=0x7fc754100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.950+0000 7fc758e61640 1 --2- 172.21.15.76:0/227171025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7541006e0 0x7fc754100b00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc748009900 tx=0x7fc74802bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.951+0000 7fc758e61640 1 -- 172.21.15.76:0/227171025 shutdown_connections 2024-03-18T14:52:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.951+0000 7fc758e61640 1 --2- 172.21.15.76:0/227171025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7541018d0 0x7fc754101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.951+0000 7fc758e61640 1 --2- 172.21.15.76:0/227171025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc754102290 0x7fc754108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.951+0000 7fc758e61640 1 --2- 172.21.15.76:0/227171025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7541006e0 0x7fc754100b00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.951+0000 7fc758e61640 1 -- 172.21.15.76:0/227171025 >> 172.21.15.76:0/227171025 conn(0x7fc7540fbc70 msgr2=0x7fc7540fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.951+0000 7fc758e61640 1 -- 172.21.15.76:0/227171025 shutdown_connections 2024-03-18T14:52:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.951+0000 7fc758e61640 1 -- 172.21.15.76:0/227171025 wait complete. 2024-03-18T14:52:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.951+0000 7fc758e61640 1 Processor -- start 2024-03-18T14:52:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.951+0000 7fc758e61640 1 -- start start 2024-03-18T14:52:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.951+0000 7fc758e61640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7541006e0 0x7fc7540694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.951+0000 7fc758e61640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7541018d0 0x7fc75419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.951+0000 7fc758e61640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc754102290 0x7fc75419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.951+0000 7fc758e61640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc75410c350 con 0x7fc754102290 2024-03-18T14:52:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.951+0000 7fc758e61640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc75410c1d0 con 0x7fc7541018d0 2024-03-18T14:52:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.951+0000 7fc758e61640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc75410c4d0 con 0x7fc7541006e0 2024-03-18T14:52:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.952+0000 7fc752d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc754102290 0x7fc75419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.952+0000 7fc752d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc754102290 0x7fc75419eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60128/0 (socket says 172.21.15.76:60128) 2024-03-18T14:52:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.952+0000 7fc752d9d640 1 -- 172.21.15.76:0/3684291155 learned_addr learned my addr 172.21.15.76:0/3684291155 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.952+0000 7fc75259c640 1 --2- 172.21.15.76:0/3684291155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7541006e0 0x7fc7540694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.952+0000 7fc751d9b640 1 --2- 172.21.15.76:0/3684291155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7541018d0 0x7fc75419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.952+0000 7fc752d9d640 1 -- 172.21.15.76:0/3684291155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7541006e0 msgr2=0x7fc7540694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.952+0000 7fc752d9d640 1 --2- 172.21.15.76:0/3684291155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7541006e0 0x7fc7540694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.952+0000 7fc752d9d640 1 -- 172.21.15.76:0/3684291155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7541018d0 msgr2=0x7fc75419c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.952+0000 7fc752d9d640 1 --2- 172.21.15.76:0/3684291155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7541018d0 0x7fc75419c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.952+0000 7fc752d9d640 1 -- 172.21.15.76:0/3684291155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75419f7b0 con 0x7fc754102290 2024-03-18T14:52:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.952+0000 7fc752d9d640 1 --2- 172.21.15.76:0/3684291155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc754102290 0x7fc75419eef0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc74c00efd0 tx=0x7fc74c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.952+0000 7fc7437fe640 1 -- 172.21.15.76:0/3684291155 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74c003b70 con 0x7fc754102290 2024-03-18T14:52:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.952+0000 7fc7437fe640 1 -- 172.21.15.76:0/3684291155 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc74c003d10 con 0x7fc754102290 2024-03-18T14:52:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.952+0000 7fc758e61640 1 -- 172.21.15.76:0/3684291155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc75419faa0 con 0x7fc754102290 2024-03-18T14:52:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.952+0000 7fc7437fe640 1 -- 172.21.15.76:0/3684291155 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74c0109f0 con 0x7fc754102290 2024-03-18T14:52:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.952+0000 7fc758e61640 1 -- 172.21.15.76:0/3684291155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc75419ffa0 con 0x7fc754102290 2024-03-18T14:52:31.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.954+0000 7fc7437fe640 1 -- 172.21.15.76:0/3684291155 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc74c004020 con 0x7fc754102290 2024-03-18T14:52:31.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.955+0000 7fc75ba52740 1 Processor -- start 2024-03-18T14:52:31.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.955+0000 7fc75ba52740 1 -- start start 2024-03-18T14:52:31.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.955+0000 7fc75ba52740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef0876abd0 0x55ef0876aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.955+0000 7fc75ba52740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876b5c0 0x55ef087700c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.955+0000 7fc75ba52740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef08770730 0x55ef08772bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.955+0000 7fc75ba52740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ef086cec80 con 0x55ef0876abd0 2024-03-18T14:52:31.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.955+0000 7fc75ba52740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ef08677aa0 con 0x55ef08770730 2024-03-18T14:52:31.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.955+0000 7fc75ba52740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ef086cad20 con 0x55ef0876b5c0 2024-03-18T14:52:31.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.955+0000 7fc7437fe640 1 --2- 172.21.15.76:0/3684291155 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc73c0a2890 0x7fc73c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.955+0000 7fc751d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef0876abd0 0x55ef0876aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.955+0000 7fc751d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef0876abd0 0x55ef0876aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60144/0 (socket says 172.21.15.76:60144) 2024-03-18T14:52:31.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.955+0000 7fc751d9b640 1 -- 172.21.15.76:0/298731353 learned_addr learned my addr 172.21.15.76:0/298731353 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:31.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.955+0000 7fc752d9d640 1 --2- 172.21.15.76:0/3684291155 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc73c0a2890 0x7fc73c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.955+0000 7fc75259c640 1 --2- 172.21.15.76:0/298731353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876b5c0 0x55ef087700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.955+0000 7fc751d9b640 1 --2- 172.21.15.76:0/298731353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef08770730 0x55ef08772bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.956+0000 7fc7437fe640 1 -- 172.21.15.76:0/3684291155 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc74c0cefc0 con 0x7fc754102290 2024-03-18T14:52:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.956+0000 7fc751d9b640 1 -- 172.21.15.76:0/298731353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876b5c0 msgr2=0x55ef087700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.956+0000 7fc751d9b640 1 --2- 172.21.15.76:0/298731353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876b5c0 0x55ef087700c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.956+0000 7fc751d9b640 1 -- 172.21.15.76:0/298731353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef08770730 msgr2=0x55ef08772bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.956+0000 7fc751d9b640 1 --2- 172.21.15.76:0/298731353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef08770730 0x55ef08772bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.956+0000 7fc751d9b640 1 -- 172.21.15.76:0/298731353 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef08699ff0 con 0x55ef0876abd0 2024-03-18T14:52:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.956+0000 7fc751d9b640 1 --2- 172.21.15.76:0/298731353 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef0876abd0 0x55ef0876aff0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc744009710 tx=0x7fc744009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d6c2d721cbc0ee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.956+0000 7fc752d9d640 1 --2- 172.21.15.76:0/3684291155 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc73c0a2890 0x7fc73c0a4d50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc74c097510 tx=0x7fc74c096290 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.956+0000 7fc740ff9640 1 -- 172.21.15.76:0/298731353 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc744013e50 con 0x55ef0876abd0 2024-03-18T14:52:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.956+0000 7fc740ff9640 1 -- 172.21.15.76:0/298731353 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc744013120 con 0x55ef0876abd0 2024-03-18T14:52:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.956+0000 7fc740ff9640 1 -- 172.21.15.76:0/298731353 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7440109d0 con 0x55ef0876abd0 2024-03-18T14:52:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 1 -- 172.21.15.76:0/298731353 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef0876abd0 msgr2=0x55ef0876aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 1 --2- 172.21.15.76:0/298731353 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef0876abd0 0x55ef0876aff0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc744009710 tx=0x7fc744009780 comp rx=0 tx=0).stop 2024-03-18T14:52:31.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 1 -- 172.21.15.76:0/298731353 shutdown_connections 2024-03-18T14:52:31.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 1 --2- 172.21.15.76:0/298731353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876b5c0 0x55ef087700c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 1 --2- 172.21.15.76:0/298731353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef08770730 0x55ef08772bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 1 --2- 172.21.15.76:0/298731353 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef0876abd0 0x55ef0876aff0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 1 -- 172.21.15.76:0/298731353 >> 172.21.15.76:0/298731353 conn(0x55ef08766140 msgr2=0x55ef08766540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:31.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 1 -- 172.21.15.76:0/298731353 shutdown_connections 2024-03-18T14:52:31.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 1 -- 172.21.15.76:0/298731353 wait complete. 2024-03-18T14:52:31.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 1 -- start start 2024-03-18T14:52:31.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 1 Processor -- start 2024-03-18T14:52:31.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876abd0 0x55ef08774d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef0876b5c0 0x55ef08775260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef0877c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ef08773680 con 0x55ef08770730 2024-03-18T14:52:31.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ef0868aef0 con 0x55ef0876b5c0 2024-03-18T14:52:31.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ef08773800 con 0x55ef0876abd0 2024-03-18T14:52:31.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc75ba52740 10 client.0 _ll_register_callbacks cb 0x55ef08685dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:52:31.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc751d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef0877c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc751d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef0877c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60154/0 (socket says 172.21.15.76:60154) 2024-03-18T14:52:31.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.957+0000 7fc751d9b640 1 -- 172.21.15.76:0/881443300 learned_addr learned my addr 172.21.15.76:0/881443300 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:31.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.958+0000 7fc751d9b640 1 --2- 172.21.15.76:0/881443300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876abd0 0x55ef08774d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.958+0000 7fc75259c640 1 --2- 172.21.15.76:0/881443300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef0876b5c0 0x55ef08775260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.958+0000 7fc751d9b640 1 -- 172.21.15.76:0/881443300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876abd0 msgr2=0x55ef08774d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:31.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.958+0000 7fc751d9b640 1 --2- 172.21.15.76:0/881443300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876abd0 0x55ef08774d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.958+0000 7fc751d9b640 1 -- 172.21.15.76:0/881443300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef0876b5c0 msgr2=0x55ef08775260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:31.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.958+0000 7fc751d9b640 1 --2- 172.21.15.76:0/881443300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef0876b5c0 0x55ef08775260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.958+0000 7fc751d9b640 1 -- 172.21.15.76:0/881443300 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef08697a30 con 0x55ef08770730 2024-03-18T14:52:31.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.958+0000 7fc751d9b640 1 --2- 172.21.15.76:0/881443300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef0877c2a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc744006010 tx=0x7fc744015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db6266e2c51b583 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:31.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.958+0000 7fc739ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:52:31.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.958+0000 7fc739ffb640 1 -- 172.21.15.76:0/881443300 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc744025020 con 0x55ef08770730 2024-03-18T14:52:31.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.958+0000 7fc739ffb640 1 -- 172.21.15.76:0/881443300 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc744004260 con 0x55ef08770730 2024-03-18T14:52:31.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.958+0000 7fc75ba52740 1 -- 172.21.15.76:0/881443300 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc744013120 con 0x55ef08770730 2024-03-18T14:52:31.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.958+0000 7fc739ffb640 1 -- 172.21.15.76:0/881443300 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc744013e50 con 0x55ef08770730 2024-03-18T14:52:31.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.958+0000 7fc75ba52740 10 client.15045 fetch_fsmap learned FSMap version 23 2024-03-18T14:52:31.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.958+0000 7fc75ba52740 1 -- 172.21.15.76:0/881443300 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ef083e32d0 con 0x55ef08770730 2024-03-18T14:52:31.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.958+0000 7fc739ffb640 1 -- 172.21.15.76:0/881443300 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc744013120 con 0x55ef08770730 2024-03-18T14:52:31.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.958+0000 7fc739ffb640 1 -- 172.21.15.76:0/881443300 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc744003780 con 0x55ef08770730 2024-03-18T14:52:31.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.959+0000 7fc75ba52740 10 client.15045 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T14:52:31.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.959+0000 7fc75ba52740 10 client.15045 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T14:52:31.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.959+0000 7fc75ba52740 20 client.15045 populate_metadata read hostname 'smithi076' 2024-03-18T14:52:31.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.959+0000 7fc75ba52740 1 --2- 172.21.15.76:0/881443300 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55ef087814e0 0x55ef08782950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.959+0000 7fc75ba52740 4 client.15045 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:52:31.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.959+0000 7fc75ba52740 1 -- 172.21.15.76:0/881443300 --> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef086dc1c0 con 0x55ef087814e0 2024-03-18T14:52:31.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.959+0000 7fc75259c640 1 --2- 172.21.15.76:0/881443300 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55ef087814e0 0x55ef08782950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.959+0000 7fc739ffb640 10 client.15045 ms_handle_connect on v2:172.21.15.174:6836/239386414 2024-03-18T14:52:31.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.959+0000 7fc75259c640 1 --2- 172.21.15.76:0/881443300 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55ef087814e0 0x55ef08782950 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8f01f8df58eead57 server_cookie=b2d672146e603c0f in_seq=0 out_seq=0 2024-03-18T14:52:31.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.960+0000 7fc739ffb640 1 -- 172.21.15.76:0/881443300 <== mds.2 v2:172.21.15.174:6836/239386414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ef086dc1c0 con 0x55ef087814e0 2024-03-18T14:52:31.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.960+0000 7fc739ffb640 10 client.15045 handle_command_reply: tid=0 2024-03-18T14:52:31.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.960+0000 7fc75ba52740 1 client.15045 shutdown 2024-03-18T14:52:31.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.960+0000 7fc75ba52740 1 -- 172.21.15.76:0/881443300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 msgr2=0x55ef0877c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.960+0000 7fc75ba52740 1 --2- 172.21.15.76:0/881443300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef0877c2a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc744006010 tx=0x7fc744015040 comp rx=0 tx=0).stop 2024-03-18T14:52:31.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.960+0000 7fc75ba52740 1 -- 172.21.15.76:0/881443300 shutdown_connections 2024-03-18T14:52:31.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.960+0000 7fc75ba52740 1 --2- 172.21.15.76:0/881443300 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55ef087814e0 0x55ef08782950 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.960+0000 7fc75ba52740 1 --2- 172.21.15.76:0/881443300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef0877c2a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.960+0000 7fc75ba52740 1 --2- 172.21.15.76:0/881443300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef0876b5c0 0x55ef08775260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.960+0000 7fc739ffb640 0 client.15045 ms_handle_reset on v2:172.21.15.174:6836/239386414 2024-03-18T14:52:31.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.960+0000 7fc75ba52740 1 --2- 172.21.15.76:0/881443300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876abd0 0x55ef08774d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.960+0000 7fc75ba52740 1 -- 172.21.15.76:0/881443300 >> 172.21.15.76:0/881443300 conn(0x55ef08766140 msgr2=0x55ef08765380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:31.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.960+0000 7fc75ba52740 1 -- 172.21.15.76:0/881443300 shutdown_connections 2024-03-18T14:52:31.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.960+0000 7fc75ba52740 1 -- 172.21.15.76:0/881443300 wait complete. 2024-03-18T14:52:31.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.960+0000 7fc75ba52740 20 client.15045 trim_cache size 0 max 16384 2024-03-18T14:52:31.971 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:31 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3724217307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:52:31.971 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:31 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1726947323' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:52:31.971 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:31 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1880568768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:52:31.971 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:31 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1732387796' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:31.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:31 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3724217307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:52:31.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:31 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1726947323' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:52:31.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:31 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1880568768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:52:31.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:31 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1732387796' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.978+0000 7fc75ba52740 1 Processor -- start 2024-03-18T14:52:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.978+0000 7fc75ba52740 1 -- start start 2024-03-18T14:52:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.978+0000 7fc75ba52740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876abd0 0x55ef0876fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.978+0000 7fc75ba52740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef0876b5c0 0x55ef08773ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.978+0000 7fc75ba52740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef087740e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.978+0000 7fc75ba52740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ef08780c90 con 0x55ef08770730 2024-03-18T14:52:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.978+0000 7fc75ba52740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ef08781360 con 0x55ef0876b5c0 2024-03-18T14:52:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.978+0000 7fc75ba52740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ef08781a30 con 0x55ef0876abd0 2024-03-18T14:52:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.978+0000 7fc751d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef087740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.978+0000 7fc751d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef087740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60166/0 (socket says 172.21.15.76:60166) 2024-03-18T14:52:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.979+0000 7fc751d9b640 1 -- 172.21.15.76:0/294420245 learned_addr learned my addr 172.21.15.76:0/294420245 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.979+0000 7fc75259c640 1 --2- 172.21.15.76:0/294420245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef0876b5c0 0x55ef08773ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.979+0000 7fc751d9b640 1 --2- 172.21.15.76:0/294420245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876abd0 0x55ef0876fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.979+0000 7fc751d9b640 1 -- 172.21.15.76:0/294420245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876abd0 msgr2=0x55ef0876fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.979+0000 7fc751d9b640 1 --2- 172.21.15.76:0/294420245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876abd0 0x55ef0876fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.979+0000 7fc751d9b640 1 -- 172.21.15.76:0/294420245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef0876b5c0 msgr2=0x55ef08773ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.979+0000 7fc751d9b640 1 --2- 172.21.15.76:0/294420245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef0876b5c0 0x55ef08773ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.979+0000 7fc751d9b640 1 -- 172.21.15.76:0/294420245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef084ad6b0 con 0x55ef08770730 2024-03-18T14:52:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.979+0000 7fc75259c640 1 --2- 172.21.15.76:0/294420245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef0876b5c0 0x55ef08773ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.979+0000 7fc751d9b640 1 --2- 172.21.15.76:0/294420245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef087740e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc74400fbd0 tx=0x7fc744021480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43993f92f5d76f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.979+0000 7fc73bfff640 1 -- 172.21.15.76:0/294420245 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc744003e10 con 0x55ef08770730 2024-03-18T14:52:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.979+0000 7fc73bfff640 1 -- 172.21.15.76:0/294420245 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc744009780 con 0x55ef08770730 2024-03-18T14:52:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.979+0000 7fc73bfff640 1 -- 172.21.15.76:0/294420245 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc744021560 con 0x55ef08770730 2024-03-18T14:52:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 1 -- 172.21.15.76:0/294420245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 msgr2=0x55ef087740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 1 --2- 172.21.15.76:0/294420245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef087740e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc74400fbd0 tx=0x7fc744021480 comp rx=0 tx=0).stop 2024-03-18T14:52:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 1 -- 172.21.15.76:0/294420245 shutdown_connections 2024-03-18T14:52:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 1 --2- 172.21.15.76:0/294420245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef087740e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 1 --2- 172.21.15.76:0/294420245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef0876b5c0 0x55ef08773ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 1 --2- 172.21.15.76:0/294420245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876abd0 0x55ef0876fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 1 -- 172.21.15.76:0/294420245 >> 172.21.15.76:0/294420245 conn(0x55ef08766140 msgr2=0x55ef08766520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 1 -- 172.21.15.76:0/294420245 shutdown_connections 2024-03-18T14:52:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 1 -- 172.21.15.76:0/294420245 wait complete. 2024-03-18T14:52:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 1 -- start start 2024-03-18T14:52:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 1 Processor -- start 2024-03-18T14:52:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876abd0 0x55ef0878a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef0876b5c0 0x55ef0878a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef087e9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ef087827d0 con 0x55ef08770730 2024-03-18T14:52:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ef08782100 con 0x55ef0876b5c0 2024-03-18T14:52:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ef08782ea0 con 0x55ef0876abd0 2024-03-18T14:52:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75ba52740 10 client.0 _ll_register_callbacks cb 0x7fc744009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:52:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc751d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef087e9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc751d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef087e9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60182/0 (socket says 172.21.15.76:60182) 2024-03-18T14:52:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc751d9b640 1 -- 172.21.15.76:0/1076597024 learned_addr learned my addr 172.21.15.76:0/1076597024 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc75259c640 1 --2- 172.21.15.76:0/1076597024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef0876b5c0 0x55ef0878a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.980+0000 7fc751d9b640 1 --2- 172.21.15.76:0/1076597024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876abd0 0x55ef0878a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc751d9b640 1 -- 172.21.15.76:0/1076597024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876abd0 msgr2=0x55ef0878a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc751d9b640 1 --2- 172.21.15.76:0/1076597024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876abd0 0x55ef0878a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc751d9b640 1 -- 172.21.15.76:0/1076597024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef0876b5c0 msgr2=0x55ef0878a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc751d9b640 1 --2- 172.21.15.76:0/1076597024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef0876b5c0 0x55ef0878a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc751d9b640 1 -- 172.21.15.76:0/1076597024 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef087e9950 con 0x55ef08770730 2024-03-18T14:52:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc751d9b640 1 --2- 172.21.15.76:0/1076597024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef087e9390 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc744009fd0 tx=0x7fc7440096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6efbf0f1015b6c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc739ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:52:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc739ffb640 1 -- 172.21.15.76:0/1076597024 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc744009b30 con 0x55ef08770730 2024-03-18T14:52:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc739ffb640 1 -- 172.21.15.76:0/1076597024 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc744021560 con 0x55ef08770730 2024-03-18T14:52:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc75ba52740 1 -- 172.21.15.76:0/1076597024 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc744009780 con 0x55ef08770730 2024-03-18T14:52:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc739ffb640 1 -- 172.21.15.76:0/1076597024 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc744032480 con 0x55ef08770730 2024-03-18T14:52:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc75ba52740 10 client.15051 fetch_fsmap learned FSMap version 23 2024-03-18T14:52:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc75ba52740 1 -- 172.21.15.76:0/1076597024 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ef087eac50 con 0x55ef08770730 2024-03-18T14:52:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc739ffb640 1 -- 172.21.15.76:0/1076597024 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc744009780 con 0x55ef08770730 2024-03-18T14:52:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc739ffb640 1 -- 172.21.15.76:0/1076597024 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc744010430 con 0x55ef08770730 2024-03-18T14:52:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc75ba52740 10 client.15051 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T14:52:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc75ba52740 10 client.15051 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T14:52:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc75ba52740 20 client.15051 populate_metadata read hostname 'smithi076' 2024-03-18T14:52:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc75ba52740 1 --2- 172.21.15.76:0/1076597024 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55ef087ef470 0x55ef087ebf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc75ba52740 4 client.15051 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T14:52:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.981+0000 7fc75ba52740 1 -- 172.21.15.76:0/1076597024 --> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] -- command(tid 0: {"prefix": "status"}) -- 0x55ef086d6510 con 0x55ef087ef470 2024-03-18T14:52:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc75259c640 1 --2- 172.21.15.76:0/1076597024 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55ef087ef470 0x55ef087ebf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc739ffb640 10 client.15051 ms_handle_connect on v2:172.21.15.174:6836/239386414 2024-03-18T14:52:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc75259c640 1 --2- 172.21.15.76:0/1076597024 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55ef087ef470 0x55ef087ebf80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=224c8ee1173f1685 server_cookie=f31422671170723f in_seq=0 out_seq=0 2024-03-18T14:52:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc739ffb640 1 -- 172.21.15.76:0/1076597024 <== mds.2 v2:172.21.15.174:6836/239386414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55ef086d6510 con 0x55ef087ef470 2024-03-18T14:52:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc739ffb640 10 client.15051 handle_command_reply: tid=0 2024-03-18T14:52:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc75ba52740 1 client.15051 shutdown 2024-03-18T14:52:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc75ba52740 1 -- 172.21.15.76:0/1076597024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 msgr2=0x55ef087e9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc75ba52740 1 --2- 172.21.15.76:0/1076597024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef087e9390 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc744009fd0 tx=0x7fc7440096a0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc75ba52740 1 -- 172.21.15.76:0/1076597024 shutdown_connections 2024-03-18T14:52:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc75ba52740 1 --2- 172.21.15.76:0/1076597024 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55ef087ef470 0x55ef087ebf80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc75ba52740 1 --2- 172.21.15.76:0/1076597024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef08770730 0x55ef087e9390 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc75ba52740 1 --2- 172.21.15.76:0/1076597024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef0876b5c0 0x55ef0878a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc739ffb640 0 client.15051 ms_handle_reset on v2:172.21.15.174:6836/239386414 2024-03-18T14:52:31.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc75ba52740 1 --2- 172.21.15.76:0/1076597024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef0876abd0 0x55ef0878a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc75ba52740 1 -- 172.21.15.76:0/1076597024 >> 172.21.15.76:0/1076597024 conn(0x55ef08766140 msgr2=0x55ef0876fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:31.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc75ba52740 1 -- 172.21.15.76:0/1076597024 shutdown_connections 2024-03-18T14:52:31.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc75ba52740 1 -- 172.21.15.76:0/1076597024 wait complete. 2024-03-18T14:52:31.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.982+0000 7fc75ba52740 20 client.15051 trim_cache size 0 max 16384 2024-03-18T14:52:31.994 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T14:52:31.994 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "9cc83b72-e535-11ee-95c9-87774f69a715", 2024-03-18T14:52:31.995 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 2, 2024-03-18T14:52:31.995 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24463, 2024-03-18T14:52:31.995 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-03-18T14:52:31.995 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-03-18T14:52:31.995 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-03-18T14:52:31.995 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 85.959079273, 2024-03-18T14:52:31.995 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 23, 2024-03-18T14:52:31.995 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 88, 2024-03-18T14:52:31.995 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T14:52:31.995 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 108.87610696900001 2024-03-18T14:52:31.995 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T14:52:31.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.983+0000 7fc758e61640 1 -- 172.21.15.76:0/3684291155 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc73c0a2890 msgr2=0x7fc73c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.983+0000 7fc758e61640 1 --2- 172.21.15.76:0/3684291155 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc73c0a2890 0x7fc73c0a4d50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc74c097510 tx=0x7fc74c096290 comp rx=0 tx=0).stop 2024-03-18T14:52:31.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.983+0000 7fc758e61640 1 -- 172.21.15.76:0/3684291155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc754102290 msgr2=0x7fc75419eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:31.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.983+0000 7fc758e61640 1 --2- 172.21.15.76:0/3684291155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc754102290 0x7fc75419eef0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc74c00efd0 tx=0x7fc74c00c6a0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.983+0000 7fc758e61640 1 -- 172.21.15.76:0/3684291155 shutdown_connections 2024-03-18T14:52:31.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.983+0000 7fc758e61640 1 --2- 172.21.15.76:0/3684291155 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc73c0a2890 0x7fc73c0a4d50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.983+0000 7fc758e61640 1 --2- 172.21.15.76:0/3684291155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc754102290 0x7fc75419eef0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.983+0000 7fc758e61640 1 --2- 172.21.15.76:0/3684291155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7541018d0 0x7fc75419c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.983+0000 7fc758e61640 1 --2- 172.21.15.76:0/3684291155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7541006e0 0x7fc7540694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:31.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.983+0000 7fc758e61640 1 -- 172.21.15.76:0/3684291155 >> 172.21.15.76:0/3684291155 conn(0x7fc7540fbc70 msgr2=0x7fc754104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:31.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.983+0000 7fc758e61640 1 -- 172.21.15.76:0/3684291155 shutdown_connections 2024-03-18T14:52:31.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:31.983+0000 7fc758e61640 1 -- 172.21.15.76:0/3684291155 wait complete. 2024-03-18T14:52:31.998 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-18T14:52:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.100+0000 7f6eb860b640 1 -- 172.21.15.76:0/3161158195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6eb0102290 msgr2=0x7f6eb0108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.100+0000 7f6eb860b640 1 --2- 172.21.15.76:0/3161158195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6eb0102290 0x7f6eb0108b20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8009900 tx=0x7f6ea802bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.100+0000 7f6eb860b640 1 -- 172.21.15.76:0/3161158195 shutdown_connections 2024-03-18T14:52:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.100+0000 7f6eb860b640 1 --2- 172.21.15.76:0/3161158195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6eb01018d0 0x7f6eb0101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.100+0000 7f6eb860b640 1 --2- 172.21.15.76:0/3161158195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6eb0102290 0x7f6eb0108b20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.100+0000 7f6eb860b640 1 --2- 172.21.15.76:0/3161158195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb01006e0 0x7f6eb0100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.100+0000 7f6eb860b640 1 -- 172.21.15.76:0/3161158195 >> 172.21.15.76:0/3161158195 conn(0x7f6eb00fbc70 msgr2=0x7f6eb00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.100+0000 7f6eb860b640 1 -- 172.21.15.76:0/3161158195 shutdown_connections 2024-03-18T14:52:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.100+0000 7f6eb860b640 1 -- 172.21.15.76:0/3161158195 wait complete. 2024-03-18T14:52:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb860b640 1 Processor -- start 2024-03-18T14:52:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb860b640 1 -- start start 2024-03-18T14:52:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb860b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6eb01006e0 0x7f6eb00694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb860b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb01018d0 0x7f6eb019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb860b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6eb0102290 0x7f6eb019ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb860b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb010c360 con 0x7f6eb0102290 2024-03-18T14:52:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb860b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb010c1e0 con 0x7f6eb01018d0 2024-03-18T14:52:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb860b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb010c4e0 con 0x7f6eb01006e0 2024-03-18T14:52:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb6ba8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6eb0102290 0x7f6eb019ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb63a7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6eb01006e0 0x7f6eb00694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb6ba8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6eb0102290 0x7f6eb019ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60202/0 (socket says 172.21.15.76:60202) 2024-03-18T14:52:32.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb6ba8640 1 -- 172.21.15.76:0/2380195383 learned_addr learned my addr 172.21.15.76:0/2380195383 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:32.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb5ba6640 1 --2- 172.21.15.76:0/2380195383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb01018d0 0x7f6eb019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb5ba6640 1 -- 172.21.15.76:0/2380195383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6eb01006e0 msgr2=0x7f6eb00694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb5ba6640 1 --2- 172.21.15.76:0/2380195383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6eb01006e0 0x7f6eb00694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb5ba6640 1 -- 172.21.15.76:0/2380195383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6eb0102290 msgr2=0x7f6eb019ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb5ba6640 1 --2- 172.21.15.76:0/2380195383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6eb0102290 0x7f6eb019ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb5ba6640 1 -- 172.21.15.76:0/2380195383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb019f750 con 0x7f6eb01018d0 2024-03-18T14:52:32.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb63a7640 1 --2- 172.21.15.76:0/2380195383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6eb01006e0 0x7f6eb00694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:32.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.101+0000 7f6eb6ba8640 1 --2- 172.21.15.76:0/2380195383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6eb0102290 0x7f6eb019ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:32.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.102+0000 7f6eb5ba6640 1 --2- 172.21.15.76:0/2380195383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb01018d0 0x7f6eb019c820 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6ea400e9f0 tx=0x7f6ea400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:32.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.102+0000 7f6eaf7fe640 1 -- 172.21.15.76:0/2380195383 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea400ccc0 con 0x7f6eb01018d0 2024-03-18T14:52:32.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.102+0000 7f6eaf7fe640 1 -- 172.21.15.76:0/2380195383 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea4005020 con 0x7f6eb01018d0 2024-03-18T14:52:32.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.102+0000 7f6eb860b640 1 -- 172.21.15.76:0/2380195383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eb019f9e0 con 0x7f6eb01018d0 2024-03-18T14:52:32.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.102+0000 7f6eaf7fe640 1 -- 172.21.15.76:0/2380195383 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea40049e0 con 0x7f6eb01018d0 2024-03-18T14:52:32.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.102+0000 7f6eb860b640 1 -- 172.21.15.76:0/2380195383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eb019fee0 con 0x7f6eb01018d0 2024-03-18T14:52:32.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.105+0000 7f6eaf7fe640 1 -- 172.21.15.76:0/2380195383 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6ea4005560 con 0x7f6eb01018d0 2024-03-18T14:52:32.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.105+0000 7f6ebb1f7740 1 Processor -- start 2024-03-18T14:52:32.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.105+0000 7f6ebb1f7740 1 -- start start 2024-03-18T14:52:32.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.105+0000 7f6ebb1f7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f0060bd0 0x55c4f0060ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.105+0000 7f6ebb1f7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f00615c0 0x55c4f00660c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.105+0000 7f6ebb1f7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f0066730 0x55c4f0068bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.105+0000 7f6ebb1f7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c4effc4c80 con 0x55c4f00615c0 2024-03-18T14:52:32.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.105+0000 7f6ebb1f7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c4eff6daa0 con 0x55c4f0060bd0 2024-03-18T14:52:32.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.105+0000 7f6ebb1f7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c4effc0d20 con 0x55c4f0066730 2024-03-18T14:52:32.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.105+0000 7f6eb63a7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f00615c0 0x55c4f00660c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.105+0000 7f6eb63a7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f00615c0 0x55c4f00660c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60208/0 (socket says 172.21.15.76:60208) 2024-03-18T14:52:32.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.105+0000 7f6eb63a7640 1 -- 172.21.15.76:0/3762347501 learned_addr learned my addr 172.21.15.76:0/3762347501 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:32.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.105+0000 7f6eb6ba8640 1 --2- 172.21.15.76:0/3762347501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f0060bd0 0x55c4f0060ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.105+0000 7f6eb5ba6640 1 --2- 172.21.15.76:0/3762347501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f0066730 0x55c4f0068bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.106+0000 7f6eb63a7640 1 -- 172.21.15.76:0/3762347501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f0066730 msgr2=0x55c4f0068bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:32.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.106+0000 7f6eb63a7640 1 --2- 172.21.15.76:0/3762347501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f0066730 0x55c4f0068bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.106+0000 7f6eb63a7640 1 -- 172.21.15.76:0/3762347501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f0060bd0 msgr2=0x55c4f0060ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:32.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.106+0000 7f6eaf7fe640 1 --2- 172.21.15.76:0/2380195383 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6e980a2890 0x7f6e980a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.106+0000 7f6eb63a7640 1 --2- 172.21.15.76:0/3762347501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f0060bd0 0x55c4f0060ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.106+0000 7f6eb63a7640 1 -- 172.21.15.76:0/3762347501 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4eff8fff0 con 0x55c4f00615c0 2024-03-18T14:52:32.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.106+0000 7f6eb6ba8640 1 --2- 172.21.15.76:0/2380195383 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6e980a2890 0x7f6e980a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.106+0000 7f6eb63a7640 1 --2- 172.21.15.76:0/3762347501 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f00615c0 0x55c4f00660c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0011820 tx=0x7f6ea0011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6654ba4d6fd5341d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:32.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.106+0000 7f6eaf7fe640 1 -- 172.21.15.76:0/2380195383 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6ea40ceb60 con 0x7f6eb01018d0 2024-03-18T14:52:32.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.106+0000 7f6eacbfe640 1 -- 172.21.15.76:0/3762347501 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0011220 con 0x55c4f00615c0 2024-03-18T14:52:32.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.106+0000 7f6eacbfe640 1 -- 172.21.15.76:0/3762347501 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea0019430 con 0x55c4f00615c0 2024-03-18T14:52:32.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.106+0000 7f6eacbfe640 1 -- 172.21.15.76:0/3762347501 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0010630 con 0x55c4f00615c0 2024-03-18T14:52:32.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/3762347501 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f00615c0 msgr2=0x55c4f00660c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6eb6ba8640 1 --2- 172.21.15.76:0/2380195383 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6e980a2890 0x7f6e980a4d50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6ea802ce70 tx=0x7f6ea802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:32.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/3762347501 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f00615c0 0x55c4f00660c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0011820 tx=0x7f6ea0011ce0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/3762347501 shutdown_connections 2024-03-18T14:52:32.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/3762347501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f0066730 0x55c4f0068bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/3762347501 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f00615c0 0x55c4f00660c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/3762347501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f0060bd0 0x55c4f0060ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/3762347501 >> 172.21.15.76:0/3762347501 conn(0x55c4f005c140 msgr2=0x55c4f005c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:32.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/3762347501 shutdown_connections 2024-03-18T14:52:32.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/3762347501 wait complete. 2024-03-18T14:52:32.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 1 -- start start 2024-03-18T14:52:32.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 1 Processor -- start 2024-03-18T14:52:32.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f0060bd0 0x55c4f006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f00615c0 0x55c4f006b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f0072320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c4f00696e0 con 0x55c4f0066730 2024-03-18T14:52:32.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c4eff80ef0 con 0x55c4f0060bd0 2024-03-18T14:52:32.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c4f0069860 con 0x55c4f00615c0 2024-03-18T14:52:32.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6ebb1f7740 10 client.0 _ll_register_callbacks cb 0x55c4eff7bdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:52:32.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.107+0000 7f6eb6ba8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f0072320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6eb5ba6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f00615c0 0x55c4f006b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6eb6ba8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f0072320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60218/0 (socket says 172.21.15.76:60218) 2024-03-18T14:52:32.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6eb63a7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f0060bd0 0x55c4f006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6eb6ba8640 1 -- 172.21.15.76:0/1461065864 learned_addr learned my addr 172.21.15.76:0/1461065864 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:32.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6eb6ba8640 1 -- 172.21.15.76:0/1461065864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f00615c0 msgr2=0x55c4f006b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:32.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6eb6ba8640 1 --2- 172.21.15.76:0/1461065864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f00615c0 0x55c4f006b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6eb6ba8640 1 -- 172.21.15.76:0/1461065864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f0060bd0 msgr2=0x55c4f006ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:32.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6eb6ba8640 1 --2- 172.21.15.76:0/1461065864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f0060bd0 0x55c4f006ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6eb6ba8640 1 -- 172.21.15.76:0/1461065864 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4eff8da30 con 0x55c4f0066730 2024-03-18T14:52:32.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6eb5ba6640 1 --2- 172.21.15.76:0/1461065864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f00615c0 0x55c4f006b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:32.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6eb63a7640 1 --2- 172.21.15.76:0/1461065864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f0060bd0 0x55c4f006ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:32.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6eb6ba8640 1 --2- 172.21.15.76:0/1461065864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f0072320 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8036040 tx=0x7f6ea802c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112fc34868c076e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:32.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6e95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:52:32.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6e95ffb640 1 -- 172.21.15.76:0/1461065864 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8050070 con 0x55c4f0066730 2024-03-18T14:52:32.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/1461065864 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ea0019430 con 0x55c4f0066730 2024-03-18T14:52:32.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6e95ffb640 1 -- 172.21.15.76:0/1461065864 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6ea8005400 con 0x55c4f0066730 2024-03-18T14:52:32.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6e95ffb640 1 -- 172.21.15.76:0/1461065864 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6ea805c050 con 0x55c4f0066730 2024-03-18T14:52:32.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6ebb1f7740 10 client.15063 fetch_fsmap learned FSMap version 23 2024-03-18T14:52:32.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/1461065864 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c4efcd92d0 con 0x55c4f0066730 2024-03-18T14:52:32.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.108+0000 7f6e95ffb640 1 -- 172.21.15.76:0/1461065864 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea0019430 con 0x55c4f0066730 2024-03-18T14:52:32.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.109+0000 7f6e95ffb640 1 -- 172.21.15.76:0/1461065864 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f6ea8034670 con 0x55c4f0066730 2024-03-18T14:52:32.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.109+0000 7f6ebb1f7740 10 client.15063 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T14:52:32.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.109+0000 7f6ebb1f7740 10 client.15063 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T14:52:32.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.109+0000 7f6ebb1f7740 20 client.15063 populate_metadata read hostname 'smithi076' 2024-03-18T14:52:32.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.109+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/1461065864 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x55c4f00778a0 0x55c4f0078d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.109+0000 7f6ebb1f7740 4 client.15063 mds_command: new command op to 24490 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:52:32.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.109+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/1461065864 --> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4effd21c0 con 0x55c4f00778a0 2024-03-18T14:52:32.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.109+0000 7f6eb63a7640 1 --2- 172.21.15.76:0/1461065864 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x55c4f00778a0 0x55c4f0078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.109+0000 7f6eb63a7640 1 --2- 172.21.15.76:0/1461065864 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x55c4f00778a0 0x55c4f0078d10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cd21b21c09e532ff server_cookie=73006a4f748727b0 in_seq=0 out_seq=0 2024-03-18T14:52:32.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.109+0000 7f6e95ffb640 10 client.15063 ms_handle_connect on v2:172.21.15.174:6838/399810652 2024-03-18T14:52:32.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.110+0000 7f6e95ffb640 1 -- 172.21.15.76:0/1461065864 <== mds.1 v2:172.21.15.174:6838/399810652 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c4effd21c0 con 0x55c4f00778a0 2024-03-18T14:52:32.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.110+0000 7f6e95ffb640 10 client.15063 handle_command_reply: tid=0 2024-03-18T14:52:32.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.110+0000 7f6ebb1f7740 1 client.15063 shutdown 2024-03-18T14:52:32.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.110+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/1461065864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 msgr2=0x55c4f0072320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.110+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/1461065864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f0072320 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8036040 tx=0x7f6ea802c880 comp rx=0 tx=0).stop 2024-03-18T14:52:32.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.110+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/1461065864 shutdown_connections 2024-03-18T14:52:32.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.110+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/1461065864 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x55c4f00778a0 0x55c4f0078d10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.110+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/1461065864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f00615c0 0x55c4f006b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.110+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/1461065864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f0072320 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.110+0000 7f6e95ffb640 0 client.15063 ms_handle_reset on v2:172.21.15.174:6838/399810652 2024-03-18T14:52:32.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.110+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/1461065864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f0060bd0 0x55c4f006ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.110+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/1461065864 >> 172.21.15.76:0/1461065864 conn(0x55c4f005c140 msgr2=0x55c4f005b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:32.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.110+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/1461065864 shutdown_connections 2024-03-18T14:52:32.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.110+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/1461065864 wait complete. 2024-03-18T14:52:32.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.110+0000 7f6ebb1f7740 20 client.15063 trim_cache size 0 max 16384 2024-03-18T14:52:32.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.128+0000 7f6ebb1f7740 1 Processor -- start 2024-03-18T14:52:32.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.128+0000 7f6ebb1f7740 1 -- start start 2024-03-18T14:52:32.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.128+0000 7f6ebb1f7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f0060bd0 0x55c4f005c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.128+0000 7f6ebb1f7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f00615c0 0x55c4f0068b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.128+0000 7f6ebb1f7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f0062060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.128+0000 7f6ebb1f7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c4f0077050 con 0x55c4f0066730 2024-03-18T14:52:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.128+0000 7f6ebb1f7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c4f0077720 con 0x55c4f00615c0 2024-03-18T14:52:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.128+0000 7f6ebb1f7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c4f0077df0 con 0x55c4f0060bd0 2024-03-18T14:52:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.128+0000 7f6eb6ba8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f0062060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.128+0000 7f6eb6ba8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f0062060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60224/0 (socket says 172.21.15.76:60224) 2024-03-18T14:52:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.128+0000 7f6eb6ba8640 1 -- 172.21.15.76:0/2815637729 learned_addr learned my addr 172.21.15.76:0/2815637729 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.128+0000 7f6eb63a7640 1 --2- 172.21.15.76:0/2815637729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f0060bd0 0x55c4f005c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.128+0000 7f6eb5ba6640 1 --2- 172.21.15.76:0/2815637729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f00615c0 0x55c4f0068b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6eb6ba8640 1 -- 172.21.15.76:0/2815637729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f0060bd0 msgr2=0x55c4f005c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6eb6ba8640 1 --2- 172.21.15.76:0/2815637729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f0060bd0 0x55c4f005c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6eb6ba8640 1 -- 172.21.15.76:0/2815637729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f00615c0 msgr2=0x55c4f0068b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6eb6ba8640 1 --2- 172.21.15.76:0/2815637729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f00615c0 0x55c4f0068b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6eb6ba8640 1 -- 172.21.15.76:0/2815637729 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4efda36b0 con 0x55c4f0066730 2024-03-18T14:52:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6eb6ba8640 1 --2- 172.21.15.76:0/2815637729 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f0062060 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6ea802c7b0 tx=0x7f6ea8004560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5d7c7115b3ff537 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6e97fff640 1 -- 172.21.15.76:0/2815637729 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8065070 con 0x55c4f0066730 2024-03-18T14:52:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6e97fff640 1 -- 172.21.15.76:0/2815637729 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea8034cd0 con 0x55c4f0066730 2024-03-18T14:52:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6e97fff640 1 -- 172.21.15.76:0/2815637729 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8050040 con 0x55c4f0066730 2024-03-18T14:52:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/2815637729 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 msgr2=0x55c4f0062060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/2815637729 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f0062060 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6ea802c7b0 tx=0x7f6ea8004560 comp rx=0 tx=0).stop 2024-03-18T14:52:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/2815637729 shutdown_connections 2024-03-18T14:52:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/2815637729 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f0062060 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/2815637729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f00615c0 0x55c4f0068b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/2815637729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f0060bd0 0x55c4f005c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/2815637729 >> 172.21.15.76:0/2815637729 conn(0x55c4f005c140 msgr2=0x55c4f006ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/2815637729 shutdown_connections 2024-03-18T14:52:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.129+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/2815637729 wait complete. 2024-03-18T14:52:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6ebb1f7740 1 -- start start 2024-03-18T14:52:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6ebb1f7740 1 Processor -- start 2024-03-18T14:52:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6ebb1f7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f0060bd0 0x55c4f00718a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6ebb1f7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f00615c0 0x55c4f0071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6ebb1f7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f00dfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6ebb1f7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c4f0078b90 con 0x55c4f0066730 2024-03-18T14:52:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6ebb1f7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c4f00784c0 con 0x55c4f0060bd0 2024-03-18T14:52:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6ebb1f7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c4f0079260 con 0x55c4f00615c0 2024-03-18T14:52:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6ebb1f7740 10 client.0 _ll_register_callbacks cb 0x7f6ea8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:52:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6eb6ba8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f00dfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6eb6ba8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f00dfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60230/0 (socket says 172.21.15.76:60230) 2024-03-18T14:52:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6eb6ba8640 1 -- 172.21.15.76:0/2873918931 learned_addr learned my addr 172.21.15.76:0/2873918931 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6eb63a7640 1 --2- 172.21.15.76:0/2873918931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f0060bd0 0x55c4f00718a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6eb5ba6640 1 --2- 172.21.15.76:0/2873918931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f00615c0 0x55c4f0071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6eb6ba8640 1 -- 172.21.15.76:0/2873918931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f00615c0 msgr2=0x55c4f0071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6eb6ba8640 1 --2- 172.21.15.76:0/2873918931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f00615c0 0x55c4f0071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6eb6ba8640 1 -- 172.21.15.76:0/2873918931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f0060bd0 msgr2=0x55c4f00718a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6eb6ba8640 1 --2- 172.21.15.76:0/2873918931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f0060bd0 0x55c4f00718a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6eb6ba8640 1 -- 172.21.15.76:0/2873918931 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4f00e0050 con 0x55c4f0066730 2024-03-18T14:52:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6eb6ba8640 1 --2- 172.21.15.76:0/2873918931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f00dfa40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8004600 tx=0x7f6ea8034f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3675fe941f620df1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.130+0000 7f6e95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:52:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.131+0000 7f6e95ffb640 1 -- 172.21.15.76:0/2873918931 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8050070 con 0x55c4f0066730 2024-03-18T14:52:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.131+0000 7f6e95ffb640 1 -- 172.21.15.76:0/2873918931 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6ea8034640 con 0x55c4f0066730 2024-03-18T14:52:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.131+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/2873918931 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ea8034cd0 con 0x55c4f0066730 2024-03-18T14:52:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.131+0000 7f6e95ffb640 1 -- 172.21.15.76:0/2873918931 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6ea8070070 con 0x55c4f0066730 2024-03-18T14:52:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.131+0000 7f6ebb1f7740 10 client.15069 fetch_fsmap learned FSMap version 23 2024-03-18T14:52:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.131+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/2873918931 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c4f00e1350 con 0x55c4f0066730 2024-03-18T14:52:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.131+0000 7f6e95ffb640 1 -- 172.21.15.76:0/2873918931 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea8034cd0 con 0x55c4f0066730 2024-03-18T14:52:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.131+0000 7f6e95ffb640 1 -- 172.21.15.76:0/2873918931 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f6ea8055770 con 0x55c4f0066730 2024-03-18T14:52:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.131+0000 7f6ebb1f7740 10 client.15069 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T14:52:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.131+0000 7f6ebb1f7740 10 client.15069 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T14:52:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.131+0000 7f6ebb1f7740 20 client.15069 populate_metadata read hostname 'smithi076' 2024-03-18T14:52:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.131+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/2873918931 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x55c4f00e5b90 0x55c4f00e48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.131+0000 7f6ebb1f7740 4 client.15069 mds_command: new command op to 24490 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T14:52:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.131+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/2873918931 --> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] -- command(tid 0: {"prefix": "status"}) -- 0x55c4effcc510 con 0x55c4f00e5b90 2024-03-18T14:52:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.131+0000 7f6eb63a7640 1 --2- 172.21.15.76:0/2873918931 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x55c4f00e5b90 0x55c4f00e48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.132+0000 7f6e95ffb640 10 client.15069 ms_handle_connect on v2:172.21.15.174:6838/399810652 2024-03-18T14:52:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.132+0000 7f6eb63a7640 1 --2- 172.21.15.76:0/2873918931 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x55c4f00e5b90 0x55c4f00e48a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d6310b0182d25f21 server_cookie=da4c8d1cc65a11cb in_seq=0 out_seq=0 2024-03-18T14:52:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.132+0000 7f6e95ffb640 1 -- 172.21.15.76:0/2873918931 <== mds.1 v2:172.21.15.174:6838/399810652 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55c4effcc510 con 0x55c4f00e5b90 2024-03-18T14:52:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.132+0000 7f6e95ffb640 10 client.15069 handle_command_reply: tid=0 2024-03-18T14:52:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.132+0000 7f6ebb1f7740 1 client.15069 shutdown 2024-03-18T14:52:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.132+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/2873918931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 msgr2=0x55c4f00dfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.132+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/2873918931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f00dfa40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8004600 tx=0x7f6ea8034f10 comp rx=0 tx=0).stop 2024-03-18T14:52:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.132+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/2873918931 shutdown_connections 2024-03-18T14:52:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.132+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/2873918931 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x55c4f00e5b90 0x55c4f00e48a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.132+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/2873918931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4f00615c0 0x55c4f0071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.132+0000 7f6e95ffb640 0 client.15069 ms_handle_reset on v2:172.21.15.174:6838/399810652 2024-03-18T14:52:32.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.132+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/2873918931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4f0066730 0x55c4f00dfa40 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.132+0000 7f6ebb1f7740 1 --2- 172.21.15.76:0/2873918931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4f0060bd0 0x55c4f00718a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.132+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/2873918931 >> 172.21.15.76:0/2873918931 conn(0x55c4f005c140 msgr2=0x55c4f005f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:32.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.132+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/2873918931 shutdown_connections 2024-03-18T14:52:32.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.132+0000 7f6ebb1f7740 1 -- 172.21.15.76:0/2873918931 wait complete. 2024-03-18T14:52:32.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.132+0000 7f6ebb1f7740 20 client.15069 trim_cache size 0 max 16384 2024-03-18T14:52:32.139 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T14:52:32.139 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "9cc83b72-e535-11ee-95c9-87774f69a715", 2024-03-18T14:52:32.139 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 1, 2024-03-18T14:52:32.140 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24490, 2024-03-18T14:52:32.140 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-03-18T14:52:32.140 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-03-18T14:52:32.140 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-03-18T14:52:32.140 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 87.115520192000005, 2024-03-18T14:52:32.140 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 23, 2024-03-18T14:52:32.140 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 88, 2024-03-18T14:52:32.140 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T14:52:32.140 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 103.40646646099999 2024-03-18T14:52:32.140 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T14:52:32.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.133+0000 7f6eb860b640 1 -- 172.21.15.76:0/2380195383 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6e980a2890 msgr2=0x7f6e980a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.133+0000 7f6eb860b640 1 --2- 172.21.15.76:0/2380195383 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6e980a2890 0x7f6e980a4d50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6ea802ce70 tx=0x7f6ea802e040 comp rx=0 tx=0).stop 2024-03-18T14:52:32.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.133+0000 7f6eb860b640 1 -- 172.21.15.76:0/2380195383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb01018d0 msgr2=0x7f6eb019c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.133+0000 7f6eb860b640 1 --2- 172.21.15.76:0/2380195383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb01018d0 0x7f6eb019c820 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6ea400e9f0 tx=0x7f6ea400eeb0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.133+0000 7f6eb860b640 1 -- 172.21.15.76:0/2380195383 shutdown_connections 2024-03-18T14:52:32.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.133+0000 7f6eb860b640 1 --2- 172.21.15.76:0/2380195383 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6e980a2890 0x7f6e980a4d50 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.133+0000 7f6eb860b640 1 --2- 172.21.15.76:0/2380195383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6eb0102290 0x7f6eb019ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.133+0000 7f6eb860b640 1 --2- 172.21.15.76:0/2380195383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb01018d0 0x7f6eb019c820 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.133+0000 7f6eb860b640 1 --2- 172.21.15.76:0/2380195383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6eb01006e0 0x7f6eb00694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.133+0000 7f6eb860b640 1 -- 172.21.15.76:0/2380195383 >> 172.21.15.76:0/2380195383 conn(0x7f6eb00fbc70 msgr2=0x7f6eb0104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:32.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.133+0000 7f6eb860b640 1 -- 172.21.15.76:0/2380195383 shutdown_connections 2024-03-18T14:52:32.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.133+0000 7f6eb860b640 1 -- 172.21.15.76:0/2380195383 wait complete. 2024-03-18T14:52:32.144 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T14:52:32.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.247+0000 7f6b61910640 1 -- 172.21.15.76:0/3940004122 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b5c100700 msgr2=0x7f6b5c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.247+0000 7f6b61910640 1 --2- 172.21.15.76:0/3940004122 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b5c100700 0x7f6b5c100b20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c009900 tx=0x7f6b4c02bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:32.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.247+0000 7f6b61910640 1 -- 172.21.15.76:0/3940004122 shutdown_connections 2024-03-18T14:52:32.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.247+0000 7f6b61910640 1 --2- 172.21.15.76:0/3940004122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b5c1018f0 0x7f6b5c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.247+0000 7f6b61910640 1 --2- 172.21.15.76:0/3940004122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b5c1022b0 0x7f6b5c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.247+0000 7f6b61910640 1 --2- 172.21.15.76:0/3940004122 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b5c100700 0x7f6b5c100b20 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.247+0000 7f6b61910640 1 -- 172.21.15.76:0/3940004122 >> 172.21.15.76:0/3940004122 conn(0x7f6b5c0fbc70 msgr2=0x7f6b5c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:32.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.247+0000 7f6b61910640 1 -- 172.21.15.76:0/3940004122 shutdown_connections 2024-03-18T14:52:32.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.247+0000 7f6b61910640 1 -- 172.21.15.76:0/3940004122 wait complete. 2024-03-18T14:52:32.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.247+0000 7f6b61910640 1 Processor -- start 2024-03-18T14:52:32.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.247+0000 7f6b61910640 1 -- start start 2024-03-18T14:52:32.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b61910640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b5c100700 0x7f6b5c196b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b61910640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b5c1018f0 0x7f6b5c197040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b61910640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b5c1022b0 0x7f6b5c19e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b61910640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5c10c3a0 con 0x7f6b5c1018f0 2024-03-18T14:52:32.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b61910640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5c10c220 con 0x7f6b5c100700 2024-03-18T14:52:32.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b61910640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5c10c520 con 0x7f6b5c1022b0 2024-03-18T14:52:32.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b53fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b5c1018f0 0x7f6b5c197040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b53fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b5c1018f0 0x7f6b5c197040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60242/0 (socket says 172.21.15.76:60242) 2024-03-18T14:52:32.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b5affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b5c100700 0x7f6b5c196b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b53fff640 1 -- 172.21.15.76:0/2069924804 learned_addr learned my addr 172.21.15.76:0/2069924804 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:32.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b5b7fe640 1 --2- 172.21.15.76:0/2069924804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b5c1022b0 0x7f6b5c19e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b53fff640 1 -- 172.21.15.76:0/2069924804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b5c1022b0 msgr2=0x7f6b5c19e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b53fff640 1 --2- 172.21.15.76:0/2069924804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b5c1022b0 0x7f6b5c19e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b53fff640 1 -- 172.21.15.76:0/2069924804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b5c100700 msgr2=0x7f6b5c196b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b53fff640 1 --2- 172.21.15.76:0/2069924804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b5c100700 0x7f6b5c196b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b53fff640 1 -- 172.21.15.76:0/2069924804 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b5c19e6d0 con 0x7f6b5c1018f0 2024-03-18T14:52:32.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b5affd640 1 --2- 172.21.15.76:0/2069924804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b5c100700 0x7f6b5c196b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:32.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b53fff640 1 --2- 172.21.15.76:0/2069924804 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b5c1018f0 0x7f6b5c197040 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400e9f0 tx=0x7f6b4400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:32.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b6090e640 1 -- 172.21.15.76:0/2069924804 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4400ccc0 con 0x7f6b5c1018f0 2024-03-18T14:52:32.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b61910640 1 -- 172.21.15.76:0/2069924804 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b5c19e9c0 con 0x7f6b5c1018f0 2024-03-18T14:52:32.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b6090e640 1 -- 172.21.15.76:0/2069924804 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b44010910 con 0x7f6b5c1018f0 2024-03-18T14:52:32.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b6090e640 1 -- 172.21.15.76:0/2069924804 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b440188a0 con 0x7f6b5c1018f0 2024-03-18T14:52:32.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.248+0000 7f6b61910640 1 -- 172.21.15.76:0/2069924804 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b5c19ef10 con 0x7f6b5c1018f0 2024-03-18T14:52:32.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.250+0000 7f6b6090e640 1 -- 172.21.15.76:0/2069924804 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6b44010ac0 con 0x7f6b5c1018f0 2024-03-18T14:52:32.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b645dc740 1 Processor -- start 2024-03-18T14:52:32.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b645dc740 1 -- start start 2024-03-18T14:52:32.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b645dc740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547abd0 0x56554547aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b645dc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547b5c0 0x5655454800c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b6090e640 1 --2- 172.21.15.76:0/2069924804 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6b480a2890 0x7f6b480a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b645dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565545480730 0x565545482bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b645dc740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5655453dec80 con 0x56554547abd0 2024-03-18T14:52:32.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b645dc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x565545387aa0 con 0x56554547b5c0 2024-03-18T14:52:32.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b645dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5655453dad20 con 0x565545480730 2024-03-18T14:52:32.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b5b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547abd0 0x56554547aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b53fff640 1 --2- 172.21.15.76:0/2069924804 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6b480a2890 0x7f6b480a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b5affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547b5c0 0x5655454800c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b5b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547abd0 0x56554547aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60244/0 (socket says 172.21.15.76:60244) 2024-03-18T14:52:32.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b5b7fe640 1 -- 172.21.15.76:0/1004482055 learned_addr learned my addr 172.21.15.76:0/1004482055 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:32.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b5b7fe640 1 --2- 172.21.15.76:0/1004482055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565545480730 0x565545482bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b6090e640 1 -- 172.21.15.76:0/2069924804 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6b440cef80 con 0x7f6b5c1018f0 2024-03-18T14:52:32.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b5b7fe640 1 -- 172.21.15.76:0/1004482055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565545480730 msgr2=0x565545482bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:32.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b5b7fe640 1 --2- 172.21.15.76:0/1004482055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565545480730 0x565545482bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b5b7fe640 1 -- 172.21.15.76:0/1004482055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547b5c0 msgr2=0x5655454800c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:32.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b5b7fe640 1 --2- 172.21.15.76:0/1004482055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547b5c0 0x5655454800c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b5b7fe640 1 -- 172.21.15.76:0/1004482055 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5655453a9ff0 con 0x56554547abd0 2024-03-18T14:52:32.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b5affd640 1 --2- 172.21.15.76:0/1004482055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547b5c0 0x5655454800c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:32.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b53fff640 1 --2- 172.21.15.76:0/2069924804 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6b480a2890 0x7f6b480a4d50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6b4409b820 tx=0x7f6b4406d510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:32.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.251+0000 7f6b5b7fe640 1 --2- 172.21.15.76:0/1004482055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547abd0 0x56554547aff0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6b54005c40 tx=0x7f6b54009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=156578e11c33c12 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:32.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.252+0000 7f6b51ffb640 1 -- 172.21.15.76:0/1004482055 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b54011ae0 con 0x56554547abd0 2024-03-18T14:52:32.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.252+0000 7f6b51ffb640 1 -- 172.21.15.76:0/1004482055 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b54011c80 con 0x56554547abd0 2024-03-18T14:52:32.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.252+0000 7f6b51ffb640 1 -- 172.21.15.76:0/1004482055 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5400f6b0 con 0x56554547abd0 2024-03-18T14:52:32.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.252+0000 7f6b645dc740 1 -- 172.21.15.76:0/1004482055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547abd0 msgr2=0x56554547aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.252+0000 7f6b645dc740 1 --2- 172.21.15.76:0/1004482055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547abd0 0x56554547aff0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6b54005c40 tx=0x7f6b54009bf0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.252+0000 7f6b645dc740 1 -- 172.21.15.76:0/1004482055 shutdown_connections 2024-03-18T14:52:32.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.252+0000 7f6b645dc740 1 --2- 172.21.15.76:0/1004482055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565545480730 0x565545482bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.252+0000 7f6b645dc740 1 --2- 172.21.15.76:0/1004482055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547b5c0 0x5655454800c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.252+0000 7f6b645dc740 1 --2- 172.21.15.76:0/1004482055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547abd0 0x56554547aff0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.252+0000 7f6b645dc740 1 -- 172.21.15.76:0/1004482055 >> 172.21.15.76:0/1004482055 conn(0x565545476140 msgr2=0x565545476540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:32.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.252+0000 7f6b645dc740 1 -- 172.21.15.76:0/1004482055 shutdown_connections 2024-03-18T14:52:32.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.252+0000 7f6b645dc740 1 -- 172.21.15.76:0/1004482055 wait complete. 2024-03-18T14:52:32.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.253+0000 7f6b645dc740 1 -- start start 2024-03-18T14:52:32.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.253+0000 7f6b645dc740 1 Processor -- start 2024-03-18T14:52:32.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.253+0000 7f6b645dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56554547abd0 0x565545484d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.253+0000 7f6b645dc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547b5c0 0x565545485260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.253+0000 7f6b645dc740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565545480730 0x56554548c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.253+0000 7f6b645dc740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5655454836e0 con 0x565545480730 2024-03-18T14:52:32.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.253+0000 7f6b645dc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56554539aef0 con 0x56554547b5c0 2024-03-18T14:52:32.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.253+0000 7f6b645dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565545483860 con 0x56554547abd0 2024-03-18T14:52:32.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.253+0000 7f6b645dc740 10 client.0 _ll_register_callbacks cb 0x565545395dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:52:32.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.253+0000 7f6b5b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565545480730 0x56554548c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.253+0000 7f6b5b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565545480730 0x56554548c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60250/0 (socket says 172.21.15.76:60250) 2024-03-18T14:52:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.253+0000 7f6b5b7fe640 1 -- 172.21.15.76:0/2289527025 learned_addr learned my addr 172.21.15.76:0/2289527025 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.253+0000 7f6b5affd640 1 --2- 172.21.15.76:0/2289527025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547b5c0 0x565545485260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.253+0000 7f6b5b7fe640 1 --2- 172.21.15.76:0/2289527025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56554547abd0 0x565545484d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b5b7fe640 1 -- 172.21.15.76:0/2289527025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56554547abd0 msgr2=0x565545484d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b5b7fe640 1 --2- 172.21.15.76:0/2289527025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56554547abd0 0x565545484d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b5b7fe640 1 -- 172.21.15.76:0/2289527025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547b5c0 msgr2=0x565545485260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b5b7fe640 1 --2- 172.21.15.76:0/2289527025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547b5c0 0x565545485260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b5b7fe640 1 -- 172.21.15.76:0/2289527025 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5655453a7a30 con 0x565545480730 2024-03-18T14:52:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b5affd640 1 --2- 172.21.15.76:0/2289527025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547b5c0 0x565545485260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b5b7fe640 1 --2- 172.21.15.76:0/2289527025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565545480730 0x56554548c2a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009560 tx=0x7f6b54005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd821046d6093c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b3affd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:52:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b3affd640 1 -- 172.21.15.76:0/2289527025 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b54003b80 con 0x565545480730 2024-03-18T14:52:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b3affd640 1 -- 172.21.15.76:0/2289527025 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6b54027040 con 0x565545480730 2024-03-18T14:52:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b645dc740 1 -- 172.21.15.76:0/2289527025 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b54011c80 con 0x565545480730 2024-03-18T14:52:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b3affd640 1 -- 172.21.15.76:0/2289527025 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6b5401f440 con 0x565545480730 2024-03-18T14:52:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b645dc740 10 client.15081 fetch_fsmap learned FSMap version 23 2024-03-18T14:52:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b645dc740 1 -- 172.21.15.76:0/2289527025 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5655450f32d0 con 0x565545480730 2024-03-18T14:52:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b3affd640 1 -- 172.21.15.76:0/2289527025 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b54011c80 con 0x565545480730 2024-03-18T14:52:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b3affd640 1 -- 172.21.15.76:0/2289527025 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f6b5401d020 con 0x565545480730 2024-03-18T14:52:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b645dc740 10 client.15081 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T14:52:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b645dc740 10 client.15081 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T14:52:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b645dc740 20 client.15081 populate_metadata read hostname 'smithi076' 2024-03-18T14:52:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b645dc740 1 --2- 172.21.15.76:0/2289527025 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x565545491440 0x5655454928b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b645dc740 4 client.15081 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:52:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.254+0000 7f6b645dc740 1 -- 172.21.15.76:0/2289527025 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5655453ec1c0 con 0x565545491440 2024-03-18T14:52:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.255+0000 7f6b5affd640 1 --2- 172.21.15.76:0/2289527025 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x565545491440 0x5655454928b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.255+0000 7f6b3affd640 10 client.15081 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:52:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.255+0000 7f6b5affd640 1 --2- 172.21.15.76:0/2289527025 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x565545491440 0x5655454928b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fa71eb7eb08d45f server_cookie=5f4cce6559d30231 in_seq=0 out_seq=0 2024-03-18T14:52:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.256+0000 7f6b3affd640 1 -- 172.21.15.76:0/2289527025 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5655453ec1c0 con 0x565545491440 2024-03-18T14:52:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.256+0000 7f6b3affd640 10 client.15081 handle_command_reply: tid=0 2024-03-18T14:52:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.256+0000 7f6b645dc740 1 client.15081 shutdown 2024-03-18T14:52:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.256+0000 7f6b645dc740 1 -- 172.21.15.76:0/2289527025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565545480730 msgr2=0x56554548c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.256+0000 7f6b645dc740 1 --2- 172.21.15.76:0/2289527025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565545480730 0x56554548c2a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009560 tx=0x7f6b54005f70 comp rx=0 tx=0).stop 2024-03-18T14:52:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.256+0000 7f6b645dc740 1 -- 172.21.15.76:0/2289527025 shutdown_connections 2024-03-18T14:52:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.256+0000 7f6b645dc740 1 --2- 172.21.15.76:0/2289527025 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x565545491440 0x5655454928b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.256+0000 7f6b645dc740 1 --2- 172.21.15.76:0/2289527025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565545480730 0x56554548c2a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.256+0000 7f6b645dc740 1 --2- 172.21.15.76:0/2289527025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547b5c0 0x565545485260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.256+0000 7f6b3affd640 0 client.15081 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:52:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.256+0000 7f6b645dc740 1 --2- 172.21.15.76:0/2289527025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56554547abd0 0x565545484d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.256+0000 7f6b645dc740 1 -- 172.21.15.76:0/2289527025 >> 172.21.15.76:0/2289527025 conn(0x565545476140 msgr2=0x565545475380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.256+0000 7f6b645dc740 1 -- 172.21.15.76:0/2289527025 shutdown_connections 2024-03-18T14:52:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.256+0000 7f6b645dc740 1 -- 172.21.15.76:0/2289527025 wait complete. 2024-03-18T14:52:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.256+0000 7f6b645dc740 20 client.15081 trim_cache size 0 max 16384 2024-03-18T14:52:32.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b645dc740 1 Processor -- start 2024-03-18T14:52:32.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b645dc740 1 -- start start 2024-03-18T14:52:32.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b645dc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547abd0 0x565545485040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b645dc740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547b5c0 0x56554548c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b645dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565545480730 0x565545474af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b645dc740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x565545490bf0 con 0x56554547b5c0 2024-03-18T14:52:32.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b645dc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5655454912c0 con 0x56554547abd0 2024-03-18T14:52:32.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b645dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565545491990 con 0x565545480730 2024-03-18T14:52:32.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b5affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547b5c0 0x56554548c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b5affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547b5c0 0x56554548c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60256/0 (socket says 172.21.15.76:60256) 2024-03-18T14:52:32.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b5affd640 1 -- 172.21.15.76:0/4054771801 learned_addr learned my addr 172.21.15.76:0/4054771801 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:32.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b5b7fe640 1 --2- 172.21.15.76:0/4054771801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547abd0 0x565545485040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b5b7fe640 1 --2- 172.21.15.76:0/4054771801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565545480730 0x565545474af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b5affd640 1 -- 172.21.15.76:0/4054771801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565545480730 msgr2=0x565545474af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:32.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b5affd640 1 --2- 172.21.15.76:0/4054771801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565545480730 0x565545474af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b5affd640 1 -- 172.21.15.76:0/4054771801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547abd0 msgr2=0x565545485040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:32.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b5affd640 1 --2- 172.21.15.76:0/4054771801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547abd0 0x565545485040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b5affd640 1 -- 172.21.15.76:0/4054771801 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5655451bd6b0 con 0x56554547b5c0 2024-03-18T14:52:32.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b5b7fe640 1 --2- 172.21.15.76:0/4054771801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547abd0 0x565545485040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:32.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b5affd640 1 --2- 172.21.15.76:0/4054771801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547b5c0 0x56554548c1c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c005ab0 tx=0x7f6b4c02cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2623f1e76b63232 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:32.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.274+0000 7f6b517fa640 1 -- 172.21.15.76:0/4054771801 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c04f070 con 0x56554547b5c0 2024-03-18T14:52:32.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.275+0000 7f6b517fa640 1 -- 172.21.15.76:0/4054771801 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b4c004410 con 0x56554547b5c0 2024-03-18T14:52:32.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.275+0000 7f6b517fa640 1 -- 172.21.15.76:0/4054771801 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c004a40 con 0x56554547b5c0 2024-03-18T14:52:32.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.275+0000 7f6b645dc740 1 -- 172.21.15.76:0/4054771801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547b5c0 msgr2=0x56554548c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.275+0000 7f6b645dc740 1 --2- 172.21.15.76:0/4054771801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547b5c0 0x56554548c1c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c005ab0 tx=0x7f6b4c02cde0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.275+0000 7f6b645dc740 1 -- 172.21.15.76:0/4054771801 shutdown_connections 2024-03-18T14:52:32.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.275+0000 7f6b645dc740 1 --2- 172.21.15.76:0/4054771801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565545480730 0x565545474af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.275+0000 7f6b645dc740 1 --2- 172.21.15.76:0/4054771801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547b5c0 0x56554548c1c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.275+0000 7f6b645dc740 1 --2- 172.21.15.76:0/4054771801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56554547abd0 0x565545485040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.275+0000 7f6b645dc740 1 -- 172.21.15.76:0/4054771801 >> 172.21.15.76:0/4054771801 conn(0x565545476140 msgr2=0x565545484930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:32.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.275+0000 7f6b645dc740 1 -- 172.21.15.76:0/4054771801 shutdown_connections 2024-03-18T14:52:32.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.275+0000 7f6b645dc740 1 -- 172.21.15.76:0/4054771801 wait complete. 2024-03-18T14:52:32.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.275+0000 7f6b645dc740 1 -- start start 2024-03-18T14:52:32.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b645dc740 1 Processor -- start 2024-03-18T14:52:32.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b645dc740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547abd0 0x56554549a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b645dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56554547b5c0 0x56554549ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b645dc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565545480730 0x5655454f51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b645dc740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x565545492730 con 0x56554547abd0 2024-03-18T14:52:32.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b645dc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x565545492060 con 0x565545480730 2024-03-18T14:52:32.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b645dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565545492e00 con 0x56554547b5c0 2024-03-18T14:52:32.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b645dc740 10 client.0 _ll_register_callbacks cb 0x7f6b44005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:52:32.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b5b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547abd0 0x56554549a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b5b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547abd0 0x56554549a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60258/0 (socket says 172.21.15.76:60258) 2024-03-18T14:52:32.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b5b7fe640 1 -- 172.21.15.76:0/69267079 learned_addr learned my addr 172.21.15.76:0/69267079 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:32.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b5affd640 1 --2- 172.21.15.76:0/69267079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56554547b5c0 0x56554549ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b5b7fe640 1 --2- 172.21.15.76:0/69267079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565545480730 0x5655454f51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b5b7fe640 1 -- 172.21.15.76:0/69267079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56554547b5c0 msgr2=0x56554549ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:32.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b5b7fe640 1 --2- 172.21.15.76:0/69267079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56554547b5c0 0x56554549ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b5b7fe640 1 -- 172.21.15.76:0/69267079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565545480730 msgr2=0x5655454f51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:32.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b5b7fe640 1 --2- 172.21.15.76:0/69267079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565545480730 0x5655454f51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b5b7fe640 1 -- 172.21.15.76:0/69267079 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5655454f56e0 con 0x56554547abd0 2024-03-18T14:52:32.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b5b7fe640 1 --2- 172.21.15.76:0/69267079 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547abd0 0x56554549a740 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009fd0 tx=0x7f6b5401fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee2f2c64fd78b47 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:32.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b3affd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:52:32.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.276+0000 7f6b3affd640 1 -- 172.21.15.76:0/69267079 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b54003d90 con 0x56554547abd0 2024-03-18T14:52:32.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.277+0000 7f6b3affd640 1 -- 172.21.15.76:0/69267079 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6b54024040 con 0x56554547abd0 2024-03-18T14:52:32.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.277+0000 7f6b645dc740 1 -- 172.21.15.76:0/69267079 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b4c004410 con 0x56554547abd0 2024-03-18T14:52:32.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.277+0000 7f6b3affd640 1 -- 172.21.15.76:0/69267079 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6b5400f3c0 con 0x56554547abd0 2024-03-18T14:52:32.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.277+0000 7f6b645dc740 10 client.15087 fetch_fsmap learned FSMap version 23 2024-03-18T14:52:32.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.277+0000 7f6b645dc740 1 -- 172.21.15.76:0/69267079 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5655454f69e0 con 0x56554547abd0 2024-03-18T14:52:32.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.277+0000 7f6b3affd640 1 -- 172.21.15.76:0/69267079 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b4c004410 con 0x56554547abd0 2024-03-18T14:52:32.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.277+0000 7f6b3affd640 1 -- 172.21.15.76:0/69267079 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f6b5401d040 con 0x56554547abd0 2024-03-18T14:52:32.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.277+0000 7f6b645dc740 10 client.15087 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T14:52:32.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.277+0000 7f6b645dc740 10 client.15087 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T14:52:32.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.277+0000 7f6b645dc740 20 client.15087 populate_metadata read hostname 'smithi076' 2024-03-18T14:52:32.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.277+0000 7f6b645dc740 1 --2- 172.21.15.76:0/69267079 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5655454fb510 0x5655454fa220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:32.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.277+0000 7f6b645dc740 4 client.15087 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T14:52:32.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.277+0000 7f6b645dc740 1 -- 172.21.15.76:0/69267079 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "status"}) -- 0x5655453e6510 con 0x5655454fb510 2024-03-18T14:52:32.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.277+0000 7f6b5affd640 1 --2- 172.21.15.76:0/69267079 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5655454fb510 0x5655454fa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:32.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b3affd640 10 client.15087 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:52:32.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b5affd640 1 --2- 172.21.15.76:0/69267079 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5655454fb510 0x5655454fa220 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dea75617fd92fcc server_cookie=c6c75b02f9363441 in_seq=0 out_seq=0 2024-03-18T14:52:32.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b3affd640 1 -- 172.21.15.76:0/69267079 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5655453e6510 con 0x5655454fb510 2024-03-18T14:52:32.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b3affd640 10 client.15087 handle_command_reply: tid=0 2024-03-18T14:52:32.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b645dc740 1 client.15087 shutdown 2024-03-18T14:52:32.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b645dc740 1 -- 172.21.15.76:0/69267079 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547abd0 msgr2=0x56554549a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b645dc740 1 --2- 172.21.15.76:0/69267079 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547abd0 0x56554549a740 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009fd0 tx=0x7f6b5401fc80 comp rx=0 tx=0).stop 2024-03-18T14:52:32.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b645dc740 1 -- 172.21.15.76:0/69267079 shutdown_connections 2024-03-18T14:52:32.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b645dc740 1 --2- 172.21.15.76:0/69267079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56554547b5c0 0x56554549ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b645dc740 1 --2- 172.21.15.76:0/69267079 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5655454fb510 0x5655454fa220 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b645dc740 1 --2- 172.21.15.76:0/69267079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565545480730 0x5655454f51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b645dc740 1 --2- 172.21.15.76:0/69267079 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56554547abd0 0x56554549a740 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b3affd640 0 client.15087 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:52:32.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b645dc740 1 -- 172.21.15.76:0/69267079 >> 172.21.15.76:0/69267079 conn(0x565545476140 msgr2=0x5655454793a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:32.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b645dc740 1 -- 172.21.15.76:0/69267079 shutdown_connections 2024-03-18T14:52:32.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b645dc740 1 -- 172.21.15.76:0/69267079 wait complete. 2024-03-18T14:52:32.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b645dc740 20 client.15087 trim_cache size 0 max 16384 2024-03-18T14:52:32.285 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T14:52:32.285 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "9cc83b72-e535-11ee-95c9-87774f69a715", 2024-03-18T14:52:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 0, 2024-03-18T14:52:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24524, 2024-03-18T14:52:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-03-18T14:52:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-03-18T14:52:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-03-18T14:52:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 90.281450848000006, 2024-03-18T14:52:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 23, 2024-03-18T14:52:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 88, 2024-03-18T14:52:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T14:52:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 94.602251582999997 2024-03-18T14:52:32.286 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T14:52:32.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b61910640 1 -- 172.21.15.76:0/2069924804 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6b480a2890 msgr2=0x7f6b480a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.278+0000 7f6b61910640 1 --2- 172.21.15.76:0/2069924804 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6b480a2890 0x7f6b480a4d50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6b4409b820 tx=0x7f6b4406d510 comp rx=0 tx=0).stop 2024-03-18T14:52:32.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.279+0000 7f6b61910640 1 -- 172.21.15.76:0/2069924804 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b5c1018f0 msgr2=0x7f6b5c197040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:32.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.279+0000 7f6b61910640 1 --2- 172.21.15.76:0/2069924804 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b5c1018f0 0x7f6b5c197040 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400e9f0 tx=0x7f6b4400edf0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.279+0000 7f6b61910640 1 -- 172.21.15.76:0/2069924804 shutdown_connections 2024-03-18T14:52:32.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.279+0000 7f6b61910640 1 --2- 172.21.15.76:0/2069924804 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6b480a2890 0x7f6b480a4d50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.279+0000 7f6b61910640 1 --2- 172.21.15.76:0/2069924804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b5c1022b0 0x7f6b5c19e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.279+0000 7f6b61910640 1 --2- 172.21.15.76:0/2069924804 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b5c1018f0 0x7f6b5c197040 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.279+0000 7f6b61910640 1 --2- 172.21.15.76:0/2069924804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b5c100700 0x7f6b5c196b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:32.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.279+0000 7f6b61910640 1 -- 172.21.15.76:0/2069924804 >> 172.21.15.76:0/2069924804 conn(0x7f6b5c0fbc70 msgr2=0x7f6b5c104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:32.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.279+0000 7f6b61910640 1 -- 172.21.15.76:0/2069924804 shutdown_connections 2024-03-18T14:52:32.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:32.279+0000 7f6b61910640 1 -- 172.21.15.76:0/2069924804 wait complete. 2024-03-18T14:52:32.290 INFO:tasks.cephadm:Setup complete, yielding 2024-03-18T14:52:32.291 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T14:52:32.300 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi076.front.sepia.ceph.com 2024-03-18T14:52:32.300 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-18T14:52:32.542 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:52:32.804 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:32 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2644441138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:52:32.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:32 smithi076 ceph-mon[32205]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-18T14:52:32.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:32 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2644441138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:52:32.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:32 smithi174 ceph-mon[36787]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-18T14:52:32.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:32 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2644441138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:52:32.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:32 smithi077 ceph-mon[35865]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-18T14:52:34.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.213+0000 7f42c10db700 1 -- 172.21.15.76:0/4036950339 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42bc101e90 msgr2=0x7f42bc109730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:34.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.213+0000 7f42c10db700 1 --2- 172.21.15.76:0/4036950339 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42bc101e90 0x7f42bc109730 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f42b400b0a0 tx=0x7f42b400b3b0 comp rx=0 tx=0).stop 2024-03-18T14:52:34.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.213+0000 7f42c10db700 1 -- 172.21.15.76:0/4036950339 shutdown_connections 2024-03-18T14:52:34.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.213+0000 7f42c10db700 1 --2- 172.21.15.76:0/4036950339 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42bc101e90 0x7f42bc109730 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:34.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.213+0000 7f42c10db700 1 --2- 172.21.15.76:0/4036950339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42bc102820 0x7f42bc102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:34.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.213+0000 7f42c10db700 1 --2- 172.21.15.76:0/4036950339 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42bc1014d0 0x7f42bc101950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:34.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.213+0000 7f42c10db700 1 -- 172.21.15.76:0/4036950339 >> 172.21.15.76:0/4036950339 conn(0x7f42bc0fbf20 msgr2=0x7f42bc0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:34.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.213+0000 7f42c10db700 1 -- 172.21.15.76:0/4036950339 shutdown_connections 2024-03-18T14:52:34.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.213+0000 7f42c10db700 1 -- 172.21.15.76:0/4036950339 wait complete. 2024-03-18T14:52:34.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42c10db700 1 Processor -- start 2024-03-18T14:52:34.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42c10db700 1 -- start start 2024-03-18T14:52:34.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42c10db700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42bc1014d0 0x7f42bc197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:34.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42c10db700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42bc101e90 0x7f42bc1978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:34.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42c10db700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42bc102820 0x7f42bc19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:34.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42c10db700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc10ce20 con 0x7f42bc1014d0 2024-03-18T14:52:34.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42c10db700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc10cca0 con 0x7f42bc102820 2024-03-18T14:52:34.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42c10db700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc10cfa0 con 0x7f42bc101e90 2024-03-18T14:52:34.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42bad9d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42bc1014d0 0x7f42bc197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:34.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42bad9d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42bc1014d0 0x7f42bc197380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60274/0 (socket says 172.21.15.76:60274) 2024-03-18T14:52:34.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42bad9d700 1 -- 172.21.15.76:0/1209081230 learned_addr learned my addr 172.21.15.76:0/1209081230 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:34.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42ba59c700 1 --2- 172.21.15.76:0/1209081230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42bc101e90 0x7f42bc1978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:34.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42bb59e700 1 --2- 172.21.15.76:0/1209081230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42bc102820 0x7f42bc19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:34.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42bad9d700 1 -- 172.21.15.76:0/1209081230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42bc101e90 msgr2=0x7f42bc1978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:34.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42bad9d700 1 --2- 172.21.15.76:0/1209081230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42bc101e90 0x7f42bc1978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:34.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42bad9d700 1 -- 172.21.15.76:0/1209081230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42bc102820 msgr2=0x7f42bc19e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:34.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42bad9d700 1 --2- 172.21.15.76:0/1209081230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42bc102820 0x7f42bc19e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:34.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.214+0000 7f42bad9d700 1 -- 172.21.15.76:0/1209081230 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42bc19efe0 con 0x7f42bc1014d0 2024-03-18T14:52:34.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.215+0000 7f42bad9d700 1 --2- 172.21.15.76:0/1209081230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42bc1014d0 0x7f42bc197380 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f42ac00e9a0 tx=0x7f42ac00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:34.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.215+0000 7f42a3fff700 1 -- 172.21.15.76:0/1209081230 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac004220 con 0x7f42bc1014d0 2024-03-18T14:52:34.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.215+0000 7f42c10db700 1 -- 172.21.15.76:0/1209081230 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42bc19f2d0 con 0x7f42bc1014d0 2024-03-18T14:52:34.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.215+0000 7f42a3fff700 1 -- 172.21.15.76:0/1209081230 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42ac0043c0 con 0x7f42bc1014d0 2024-03-18T14:52:34.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.215+0000 7f42a3fff700 1 -- 172.21.15.76:0/1209081230 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac003df0 con 0x7f42bc1014d0 2024-03-18T14:52:34.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.215+0000 7f42c10db700 1 -- 172.21.15.76:0/1209081230 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42bc19f860 con 0x7f42bc1014d0 2024-03-18T14:52:34.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.216+0000 7f42c10db700 1 -- 172.21.15.76:0/1209081230 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42bc068f90 con 0x7f42bc1014d0 2024-03-18T14:52:34.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.218+0000 7f42a3fff700 1 -- 172.21.15.76:0/1209081230 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f42ac00c810 con 0x7f42bc1014d0 2024-03-18T14:52:34.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.218+0000 7f42a3fff700 1 --2- 172.21.15.76:0/1209081230 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f42a40a2920 0x7f42a40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:34.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.218+0000 7f42ba59c700 1 --2- 172.21.15.76:0/1209081230 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f42a40a2920 0x7f42a40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:34.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.218+0000 7f42a3fff700 1 -- 172.21.15.76:0/1209081230 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f42ac0ceea0 con 0x7f42bc1014d0 2024-03-18T14:52:34.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.219+0000 7f42ba59c700 1 --2- 172.21.15.76:0/1209081230 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f42a40a2920 0x7f42a40a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f42bc102550 tx=0x7f42b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:34.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.222+0000 7f42a3fff700 1 -- 172.21.15.76:0/1209081230 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f42ac09bea0 con 0x7f42bc1014d0 2024-03-18T14:52:34.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.443+0000 7f42c10db700 1 -- 172.21.15.76:0/1209081230 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f42bc062710 con 0x7f42a40a2920 2024-03-18T14:52:34.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.444+0000 7f42a3fff700 1 -- 172.21.15.76:0/1209081230 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f42bc062710 con 0x7f42a40a2920 2024-03-18T14:52:34.444 INFO:teuthology.orchestra.run.smithi076.stdout:Backend: cephadm 2024-03-18T14:52:34.444 INFO:teuthology.orchestra.run.smithi076.stdout:Available: Yes 2024-03-18T14:52:34.445 INFO:teuthology.orchestra.run.smithi076.stdout:Paused: No 2024-03-18T14:52:34.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.446+0000 7f42c10db700 1 -- 172.21.15.76:0/1209081230 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f42a40a2920 msgr2=0x7f42a40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:34.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.446+0000 7f42c10db700 1 --2- 172.21.15.76:0/1209081230 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f42a40a2920 0x7f42a40a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f42bc102550 tx=0x7f42b0009260 comp rx=0 tx=0).stop 2024-03-18T14:52:34.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.446+0000 7f42c10db700 1 -- 172.21.15.76:0/1209081230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42bc1014d0 msgr2=0x7f42bc197380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:34.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.446+0000 7f42c10db700 1 --2- 172.21.15.76:0/1209081230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42bc1014d0 0x7f42bc197380 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f42ac00e9a0 tx=0x7f42ac00ecb0 comp rx=0 tx=0).stop 2024-03-18T14:52:34.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.446+0000 7f42c10db700 1 -- 172.21.15.76:0/1209081230 shutdown_connections 2024-03-18T14:52:34.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.446+0000 7f42c10db700 1 --2- 172.21.15.76:0/1209081230 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f42a40a2920 0x7f42a40a4de0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:34.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.446+0000 7f42c10db700 1 --2- 172.21.15.76:0/1209081230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42bc102820 0x7f42bc19e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:34.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.446+0000 7f42c10db700 1 --2- 172.21.15.76:0/1209081230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42bc1014d0 0x7f42bc197380 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:34.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.446+0000 7f42c10db700 1 --2- 172.21.15.76:0/1209081230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42bc101e90 0x7f42bc1978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:34.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.446+0000 7f42c10db700 1 -- 172.21.15.76:0/1209081230 >> 172.21.15.76:0/1209081230 conn(0x7f42bc0fbf20 msgr2=0x7f42bc105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:34.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.446+0000 7f42c10db700 1 -- 172.21.15.76:0/1209081230 shutdown_connections 2024-03-18T14:52:34.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:34.446+0000 7f42c10db700 1 -- 172.21.15.76:0/1209081230 wait complete. 2024-03-18T14:52:35.176 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-18T14:52:35.424 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:52:35.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:35 smithi076 ceph-mon[32205]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:35.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:35 smithi076 ceph-mon[32205]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:52:35.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:35 smithi174 ceph-mon[36787]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:35.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:35 smithi174 ceph-mon[36787]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:52:35.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:35 smithi077 ceph-mon[35865]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:35.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:35 smithi077 ceph-mon[35865]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:52:37.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.109+0000 7fe93c1a5700 1 -- 172.21.15.76:0/2435819331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe934101200 msgr2=0x7fe934101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:37.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.109+0000 7fe93c1a5700 1 --2- 172.21.15.76:0/2435819331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe934101200 0x7fe934101620 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe9240099c0 tx=0x7fe924009cd0 comp rx=0 tx=0).stop 2024-03-18T14:52:37.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.109+0000 7fe93c1a5700 1 -- 172.21.15.76:0/2435819331 shutdown_connections 2024-03-18T14:52:37.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.109+0000 7fe93c1a5700 1 --2- 172.21.15.76:0/2435819331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe934102da0 0x7fe934109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:37.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.109+0000 7fe93c1a5700 1 --2- 172.21.15.76:0/2435819331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe934101200 0x7fe934101620 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:37.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.109+0000 7fe93c1a5700 1 --2- 172.21.15.76:0/2435819331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe934102400 0x7fe934102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:37.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.109+0000 7fe93c1a5700 1 -- 172.21.15.76:0/2435819331 >> 172.21.15.76:0/2435819331 conn(0x7fe9340fc7c0 msgr2=0x7fe9340febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:37.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.109+0000 7fe93c1a5700 1 -- 172.21.15.76:0/2435819331 shutdown_connections 2024-03-18T14:52:37.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.109+0000 7fe93c1a5700 1 -- 172.21.15.76:0/2435819331 wait complete. 2024-03-18T14:52:37.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe93c1a5700 1 Processor -- start 2024-03-18T14:52:37.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe93c1a5700 1 -- start start 2024-03-18T14:52:37.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe93c1a5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe934101200 0x7fe93410f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:37.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe93c1a5700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe934102400 0x7fe93410d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:37.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe93c1a5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe934102da0 0x7fe93410dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:37.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe93c1a5700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe93410cdf0 con 0x7fe934102400 2024-03-18T14:52:37.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe93c1a5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe93410cc70 con 0x7fe934102da0 2024-03-18T14:52:37.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe93c1a5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe93410cf70 con 0x7fe934101200 2024-03-18T14:52:37.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe939740700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe934102400 0x7fe93410d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:37.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe939740700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe934102400 0x7fe93410d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60292/0 (socket says 172.21.15.76:60292) 2024-03-18T14:52:37.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe939740700 1 -- 172.21.15.76:0/161201846 learned_addr learned my addr 172.21.15.76:0/161201846 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:37.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe93a742700 1 --2- 172.21.15.76:0/161201846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe934102da0 0x7fe93410dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:37.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe939f41700 1 --2- 172.21.15.76:0/161201846 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe934101200 0x7fe93410f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:37.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe93a742700 1 -- 172.21.15.76:0/161201846 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe934101200 msgr2=0x7fe93410f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:37.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe93a742700 1 --2- 172.21.15.76:0/161201846 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe934101200 0x7fe93410f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:37.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe93a742700 1 -- 172.21.15.76:0/161201846 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe934102400 msgr2=0x7fe93410d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:37.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe93a742700 1 --2- 172.21.15.76:0/161201846 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe934102400 0x7fe93410d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:37.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.110+0000 7fe93a742700 1 -- 172.21.15.76:0/161201846 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe93410e770 con 0x7fe934102da0 2024-03-18T14:52:37.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.111+0000 7fe939740700 1 --2- 172.21.15.76:0/161201846 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe934102400 0x7fe93410d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:37.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.111+0000 7fe93a742700 1 --2- 172.21.15.76:0/161201846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe934102da0 0x7fe93410dec0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe92c00efc0 tx=0x7fe92c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:37.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.111+0000 7fe92affd700 1 -- 172.21.15.76:0/161201846 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe92c0092e0 con 0x7fe934102da0 2024-03-18T14:52:37.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.111+0000 7fe93c1a5700 1 -- 172.21.15.76:0/161201846 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9341a2660 con 0x7fe934102da0 2024-03-18T14:52:37.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.111+0000 7fe92affd700 1 -- 172.21.15.76:0/161201846 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe92c004bb0 con 0x7fe934102da0 2024-03-18T14:52:37.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.111+0000 7fe92affd700 1 -- 172.21.15.76:0/161201846 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe92c004280 con 0x7fe934102da0 2024-03-18T14:52:37.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.111+0000 7fe93c1a5700 1 -- 172.21.15.76:0/161201846 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9341a2ba0 con 0x7fe934102da0 2024-03-18T14:52:37.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.112+0000 7fe93c1a5700 1 -- 172.21.15.76:0/161201846 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe934068f90 con 0x7fe934102da0 2024-03-18T14:52:37.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.113+0000 7fe92affd700 1 -- 172.21.15.76:0/161201846 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe92c01b030 con 0x7fe934102da0 2024-03-18T14:52:37.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.114+0000 7fe92affd700 1 --2- 172.21.15.76:0/161201846 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe9200a2920 0x7fe9200a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:37.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.114+0000 7fe939f41700 1 --2- 172.21.15.76:0/161201846 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe9200a2920 0x7fe9200a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:37.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.114+0000 7fe92affd700 1 -- 172.21.15.76:0/161201846 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fe92c0cf0e0 con 0x7fe934102da0 2024-03-18T14:52:37.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.114+0000 7fe939f41700 1 --2- 172.21.15.76:0/161201846 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe9200a2920 0x7fe9200a4de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe92400b580 tx=0x7fe924005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:37.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.122+0000 7fe92affd700 1 -- 172.21.15.76:0/161201846 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe92c022020 con 0x7fe934102da0 2024-03-18T14:52:37.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.329+0000 7fe93c1a5700 1 -- 172.21.15.76:0/161201846 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fe9340fd5c0 con 0x7fe9200a2920 2024-03-18T14:52:37.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.342+0000 7fe92affd700 1 -- 172.21.15.76:0/161201846 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7fe9340fd5c0 con 0x7fe9200a2920 2024-03-18T14:52:37.343 INFO:teuthology.orchestra.run.smithi076.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-18T14:52:37.343 INFO:teuthology.orchestra.run.smithi076.stdout:mds.a smithi076 running (103s) 15s ago 102s 16.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b bf67b06ccf43 2024-03-18T14:52:37.343 INFO:teuthology.orchestra.run.smithi076.stdout:mds.b smithi077 running (100s) 32s ago 99s 25.5M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 7709a3c15615 2024-03-18T14:52:37.343 INFO:teuthology.orchestra.run.smithi076.stdout:mds.c smithi174 running (114s) 32s ago 114s 19.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 717e2533657d 2024-03-18T14:52:37.343 INFO:teuthology.orchestra.run.smithi076.stdout:mds.d smithi076 running (2m) 15s ago 2m 16.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 3e7989df5e0c 2024-03-18T14:52:37.343 INFO:teuthology.orchestra.run.smithi076.stdout:mds.e smithi077 running (2m) 32s ago 119s 19.4M - 19.0.0-2162-g3d78f5ba be86dc8cb22b d4bb2c6604ed 2024-03-18T14:52:37.344 INFO:teuthology.orchestra.run.smithi076.stdout:mds.f smithi174 running (117s) 32s ago 116s 19.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 45a90165cc16 2024-03-18T14:52:37.344 INFO:teuthology.orchestra.run.smithi076.stdout:mds.g smithi076 running (106s) 15s ago 105s 13.9M - 19.0.0-2162-g3d78f5ba be86dc8cb22b d5e4e51ec334 2024-03-18T14:52:37.344 INFO:teuthology.orchestra.run.smithi076.stdout:mds.h smithi077 running (111s) 32s ago 111s 15.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b e9e30dd1c76b 2024-03-18T14:52:37.344 INFO:teuthology.orchestra.run.smithi076.stdout:mds.i smithi174 running (108s) 32s ago 108s 19.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 6ed05e4295d7 2024-03-18T14:52:37.344 INFO:teuthology.orchestra.run.smithi076.stdout:mds.j smithi076 running (2m) 15s ago 2m 14.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 4e1a88725374 2024-03-18T14:52:37.344 INFO:teuthology.orchestra.run.smithi076.stdout:mds.k smithi077 running (2m) 32s ago 2m 16.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 1f0f58e39456 2024-03-18T14:52:37.344 INFO:teuthology.orchestra.run.smithi076.stdout:mds.l smithi174 running (2m) 32s ago 2m 15.4M - 19.0.0-2162-g3d78f5ba be86dc8cb22b a4e76b74a47b 2024-03-18T14:52:37.344 INFO:teuthology.orchestra.run.smithi076.stdout:mgr.x smithi076 *:9283,8765 running (9m) 15s ago 9m 512M - 19.0.0-2162-g3d78f5ba be86dc8cb22b bb92b6931123 2024-03-18T14:52:37.344 INFO:teuthology.orchestra.run.smithi076.stdout:mgr.y smithi077 *:8443,8765 running (7m) 32s ago 7m 454M - 19.0.0-2162-g3d78f5ba be86dc8cb22b c90a05c8a55e 2024-03-18T14:52:37.344 INFO:teuthology.orchestra.run.smithi076.stdout:mgr.z smithi174 *:8443,8765 running (7m) 32s ago 7m 452M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 2a8836b2bbaa 2024-03-18T14:52:37.344 INFO:teuthology.orchestra.run.smithi076.stdout:mon.a smithi076 running (9m) 15s ago 9m 52.4M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 3df3aa06ebbe 2024-03-18T14:52:37.344 INFO:teuthology.orchestra.run.smithi076.stdout:mon.b smithi077 running (7m) 32s ago 7m 41.7M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b a1709407081a 2024-03-18T14:52:37.344 INFO:teuthology.orchestra.run.smithi076.stdout:mon.c smithi174 running (7m) 32s ago 7m 40.9M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 5e438e85c48f 2024-03-18T14:52:37.344 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0 smithi076 running (7m) 15s ago 7m 55.4M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 32d7fab3190b 2024-03-18T14:52:37.345 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1 smithi077 running (6m) 32s ago 6m 48.1M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 24b2458cfd9b 2024-03-18T14:52:37.345 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2 smithi174 running (6m) 32s ago 6m 46.2M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 0590c58a8ebb 2024-03-18T14:52:37.345 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3 smithi076 running (5m) 15s ago 5m 55.1M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 17aa4f914ff7 2024-03-18T14:52:37.345 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4 smithi077 running (5m) 32s ago 5m 55.2M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b a1bd98f801a8 2024-03-18T14:52:37.345 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5 smithi174 running (5m) 32s ago 5m 58.0M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 8693dd595fd8 2024-03-18T14:52:37.345 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6 smithi076 running (4m) 15s ago 4m 51.8M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b bec12d32e07b 2024-03-18T14:52:37.345 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7 smithi077 running (4m) 32s ago 4m 43.2M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 9c6250ba0541 2024-03-18T14:52:37.345 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8 smithi174 running (3m) 32s ago 3m 51.8M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b caf831b8a8bf 2024-03-18T14:52:37.345 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9 smithi076 running (3m) 15s ago 3m 43.1M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 18079dfe980f 2024-03-18T14:52:37.345 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10 smithi077 running (2m) 32s ago 2m 53.5M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 352db622b1b7 2024-03-18T14:52:37.345 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11 smithi174 running (2m) 32s ago 2m 51.1M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 5f547efce716 2024-03-18T14:52:37.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.345+0000 7fe93c1a5700 1 -- 172.21.15.76:0/161201846 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe9200a2920 msgr2=0x7fe9200a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:37.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.345+0000 7fe93c1a5700 1 --2- 172.21.15.76:0/161201846 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe9200a2920 0x7fe9200a4de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe92400b580 tx=0x7fe924005fb0 comp rx=0 tx=0).stop 2024-03-18T14:52:37.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.345+0000 7fe93c1a5700 1 -- 172.21.15.76:0/161201846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe934102da0 msgr2=0x7fe93410dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:37.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.345+0000 7fe93c1a5700 1 --2- 172.21.15.76:0/161201846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe934102da0 0x7fe93410dec0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe92c00efc0 tx=0x7fe92c00c3a0 comp rx=0 tx=0).stop 2024-03-18T14:52:37.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.345+0000 7fe93c1a5700 1 -- 172.21.15.76:0/161201846 shutdown_connections 2024-03-18T14:52:37.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.345+0000 7fe93c1a5700 1 --2- 172.21.15.76:0/161201846 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe9200a2920 0x7fe9200a4de0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:37.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.345+0000 7fe93c1a5700 1 --2- 172.21.15.76:0/161201846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe934102da0 0x7fe93410dec0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:37.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.345+0000 7fe93c1a5700 1 --2- 172.21.15.76:0/161201846 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe934101200 0x7fe93410f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:37.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.345+0000 7fe93c1a5700 1 --2- 172.21.15.76:0/161201846 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe934102400 0x7fe93410d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:37.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.345+0000 7fe93c1a5700 1 -- 172.21.15.76:0/161201846 >> 172.21.15.76:0/161201846 conn(0x7fe9340fc7c0 msgr2=0x7fe9340fceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:37.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.345+0000 7fe93c1a5700 1 -- 172.21.15.76:0/161201846 shutdown_connections 2024-03-18T14:52:37.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:37.345+0000 7fe93c1a5700 1 -- 172.21.15.76:0/161201846 wait complete. 2024-03-18T14:52:37.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:37 smithi076 ceph-mon[32205]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:52:37.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:37 smithi174 ceph-mon[36787]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:52:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:37 smithi077 ceph-mon[35865]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:52:38.147 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-18T14:52:38.394 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:52:38.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:38 smithi174 ceph-mon[36787]: from='client.24971 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:52:38.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:38 smithi174 ceph-mon[36787]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-18T14:52:38.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:38 smithi077 ceph-mon[35865]: from='client.24971 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:52:38.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:38 smithi077 ceph-mon[35865]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-18T14:52:38.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:38 smithi076 ceph-mon[32205]: from='client.24971 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:52:38.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:38 smithi076 ceph-mon[32205]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-18T14:52:39.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:39 smithi076 ceph-mon[32205]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T14:52:39.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:39 smithi174 ceph-mon[36787]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T14:52:39.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:39 smithi077 ceph-mon[35865]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T14:52:40.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.357+0000 7f020e329700 1 -- 172.21.15.76:0/512635025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0208102d90 msgr2=0x7f0208105180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:40.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.357+0000 7f020e329700 1 --2- 172.21.15.76:0/512635025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0208102d90 0x7f0208105180 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f01f8009990 tx=0x7f01f8009ca0 comp rx=0 tx=0).stop 2024-03-18T14:52:40.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.358+0000 7f020e329700 1 -- 172.21.15.76:0/512635025 shutdown_connections 2024-03-18T14:52:40.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.358+0000 7f020e329700 1 --2- 172.21.15.76:0/512635025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0208107ff0 0x7f020810a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:40.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.358+0000 7f020e329700 1 --2- 172.21.15.76:0/512635025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0208102d90 0x7f0208105180 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:40.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.358+0000 7f020e329700 1 --2- 172.21.15.76:0/512635025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02081056c0 0x7f0208107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:40.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.358+0000 7f020e329700 1 -- 172.21.15.76:0/512635025 >> 172.21.15.76:0/512635025 conn(0x7f02080fc7c0 msgr2=0x7f02080febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:40.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.358+0000 7f020e329700 1 -- 172.21.15.76:0/512635025 shutdown_connections 2024-03-18T14:52:40.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.358+0000 7f020e329700 1 -- 172.21.15.76:0/512635025 wait complete. 2024-03-18T14:52:40.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.358+0000 7f020e329700 1 Processor -- start 2024-03-18T14:52:40.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.358+0000 7f020e329700 1 -- start start 2024-03-18T14:52:40.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.358+0000 7f020e329700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0208102d90 0x7f0208198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:40.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.358+0000 7f020e329700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02081056c0 0x7f0208198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:40.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.358+0000 7f020e329700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0208107ff0 0x7f020819fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:40.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.358+0000 7f020e329700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f020810df10 con 0x7f0208102d90 2024-03-18T14:52:40.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.358+0000 7f020e329700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f020810dd90 con 0x7f02081056c0 2024-03-18T14:52:40.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.358+0000 7f020e329700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f020810e090 con 0x7f0208107ff0 2024-03-18T14:52:40.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f0207fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0208102d90 0x7f0208198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:40.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f0207fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0208102d90 0x7f0208198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40778/0 (socket says 172.21.15.76:40778) 2024-03-18T14:52:40.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f0207fff700 1 -- 172.21.15.76:0/1551284788 learned_addr learned my addr 172.21.15.76:0/1551284788 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:40.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f020c8c6700 1 --2- 172.21.15.76:0/1551284788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0208107ff0 0x7f020819fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:40.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f02077fe700 1 --2- 172.21.15.76:0/1551284788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02081056c0 0x7f0208198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:40.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f0207fff700 1 -- 172.21.15.76:0/1551284788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0208107ff0 msgr2=0x7f020819fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:40.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f0207fff700 1 --2- 172.21.15.76:0/1551284788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0208107ff0 0x7f020819fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:40.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f0207fff700 1 -- 172.21.15.76:0/1551284788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02081056c0 msgr2=0x7f0208198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:40.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f0207fff700 1 --2- 172.21.15.76:0/1551284788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02081056c0 0x7f0208198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:40.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f0207fff700 1 -- 172.21.15.76:0/1551284788 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02081a0160 con 0x7f0208102d90 2024-03-18T14:52:40.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f020c8c6700 1 --2- 172.21.15.76:0/1551284788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0208107ff0 0x7f020819fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:40.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f02077fe700 1 --2- 172.21.15.76:0/1551284788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02081056c0 0x7f0208198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:40.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f0207fff700 1 --2- 172.21.15.76:0/1551284788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0208102d90 0x7f0208198500 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f01f800bbb0 tx=0x7f01f800bbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:40.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f02057fa700 1 -- 172.21.15.76:0/1551284788 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f8026070 con 0x7f0208102d90 2024-03-18T14:52:40.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f020e329700 1 -- 172.21.15.76:0/1551284788 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02081a03f0 con 0x7f0208102d90 2024-03-18T14:52:40.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f02057fa700 1 -- 172.21.15.76:0/1551284788 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01f800f460 con 0x7f0208102d90 2024-03-18T14:52:40.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f02057fa700 1 -- 172.21.15.76:0/1551284788 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f8021650 con 0x7f0208102d90 2024-03-18T14:52:40.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.359+0000 7f020e329700 1 -- 172.21.15.76:0/1551284788 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02081a0840 con 0x7f0208102d90 2024-03-18T14:52:40.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.360+0000 7f020e329700 1 -- 172.21.15.76:0/1551284788 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0208068f90 con 0x7f0208102d90 2024-03-18T14:52:40.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.361+0000 7f02057fa700 1 -- 172.21.15.76:0/1551284788 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f01f8029070 con 0x7f0208102d90 2024-03-18T14:52:40.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.362+0000 7f02057fa700 1 --2- 172.21.15.76:0/1551284788 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f01f00a2890 0x7f01f00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:40.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.362+0000 7f02077fe700 1 --2- 172.21.15.76:0/1551284788 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f01f00a2890 0x7f01f00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:40.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.362+0000 7f02057fa700 1 -- 172.21.15.76:0/1551284788 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f01f80cf810 con 0x7f0208102d90 2024-03-18T14:52:40.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.362+0000 7f02077fe700 1 --2- 172.21.15.76:0/1551284788 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f01f00a2890 0x7f01f00a4d50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f01fc011490 tx=0x7f01fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:40.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.367+0000 7f02057fa700 1 -- 172.21.15.76:0/1551284788 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f01f805d740 con 0x7f0208102d90 2024-03-18T14:52:40.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.574+0000 7f020e329700 1 -- 172.21.15.76:0/1551284788 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f0208062710 con 0x7f01f00a2890 2024-03-18T14:52:40.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.580+0000 7f02057fa700 1 -- 172.21.15.76:0/1551284788 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f0208062710 con 0x7f01f00a2890 2024-03-18T14:52:40.580 INFO:teuthology.orchestra.run.smithi076.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-18T14:52:40.581 INFO:teuthology.orchestra.run.smithi076.stdout:mds.all 12/12 36s ago 2m smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l;count:12 2024-03-18T14:52:40.581 INFO:teuthology.orchestra.run.smithi076.stdout:mgr 3/3 36s ago 7m smithi076=x;smithi077=y;smithi174=z;count:3 2024-03-18T14:52:40.581 INFO:teuthology.orchestra.run.smithi076.stdout:mon 3/3 36s ago 7m smithi076:172.21.15.76=a;smithi077:172.21.15.77=b;smithi174:172.21.15.174=c;count:3 2024-03-18T14:52:40.581 INFO:teuthology.orchestra.run.smithi076.stdout:osd 12 36s ago - 2024-03-18T14:52:40.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.582+0000 7f020e329700 1 -- 172.21.15.76:0/1551284788 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f01f00a2890 msgr2=0x7f01f00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:40.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.582+0000 7f020e329700 1 --2- 172.21.15.76:0/1551284788 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f01f00a2890 0x7f01f00a4d50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f01fc011490 tx=0x7f01fc009260 comp rx=0 tx=0).stop 2024-03-18T14:52:40.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.582+0000 7f020e329700 1 -- 172.21.15.76:0/1551284788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0208102d90 msgr2=0x7f0208198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:40.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.582+0000 7f020e329700 1 --2- 172.21.15.76:0/1551284788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0208102d90 0x7f0208198500 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f01f800bbb0 tx=0x7f01f800bbe0 comp rx=0 tx=0).stop 2024-03-18T14:52:40.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.582+0000 7f020e329700 1 -- 172.21.15.76:0/1551284788 shutdown_connections 2024-03-18T14:52:40.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.582+0000 7f020e329700 1 --2- 172.21.15.76:0/1551284788 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f01f00a2890 0x7f01f00a4d50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:40.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.582+0000 7f020e329700 1 --2- 172.21.15.76:0/1551284788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0208107ff0 0x7f020819fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:40.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.582+0000 7f020e329700 1 --2- 172.21.15.76:0/1551284788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0208102d90 0x7f0208198500 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:40.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.582+0000 7f020e329700 1 --2- 172.21.15.76:0/1551284788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02081056c0 0x7f0208198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:40.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.582+0000 7f020e329700 1 -- 172.21.15.76:0/1551284788 >> 172.21.15.76:0/1551284788 conn(0x7f02080fc7c0 msgr2=0x7f0208103810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:40.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.582+0000 7f020e329700 1 -- 172.21.15.76:0/1551284788 shutdown_connections 2024-03-18T14:52:40.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:40.582+0000 7f020e329700 1 -- 172.21.15.76:0/1551284788 wait complete. 2024-03-18T14:52:41.390 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-18T14:52:41.631 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:52:41.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:41 smithi076 ceph-mon[32205]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T14:52:41.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:41 smithi174 ceph-mon[36787]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T14:52:41.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:41 smithi077 ceph-mon[35865]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T14:52:42.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:42 smithi174 ceph-mon[36787]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:52:42.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:42 smithi077 ceph-mon[35865]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:52:42.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:42 smithi076 ceph-mon[32205]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:52:43.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.372+0000 7f215e1b0700 1 -- 172.21.15.76:0/2581126883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2158102340 msgr2=0x7f2158102760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:43.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.372+0000 7f215e1b0700 1 --2- 172.21.15.76:0/2581126883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2158102340 0x7f2158102760 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f21480099c0 tx=0x7f2148009cd0 comp rx=0 tx=0).stop 2024-03-18T14:52:43.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.372+0000 7f215e1b0700 1 -- 172.21.15.76:0/2581126883 shutdown_connections 2024-03-18T14:52:43.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.372+0000 7f215e1b0700 1 --2- 172.21.15.76:0/2581126883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2158103ee0 0x7f215810a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:43.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.372+0000 7f215e1b0700 1 --2- 172.21.15.76:0/2581126883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2158102340 0x7f2158102760 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:43.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.372+0000 7f215e1b0700 1 --2- 172.21.15.76:0/2581126883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2158103540 0x7f21581039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:43.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.372+0000 7f215e1b0700 1 -- 172.21.15.76:0/2581126883 >> 172.21.15.76:0/2581126883 conn(0x7f21580fd920 msgr2=0x7f21580ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:43.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.372+0000 7f215e1b0700 1 -- 172.21.15.76:0/2581126883 shutdown_connections 2024-03-18T14:52:43.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.372+0000 7f215e1b0700 1 -- 172.21.15.76:0/2581126883 wait complete. 2024-03-18T14:52:43.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.373+0000 7f215e1b0700 1 Processor -- start 2024-03-18T14:52:43.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.373+0000 7f215e1b0700 1 -- start start 2024-03-18T14:52:43.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.373+0000 7f215e1b0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2158102340 0x7f2158198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:43.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.373+0000 7f215e1b0700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2158103540 0x7f2158198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:43.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.373+0000 7f215e1b0700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2158103ee0 0x7f215819fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:43.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.373+0000 7f215e1b0700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f215810df30 con 0x7f2158103540 2024-03-18T14:52:43.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.373+0000 7f215e1b0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f215810ddb0 con 0x7f2158102340 2024-03-18T14:52:43.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.373+0000 7f215e1b0700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f215810e0b0 con 0x7f2158103ee0 2024-03-18T14:52:43.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.373+0000 7f2156ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2158103540 0x7f2158198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:43.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.373+0000 7f2156ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2158103540 0x7f2158198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40810/0 (socket says 172.21.15.76:40810) 2024-03-18T14:52:43.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.373+0000 7f2156ffd700 1 -- 172.21.15.76:0/4269654838 learned_addr learned my addr 172.21.15.76:0/4269654838 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:43.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.373+0000 7f21577fe700 1 --2- 172.21.15.76:0/4269654838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2158102340 0x7f2158198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:43.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.373+0000 7f2157fff700 1 --2- 172.21.15.76:0/4269654838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2158103ee0 0x7f215819fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:43.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.374+0000 7f2156ffd700 1 -- 172.21.15.76:0/4269654838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2158103ee0 msgr2=0x7f215819fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:43.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.374+0000 7f2156ffd700 1 --2- 172.21.15.76:0/4269654838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2158103ee0 0x7f215819fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:43.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.374+0000 7f2156ffd700 1 -- 172.21.15.76:0/4269654838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2158102340 msgr2=0x7f2158198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:43.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.374+0000 7f2156ffd700 1 --2- 172.21.15.76:0/4269654838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2158102340 0x7f2158198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:43.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.374+0000 7f2156ffd700 1 -- 172.21.15.76:0/4269654838 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21581a0170 con 0x7f2158103540 2024-03-18T14:52:43.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.374+0000 7f21577fe700 1 --2- 172.21.15.76:0/4269654838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2158102340 0x7f2158198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:43.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.374+0000 7f2156ffd700 1 --2- 172.21.15.76:0/4269654838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2158103540 0x7f2158198a50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f214c00e9f0 tx=0x7f214c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:43.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.374+0000 7f2154ff9700 1 -- 172.21.15.76:0/4269654838 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214c00cbd0 con 0x7f2158103540 2024-03-18T14:52:43.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.374+0000 7f2154ff9700 1 -- 172.21.15.76:0/4269654838 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f214c00ee90 con 0x7f2158103540 2024-03-18T14:52:43.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.374+0000 7f215e1b0700 1 -- 172.21.15.76:0/4269654838 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21581a0460 con 0x7f2158103540 2024-03-18T14:52:43.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.374+0000 7f2154ff9700 1 -- 172.21.15.76:0/4269654838 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214c0056c0 con 0x7f2158103540 2024-03-18T14:52:43.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.374+0000 7f215e1b0700 1 -- 172.21.15.76:0/4269654838 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21581a0950 con 0x7f2158103540 2024-03-18T14:52:43.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.375+0000 7f215e1b0700 1 -- 172.21.15.76:0/4269654838 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2158068f90 con 0x7f2158103540 2024-03-18T14:52:43.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.376+0000 7f2154ff9700 1 -- 172.21.15.76:0/4269654838 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f214c004750 con 0x7f2158103540 2024-03-18T14:52:43.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.377+0000 7f2154ff9700 1 --2- 172.21.15.76:0/4269654838 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f21400a2890 0x7f21400a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:43.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.377+0000 7f21577fe700 1 --2- 172.21.15.76:0/4269654838 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f21400a2890 0x7f21400a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:43.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.377+0000 7f2154ff9700 1 -- 172.21.15.76:0/4269654838 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f214c0ceca0 con 0x7f2158103540 2024-03-18T14:52:43.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.377+0000 7f21577fe700 1 --2- 172.21.15.76:0/4269654838 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f21400a2890 0x7f21400a4d50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f214800b580 tx=0x7f2148005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:43.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.382+0000 7f2154ff9700 1 -- 172.21.15.76:0/4269654838 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f214c09bd70 con 0x7f2158103540 2024-03-18T14:52:43.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.587+0000 7f215e1b0700 1 -- 172.21.15.76:0/4269654838 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f2158062710 con 0x7f21400a2890 2024-03-18T14:52:43.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.589+0000 7f2154ff9700 1 -- 172.21.15.76:0/4269654838 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f2158062710 con 0x7f21400a2890 2024-03-18T14:52:43.589 INFO:teuthology.orchestra.run.smithi076.stdout:HOST ADDR LABELS STATUS 2024-03-18T14:52:43.590 INFO:teuthology.orchestra.run.smithi076.stdout:smithi076 172.21.15.76 2024-03-18T14:52:43.590 INFO:teuthology.orchestra.run.smithi076.stdout:smithi077 172.21.15.77 2024-03-18T14:52:43.590 INFO:teuthology.orchestra.run.smithi076.stdout:smithi174 172.21.15.174 2024-03-18T14:52:43.590 INFO:teuthology.orchestra.run.smithi076.stdout:3 hosts in cluster 2024-03-18T14:52:43.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.591+0000 7f215e1b0700 1 -- 172.21.15.76:0/4269654838 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f21400a2890 msgr2=0x7f21400a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:43.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.591+0000 7f215e1b0700 1 --2- 172.21.15.76:0/4269654838 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f21400a2890 0x7f21400a4d50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f214800b580 tx=0x7f2148005fb0 comp rx=0 tx=0).stop 2024-03-18T14:52:43.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.591+0000 7f215e1b0700 1 -- 172.21.15.76:0/4269654838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2158103540 msgr2=0x7f2158198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:43.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.591+0000 7f215e1b0700 1 --2- 172.21.15.76:0/4269654838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2158103540 0x7f2158198a50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f214c00e9f0 tx=0x7f214c00ed00 comp rx=0 tx=0).stop 2024-03-18T14:52:43.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.591+0000 7f215e1b0700 1 -- 172.21.15.76:0/4269654838 shutdown_connections 2024-03-18T14:52:43.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.591+0000 7f215e1b0700 1 --2- 172.21.15.76:0/4269654838 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f21400a2890 0x7f21400a4d50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:43.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.591+0000 7f215e1b0700 1 --2- 172.21.15.76:0/4269654838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2158103ee0 0x7f215819fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:43.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.591+0000 7f215e1b0700 1 --2- 172.21.15.76:0/4269654838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2158102340 0x7f2158198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:43.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.591+0000 7f215e1b0700 1 --2- 172.21.15.76:0/4269654838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2158103540 0x7f2158198a50 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:43.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.591+0000 7f215e1b0700 1 -- 172.21.15.76:0/4269654838 >> 172.21.15.76:0/4269654838 conn(0x7f21580fd920 msgr2=0x7f2158106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:43.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.591+0000 7f215e1b0700 1 -- 172.21.15.76:0/4269654838 shutdown_connections 2024-03-18T14:52:43.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:43.591+0000 7f215e1b0700 1 -- 172.21.15.76:0/4269654838 wait complete. 2024-03-18T14:52:43.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:43 smithi076 ceph-mon[32205]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-18T14:52:43.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:43 smithi174 ceph-mon[36787]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-18T14:52:43.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:43 smithi077 ceph-mon[35865]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-18T14:52:44.417 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-18T14:52:44.660 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:52:45.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:45 smithi174 ceph-mon[36787]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:52:45.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:45 smithi174 ceph-mon[36787]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-18T14:52:45.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:45 smithi077 ceph-mon[35865]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:52:45.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:45 smithi077 ceph-mon[35865]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-18T14:52:45.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:45 smithi076 ceph-mon[32205]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:52:45.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:45 smithi076 ceph-mon[32205]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-18T14:52:46.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.345+0000 7f4b55516700 1 -- 172.21.15.76:0/311109298 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b50101200 msgr2=0x7f4b50101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:46.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.345+0000 7f4b55516700 1 --2- 172.21.15.76:0/311109298 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b50101200 0x7f4b50101620 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4b440099c0 tx=0x7f4b44009cd0 comp rx=0 tx=0).stop 2024-03-18T14:52:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.345+0000 7f4b55516700 1 -- 172.21.15.76:0/311109298 shutdown_connections 2024-03-18T14:52:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.345+0000 7f4b55516700 1 --2- 172.21.15.76:0/311109298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b50102da0 0x7f4b50109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.345+0000 7f4b55516700 1 --2- 172.21.15.76:0/311109298 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b50101200 0x7f4b50101620 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.345+0000 7f4b55516700 1 --2- 172.21.15.76:0/311109298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b50102400 0x7f4b50102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.345+0000 7f4b55516700 1 -- 172.21.15.76:0/311109298 >> 172.21.15.76:0/311109298 conn(0x7f4b500fc7a0 msgr2=0x7f4b500febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.345+0000 7f4b55516700 1 -- 172.21.15.76:0/311109298 shutdown_connections 2024-03-18T14:52:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.345+0000 7f4b55516700 1 -- 172.21.15.76:0/311109298 wait complete. 2024-03-18T14:52:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.346+0000 7f4b55516700 1 Processor -- start 2024-03-18T14:52:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.346+0000 7f4b55516700 1 -- start start 2024-03-18T14:52:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.346+0000 7f4b55516700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b50101200 0x7f4b5006aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.346+0000 7f4b55516700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b50102400 0x7f4b5019d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.346+0000 7f4b55516700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b50102da0 0x7f4b5019f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.346+0000 7f4b55516700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5010ce00 con 0x7f4b50102da0 2024-03-18T14:52:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.346+0000 7f4b55516700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5010cc80 con 0x7f4b50101200 2024-03-18T14:52:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.346+0000 7f4b55516700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5010cf80 con 0x7f4b50102400 2024-03-18T14:52:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.346+0000 7f4b4f7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b50102da0 0x7f4b5019f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.346+0000 7f4b4effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b50101200 0x7f4b5006aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.346+0000 7f4b4f7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b50102da0 0x7f4b5019f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40834/0 (socket says 172.21.15.76:40834) 2024-03-18T14:52:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.346+0000 7f4b4f7fe700 1 -- 172.21.15.76:0/3955557426 learned_addr learned my addr 172.21.15.76:0/3955557426 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.346+0000 7f4b4e7fc700 1 --2- 172.21.15.76:0/3955557426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b50102400 0x7f4b5019d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.347+0000 7f4b4effd700 1 -- 172.21.15.76:0/3955557426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b50102400 msgr2=0x7f4b5019d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.347+0000 7f4b4effd700 1 --2- 172.21.15.76:0/3955557426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b50102400 0x7f4b5019d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.347+0000 7f4b4effd700 1 -- 172.21.15.76:0/3955557426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b50102da0 msgr2=0x7f4b5019f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.347+0000 7f4b4effd700 1 --2- 172.21.15.76:0/3955557426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b50102da0 0x7f4b5019f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.347+0000 7f4b4effd700 1 -- 172.21.15.76:0/3955557426 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b501a0200 con 0x7f4b50101200 2024-03-18T14:52:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.347+0000 7f4b4effd700 1 --2- 172.21.15.76:0/3955557426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b50101200 0x7f4b5006aae0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4b4400b580 tx=0x7f4b4400f710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.347+0000 7f4b37fff700 1 -- 172.21.15.76:0/3955557426 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44009e60 con 0x7f4b50101200 2024-03-18T14:52:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.347+0000 7f4b37fff700 1 -- 172.21.15.76:0/3955557426 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b44003e40 con 0x7f4b50101200 2024-03-18T14:52:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.347+0000 7f4b37fff700 1 -- 172.21.15.76:0/3955557426 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44005490 con 0x7f4b50101200 2024-03-18T14:52:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.347+0000 7f4b55516700 1 -- 172.21.15.76:0/3955557426 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b501970c0 con 0x7f4b50101200 2024-03-18T14:52:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.347+0000 7f4b55516700 1 -- 172.21.15.76:0/3955557426 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b501975a0 con 0x7f4b50101200 2024-03-18T14:52:46.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.348+0000 7f4b55516700 1 -- 172.21.15.76:0/3955557426 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b50068f90 con 0x7f4b50101200 2024-03-18T14:52:46.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.349+0000 7f4b37fff700 1 -- 172.21.15.76:0/3955557426 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4b4400fab0 con 0x7f4b50101200 2024-03-18T14:52:46.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.349+0000 7f4b37fff700 1 --2- 172.21.15.76:0/3955557426 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f4b380a2890 0x7f4b380a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:46.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.350+0000 7f4b4e7fc700 1 --2- 172.21.15.76:0/3955557426 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f4b380a2890 0x7f4b380a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:46.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.350+0000 7f4b37fff700 1 -- 172.21.15.76:0/3955557426 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f4b440cf630 con 0x7f4b50101200 2024-03-18T14:52:46.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.350+0000 7f4b4e7fc700 1 --2- 172.21.15.76:0/3955557426 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f4b380a2890 0x7f4b380a4d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4b5006b060 tx=0x7f4b40009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.353+0000 7f4b37fff700 1 -- 172.21.15.76:0/3955557426 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4b44013070 con 0x7f4b50101200 2024-03-18T14:52:46.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.559+0000 7f4b55516700 1 -- 172.21.15.76:0/3955557426 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f4b50062710 con 0x7f4b380a2890 2024-03-18T14:52:46.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.561+0000 7f4b37fff700 1 -- 172.21.15.76:0/3955557426 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f4b50062710 con 0x7f4b380a2890 2024-03-18T14:52:46.562 INFO:teuthology.orchestra.run.smithi076.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-18T14:52:46.562 INFO:teuthology.orchestra.run.smithi076.stdout:smithi076 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400U4400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-18T14:52:46.562 INFO:teuthology.orchestra.run.smithi076.stdout:smithi077 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400ZJ400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-18T14:52:46.562 INFO:teuthology.orchestra.run.smithi076.stdout:smithi174 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62050074400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-18T14:52:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.563+0000 7f4b55516700 1 -- 172.21.15.76:0/3955557426 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f4b380a2890 msgr2=0x7f4b380a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.563+0000 7f4b55516700 1 --2- 172.21.15.76:0/3955557426 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f4b380a2890 0x7f4b380a4d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4b5006b060 tx=0x7f4b40009260 comp rx=0 tx=0).stop 2024-03-18T14:52:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.563+0000 7f4b55516700 1 -- 172.21.15.76:0/3955557426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b50101200 msgr2=0x7f4b5006aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.563+0000 7f4b55516700 1 --2- 172.21.15.76:0/3955557426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b50101200 0x7f4b5006aae0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4b4400b580 tx=0x7f4b4400f710 comp rx=0 tx=0).stop 2024-03-18T14:52:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.563+0000 7f4b55516700 1 -- 172.21.15.76:0/3955557426 shutdown_connections 2024-03-18T14:52:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.563+0000 7f4b55516700 1 --2- 172.21.15.76:0/3955557426 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f4b380a2890 0x7f4b380a4d50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.563+0000 7f4b55516700 1 --2- 172.21.15.76:0/3955557426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b50102da0 0x7f4b5019f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.563+0000 7f4b55516700 1 --2- 172.21.15.76:0/3955557426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b50101200 0x7f4b5006aae0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.563+0000 7f4b55516700 1 --2- 172.21.15.76:0/3955557426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b50102400 0x7f4b5019d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.563+0000 7f4b55516700 1 -- 172.21.15.76:0/3955557426 >> 172.21.15.76:0/3955557426 conn(0x7f4b500fc7a0 msgr2=0x7f4b501057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.563+0000 7f4b55516700 1 -- 172.21.15.76:0/3955557426 shutdown_connections 2024-03-18T14:52:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:46.563+0000 7f4b55516700 1 -- 172.21.15.76:0/3955557426 wait complete. 2024-03-18T14:52:47.309 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T14:52:47.317 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi076.front.sepia.ceph.com 2024-03-18T14:52:47.317 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-18T14:52:47.568 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:52:47.598 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:47 smithi076 ceph-mon[32205]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T14:52:47.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:47 smithi174 ceph-mon[36787]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T14:52:47.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:47 smithi077 ceph-mon[35865]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T14:52:48.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:48 smithi174 ceph-mon[36787]: from='client.24989 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:52:48.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:48 smithi077 ceph-mon[35865]: from='client.24989 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:52:48.878 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:48 smithi076 ceph-mon[32205]: from='client.24989 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:52:49.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.450+0000 7f40a45b4700 1 -- 172.21.15.76:0/1068511886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f409c102d90 msgr2=0x7f409c109640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:49.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.450+0000 7f40a45b4700 1 --2- 172.21.15.76:0/1068511886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f409c102d90 0x7f409c109640 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f409400b0d0 tx=0x7f409400b3e0 comp rx=0 tx=0).stop 2024-03-18T14:52:49.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.450+0000 7f40a45b4700 1 -- 172.21.15.76:0/1068511886 shutdown_connections 2024-03-18T14:52:49.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.450+0000 7f40a45b4700 1 --2- 172.21.15.76:0/1068511886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f409c102d90 0x7f409c109640 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:49.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.450+0000 7f40a45b4700 1 --2- 172.21.15.76:0/1068511886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f409c1011f0 0x7f409c101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:49.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.450+0000 7f40a45b4700 1 --2- 172.21.15.76:0/1068511886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f409c1023f0 0x7f409c102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:49.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.450+0000 7f40a45b4700 1 -- 172.21.15.76:0/1068511886 >> 172.21.15.76:0/1068511886 conn(0x7f409c0fc790 msgr2=0x7f409c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:49.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.450+0000 7f40a45b4700 1 -- 172.21.15.76:0/1068511886 shutdown_connections 2024-03-18T14:52:49.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.450+0000 7f40a45b4700 1 -- 172.21.15.76:0/1068511886 wait complete. 2024-03-18T14:52:49.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a45b4700 1 Processor -- start 2024-03-18T14:52:49.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a45b4700 1 -- start start 2024-03-18T14:52:49.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a45b4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f409c1011f0 0x7f409c197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:49.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a45b4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f409c1023f0 0x7f409c1978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:49.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a45b4700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f409c102d90 0x7f409c19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:49.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a45b4700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f409c10cdf0 con 0x7f409c102d90 2024-03-18T14:52:49.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a45b4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f409c10cc70 con 0x7f409c1023f0 2024-03-18T14:52:49.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a45b4700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f409c10cf70 con 0x7f409c1011f0 2024-03-18T14:52:49.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a2b51700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f409c102d90 0x7f409c19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:49.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a2b51700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f409c102d90 0x7f409c19e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50856/0 (socket says 172.21.15.76:50856) 2024-03-18T14:52:49.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a2b51700 1 -- 172.21.15.76:0/669538386 learned_addr learned my addr 172.21.15.76:0/669538386 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a2350700 1 --2- 172.21.15.76:0/669538386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f409c1011f0 0x7f409c197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a1b4f700 1 --2- 172.21.15.76:0/669538386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f409c1023f0 0x7f409c1978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a2b51700 1 -- 172.21.15.76:0/669538386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f409c1011f0 msgr2=0x7f409c197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a2b51700 1 --2- 172.21.15.76:0/669538386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f409c1011f0 0x7f409c197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a2b51700 1 -- 172.21.15.76:0/669538386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f409c1023f0 msgr2=0x7f409c1978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a2b51700 1 --2- 172.21.15.76:0/669538386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f409c1023f0 0x7f409c1978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a2b51700 1 -- 172.21.15.76:0/669538386 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409c19efe0 con 0x7f409c102d90 2024-03-18T14:52:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.451+0000 7f40a2b51700 1 --2- 172.21.15.76:0/669538386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f409c102d90 0x7f409c19e940 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f409400b770 tx=0x7f4094007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.452+0000 7f40937fe700 1 -- 172.21.15.76:0/669538386 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094003c00 con 0x7f409c102d90 2024-03-18T14:52:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.452+0000 7f40937fe700 1 -- 172.21.15.76:0/669538386 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f409401e070 con 0x7f409c102d90 2024-03-18T14:52:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.452+0000 7f40a45b4700 1 -- 172.21.15.76:0/669538386 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f409c06a810 con 0x7f409c102d90 2024-03-18T14:52:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.452+0000 7f40937fe700 1 -- 172.21.15.76:0/669538386 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40940053b0 con 0x7f409c102d90 2024-03-18T14:52:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.452+0000 7f40a45b4700 1 -- 172.21.15.76:0/669538386 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f409c06ad50 con 0x7f409c102d90 2024-03-18T14:52:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.452+0000 7f40a45b4700 1 -- 172.21.15.76:0/669538386 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f409c10d5b0 con 0x7f409c102d90 2024-03-18T14:52:49.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.454+0000 7f40937fe700 1 -- 172.21.15.76:0/669538386 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f409401a4c0 con 0x7f409c102d90 2024-03-18T14:52:49.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.454+0000 7f40937fe700 1 --2- 172.21.15.76:0/669538386 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f40880a2890 0x7f40880a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:49.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.454+0000 7f40a2350700 1 --2- 172.21.15.76:0/669538386 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f40880a2890 0x7f40880a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:49.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.455+0000 7f40937fe700 1 -- 172.21.15.76:0/669538386 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f409400f020 con 0x7f409c102d90 2024-03-18T14:52:49.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.455+0000 7f40a2350700 1 --2- 172.21.15.76:0/669538386 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f40880a2890 0x7f40880a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f409c102250 tx=0x7f4098009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:49.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.458+0000 7f40937fe700 1 -- 172.21.15.76:0/669538386 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f409402a020 con 0x7f409c102d90 2024-03-18T14:52:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:49 smithi076 ceph-mon[32205]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T14:52:49.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.700+0000 7f40a45b4700 1 -- 172.21.15.76:0/669538386 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f409c068f90 con 0x7f409c102d90 2024-03-18T14:52:49.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.701+0000 7f40937fe700 1 -- 172.21.15.76:0/669538386 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) ==== 76+0+3044 (secure 0 0 0) 0x7f409409ed10 con 0x7f409c102d90 2024-03-18T14:52:49.702 INFO:teuthology.orchestra.run.smithi076.stdout:e23 2024-03-18T14:52:49.702 INFO:teuthology.orchestra.run.smithi076.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T14:52:49.702 INFO:teuthology.orchestra.run.smithi076.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T14:52:49.702 INFO:teuthology.orchestra.run.smithi076.stdout:legacy client fscid: 1 2024-03-18T14:52:49.702 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:49.702 INFO:teuthology.orchestra.run.smithi076.stdout:Filesystem 'cephfs' (1) 2024-03-18T14:52:49.702 INFO:teuthology.orchestra.run.smithi076.stdout:fs_name cephfs 2024-03-18T14:52:49.702 INFO:teuthology.orchestra.run.smithi076.stdout:epoch 23 2024-03-18T14:52:49.702 INFO:teuthology.orchestra.run.smithi076.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-18T14:52:49.702 INFO:teuthology.orchestra.run.smithi076.stdout:created 2024-03-18T14:51:01.293009+0000 2024-03-18T14:52:49.703 INFO:teuthology.orchestra.run.smithi076.stdout:modified 2024-03-18T14:51:07.032425+0000 2024-03-18T14:52:49.703 INFO:teuthology.orchestra.run.smithi076.stdout:tableserver 0 2024-03-18T14:52:49.703 INFO:teuthology.orchestra.run.smithi076.stdout:root 0 2024-03-18T14:52:49.703 INFO:teuthology.orchestra.run.smithi076.stdout:session_timeout 300 2024-03-18T14:52:49.703 INFO:teuthology.orchestra.run.smithi076.stdout:session_autoclose 300 2024-03-18T14:52:49.703 INFO:teuthology.orchestra.run.smithi076.stdout:max_file_size 1099511627776 2024-03-18T14:52:49.703 INFO:teuthology.orchestra.run.smithi076.stdout:max_xattr_size 65536 2024-03-18T14:52:49.703 INFO:teuthology.orchestra.run.smithi076.stdout:required_client_features {} 2024-03-18T14:52:49.703 INFO:teuthology.orchestra.run.smithi076.stdout:last_failure 0 2024-03-18T14:52:49.703 INFO:teuthology.orchestra.run.smithi076.stdout:last_failure_osd_epoch 0 2024-03-18T14:52:49.703 INFO:teuthology.orchestra.run.smithi076.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T14:52:49.703 INFO:teuthology.orchestra.run.smithi076.stdout:max_mds 3 2024-03-18T14:52:49.704 INFO:teuthology.orchestra.run.smithi076.stdout:in 0,1,2 2024-03-18T14:52:49.704 INFO:teuthology.orchestra.run.smithi076.stdout:up {0=24524,1=24490,2=24463} 2024-03-18T14:52:49.704 INFO:teuthology.orchestra.run.smithi076.stdout:failed 2024-03-18T14:52:49.704 INFO:teuthology.orchestra.run.smithi076.stdout:damaged 2024-03-18T14:52:49.704 INFO:teuthology.orchestra.run.smithi076.stdout:stopped 2024-03-18T14:52:49.704 INFO:teuthology.orchestra.run.smithi076.stdout:data_pools [3] 2024-03-18T14:52:49.704 INFO:teuthology.orchestra.run.smithi076.stdout:metadata_pool 2 2024-03-18T14:52:49.704 INFO:teuthology.orchestra.run.smithi076.stdout:inline_data disabled 2024-03-18T14:52:49.704 INFO:teuthology.orchestra.run.smithi076.stdout:balancer 2024-03-18T14:52:49.704 INFO:teuthology.orchestra.run.smithi076.stdout:bal_rank_mask -1 2024-03-18T14:52:49.704 INFO:teuthology.orchestra.run.smithi076.stdout:standby_count_wanted 0 2024-03-18T14:52:49.704 INFO:teuthology.orchestra.run.smithi076.stdout:qdb_cluster leader: 24524 members: 24463,24490,24524 2024-03-18T14:52:49.704 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.b{0:24524} state up:active seq 3 addr [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T14:52:49.705 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.e{0:24455} state up:standby-replay seq 1 addr [v2:172.21.15.77:6834/3551464902,v1:172.21.15.77:6835/3551464902] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T14:52:49.705 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.i{1:24490} state up:active seq 6 addr [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T14:52:49.705 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.f{1:24451} state up:standby-replay seq 1 addr [v2:172.21.15.174:6834/603893267,v1:172.21.15.174:6835/603893267] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T14:52:49.705 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.c{2:24463} state up:active seq 7 addr [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T14:52:49.705 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.k{2:24422} state up:standby-replay seq 1 addr [v2:172.21.15.77:6832/2726911960,v1:172.21.15.77:6833/2726911960] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T14:52:49.705 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:49.705 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:49.705 INFO:teuthology.orchestra.run.smithi076.stdout:Standby daemons: 2024-03-18T14:52:49.705 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:49.705 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.j{-1:14535} state up:standby seq 1 addr [v2:172.21.15.76:6834/3585848317,v1:172.21.15.76:6835/3585848317] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T14:52:49.705 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.d{-1:14547} state up:standby seq 1 addr [v2:172.21.15.76:6836/3302809366,v1:172.21.15.76:6837/3302809366] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T14:52:49.705 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.h{-1:14583} state up:standby seq 1 addr [v2:172.21.15.77:6836/2568866075,v1:172.21.15.77:6837/2568866075] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T14:52:49.705 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.g{-1:14613} state up:standby seq 1 addr [v2:172.21.15.76:6838/2801069369,v1:172.21.15.76:6839/2801069369] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T14:52:49.705 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.a{-1:14625} state up:standby seq 1 addr [v2:172.21.15.76:6840/2645229232,v1:172.21.15.76:6841/2645229232] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T14:52:49.705 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.l{-1:24400} state up:standby seq 1 addr [v2:172.21.15.174:6832/3569060524,v1:172.21.15.174:6833/3569060524] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T14:52:49.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.703+0000 7f40a45b4700 1 -- 172.21.15.76:0/669538386 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f40880a2890 msgr2=0x7f40880a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:49.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.703+0000 7f40a45b4700 1 --2- 172.21.15.76:0/669538386 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f40880a2890 0x7f40880a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f409c102250 tx=0x7f4098009260 comp rx=0 tx=0).stop 2024-03-18T14:52:49.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.703+0000 7f40a45b4700 1 -- 172.21.15.76:0/669538386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f409c102d90 msgr2=0x7f409c19e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:49.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.703+0000 7f40a45b4700 1 --2- 172.21.15.76:0/669538386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f409c102d90 0x7f409c19e940 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f409400b770 tx=0x7f4094007a00 comp rx=0 tx=0).stop 2024-03-18T14:52:49.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.703+0000 7f40a45b4700 1 -- 172.21.15.76:0/669538386 shutdown_connections 2024-03-18T14:52:49.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.703+0000 7f40a45b4700 1 --2- 172.21.15.76:0/669538386 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f40880a2890 0x7f40880a4d50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:49.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.703+0000 7f40a45b4700 1 --2- 172.21.15.76:0/669538386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f409c102d90 0x7f409c19e940 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:49.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.703+0000 7f40a45b4700 1 --2- 172.21.15.76:0/669538386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f409c1011f0 0x7f409c197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:49.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.703+0000 7f40a45b4700 1 --2- 172.21.15.76:0/669538386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f409c1023f0 0x7f409c1978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:49.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.703+0000 7f40a45b4700 1 -- 172.21.15.76:0/669538386 >> 172.21.15.76:0/669538386 conn(0x7f409c0fc790 msgr2=0x7f409c1056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:49.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.703+0000 7f40a45b4700 1 -- 172.21.15.76:0/669538386 shutdown_connections 2024-03-18T14:52:49.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:49.703+0000 7f40a45b4700 1 -- 172.21.15.76:0/669538386 wait complete. 2024-03-18T14:52:49.707 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 23 2024-03-18T14:52:49.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:49 smithi174 ceph-mon[36787]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T14:52:49.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:49 smithi077 ceph-mon[35865]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T14:52:50.492 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-18T14:52:50.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:50 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/669538386' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T14:52:50.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:50 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/669538386' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T14:52:50.735 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:52:50.856 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:50 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/669538386' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T14:52:51.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:51 smithi174 ceph-mon[36787]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-18T14:52:51.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:51 smithi077 ceph-mon[35865]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-18T14:52:51.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:51 smithi076 ceph-mon[32205]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-18T14:52:52.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.540+0000 7f26f08cb700 1 -- 172.21.15.76:0/1018981398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f26ec102400 msgr2=0x7f26ec102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:52.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.540+0000 7f26f08cb700 1 --2- 172.21.15.76:0/1018981398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f26ec102400 0x7f26ec102860 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f26d40099c0 tx=0x7f26d4009cd0 comp rx=0 tx=0).stop 2024-03-18T14:52:52.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.540+0000 7f26f08cb700 1 -- 172.21.15.76:0/1018981398 shutdown_connections 2024-03-18T14:52:52.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.540+0000 7f26f08cb700 1 --2- 172.21.15.76:0/1018981398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26ec102da0 0x7f26ec109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:52.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.540+0000 7f26f08cb700 1 --2- 172.21.15.76:0/1018981398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26ec101200 0x7f26ec101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:52.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.540+0000 7f26f08cb700 1 --2- 172.21.15.76:0/1018981398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f26ec102400 0x7f26ec102860 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:52.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.540+0000 7f26f08cb700 1 -- 172.21.15.76:0/1018981398 >> 172.21.15.76:0/1018981398 conn(0x7f26ec0fc7a0 msgr2=0x7f26ec0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:52.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.540+0000 7f26f08cb700 1 -- 172.21.15.76:0/1018981398 shutdown_connections 2024-03-18T14:52:52.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.540+0000 7f26f08cb700 1 -- 172.21.15.76:0/1018981398 wait complete. 2024-03-18T14:52:52.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.540+0000 7f26f08cb700 1 Processor -- start 2024-03-18T14:52:52.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.540+0000 7f26f08cb700 1 -- start start 2024-03-18T14:52:52.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26f08cb700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f26ec101200 0x7f26ec197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:52.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26f08cb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26ec102400 0x7f26ec197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:52.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26f08cb700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26ec102da0 0x7f26ec19ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:52.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26f08cb700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec10cdc0 con 0x7f26ec101200 2024-03-18T14:52:52.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26f08cb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec10cc40 con 0x7f26ec102400 2024-03-18T14:52:52.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26f08cb700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec10cf40 con 0x7f26ec102da0 2024-03-18T14:52:52.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26ea59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f26ec101200 0x7f26ec197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:52.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26e9d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26ec102400 0x7f26ec197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:52.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26ea59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f26ec101200 0x7f26ec197440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50876/0 (socket says 172.21.15.76:50876) 2024-03-18T14:52:52.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26ea59c700 1 -- 172.21.15.76:0/2700027179 learned_addr learned my addr 172.21.15.76:0/2700027179 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:52.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26ead9d700 1 --2- 172.21.15.76:0/2700027179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26ec102da0 0x7f26ec19ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:52.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26e9d9b700 1 -- 172.21.15.76:0/2700027179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26ec102da0 msgr2=0x7f26ec19ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:52.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26e9d9b700 1 --2- 172.21.15.76:0/2700027179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26ec102da0 0x7f26ec19ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:52.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26e9d9b700 1 -- 172.21.15.76:0/2700027179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f26ec101200 msgr2=0x7f26ec197440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:52.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26e9d9b700 1 --2- 172.21.15.76:0/2700027179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f26ec101200 0x7f26ec197440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:52.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26e9d9b700 1 -- 172.21.15.76:0/2700027179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26ec19f0a0 con 0x7f26ec102400 2024-03-18T14:52:52.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26e9d9b700 1 --2- 172.21.15.76:0/2700027179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26ec102400 0x7f26ec197980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f26d400b580 tx=0x7f26d400bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:52.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26e37fe700 1 -- 172.21.15.76:0/2700027179 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d4009e60 con 0x7f26ec102400 2024-03-18T14:52:52.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26e37fe700 1 -- 172.21.15.76:0/2700027179 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26d400f460 con 0x7f26ec102400 2024-03-18T14:52:52.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26f08cb700 1 -- 172.21.15.76:0/2700027179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26ec19f330 con 0x7f26ec102400 2024-03-18T14:52:52.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26e37fe700 1 -- 172.21.15.76:0/2700027179 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d4017630 con 0x7f26ec102400 2024-03-18T14:52:52.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.541+0000 7f26f08cb700 1 -- 172.21.15.76:0/2700027179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26ec19f780 con 0x7f26ec102400 2024-03-18T14:52:52.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.542+0000 7f26f08cb700 1 -- 172.21.15.76:0/2700027179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26ec068f90 con 0x7f26ec102400 2024-03-18T14:52:52.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.543+0000 7f26e37fe700 1 -- 172.21.15.76:0/2700027179 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f26d400f610 con 0x7f26ec102400 2024-03-18T14:52:52.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.544+0000 7f26e37fe700 1 --2- 172.21.15.76:0/2700027179 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f26d80a2890 0x7f26d80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:52.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.544+0000 7f26ea59c700 1 --2- 172.21.15.76:0/2700027179 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f26d80a2890 0x7f26d80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:52.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.544+0000 7f26e37fe700 1 -- 172.21.15.76:0/2700027179 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f26d40cf7b0 con 0x7f26ec102400 2024-03-18T14:52:52.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.544+0000 7f26ea59c700 1 --2- 172.21.15.76:0/2700027179 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f26d80a2890 0x7f26d80a4d50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f26dc005950 tx=0x7f26dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:52.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.548+0000 7f26e37fe700 1 -- 172.21.15.76:0/2700027179 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f26d4013070 con 0x7f26ec102400 2024-03-18T14:52:52.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.744+0000 7f26f08cb700 1 -- 172.21.15.76:0/2700027179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f26ec062ee0 con 0x7f26ec102400 2024-03-18T14:52:52.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.745+0000 7f26e37fe700 1 -- 172.21.15.76:0/2700027179 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) ==== 56+0+5248 (secure 0 0 0) 0x7f26d409c910 con 0x7f26ec102400 2024-03-18T14:52:52.746 INFO:teuthology.orchestra.run.smithi076.stdout:epoch 88 2024-03-18T14:52:52.746 INFO:teuthology.orchestra.run.smithi076.stdout:fsid 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T14:52:52.746 INFO:teuthology.orchestra.run.smithi076.stdout:created 2024-03-18T14:43:12.899171+0000 2024-03-18T14:52:52.747 INFO:teuthology.orchestra.run.smithi076.stdout:modified 2024-03-18T14:52:07.361301+0000 2024-03-18T14:52:52.747 INFO:teuthology.orchestra.run.smithi076.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-18T14:52:52.747 INFO:teuthology.orchestra.run.smithi076.stdout:crush_version 26 2024-03-18T14:52:52.747 INFO:teuthology.orchestra.run.smithi076.stdout:full_ratio 0.95 2024-03-18T14:52:52.747 INFO:teuthology.orchestra.run.smithi076.stdout:backfillfull_ratio 0.9 2024-03-18T14:52:52.747 INFO:teuthology.orchestra.run.smithi076.stdout:nearfull_ratio 0.85 2024-03-18T14:52:52.748 INFO:teuthology.orchestra.run.smithi076.stdout:require_min_compat_client luminous 2024-03-18T14:52:52.748 INFO:teuthology.orchestra.run.smithi076.stdout:min_compat_client luminous 2024-03-18T14:52:52.748 INFO:teuthology.orchestra.run.smithi076.stdout:require_osd_release squid 2024-03-18T14:52:52.748 INFO:teuthology.orchestra.run.smithi076.stdout:stretch_mode_enabled false 2024-03-18T14:52:52.748 INFO:teuthology.orchestra.run.smithi076.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-18T14:52:52.748 INFO:teuthology.orchestra.run.smithi076.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-18T14:52:52.749 INFO:teuthology.orchestra.run.smithi076.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-18T14:52:52.749 INFO:teuthology.orchestra.run.smithi076.stdout:max_osd 12 2024-03-18T14:52:52.749 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6802/2682004861,v1:172.21.15.76:6803/2682004861] [v2:172.21.15.76:6804/2682004861,v1:172.21.15.76:6805/2682004861] exists,up 7b8ee0a0-29f7-4333-978a-08c4db54db0d 2024-03-18T14:52:52.749 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1 up in weight 1 up_from 13 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6800/3243872836,v1:172.21.15.77:6801/3243872836] [v2:172.21.15.77:6802/3243872836,v1:172.21.15.77:6803/3243872836] exists,up 71422220-775b-4b49-a9a4-ae0a7a16c2e5 2024-03-18T14:52:52.749 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6800/3812090370,v1:172.21.15.174:6801/3812090370] [v2:172.21.15.174:6802/3812090370,v1:172.21.15.174:6803/3812090370] exists,up 3a5bf599-bc62-4986-9bf3-6486a08857a2 2024-03-18T14:52:52.749 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3 up in weight 1 up_from 26 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6810/140675439,v1:172.21.15.76:6811/140675439] [v2:172.21.15.76:6812/140675439,v1:172.21.15.76:6813/140675439] exists,up 5060b19a-4b1f-4a15-bd07-a920a267bcfd 2024-03-18T14:52:52.750 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4 up in weight 1 up_from 34 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6808/1533069359,v1:172.21.15.77:6809/1533069359] [v2:172.21.15.77:6810/1533069359,v1:172.21.15.77:6811/1533069359] exists,up b88f6022-4690-4c54-9072-8a1f0febddd3 2024-03-18T14:52:52.750 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5 up in weight 1 up_from 39 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6808/1607563490,v1:172.21.15.174:6809/1607563490] [v2:172.21.15.174:6810/1607563490,v1:172.21.15.174:6811/1607563490] exists,up 53063e2c-b521-41d8-859d-7a4bcf470fa2 2024-03-18T14:52:52.750 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6 up in weight 1 up_from 47 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6818/1891318352,v1:172.21.15.76:6819/1891318352] [v2:172.21.15.76:6820/1891318352,v1:172.21.15.76:6821/1891318352] exists,up f008e913-20ac-4df7-8bed-fb345d6d84bb 2024-03-18T14:52:52.750 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7 up in weight 1 up_from 52 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6816/3828133263,v1:172.21.15.77:6817/3828133263] [v2:172.21.15.77:6818/3828133263,v1:172.21.15.77:6819/3828133263] exists,up 7354ce4c-94d9-4d70-9134-b01836fbfc90 2024-03-18T14:52:52.750 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8 up in weight 1 up_from 60 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6816/2665844766,v1:172.21.15.174:6817/2665844766] [v2:172.21.15.174:6818/2665844766,v1:172.21.15.174:6819/2665844766] exists,up 955ffff4-1c26-4dbd-82f7-aaf6ceb7da21 2024-03-18T14:52:52.750 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9 up in weight 1 up_from 68 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6826/3937900663,v1:172.21.15.76:6827/3937900663] [v2:172.21.15.76:6828/3937900663,v1:172.21.15.76:6829/3937900663] exists,up 1a818122-b7a3-4f62-adaf-0ec35156cf1c 2024-03-18T14:52:52.751 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10 up in weight 1 up_from 73 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6824/4105457096,v1:172.21.15.77:6825/4105457096] [v2:172.21.15.77:6826/4105457096,v1:172.21.15.77:6827/4105457096] exists,up ee1f73e8-94b3-4969-9944-da77e7f46bec 2024-03-18T14:52:52.751 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11 up in weight 1 up_from 81 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6824/1489039553,v1:172.21.15.174:6825/1489039553] [v2:172.21.15.174:6826/1489039553,v1:172.21.15.174:6827/1489039553] exists,up 35839388-3d56-497b-a49c-0db8686e3a97 2024-03-18T14:52:52.751 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 2.a [2,11] 2024-03-18T14:52:52.751 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 2.14 [2,4] 2024-03-18T14:52:52.751 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 2.1d [8,11] 2024-03-18T14:52:52.751 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 3.21 [5,9] 2024-03-18T14:52:52.752 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 3.2c [4,9] 2024-03-18T14:52:52.752 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 3.31 [4,8] 2024-03-18T14:52:52.752 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 3.3c [4,9] 2024-03-18T14:52:52.752 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/3122154733 expires 2024-03-19T14:44:06.251768+0000 2024-03-18T14:52:52.752 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/752916178 expires 2024-03-19T14:44:06.251768+0000 2024-03-18T14:52:52.752 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/3261047930 expires 2024-03-19T14:44:06.251768+0000 2024-03-18T14:52:52.753 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:6801/3054039863 expires 2024-03-19T14:44:06.251768+0000 2024-03-18T14:52:52.753 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/600153683 expires 2024-03-19T14:43:39.730762+0000 2024-03-18T14:52:52.753 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:6801/216126402 expires 2024-03-19T14:43:39.730762+0000 2024-03-18T14:52:52.753 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/1895213347 expires 2024-03-19T14:43:39.730762+0000 2024-03-18T14:52:52.753 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/918193033 expires 2024-03-19T14:43:39.730762+0000 2024-03-18T14:52:52.753 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:6800/216126402 expires 2024-03-19T14:43:39.730762+0000 2024-03-18T14:52:52.754 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:6800/3054039863 expires 2024-03-19T14:44:06.251768+0000 2024-03-18T14:52:52.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.747+0000 7f26f08cb700 1 -- 172.21.15.76:0/2700027179 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f26d80a2890 msgr2=0x7f26d80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:52.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.747+0000 7f26f08cb700 1 --2- 172.21.15.76:0/2700027179 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f26d80a2890 0x7f26d80a4d50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f26dc005950 tx=0x7f26dc009260 comp rx=0 tx=0).stop 2024-03-18T14:52:52.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.747+0000 7f26f08cb700 1 -- 172.21.15.76:0/2700027179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26ec102400 msgr2=0x7f26ec197980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:52.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.747+0000 7f26f08cb700 1 --2- 172.21.15.76:0/2700027179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26ec102400 0x7f26ec197980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f26d400b580 tx=0x7f26d400bee0 comp rx=0 tx=0).stop 2024-03-18T14:52:52.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.747+0000 7f26f08cb700 1 -- 172.21.15.76:0/2700027179 shutdown_connections 2024-03-18T14:52:52.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.747+0000 7f26f08cb700 1 --2- 172.21.15.76:0/2700027179 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f26d80a2890 0x7f26d80a4d50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:52.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.747+0000 7f26f08cb700 1 --2- 172.21.15.76:0/2700027179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26ec102da0 0x7f26ec19ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:52.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.747+0000 7f26f08cb700 1 --2- 172.21.15.76:0/2700027179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f26ec101200 0x7f26ec197440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:52.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.747+0000 7f26f08cb700 1 --2- 172.21.15.76:0/2700027179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26ec102400 0x7f26ec197980 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:52.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.747+0000 7f26f08cb700 1 -- 172.21.15.76:0/2700027179 >> 172.21.15.76:0/2700027179 conn(0x7f26ec0fc7a0 msgr2=0x7f26ec105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:52.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.747+0000 7f26f08cb700 1 -- 172.21.15.76:0/2700027179 shutdown_connections 2024-03-18T14:52:52.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:52.747+0000 7f26f08cb700 1 -- 172.21.15.76:0/2700027179 wait complete. 2024-03-18T14:52:53.582 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-18T14:52:53.593 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:52:53.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.697+0000 7f165633b640 1 -- 172.21.15.76:0/983966887 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1650100700 msgr2=0x7f1650100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:53.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.697+0000 7f165633b640 1 --2- 172.21.15.76:0/983966887 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1650100700 0x7f1650100b20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1644009900 tx=0x7f164402bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:53.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.697+0000 7f165633b640 1 -- 172.21.15.76:0/983966887 shutdown_connections 2024-03-18T14:52:53.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.697+0000 7f165633b640 1 --2- 172.21.15.76:0/983966887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16501018f0 0x7f1650101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:53.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.697+0000 7f165633b640 1 --2- 172.21.15.76:0/983966887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16501022b0 0x7f1650108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:53.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.697+0000 7f165633b640 1 --2- 172.21.15.76:0/983966887 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1650100700 0x7f1650100b20 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:53.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.697+0000 7f165633b640 1 -- 172.21.15.76:0/983966887 >> 172.21.15.76:0/983966887 conn(0x7f16500fbc70 msgr2=0x7f16500fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:53.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.697+0000 7f165633b640 1 -- 172.21.15.76:0/983966887 shutdown_connections 2024-03-18T14:52:53.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.697+0000 7f165633b640 1 -- 172.21.15.76:0/983966887 wait complete. 2024-03-18T14:52:53.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.698+0000 7f165633b640 1 Processor -- start 2024-03-18T14:52:53.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.698+0000 7f165633b640 1 -- start start 2024-03-18T14:52:53.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.698+0000 7f165633b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1650100700 0x7f165006aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:53.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.698+0000 7f165633b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16501018f0 0x7f165019c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:53.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.698+0000 7f165633b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16501022b0 0x7f165019ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:53.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.698+0000 7f165633b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f165010c350 con 0x7f1650100700 2024-03-18T14:52:53.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.698+0000 7f165633b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f165010c1d0 con 0x7f16501022b0 2024-03-18T14:52:53.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.698+0000 7f165633b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f165010c4d0 con 0x7f16501018f0 2024-03-18T14:52:53.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.698+0000 7f164ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1650100700 0x7f165006aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:53.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.698+0000 7f164ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1650100700 0x7f165006aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50906/0 (socket says 172.21.15.76:50906) 2024-03-18T14:52:53.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.698+0000 7f164ffff640 1 -- 172.21.15.76:0/3162359530 learned_addr learned my addr 172.21.15.76:0/3162359530 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:53.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.698+0000 7f16548d8640 1 --2- 172.21.15.76:0/3162359530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16501022b0 0x7f165019ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:53.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.698+0000 7f164f7fe640 1 --2- 172.21.15.76:0/3162359530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16501018f0 0x7f165019c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:53.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.699+0000 7f164ffff640 1 -- 172.21.15.76:0/3162359530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16501018f0 msgr2=0x7f165019c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:53.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.699+0000 7f164ffff640 1 --2- 172.21.15.76:0/3162359530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16501018f0 0x7f165019c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:53.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.699+0000 7f164ffff640 1 -- 172.21.15.76:0/3162359530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16501022b0 msgr2=0x7f165019ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:53.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.699+0000 7f164ffff640 1 --2- 172.21.15.76:0/3162359530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16501022b0 0x7f165019ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:53.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.699+0000 7f164ffff640 1 -- 172.21.15.76:0/3162359530 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f165019f710 con 0x7f1650100700 2024-03-18T14:52:53.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.699+0000 7f164f7fe640 1 --2- 172.21.15.76:0/3162359530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16501018f0 0x7f165019c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:53.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.699+0000 7f164ffff640 1 --2- 172.21.15.76:0/3162359530 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1650100700 0x7f165006aa20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1644000c00 tx=0x7f164402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:53.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.699+0000 7f164d7fa640 1 -- 172.21.15.76:0/3162359530 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1644041070 con 0x7f1650100700 2024-03-18T14:52:53.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.699+0000 7f164d7fa640 1 -- 172.21.15.76:0/3162359530 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1644004450 con 0x7f1650100700 2024-03-18T14:52:53.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.699+0000 7f165633b640 1 -- 172.21.15.76:0/3162359530 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f165019f9a0 con 0x7f1650100700 2024-03-18T14:52:53.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.699+0000 7f164d7fa640 1 -- 172.21.15.76:0/3162359530 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1644034440 con 0x7f1650100700 2024-03-18T14:52:53.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.699+0000 7f165633b640 1 -- 172.21.15.76:0/3162359530 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f165019fe90 con 0x7f1650100700 2024-03-18T14:52:53.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.700+0000 7f165633b640 1 -- 172.21.15.76:0/3162359530 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1650107a20 con 0x7f1650100700 2024-03-18T14:52:53.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.701+0000 7f164d7fa640 1 -- 172.21.15.76:0/3162359530 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1644044070 con 0x7f1650100700 2024-03-18T14:52:53.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.701+0000 7f164d7fa640 1 --2- 172.21.15.76:0/3162359530 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f16380a2800 0x7f16380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:53.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.702+0000 7f164f7fe640 1 --2- 172.21.15.76:0/3162359530 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f16380a2800 0x7f16380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:53.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.702+0000 7f164d7fa640 1 -- 172.21.15.76:0/3162359530 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f16440f04f0 con 0x7f1650100700 2024-03-18T14:52:53.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.702+0000 7f164f7fe640 1 --2- 172.21.15.76:0/3162359530 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f16380a2800 0x7f16380a4cc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f165006a790 tx=0x7f1640005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:53.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.706+0000 7f164d7fa640 1 -- 172.21.15.76:0/3162359530 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f16440bd000 con 0x7f1650100700 2024-03-18T14:52:53.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:53 smithi174 ceph-mon[36787]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-18T14:52:53.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:53 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2700027179' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T14:52:53.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:53 smithi077 ceph-mon[35865]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-18T14:52:53.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:53 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2700027179' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T14:52:53.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.883+0000 7f165633b640 1 -- 172.21.15.76:0/3162359530 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1650062410 con 0x7f1650100700 2024-03-18T14:52:53.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.884+0000 7f164d7fa640 1 -- 172.21.15.76:0/3162359530 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22580 (secure 0 0 0) 0x7f16440c10a0 con 0x7f1650100700 2024-03-18T14:52:53.885 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:53.886 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:52:07.361301+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:52:53.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.886+0000 7f165633b640 1 -- 172.21.15.76:0/3162359530 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f16380a2800 msgr2=0x7f16380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:53.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.886+0000 7f165633b640 1 --2- 172.21.15.76:0/3162359530 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f16380a2800 0x7f16380a4cc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f165006a790 tx=0x7f1640005f90 comp rx=0 tx=0).stop 2024-03-18T14:52:53.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.886+0000 7f165633b640 1 -- 172.21.15.76:0/3162359530 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1650100700 msgr2=0x7f165006aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:53.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.886+0000 7f165633b640 1 --2- 172.21.15.76:0/3162359530 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1650100700 0x7f165006aa20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1644000c00 tx=0x7f164402c640 comp rx=0 tx=0).stop 2024-03-18T14:52:53.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.887+0000 7f165633b640 1 -- 172.21.15.76:0/3162359530 shutdown_connections 2024-03-18T14:52:53.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.887+0000 7f165633b640 1 --2- 172.21.15.76:0/3162359530 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f16380a2800 0x7f16380a4cc0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:53.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.887+0000 7f165633b640 1 --2- 172.21.15.76:0/3162359530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16501018f0 0x7f165019c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:53.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.887+0000 7f165633b640 1 --2- 172.21.15.76:0/3162359530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16501022b0 0x7f165019ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:53.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.887+0000 7f165633b640 1 --2- 172.21.15.76:0/3162359530 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1650100700 0x7f165006aa20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:53.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.887+0000 7f165633b640 1 -- 172.21.15.76:0/3162359530 >> 172.21.15.76:0/3162359530 conn(0x7f16500fbc70 msgr2=0x7f1650104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:53.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.887+0000 7f165633b640 1 -- 172.21.15.76:0/3162359530 shutdown_connections 2024-03-18T14:52:53.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:53.887+0000 7f165633b640 1 -- 172.21.15.76:0/3162359530 wait complete. 2024-03-18T14:52:53.898 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:53 smithi076 ceph-mon[32205]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-18T14:52:53.898 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:53 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2700027179' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T14:52:53.899 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:46:26.588384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:50:59.606852+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:51:00.465325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:52:53.900 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:52:54.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.002+0000 7f5ccc8ea640 1 -- 172.21.15.76:0/1058603432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc81018f0 msgr2=0x7f5cc8101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.003+0000 7f5ccc8ea640 1 --2- 172.21.15.76:0/1058603432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc81018f0 0x7f5cc8101d70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0009900 tx=0x7f5cb002bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:54.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.003+0000 7f5ccc8ea640 1 -- 172.21.15.76:0/1058603432 shutdown_connections 2024-03-18T14:52:54.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.003+0000 7f5ccc8ea640 1 --2- 172.21.15.76:0/1058603432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5cc81022b0 0x7f5cc8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.003+0000 7f5ccc8ea640 1 --2- 172.21.15.76:0/1058603432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc81018f0 0x7f5cc8101d70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.003+0000 7f5ccc8ea640 1 --2- 172.21.15.76:0/1058603432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc8100700 0x7f5cc8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.003+0000 7f5ccc8ea640 1 -- 172.21.15.76:0/1058603432 >> 172.21.15.76:0/1058603432 conn(0x7f5cc80fbc70 msgr2=0x7f5cc80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:54.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.003+0000 7f5ccc8ea640 1 -- 172.21.15.76:0/1058603432 shutdown_connections 2024-03-18T14:52:54.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.003+0000 7f5ccc8ea640 1 -- 172.21.15.76:0/1058603432 wait complete. 2024-03-18T14:52:54.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.003+0000 7f5ccc8ea640 1 Processor -- start 2024-03-18T14:52:54.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.003+0000 7f5ccc8ea640 1 -- start start 2024-03-18T14:52:54.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.003+0000 7f5ccc8ea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc8100700 0x7f5cc8196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:54.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5ccc8ea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc81018f0 0x7f5cc8196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:54.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5ccc8ea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5cc81022b0 0x7f5cc819e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:54.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5ccc8ea640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc810c380 con 0x7f5cc8100700 2024-03-18T14:52:54.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5ccc8ea640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc810c200 con 0x7f5cc81018f0 2024-03-18T14:52:54.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5ccc8ea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc810c500 con 0x7f5cc81022b0 2024-03-18T14:52:54.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5cc659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc8100700 0x7f5cc8196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:54.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5cc659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc8100700 0x7f5cc8196ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50936/0 (socket says 172.21.15.76:50936) 2024-03-18T14:52:54.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5cc659c640 1 -- 172.21.15.76:0/1052822675 learned_addr learned my addr 172.21.15.76:0/1052822675 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:54.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5cc5d9b640 1 --2- 172.21.15.76:0/1052822675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc81018f0 0x7f5cc8196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:54.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5cc6d9d640 1 --2- 172.21.15.76:0/1052822675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5cc81022b0 0x7f5cc819e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:54.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5cc659c640 1 -- 172.21.15.76:0/1052822675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5cc81022b0 msgr2=0x7f5cc819e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5cc659c640 1 --2- 172.21.15.76:0/1052822675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5cc81022b0 0x7f5cc819e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5cc659c640 1 -- 172.21.15.76:0/1052822675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc81018f0 msgr2=0x7f5cc8196ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5cc659c640 1 --2- 172.21.15.76:0/1052822675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc81018f0 0x7f5cc8196ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5cc659c640 1 -- 172.21.15.76:0/1052822675 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc819e6d0 con 0x7f5cc8100700 2024-03-18T14:52:54.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5cc5d9b640 1 --2- 172.21.15.76:0/1052822675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc81018f0 0x7f5cc8196ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:54.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5cc659c640 1 --2- 172.21.15.76:0/1052822675 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc8100700 0x7f5cc8196ab0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5cb800e9c0 tx=0x7f5cb800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:54.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5cbf7fe640 1 -- 172.21.15.76:0/1052822675 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb800cde0 con 0x7f5cc8100700 2024-03-18T14:52:54.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5cbf7fe640 1 -- 172.21.15.76:0/1052822675 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cb8010910 con 0x7f5cc8100700 2024-03-18T14:52:54.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5ccc8ea640 1 -- 172.21.15.76:0/1052822675 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cc819e9c0 con 0x7f5cc8100700 2024-03-18T14:52:54.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5cbf7fe640 1 -- 172.21.15.76:0/1052822675 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8018a10 con 0x7f5cc8100700 2024-03-18T14:52:54.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.004+0000 7f5ccc8ea640 1 -- 172.21.15.76:0/1052822675 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cc819eee0 con 0x7f5cc8100700 2024-03-18T14:52:54.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.005+0000 7f5ccc8ea640 1 -- 172.21.15.76:0/1052822675 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cc8107b40 con 0x7f5cc8100700 2024-03-18T14:52:54.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.006+0000 7f5cbf7fe640 1 -- 172.21.15.76:0/1052822675 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5cb801f020 con 0x7f5cc8100700 2024-03-18T14:52:54.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.007+0000 7f5cbf7fe640 1 --2- 172.21.15.76:0/1052822675 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5cb40a2800 0x7f5cb40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:54.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.007+0000 7f5cc5d9b640 1 --2- 172.21.15.76:0/1052822675 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5cb40a2800 0x7f5cb40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:54.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.007+0000 7f5cbf7fe640 1 -- 172.21.15.76:0/1052822675 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f5cb80cebf0 con 0x7f5cc8100700 2024-03-18T14:52:54.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.007+0000 7f5cc5d9b640 1 --2- 172.21.15.76:0/1052822675 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5cb40a2800 0x7f5cb40a4cc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0009230 tx=0x7f5cb0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:54.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.011+0000 7f5cbf7fe640 1 -- 172.21.15.76:0/1052822675 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5cb809bcc0 con 0x7f5cc8100700 2024-03-18T14:52:54.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.185+0000 7f5ccc8ea640 1 -- 172.21.15.76:0/1052822675 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5cc8062410 con 0x7f5cc8100700 2024-03-18T14:52:54.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.186+0000 7f5cbf7fe640 1 -- 172.21.15.76:0/1052822675 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f5cb809b5c0 con 0x7f5cc8100700 2024-03-18T14:52:54.186 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-18T14:52:54.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.188+0000 7f5ccc8ea640 1 -- 172.21.15.76:0/1052822675 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5cb40a2800 msgr2=0x7f5cb40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.188+0000 7f5ccc8ea640 1 --2- 172.21.15.76:0/1052822675 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5cb40a2800 0x7f5cb40a4cc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0009230 tx=0x7f5cb0005fd0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.188+0000 7f5ccc8ea640 1 -- 172.21.15.76:0/1052822675 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc8100700 msgr2=0x7f5cc8196ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.188+0000 7f5ccc8ea640 1 --2- 172.21.15.76:0/1052822675 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc8100700 0x7f5cc8196ab0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5cb800e9c0 tx=0x7f5cb800ee80 comp rx=0 tx=0).stop 2024-03-18T14:52:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.188+0000 7f5ccc8ea640 1 -- 172.21.15.76:0/1052822675 shutdown_connections 2024-03-18T14:52:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.188+0000 7f5ccc8ea640 1 --2- 172.21.15.76:0/1052822675 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f5cb40a2800 0x7f5cb40a4cc0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.188+0000 7f5ccc8ea640 1 --2- 172.21.15.76:0/1052822675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5cc81022b0 0x7f5cc819e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.188+0000 7f5ccc8ea640 1 --2- 172.21.15.76:0/1052822675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc81018f0 0x7f5cc8196ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.188+0000 7f5ccc8ea640 1 --2- 172.21.15.76:0/1052822675 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc8100700 0x7f5cc8196ab0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.188+0000 7f5ccc8ea640 1 -- 172.21.15.76:0/1052822675 >> 172.21.15.76:0/1052822675 conn(0x7f5cc80fbc70 msgr2=0x7f5cc8104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.188+0000 7f5ccc8ea640 1 -- 172.21.15.76:0/1052822675 shutdown_connections 2024-03-18T14:52:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.188+0000 7f5ccc8ea640 1 -- 172.21.15.76:0/1052822675 wait complete. 2024-03-18T14:52:54.200 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:52:54.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.303+0000 7f497859d640 1 -- 172.21.15.76:0/3699192971 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49701029e0 msgr2=0x7f4970102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.303+0000 7f497859d640 1 --2- 172.21.15.76:0/3699192971 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49701029e0 0x7f4970102e60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4960009900 tx=0x7f496002bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:54.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.303+0000 7f497859d640 1 -- 172.21.15.76:0/3699192971 shutdown_connections 2024-03-18T14:52:54.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.303+0000 7f497859d640 1 --2- 172.21.15.76:0/3699192971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49701033a0 0x7f4970109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.303+0000 7f497859d640 1 --2- 172.21.15.76:0/3699192971 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49701029e0 0x7f4970102e60 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.303+0000 7f497859d640 1 --2- 172.21.15.76:0/3699192971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49701017f0 0x7f4970101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.303+0000 7f497859d640 1 -- 172.21.15.76:0/3699192971 >> 172.21.15.76:0/3699192971 conn(0x7f49700fcd80 msgr2=0x7f49700ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:54.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.304+0000 7f497859d640 1 -- 172.21.15.76:0/3699192971 shutdown_connections 2024-03-18T14:52:54.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.304+0000 7f497859d640 1 -- 172.21.15.76:0/3699192971 wait complete. 2024-03-18T14:52:54.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.304+0000 7f497859d640 1 Processor -- start 2024-03-18T14:52:54.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.304+0000 7f497859d640 1 -- start start 2024-03-18T14:52:54.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.304+0000 7f497859d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49701017f0 0x7f4970199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:54.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.304+0000 7f497859d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49701029e0 0x7f497019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:54.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.304+0000 7f497859d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49701033a0 0x7f49701a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:54.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.304+0000 7f497859d640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f497010d470 con 0x7f49701017f0 2024-03-18T14:52:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.304+0000 7f497859d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f497010d2f0 con 0x7f49701033a0 2024-03-18T14:52:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.304+0000 7f497859d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f497010d5f0 con 0x7f49701029e0 2024-03-18T14:52:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.304+0000 7f4976339640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49701017f0 0x7f4970199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.304+0000 7f4976339640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49701017f0 0x7f4970199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50960/0 (socket says 172.21.15.76:50960) 2024-03-18T14:52:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.304+0000 7f4976339640 1 -- 172.21.15.76:0/3536461772 learned_addr learned my addr 172.21.15.76:0/3536461772 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.304+0000 7f4976b3a640 1 --2- 172.21.15.76:0/3536461772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49701033a0 0x7f49701a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.304+0000 7f4975b38640 1 --2- 172.21.15.76:0/3536461772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49701029e0 0x7f497019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.305+0000 7f4976339640 1 -- 172.21.15.76:0/3536461772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49701029e0 msgr2=0x7f497019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.305+0000 7f4976339640 1 --2- 172.21.15.76:0/3536461772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49701029e0 0x7f497019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.305+0000 7f4976339640 1 -- 172.21.15.76:0/3536461772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49701033a0 msgr2=0x7f49701a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.305+0000 7f4976339640 1 --2- 172.21.15.76:0/3536461772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49701033a0 0x7f49701a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.305+0000 7f4976339640 1 -- 172.21.15.76:0/3536461772 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49701a1950 con 0x7f49701017f0 2024-03-18T14:52:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.305+0000 7f4976b3a640 1 --2- 172.21.15.76:0/3536461772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49701033a0 0x7f49701a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.305+0000 7f4976339640 1 --2- 172.21.15.76:0/3536461772 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49701017f0 0x7f4970199d40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f496c00eb70 tx=0x7f496c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.305+0000 7f49677fe640 1 -- 172.21.15.76:0/3536461772 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c003b70 con 0x7f49701017f0 2024-03-18T14:52:54.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.305+0000 7f497859d640 1 -- 172.21.15.76:0/3536461772 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49701a1c40 con 0x7f49701017f0 2024-03-18T14:52:54.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.305+0000 7f49677fe640 1 -- 172.21.15.76:0/3536461772 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f496c00ee70 con 0x7f49701017f0 2024-03-18T14:52:54.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.305+0000 7f49677fe640 1 -- 172.21.15.76:0/3536461772 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c010830 con 0x7f49701017f0 2024-03-18T14:52:54.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.305+0000 7f497859d640 1 -- 172.21.15.76:0/3536461772 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49701a2190 con 0x7f49701017f0 2024-03-18T14:52:54.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.306+0000 7f497859d640 1 -- 172.21.15.76:0/3536461772 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49701a2360 con 0x7f49701017f0 2024-03-18T14:52:54.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.307+0000 7f49677fe640 1 -- 172.21.15.76:0/3536461772 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f496c004a40 con 0x7f49701017f0 2024-03-18T14:52:54.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.308+0000 7f49677fe640 1 --2- 172.21.15.76:0/3536461772 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f495c0a2800 0x7f495c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:54.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.308+0000 7f4975b38640 1 --2- 172.21.15.76:0/3536461772 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f495c0a2800 0x7f495c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:54.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.308+0000 7f49677fe640 1 -- 172.21.15.76:0/3536461772 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f496c0cf1c0 con 0x7f49701017f0 2024-03-18T14:52:54.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.308+0000 7f4975b38640 1 --2- 172.21.15.76:0/3536461772 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f495c0a2800 0x7f495c0a4cc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f496002c7b0 tx=0x7f4960005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:54.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.311+0000 7f49677fe640 1 -- 172.21.15.76:0/3536461772 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f496c09c290 con 0x7f49701017f0 2024-03-18T14:52:54.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.489+0000 7f497859d640 1 -- 172.21.15.76:0/3536461772 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f49701a2530 con 0x7f49701017f0 2024-03-18T14:52:54.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.490+0000 7f49677fe640 1 -- 172.21.15.76:0/3536461772 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f496c09bb90 con 0x7f49701017f0 2024-03-18T14:52:54.490 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-18T14:52:54.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.492+0000 7f497859d640 1 -- 172.21.15.76:0/3536461772 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f495c0a2800 msgr2=0x7f495c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.492+0000 7f497859d640 1 --2- 172.21.15.76:0/3536461772 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f495c0a2800 0x7f495c0a4cc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f496002c7b0 tx=0x7f4960005fd0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.492+0000 7f497859d640 1 -- 172.21.15.76:0/3536461772 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49701017f0 msgr2=0x7f4970199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.492+0000 7f497859d640 1 --2- 172.21.15.76:0/3536461772 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49701017f0 0x7f4970199d40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f496c00eb70 tx=0x7f496c00c6a0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.492+0000 7f497859d640 1 -- 172.21.15.76:0/3536461772 shutdown_connections 2024-03-18T14:52:54.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.492+0000 7f497859d640 1 --2- 172.21.15.76:0/3536461772 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f495c0a2800 0x7f495c0a4cc0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.492+0000 7f497859d640 1 --2- 172.21.15.76:0/3536461772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49701029e0 0x7f497019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.492+0000 7f497859d640 1 --2- 172.21.15.76:0/3536461772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49701033a0 0x7f49701a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.492+0000 7f497859d640 1 --2- 172.21.15.76:0/3536461772 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49701017f0 0x7f4970199d40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.492+0000 7f497859d640 1 -- 172.21.15.76:0/3536461772 >> 172.21.15.76:0/3536461772 conn(0x7f49700fcd80 msgr2=0x7f4970073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:54.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.492+0000 7f497859d640 1 -- 172.21.15.76:0/3536461772 shutdown_connections 2024-03-18T14:52:54.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.492+0000 7f497859d640 1 -- 172.21.15.76:0/3536461772 wait complete. 2024-03-18T14:52:54.503 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:52:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.607+0000 7fce3a1b9640 1 -- 172.21.15.76:0/1285264712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fce341006e0 msgr2=0x7fce34100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.607+0000 7fce3a1b9640 1 --2- 172.21.15.76:0/1285264712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fce341006e0 0x7fce34100b00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fce24009900 tx=0x7fce2402bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.607+0000 7fce3a1b9640 1 -- 172.21.15.76:0/1285264712 shutdown_connections 2024-03-18T14:52:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.607+0000 7fce3a1b9640 1 --2- 172.21.15.76:0/1285264712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce341018d0 0x7fce34101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.607+0000 7fce3a1b9640 1 --2- 172.21.15.76:0/1285264712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce34102290 0x7fce34108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.607+0000 7fce3a1b9640 1 --2- 172.21.15.76:0/1285264712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fce341006e0 0x7fce34100b00 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.607+0000 7fce3a1b9640 1 -- 172.21.15.76:0/1285264712 >> 172.21.15.76:0/1285264712 conn(0x7fce340fbc50 msgr2=0x7fce340fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.607+0000 7fce3a1b9640 1 -- 172.21.15.76:0/1285264712 shutdown_connections 2024-03-18T14:52:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.607+0000 7fce3a1b9640 1 -- 172.21.15.76:0/1285264712 wait complete. 2024-03-18T14:52:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce3a1b9640 1 Processor -- start 2024-03-18T14:52:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce3a1b9640 1 -- start start 2024-03-18T14:52:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce3a1b9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fce341006e0 0x7fce34197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce3a1b9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce341018d0 0x7fce341980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce3a1b9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce34102290 0x7fce3419f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce3a1b9640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fce3410d480 con 0x7fce341006e0 2024-03-18T14:52:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce3a1b9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fce3410d300 con 0x7fce34102290 2024-03-18T14:52:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce3a1b9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fce3410d600 con 0x7fce341018d0 2024-03-18T14:52:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce337fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fce341006e0 0x7fce34197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce337fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fce341006e0 0x7fce34197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50978/0 (socket says 172.21.15.76:50978) 2024-03-18T14:52:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce337fe640 1 -- 172.21.15.76:0/2658997245 learned_addr learned my addr 172.21.15.76:0/2658997245 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce33fff640 1 --2- 172.21.15.76:0/2658997245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce34102290 0x7fce3419f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce2bfff640 1 --2- 172.21.15.76:0/2658997245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce341018d0 0x7fce341980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce337fe640 1 -- 172.21.15.76:0/2658997245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce341018d0 msgr2=0x7fce341980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce337fe640 1 --2- 172.21.15.76:0/2658997245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce341018d0 0x7fce341980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce337fe640 1 -- 172.21.15.76:0/2658997245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce34102290 msgr2=0x7fce3419f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce337fe640 1 --2- 172.21.15.76:0/2658997245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce34102290 0x7fce3419f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce337fe640 1 -- 172.21.15.76:0/2658997245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce3419f780 con 0x7fce341006e0 2024-03-18T14:52:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.608+0000 7fce337fe640 1 --2- 172.21.15.76:0/2658997245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fce341006e0 0x7fce34197b60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fce24000c00 tx=0x7fce2402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.609+0000 7fce317fa640 1 -- 172.21.15.76:0/2658997245 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce24041070 con 0x7fce341006e0 2024-03-18T14:52:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.609+0000 7fce317fa640 1 -- 172.21.15.76:0/2658997245 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce24004450 con 0x7fce341006e0 2024-03-18T14:52:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.609+0000 7fce317fa640 1 -- 172.21.15.76:0/2658997245 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce24034440 con 0x7fce341006e0 2024-03-18T14:52:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.609+0000 7fce3a1b9640 1 -- 172.21.15.76:0/2658997245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce3419fa10 con 0x7fce341006e0 2024-03-18T14:52:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.609+0000 7fce3a1b9640 1 -- 172.21.15.76:0/2658997245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce3419ff50 con 0x7fce341006e0 2024-03-18T14:52:54.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.610+0000 7fce3a1b9640 1 -- 172.21.15.76:0/2658997245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce34107a80 con 0x7fce341006e0 2024-03-18T14:52:54.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.611+0000 7fce317fa640 1 -- 172.21.15.76:0/2658997245 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fce24044070 con 0x7fce341006e0 2024-03-18T14:52:54.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.611+0000 7fce317fa640 1 --2- 172.21.15.76:0/2658997245 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fce200a2880 0x7fce200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:54.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.611+0000 7fce2bfff640 1 --2- 172.21.15.76:0/2658997245 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fce200a2880 0x7fce200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:54.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.611+0000 7fce317fa640 1 -- 172.21.15.76:0/2658997245 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fce240f0680 con 0x7fce341006e0 2024-03-18T14:52:54.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.612+0000 7fce2bfff640 1 --2- 172.21.15.76:0/2658997245 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fce200a2880 0x7fce200a4d40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fce34199070 tx=0x7fce1c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:54.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.615+0000 7fce317fa640 1 -- 172.21.15.76:0/2658997245 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fce240bd190 con 0x7fce341006e0 2024-03-18T14:52:54.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:54 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3162359530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:54.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:54 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1052822675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:52:54.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:54 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3162359530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:54.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:54 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1052822675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:52:54.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.790+0000 7fce3a1b9640 1 -- 172.21.15.76:0/2658997245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fce3402d1b0 con 0x7fce341006e0 2024-03-18T14:52:54.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.790+0000 7fce317fa640 1 -- 172.21.15.76:0/2658997245 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fce240c1230 con 0x7fce341006e0 2024-03-18T14:52:54.791 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-18T14:52:54.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.792+0000 7fce3a1b9640 1 -- 172.21.15.76:0/2658997245 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fce200a2880 msgr2=0x7fce200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.792+0000 7fce3a1b9640 1 --2- 172.21.15.76:0/2658997245 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fce200a2880 0x7fce200a4d40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fce34199070 tx=0x7fce1c005f90 comp rx=0 tx=0).stop 2024-03-18T14:52:54.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.792+0000 7fce3a1b9640 1 -- 172.21.15.76:0/2658997245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fce341006e0 msgr2=0x7fce34197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.792+0000 7fce3a1b9640 1 --2- 172.21.15.76:0/2658997245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fce341006e0 0x7fce34197b60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fce24000c00 tx=0x7fce2402c640 comp rx=0 tx=0).stop 2024-03-18T14:52:54.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.792+0000 7fce3a1b9640 1 -- 172.21.15.76:0/2658997245 shutdown_connections 2024-03-18T14:52:54.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.792+0000 7fce3a1b9640 1 --2- 172.21.15.76:0/2658997245 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fce200a2880 0x7fce200a4d40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.792+0000 7fce3a1b9640 1 --2- 172.21.15.76:0/2658997245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce341018d0 0x7fce341980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.792+0000 7fce3a1b9640 1 --2- 172.21.15.76:0/2658997245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce34102290 0x7fce3419f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.792+0000 7fce3a1b9640 1 --2- 172.21.15.76:0/2658997245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fce341006e0 0x7fce34197b60 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.792+0000 7fce3a1b9640 1 -- 172.21.15.76:0/2658997245 >> 172.21.15.76:0/2658997245 conn(0x7fce340fbc50 msgr2=0x7fce34104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:54.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.792+0000 7fce3a1b9640 1 -- 172.21.15.76:0/2658997245 shutdown_connections 2024-03-18T14:52:54.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.792+0000 7fce3a1b9640 1 -- 172.21.15.76:0/2658997245 wait complete. 2024-03-18T14:52:54.803 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:54 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3162359530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:54.803 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:54 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1052822675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:52:54.804 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:52:54.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.905+0000 7f6b9f93b640 1 -- 172.21.15.76:0/3271595727 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b98104c30 msgr2=0x7f6b98107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.905+0000 7f6b9f93b640 1 --2- 172.21.15.76:0/3271595727 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b98104c30 0x7f6b98107020 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6b88009900 tx=0x7f6b8802bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:54.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.905+0000 7f6b9f93b640 1 -- 172.21.15.76:0/3271595727 shutdown_connections 2024-03-18T14:52:54.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.905+0000 7f6b9f93b640 1 --2- 172.21.15.76:0/3271595727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b98107560 0x7f6b981099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.905+0000 7f6b9f93b640 1 --2- 172.21.15.76:0/3271595727 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b98104c30 0x7f6b98107020 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.905+0000 7f6b9f93b640 1 --2- 172.21.15.76:0/3271595727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b98102300 0x7f6b981046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.905+0000 7f6b9f93b640 1 -- 172.21.15.76:0/3271595727 >> 172.21.15.76:0/3271595727 conn(0x7f6b980fbd10 msgr2=0x7f6b980fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:54.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.905+0000 7f6b9f93b640 1 -- 172.21.15.76:0/3271595727 shutdown_connections 2024-03-18T14:52:54.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.905+0000 7f6b9f93b640 1 -- 172.21.15.76:0/3271595727 wait complete. 2024-03-18T14:52:54.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9f93b640 1 Processor -- start 2024-03-18T14:52:54.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9f93b640 1 -- start start 2024-03-18T14:52:54.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9f93b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b98102300 0x7f6b98197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:54.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9f93b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b98104c30 0x7f6b98198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:54.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9f93b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b98107560 0x7f6b9819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:54.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9f93b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9810d4a0 con 0x7f6b98102300 2024-03-18T14:52:54.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9f93b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9810d320 con 0x7f6b98107560 2024-03-18T14:52:54.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9f93b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9810d620 con 0x7f6b98104c30 2024-03-18T14:52:54.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9d6d7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b98102300 0x7f6b98197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:54.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9d6d7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b98102300 0x7f6b98197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50988/0 (socket says 172.21.15.76:50988) 2024-03-18T14:52:54.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9d6d7640 1 -- 172.21.15.76:0/220099276 learned_addr learned my addr 172.21.15.76:0/220099276 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:54.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9ded8640 1 --2- 172.21.15.76:0/220099276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b98107560 0x7f6b9819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:54.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9ced6640 1 --2- 172.21.15.76:0/220099276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b98104c30 0x7f6b98198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:54.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9d6d7640 1 -- 172.21.15.76:0/220099276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b98104c30 msgr2=0x7f6b98198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9d6d7640 1 --2- 172.21.15.76:0/220099276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b98104c30 0x7f6b98198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9d6d7640 1 -- 172.21.15.76:0/220099276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b98107560 msgr2=0x7f6b9819f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:54.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9d6d7640 1 --2- 172.21.15.76:0/220099276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b98107560 0x7f6b9819f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:54.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9d6d7640 1 -- 172.21.15.76:0/220099276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9819f760 con 0x7f6b98102300 2024-03-18T14:52:54.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9ced6640 1 --2- 172.21.15.76:0/220099276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b98104c30 0x7f6b98198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:52:54.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9d6d7640 1 --2- 172.21.15.76:0/220099276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b98102300 0x7f6b98197b40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400e9f0 tx=0x7f6b9400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:54.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b8e7fc640 1 -- 172.21.15.76:0/220099276 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9400ccc0 con 0x7f6b98102300 2024-03-18T14:52:54.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b8e7fc640 1 -- 172.21.15.76:0/220099276 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b9400ce60 con 0x7f6b98102300 2024-03-18T14:52:54.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9f93b640 1 -- 172.21.15.76:0/220099276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b9819fa50 con 0x7f6b98102300 2024-03-18T14:52:54.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b8e7fc640 1 -- 172.21.15.76:0/220099276 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b94010690 con 0x7f6b98102300 2024-03-18T14:52:54.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.906+0000 7f6b9f93b640 1 -- 172.21.15.76:0/220099276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b9819ff20 con 0x7f6b98102300 2024-03-18T14:52:54.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.907+0000 7f6b9f93b640 1 -- 172.21.15.76:0/220099276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b980fd800 con 0x7f6b98102300 2024-03-18T14:52:54.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.909+0000 7f6b8e7fc640 1 -- 172.21.15.76:0/220099276 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6b940046c0 con 0x7f6b98102300 2024-03-18T14:52:54.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.909+0000 7f6b8e7fc640 1 --2- 172.21.15.76:0/220099276 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6b840a2800 0x7f6b840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:54.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.909+0000 7f6b9ced6640 1 --2- 172.21.15.76:0/220099276 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6b840a2800 0x7f6b840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:54.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.909+0000 7f6b8e7fc640 1 -- 172.21.15.76:0/220099276 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6b940cf080 con 0x7f6b98102300 2024-03-18T14:52:54.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.910+0000 7f6b9ced6640 1 --2- 172.21.15.76:0/220099276 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6b840a2800 0x7f6b840a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6b8802c7b0 tx=0x7f6b88005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:54.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:54.913+0000 7f6b8e7fc640 1 -- 172.21.15.76:0/220099276 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b94053740 con 0x7f6b98102300 2024-03-18T14:52:55.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.125+0000 7f6b9f93b640 1 -- 172.21.15.76:0/220099276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b9802d1b0 con 0x7f6b98102300 2024-03-18T14:52:55.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.126+0000 7f6b8e7fc640 1 -- 172.21.15.76:0/220099276 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11863 (secure 0 0 0) 0x7f6b9409c150 con 0x7f6b98102300 2024-03-18T14:52:55.127 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:55.127 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/2801069369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":2801069369},{"type":"v1","addr":"172.21.15.76:6839","nonce":2801069369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2645229232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2645229232},{"type":"v1","addr":"172.21.15.76:6841","nonce":2645229232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T14:51:01.293009+0000","modified":"2024-03-18T14:51:07.032425+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24490,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6839/399810652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":399810652},{"type":"v1","addr":"172.21.15.174:6839","nonce":399810652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1685041985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1685041985},{"type":"v1","addr":"172.21.15.77:6839","nonce":1685041985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24463,24490,24524]},"id":1}]} 2024-03-18T14:52:55.127 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 23 2024-03-18T14:52:55.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.128+0000 7f6b9f93b640 1 -- 172.21.15.76:0/220099276 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6b840a2800 msgr2=0x7f6b840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:55.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.128+0000 7f6b9f93b640 1 --2- 172.21.15.76:0/220099276 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6b840a2800 0x7f6b840a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6b8802c7b0 tx=0x7f6b88005fd0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.128+0000 7f6b9f93b640 1 -- 172.21.15.76:0/220099276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b98102300 msgr2=0x7f6b98197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:55.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.128+0000 7f6b9f93b640 1 --2- 172.21.15.76:0/220099276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b98102300 0x7f6b98197b40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400e9f0 tx=0x7f6b9400edf0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.128+0000 7f6b9f93b640 1 -- 172.21.15.76:0/220099276 shutdown_connections 2024-03-18T14:52:55.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.128+0000 7f6b9f93b640 1 --2- 172.21.15.76:0/220099276 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6b840a2800 0x7f6b840a4cc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.128+0000 7f6b9f93b640 1 --2- 172.21.15.76:0/220099276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b98104c30 0x7f6b98198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.128+0000 7f6b9f93b640 1 --2- 172.21.15.76:0/220099276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b98107560 0x7f6b9819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.128+0000 7f6b9f93b640 1 --2- 172.21.15.76:0/220099276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b98102300 0x7f6b98197b40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.128+0000 7f6b9f93b640 1 -- 172.21.15.76:0/220099276 >> 172.21.15.76:0/220099276 conn(0x7f6b980fbd10 msgr2=0x7f6b98102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:55.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.128+0000 7f6b9f93b640 1 -- 172.21.15.76:0/220099276 shutdown_connections 2024-03-18T14:52:55.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.128+0000 7f6b9f93b640 1 -- 172.21.15.76:0/220099276 wait complete. 2024-03-18T14:52:55.140 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:52:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.241+0000 7fda0d3ff640 1 -- 172.21.15.76:0/3737866504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda08102100 msgr2=0x7fda08108990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.241+0000 7fda0d3ff640 1 --2- 172.21.15.76:0/3737866504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda08102100 0x7fda08108990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fda00009be0 tx=0x7fda0002bef0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.241+0000 7fda0d3ff640 1 -- 172.21.15.76:0/3737866504 shutdown_connections 2024-03-18T14:52:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.241+0000 7fda0d3ff640 1 --2- 172.21.15.76:0/3737866504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda08101740 0x7fda08101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.241+0000 7fda0d3ff640 1 --2- 172.21.15.76:0/3737866504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda08102100 0x7fda08108990 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.241+0000 7fda0d3ff640 1 --2- 172.21.15.76:0/3737866504 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda08100550 0x7fda08100970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.241+0000 7fda0d3ff640 1 -- 172.21.15.76:0/3737866504 >> 172.21.15.76:0/3737866504 conn(0x7fda080fbb00 msgr2=0x7fda080fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.241+0000 7fda0d3ff640 1 -- 172.21.15.76:0/3737866504 shutdown_connections 2024-03-18T14:52:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.241+0000 7fda0d3ff640 1 -- 172.21.15.76:0/3737866504 wait complete. 2024-03-18T14:52:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.242+0000 7fda0d3ff640 1 Processor -- start 2024-03-18T14:52:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.242+0000 7fda0d3ff640 1 -- start start 2024-03-18T14:52:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.242+0000 7fda0d3ff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda08100550 0x7fda08197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.242+0000 7fda0d3ff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda08101740 0x7fda08197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.242+0000 7fda0d3ff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda08102100 0x7fda0819eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.242+0000 7fda0d3ff640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fda0810c1e0 con 0x7fda08101740 2024-03-18T14:52:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.242+0000 7fda0d3ff640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fda0810c060 con 0x7fda08102100 2024-03-18T14:52:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.242+0000 7fda0d3ff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fda0810c360 con 0x7fda08100550 2024-03-18T14:52:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.242+0000 7fda067fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda08101740 0x7fda08197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.242+0000 7fda067fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda08101740 0x7fda08197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51006/0 (socket says 172.21.15.76:51006) 2024-03-18T14:52:55.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.242+0000 7fda067fc640 1 -- 172.21.15.76:0/1691584037 learned_addr learned my addr 172.21.15.76:0/1691584037 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:55.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.243+0000 7fda077fe640 1 --2- 172.21.15.76:0/1691584037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda08102100 0x7fda0819eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:55.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.243+0000 7fda06ffd640 1 --2- 172.21.15.76:0/1691584037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda08100550 0x7fda08197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:55.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.243+0000 7fda067fc640 1 -- 172.21.15.76:0/1691584037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda08100550 msgr2=0x7fda08197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:55.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.243+0000 7fda067fc640 1 --2- 172.21.15.76:0/1691584037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda08100550 0x7fda08197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.243+0000 7fda067fc640 1 -- 172.21.15.76:0/1691584037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda08102100 msgr2=0x7fda0819eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:55.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.243+0000 7fda067fc640 1 --2- 172.21.15.76:0/1691584037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda08102100 0x7fda0819eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.243+0000 7fda067fc640 1 -- 172.21.15.76:0/1691584037 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda0819f540 con 0x7fda08101740 2024-03-18T14:52:55.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.243+0000 7fda077fe640 1 --2- 172.21.15.76:0/1691584037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda08102100 0x7fda0819eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:55.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.243+0000 7fda067fc640 1 --2- 172.21.15.76:0/1691584037 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda08101740 0x7fda08197e60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd9f800e9f0 tx=0x7fd9f800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:55.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.243+0000 7fd9effff640 1 -- 172.21.15.76:0/1691584037 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f800cd80 con 0x7fda08101740 2024-03-18T14:52:55.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.243+0000 7fda0d3ff640 1 -- 172.21.15.76:0/1691584037 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda0819f830 con 0x7fda08101740 2024-03-18T14:52:55.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.243+0000 7fd9effff640 1 -- 172.21.15.76:0/1691584037 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9f8005500 con 0x7fda08101740 2024-03-18T14:52:55.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.243+0000 7fd9effff640 1 -- 172.21.15.76:0/1691584037 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f8010690 con 0x7fda08101740 2024-03-18T14:52:55.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.243+0000 7fda0d3ff640 1 -- 172.21.15.76:0/1691584037 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda0819fd30 con 0x7fda08101740 2024-03-18T14:52:55.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.244+0000 7fda0d3ff640 1 -- 172.21.15.76:0/1691584037 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda0819fff0 con 0x7fda08101740 2024-03-18T14:52:55.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.245+0000 7fd9effff640 1 -- 172.21.15.76:0/1691584037 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd9f80046c0 con 0x7fda08101740 2024-03-18T14:52:55.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.246+0000 7fd9effff640 1 --2- 172.21.15.76:0/1691584037 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd9f00a2800 0x7fd9f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:55.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.246+0000 7fda06ffd640 1 --2- 172.21.15.76:0/1691584037 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd9f00a2800 0x7fd9f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:55.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.246+0000 7fd9effff640 1 -- 172.21.15.76:0/1691584037 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fd9f80cee20 con 0x7fda08101740 2024-03-18T14:52:55.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.246+0000 7fda06ffd640 1 --2- 172.21.15.76:0/1691584037 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd9f00a2800 0x7fd9f00a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009530 tx=0x7fd9fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:55.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.250+0000 7fd9effff640 1 -- 172.21.15.76:0/1691584037 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd9f809bef0 con 0x7fda08101740 2024-03-18T14:52:55.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.428+0000 7fda0d3ff640 1 -- 172.21.15.76:0/1691584037 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda081a01c0 con 0x7fda08101740 2024-03-18T14:52:55.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.430+0000 7fd9effff640 1 -- 172.21.15.76:0/1691584037 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22580 (secure 0 0 0) 0x7fd9f809b7f0 con 0x7fda08101740 2024-03-18T14:52:55.431 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:55.431 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:52:07.361301+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:52:55.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.432+0000 7fda0d3ff640 1 -- 172.21.15.76:0/1691584037 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd9f00a2800 msgr2=0x7fd9f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:55.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.432+0000 7fda0d3ff640 1 --2- 172.21.15.76:0/1691584037 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd9f00a2800 0x7fd9f00a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009530 tx=0x7fd9fc009260 comp rx=0 tx=0).stop 2024-03-18T14:52:55.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.432+0000 7fda0d3ff640 1 -- 172.21.15.76:0/1691584037 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda08101740 msgr2=0x7fda08197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:55.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.432+0000 7fda0d3ff640 1 --2- 172.21.15.76:0/1691584037 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda08101740 0x7fda08197e60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd9f800e9f0 tx=0x7fd9f800edf0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.432+0000 7fda0d3ff640 1 -- 172.21.15.76:0/1691584037 shutdown_connections 2024-03-18T14:52:55.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.432+0000 7fda0d3ff640 1 --2- 172.21.15.76:0/1691584037 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd9f00a2800 0x7fd9f00a4cc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.432+0000 7fda0d3ff640 1 --2- 172.21.15.76:0/1691584037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda08102100 0x7fda0819eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.432+0000 7fda0d3ff640 1 --2- 172.21.15.76:0/1691584037 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda08101740 0x7fda08197e60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.432+0000 7fda0d3ff640 1 --2- 172.21.15.76:0/1691584037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda08100550 0x7fda08197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.432+0000 7fda0d3ff640 1 -- 172.21.15.76:0/1691584037 >> 172.21.15.76:0/1691584037 conn(0x7fda080fbb00 msgr2=0x7fda080fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:55.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.432+0000 7fda0d3ff640 1 -- 172.21.15.76:0/1691584037 shutdown_connections 2024-03-18T14:52:55.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.432+0000 7fda0d3ff640 1 -- 172.21.15.76:0/1691584037 wait complete. 2024-03-18T14:52:55.444 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:46:26.588384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:50:59.606852+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:51:00.465325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:52:55.444 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:52:55.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.548+0000 7f89fa9cd640 1 -- 172.21.15.76:0/1799919527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89f41006f0 msgr2=0x7f89f4100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:55.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.548+0000 7f89fa9cd640 1 --2- 172.21.15.76:0/1799919527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89f41006f0 0x7f89f4100b10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f89e8009900 tx=0x7f89e802bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:55.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.548+0000 7f89fa9cd640 1 -- 172.21.15.76:0/1799919527 shutdown_connections 2024-03-18T14:52:55.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.548+0000 7f89fa9cd640 1 --2- 172.21.15.76:0/1799919527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89f41022a0 0x7f89f4108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.548+0000 7f89fa9cd640 1 --2- 172.21.15.76:0/1799919527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89f41018e0 0x7f89f4101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.548+0000 7f89fa9cd640 1 --2- 172.21.15.76:0/1799919527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89f41006f0 0x7f89f4100b10 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.548+0000 7f89fa9cd640 1 -- 172.21.15.76:0/1799919527 >> 172.21.15.76:0/1799919527 conn(0x7f89f40fbc80 msgr2=0x7f89f40fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:55.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.548+0000 7f89fa9cd640 1 -- 172.21.15.76:0/1799919527 shutdown_connections 2024-03-18T14:52:55.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.548+0000 7f89fa9cd640 1 -- 172.21.15.76:0/1799919527 wait complete. 2024-03-18T14:52:55.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89fa9cd640 1 Processor -- start 2024-03-18T14:52:55.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89fa9cd640 1 -- start start 2024-03-18T14:52:55.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89fa9cd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89f41006f0 0x7f89f406a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89fa9cd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89f41018e0 0x7f89f419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89fa9cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89f41022a0 0x7f89f419eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89fa9cd640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f89f410c370 con 0x7f89f41006f0 2024-03-18T14:52:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89fa9cd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f89f410c1f0 con 0x7f89f41022a0 2024-03-18T14:52:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89fa9cd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f89f410c4f0 con 0x7f89f41018e0 2024-03-18T14:52:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89f3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89f41006f0 0x7f89f406a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89f3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89f41006f0 0x7f89f406a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51016/0 (socket says 172.21.15.76:51016) 2024-03-18T14:52:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89f3fff640 1 -- 172.21.15.76:0/1840717422 learned_addr learned my addr 172.21.15.76:0/1840717422 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89f37fe640 1 --2- 172.21.15.76:0/1840717422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89f41018e0 0x7f89f419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89f8f6a640 1 --2- 172.21.15.76:0/1840717422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89f41022a0 0x7f89f419eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89f3fff640 1 -- 172.21.15.76:0/1840717422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89f41018e0 msgr2=0x7f89f419c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89f3fff640 1 --2- 172.21.15.76:0/1840717422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89f41018e0 0x7f89f419c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89f3fff640 1 -- 172.21.15.76:0/1840717422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89f41022a0 msgr2=0x7f89f419eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89f3fff640 1 --2- 172.21.15.76:0/1840717422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89f41022a0 0x7f89f419eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89f3fff640 1 -- 172.21.15.76:0/1840717422 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89f419f7b0 con 0x7f89f41006f0 2024-03-18T14:52:55.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.549+0000 7f89f3fff640 1 --2- 172.21.15.76:0/1840717422 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89f41006f0 0x7f89f406a920 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f89e8000c00 tx=0x7f89e802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:55.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.550+0000 7f89f17fa640 1 -- 172.21.15.76:0/1840717422 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e8041070 con 0x7f89f41006f0 2024-03-18T14:52:55.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.550+0000 7f89fa9cd640 1 -- 172.21.15.76:0/1840717422 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89f419f9e0 con 0x7f89f41006f0 2024-03-18T14:52:55.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.550+0000 7f89f17fa640 1 -- 172.21.15.76:0/1840717422 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89e8004450 con 0x7f89f41006f0 2024-03-18T14:52:55.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.550+0000 7f89f17fa640 1 -- 172.21.15.76:0/1840717422 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e8034440 con 0x7f89f41006f0 2024-03-18T14:52:55.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.550+0000 7f89fa9cd640 1 -- 172.21.15.76:0/1840717422 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89f419ff20 con 0x7f89f41006f0 2024-03-18T14:52:55.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.551+0000 7f89fa9cd640 1 -- 172.21.15.76:0/1840717422 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89f4107ae0 con 0x7f89f41006f0 2024-03-18T14:52:55.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.552+0000 7f89f17fa640 1 -- 172.21.15.76:0/1840717422 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f89e8044070 con 0x7f89f41006f0 2024-03-18T14:52:55.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.552+0000 7f89f17fa640 1 --2- 172.21.15.76:0/1840717422 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89dc0a2800 0x7f89dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:55.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.552+0000 7f89f37fe640 1 --2- 172.21.15.76:0/1840717422 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89dc0a2800 0x7f89dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:55.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.552+0000 7f89f17fa640 1 -- 172.21.15.76:0/1840717422 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f89e80f0700 con 0x7f89f41006f0 2024-03-18T14:52:55.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.553+0000 7f89f37fe640 1 --2- 172.21.15.76:0/1840717422 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89dc0a2800 0x7f89dc0a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f89f406a690 tx=0x7f89e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:55.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.556+0000 7f89f17fa640 1 -- 172.21.15.76:0/1840717422 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f89e80bd210 con 0x7f89f41006f0 2024-03-18T14:52:55.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:55 smithi174 ceph-mon[36787]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:55.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:55 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3536461772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:52:55.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:55 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2658997245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:52:55.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:55 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/220099276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:52:55.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:55 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1691584037' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:55.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:55 smithi077 ceph-mon[35865]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:55.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:55 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3536461772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:52:55.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:55 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2658997245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:52:55.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:55 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/220099276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:52:55.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:55 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1691584037' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:55.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.731+0000 7f89fa9cd640 1 -- 172.21.15.76:0/1840717422 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f89f402d340 con 0x7f89f41006f0 2024-03-18T14:52:55.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.731+0000 7f89f17fa640 1 -- 172.21.15.76:0/1840717422 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f89e80c12b0 con 0x7f89f41006f0 2024-03-18T14:52:55.732 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-18T14:52:55.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.733+0000 7f89fa9cd640 1 -- 172.21.15.76:0/1840717422 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89dc0a2800 msgr2=0x7f89dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:55.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.733+0000 7f89fa9cd640 1 --2- 172.21.15.76:0/1840717422 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89dc0a2800 0x7f89dc0a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f89f406a690 tx=0x7f89e4005f90 comp rx=0 tx=0).stop 2024-03-18T14:52:55.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.733+0000 7f89fa9cd640 1 -- 172.21.15.76:0/1840717422 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89f41006f0 msgr2=0x7f89f406a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:55.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.733+0000 7f89fa9cd640 1 --2- 172.21.15.76:0/1840717422 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89f41006f0 0x7f89f406a920 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f89e8000c00 tx=0x7f89e802c640 comp rx=0 tx=0).stop 2024-03-18T14:52:55.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.734+0000 7f89fa9cd640 1 -- 172.21.15.76:0/1840717422 shutdown_connections 2024-03-18T14:52:55.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.734+0000 7f89fa9cd640 1 --2- 172.21.15.76:0/1840717422 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f89dc0a2800 0x7f89dc0a4cc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.734+0000 7f89fa9cd640 1 --2- 172.21.15.76:0/1840717422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89f41018e0 0x7f89f419c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.734+0000 7f89fa9cd640 1 --2- 172.21.15.76:0/1840717422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89f41022a0 0x7f89f419eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.734+0000 7f89fa9cd640 1 --2- 172.21.15.76:0/1840717422 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89f41006f0 0x7f89f406a920 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.734+0000 7f89fa9cd640 1 -- 172.21.15.76:0/1840717422 >> 172.21.15.76:0/1840717422 conn(0x7f89f40fbc80 msgr2=0x7f89f4104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:55.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.734+0000 7f89fa9cd640 1 -- 172.21.15.76:0/1840717422 shutdown_connections 2024-03-18T14:52:55.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.734+0000 7f89fa9cd640 1 -- 172.21.15.76:0/1840717422 wait complete. 2024-03-18T14:52:55.744 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:55 smithi076 ceph-mon[32205]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:52:55.744 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:55 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3536461772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:52:55.745 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:55 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2658997245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:52:55.745 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:55 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/220099276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:52:55.745 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:55 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1691584037' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:55.746 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:52:55.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.849+0000 7f1863836640 1 -- 172.21.15.76:0/2012465524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f185c1018d0 msgr2=0x7f185c101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:55.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.849+0000 7f1863836640 1 --2- 172.21.15.76:0/2012465524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f185c1018d0 0x7f185c101d50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f184c009900 tx=0x7f184c02bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:55.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.849+0000 7f1863836640 1 -- 172.21.15.76:0/2012465524 shutdown_connections 2024-03-18T14:52:55.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.849+0000 7f1863836640 1 --2- 172.21.15.76:0/2012465524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f185c102290 0x7f185c108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.849+0000 7f1863836640 1 --2- 172.21.15.76:0/2012465524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f185c1018d0 0x7f185c101d50 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.849+0000 7f1863836640 1 --2- 172.21.15.76:0/2012465524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c1006e0 0x7f185c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.849+0000 7f1863836640 1 -- 172.21.15.76:0/2012465524 >> 172.21.15.76:0/2012465524 conn(0x7f185c0fbc70 msgr2=0x7f185c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:55.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.850+0000 7f1863836640 1 -- 172.21.15.76:0/2012465524 shutdown_connections 2024-03-18T14:52:55.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.850+0000 7f1863836640 1 -- 172.21.15.76:0/2012465524 wait complete. 2024-03-18T14:52:55.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.850+0000 7f1863836640 1 Processor -- start 2024-03-18T14:52:55.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.850+0000 7f1863836640 1 -- start start 2024-03-18T14:52:55.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.850+0000 7f1863836640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f185c1006e0 0x7f185c196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:55.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.850+0000 7f1863836640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f185c1018d0 0x7f185c196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:55.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.850+0000 7f1863836640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c102290 0x7f185c19dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:55.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.850+0000 7f1863836640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f185c10c360 con 0x7f185c1006e0 2024-03-18T14:52:55.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.850+0000 7f1863836640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f185c10c1e0 con 0x7f185c102290 2024-03-18T14:52:55.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.850+0000 7f1863836640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f185c10c4e0 con 0x7f185c1018d0 2024-03-18T14:52:55.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.850+0000 7f18615d2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f185c1006e0 0x7f185c196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:55.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.850+0000 7f18615d2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f185c1006e0 0x7f185c196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51026/0 (socket says 172.21.15.76:51026) 2024-03-18T14:52:55.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.850+0000 7f18615d2640 1 -- 172.21.15.76:0/814280969 learned_addr learned my addr 172.21.15.76:0/814280969 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:55.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.851+0000 7f1861dd3640 1 --2- 172.21.15.76:0/814280969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c102290 0x7f185c19dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:55.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.851+0000 7f1860dd1640 1 --2- 172.21.15.76:0/814280969 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f185c1018d0 0x7f185c196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:55.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.851+0000 7f18615d2640 1 -- 172.21.15.76:0/814280969 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f185c1018d0 msgr2=0x7f185c196f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:55.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.851+0000 7f18615d2640 1 --2- 172.21.15.76:0/814280969 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f185c1018d0 0x7f185c196f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.851+0000 7f18615d2640 1 -- 172.21.15.76:0/814280969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c102290 msgr2=0x7f185c19dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:55.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.851+0000 7f18615d2640 1 --2- 172.21.15.76:0/814280969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c102290 0x7f185c19dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:55.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.851+0000 7f18615d2640 1 -- 172.21.15.76:0/814280969 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f185c19e620 con 0x7f185c1006e0 2024-03-18T14:52:55.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.851+0000 7f1861dd3640 1 --2- 172.21.15.76:0/814280969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c102290 0x7f185c19dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:52:55.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.851+0000 7f18615d2640 1 --2- 172.21.15.76:0/814280969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f185c1006e0 0x7f185c196a00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f185800e9c0 tx=0x7f185800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:55.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.851+0000 7f18527fc640 1 -- 172.21.15.76:0/814280969 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185800cd80 con 0x7f185c1006e0 2024-03-18T14:52:55.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.851+0000 7f1863836640 1 -- 172.21.15.76:0/814280969 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f185c19e910 con 0x7f185c1006e0 2024-03-18T14:52:55.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.851+0000 7f18527fc640 1 -- 172.21.15.76:0/814280969 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18580035f0 con 0x7f185c1006e0 2024-03-18T14:52:55.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.851+0000 7f18527fc640 1 -- 172.21.15.76:0/814280969 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1858010690 con 0x7f185c1006e0 2024-03-18T14:52:55.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.851+0000 7f1863836640 1 -- 172.21.15.76:0/814280969 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f185c19ee30 con 0x7f185c1006e0 2024-03-18T14:52:55.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.852+0000 7f1863836640 1 -- 172.21.15.76:0/814280969 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f185c0685d0 con 0x7f185c1006e0 2024-03-18T14:52:55.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.853+0000 7f18527fc640 1 -- 172.21.15.76:0/814280969 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f18580037a0 con 0x7f185c1006e0 2024-03-18T14:52:55.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.853+0000 7f18527fc640 1 --2- 172.21.15.76:0/814280969 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f18480a2800 0x7f18480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:55.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.854+0000 7f1860dd1640 1 --2- 172.21.15.76:0/814280969 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f18480a2800 0x7f18480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:55.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.854+0000 7f18527fc640 1 -- 172.21.15.76:0/814280969 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f18580cef80 con 0x7f185c1006e0 2024-03-18T14:52:55.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.854+0000 7f1860dd1640 1 --2- 172.21.15.76:0/814280969 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f18480a2800 0x7f18480a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f184c006010 tx=0x7f184c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:55.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:55.857+0000 7f18527fc640 1 -- 172.21.15.76:0/814280969 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f185809c180 con 0x7f185c1006e0 2024-03-18T14:52:56.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.033+0000 7f1863836640 1 -- 172.21.15.76:0/814280969 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f185c02d270 con 0x7f185c1006e0 2024-03-18T14:52:56.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.034+0000 7f18527fc640 1 -- 172.21.15.76:0/814280969 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f185809ba80 con 0x7f185c1006e0 2024-03-18T14:52:56.035 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-18T14:52:56.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.036+0000 7f1863836640 1 -- 172.21.15.76:0/814280969 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f18480a2800 msgr2=0x7f18480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:56.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.036+0000 7f1863836640 1 --2- 172.21.15.76:0/814280969 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f18480a2800 0x7f18480a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f184c006010 tx=0x7f184c000bc0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.036+0000 7f1863836640 1 -- 172.21.15.76:0/814280969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f185c1006e0 msgr2=0x7f185c196a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:56.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.036+0000 7f1863836640 1 --2- 172.21.15.76:0/814280969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f185c1006e0 0x7f185c196a00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f185800e9c0 tx=0x7f185800ee80 comp rx=0 tx=0).stop 2024-03-18T14:52:56.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.036+0000 7f1863836640 1 -- 172.21.15.76:0/814280969 shutdown_connections 2024-03-18T14:52:56.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.036+0000 7f1863836640 1 --2- 172.21.15.76:0/814280969 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f18480a2800 0x7f18480a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.036+0000 7f1863836640 1 --2- 172.21.15.76:0/814280969 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f185c1018d0 0x7f185c196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.036+0000 7f1863836640 1 --2- 172.21.15.76:0/814280969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c102290 0x7f185c19dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.036+0000 7f1863836640 1 --2- 172.21.15.76:0/814280969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f185c1006e0 0x7f185c196a00 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.036+0000 7f1863836640 1 -- 172.21.15.76:0/814280969 >> 172.21.15.76:0/814280969 conn(0x7f185c0fbc70 msgr2=0x7f185c104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:56.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.036+0000 7f1863836640 1 -- 172.21.15.76:0/814280969 shutdown_connections 2024-03-18T14:52:56.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.036+0000 7f1863836640 1 -- 172.21.15.76:0/814280969 wait complete. 2024-03-18T14:52:56.048 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:52:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.151+0000 7fc805777640 1 -- 172.21.15.76:0/797479067 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc800102290 msgr2=0x7fc800108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.151+0000 7fc805777640 1 --2- 172.21.15.76:0/797479067 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc800102290 0x7fc800108b20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8009900 tx=0x7fc7f802bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.151+0000 7fc805777640 1 -- 172.21.15.76:0/797479067 shutdown_connections 2024-03-18T14:52:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.151+0000 7fc805777640 1 --2- 172.21.15.76:0/797479067 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc800102290 0x7fc800108b20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.151+0000 7fc805777640 1 --2- 172.21.15.76:0/797479067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8001018d0 0x7fc800101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.151+0000 7fc805777640 1 --2- 172.21.15.76:0/797479067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8001006e0 0x7fc800100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.151+0000 7fc805777640 1 -- 172.21.15.76:0/797479067 >> 172.21.15.76:0/797479067 conn(0x7fc8000fbc70 msgr2=0x7fc8000fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.151+0000 7fc805777640 1 -- 172.21.15.76:0/797479067 shutdown_connections 2024-03-18T14:52:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.151+0000 7fc805777640 1 -- 172.21.15.76:0/797479067 wait complete. 2024-03-18T14:52:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.152+0000 7fc805777640 1 Processor -- start 2024-03-18T14:52:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.152+0000 7fc805777640 1 -- start start 2024-03-18T14:52:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.152+0000 7fc805777640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8001006e0 0x7fc800197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.152+0000 7fc805777640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8001018d0 0x7fc800198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.152+0000 7fc805777640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc800102290 0x7fc80019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.152+0000 7fc805777640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc80010d480 con 0x7fc8001018d0 2024-03-18T14:52:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.152+0000 7fc805777640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc80010d300 con 0x7fc8001006e0 2024-03-18T14:52:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.152+0000 7fc805777640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc80010d600 con 0x7fc800102290 2024-03-18T14:52:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.152+0000 7fc7fe7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8001018d0 0x7fc800198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.152+0000 7fc7fe7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8001018d0 0x7fc800198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51046/0 (socket says 172.21.15.76:51046) 2024-03-18T14:52:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.152+0000 7fc7fe7fc640 1 -- 172.21.15.76:0/1162777154 learned_addr learned my addr 172.21.15.76:0/1162777154 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.152+0000 7fc7ff7fe640 1 --2- 172.21.15.76:0/1162777154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc800102290 0x7fc80019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.152+0000 7fc7feffd640 1 --2- 172.21.15.76:0/1162777154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8001006e0 0x7fc800197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.153+0000 7fc7fe7fc640 1 -- 172.21.15.76:0/1162777154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc800102290 msgr2=0x7fc80019f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.153+0000 7fc7fe7fc640 1 --2- 172.21.15.76:0/1162777154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc800102290 0x7fc80019f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.153+0000 7fc7fe7fc640 1 -- 172.21.15.76:0/1162777154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8001006e0 msgr2=0x7fc800197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.153+0000 7fc7fe7fc640 1 --2- 172.21.15.76:0/1162777154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8001006e0 0x7fc800197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.153+0000 7fc7fe7fc640 1 -- 172.21.15.76:0/1162777154 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80019f6f0 con 0x7fc8001018d0 2024-03-18T14:52:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.153+0000 7fc7ff7fe640 1 --2- 172.21.15.76:0/1162777154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc800102290 0x7fc80019f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.153+0000 7fc7fe7fc640 1 --2- 172.21.15.76:0/1162777154 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8001018d0 0x7fc800198010 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc7f400e9f0 tx=0x7fc7f400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.153+0000 7fc7e7fff640 1 -- 172.21.15.76:0/1162777154 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f400cd80 con 0x7fc8001018d0 2024-03-18T14:52:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.153+0000 7fc7e7fff640 1 -- 172.21.15.76:0/1162777154 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7f4010910 con 0x7fc8001018d0 2024-03-18T14:52:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.153+0000 7fc805777640 1 -- 172.21.15.76:0/1162777154 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc80019f9e0 con 0x7fc8001018d0 2024-03-18T14:52:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.153+0000 7fc7e7fff640 1 -- 172.21.15.76:0/1162777154 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4018990 con 0x7fc8001018d0 2024-03-18T14:52:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.153+0000 7fc805777640 1 -- 172.21.15.76:0/1162777154 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc80019ff30 con 0x7fc8001018d0 2024-03-18T14:52:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.154+0000 7fc805777640 1 -- 172.21.15.76:0/1162777154 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc800107a00 con 0x7fc8001018d0 2024-03-18T14:52:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.155+0000 7fc7e7fff640 1 -- 172.21.15.76:0/1162777154 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc7f4010ac0 con 0x7fc8001018d0 2024-03-18T14:52:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.156+0000 7fc7e7fff640 1 --2- 172.21.15.76:0/1162777154 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc7e80a2800 0x7fc7e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.156+0000 7fc7feffd640 1 --2- 172.21.15.76:0/1162777154 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc7e80a2800 0x7fc7e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.156+0000 7fc7e7fff640 1 -- 172.21.15.76:0/1162777154 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc7f40ced60 con 0x7fc8001018d0 2024-03-18T14:52:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.156+0000 7fc7feffd640 1 --2- 172.21.15.76:0/1162777154 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc7e80a2800 0x7fc7e80a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc800101730 tx=0x7fc7f0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:56.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.160+0000 7fc7e7fff640 1 -- 172.21.15.76:0/1162777154 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc7f409b7e0 con 0x7fc8001018d0 2024-03-18T14:52:56.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.335+0000 7fc805777640 1 -- 172.21.15.76:0/1162777154 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc80002d1b0 con 0x7fc8001018d0 2024-03-18T14:52:56.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.336+0000 7fc7e7fff640 1 -- 172.21.15.76:0/1162777154 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fc7f409f880 con 0x7fc8001018d0 2024-03-18T14:52:56.336 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-18T14:52:56.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.338+0000 7fc805777640 1 -- 172.21.15.76:0/1162777154 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc7e80a2800 msgr2=0x7fc7e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:56.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.338+0000 7fc805777640 1 --2- 172.21.15.76:0/1162777154 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc7e80a2800 0x7fc7e80a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc800101730 tx=0x7fc7f0009500 comp rx=0 tx=0).stop 2024-03-18T14:52:56.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.338+0000 7fc805777640 1 -- 172.21.15.76:0/1162777154 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8001018d0 msgr2=0x7fc800198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:56.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.338+0000 7fc805777640 1 --2- 172.21.15.76:0/1162777154 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8001018d0 0x7fc800198010 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc7f400e9f0 tx=0x7fc7f400eeb0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.338+0000 7fc805777640 1 -- 172.21.15.76:0/1162777154 shutdown_connections 2024-03-18T14:52:56.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.338+0000 7fc805777640 1 --2- 172.21.15.76:0/1162777154 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc7e80a2800 0x7fc7e80a4cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.338+0000 7fc805777640 1 --2- 172.21.15.76:0/1162777154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc800102290 0x7fc80019f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.338+0000 7fc805777640 1 --2- 172.21.15.76:0/1162777154 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8001018d0 0x7fc800198010 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.338+0000 7fc805777640 1 --2- 172.21.15.76:0/1162777154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8001006e0 0x7fc800197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.338+0000 7fc805777640 1 -- 172.21.15.76:0/1162777154 >> 172.21.15.76:0/1162777154 conn(0x7fc8000fbc70 msgr2=0x7fc800104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:56.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.338+0000 7fc805777640 1 -- 172.21.15.76:0/1162777154 shutdown_connections 2024-03-18T14:52:56.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.338+0000 7fc805777640 1 -- 172.21.15.76:0/1162777154 wait complete. 2024-03-18T14:52:56.350 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:52:56.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.451+0000 7f2574a54640 1 -- 172.21.15.76:0/3123837828 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2570100700 msgr2=0x7f2570100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:56.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.451+0000 7f2574a54640 1 --2- 172.21.15.76:0/3123837828 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2570100700 0x7f2570100b20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2568009900 tx=0x7f256802bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:56.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.451+0000 7f2574a54640 1 -- 172.21.15.76:0/3123837828 shutdown_connections 2024-03-18T14:52:56.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.451+0000 7f2574a54640 1 --2- 172.21.15.76:0/3123837828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25701022b0 0x7f2570108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.451+0000 7f2574a54640 1 --2- 172.21.15.76:0/3123837828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25701018f0 0x7f2570101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.451+0000 7f2574a54640 1 --2- 172.21.15.76:0/3123837828 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2570100700 0x7f2570100b20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.451+0000 7f2574a54640 1 -- 172.21.15.76:0/3123837828 >> 172.21.15.76:0/3123837828 conn(0x7f25700fbc70 msgr2=0x7f25700fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:56.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.451+0000 7f2574a54640 1 -- 172.21.15.76:0/3123837828 shutdown_connections 2024-03-18T14:52:56.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.451+0000 7f2574a54640 1 -- 172.21.15.76:0/3123837828 wait complete. 2024-03-18T14:52:56.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.452+0000 7f2574a54640 1 Processor -- start 2024-03-18T14:52:56.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.452+0000 7f2574a54640 1 -- start start 2024-03-18T14:52:56.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.452+0000 7f2574a54640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2570100700 0x7f2570197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:56.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.452+0000 7f2574a54640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25701018f0 0x7f2570198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:56.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.452+0000 7f2574a54640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25701022b0 0x7f257019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:56.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.452+0000 7f2574a54640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f257010c380 con 0x7f25701018f0 2024-03-18T14:52:56.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.452+0000 7f2574a54640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f257010c200 con 0x7f2570100700 2024-03-18T14:52:56.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.452+0000 7f2574a54640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f257010c500 con 0x7f25701022b0 2024-03-18T14:52:56.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.452+0000 7f256dd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25701018f0 0x7f2570198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:56.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.453+0000 7f256dd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25701018f0 0x7f2570198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51060/0 (socket says 172.21.15.76:51060) 2024-03-18T14:52:56.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.453+0000 7f256dd9b640 1 -- 172.21.15.76:0/760175264 learned_addr learned my addr 172.21.15.76:0/760175264 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:56.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.453+0000 7f256e59c640 1 --2- 172.21.15.76:0/760175264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2570100700 0x7f2570197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:56.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.453+0000 7f256ed9d640 1 --2- 172.21.15.76:0/760175264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25701022b0 0x7f257019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:56.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.453+0000 7f256dd9b640 1 -- 172.21.15.76:0/760175264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25701022b0 msgr2=0x7f257019f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:56.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.453+0000 7f256dd9b640 1 --2- 172.21.15.76:0/760175264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25701022b0 0x7f257019f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.453+0000 7f256dd9b640 1 -- 172.21.15.76:0/760175264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2570100700 msgr2=0x7f2570197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:56.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.453+0000 7f256dd9b640 1 --2- 172.21.15.76:0/760175264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2570100700 0x7f2570197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.453+0000 7f256dd9b640 1 -- 172.21.15.76:0/760175264 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f257019f760 con 0x7f25701018f0 2024-03-18T14:52:56.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.453+0000 7f256e59c640 1 --2- 172.21.15.76:0/760175264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2570100700 0x7f2570197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:56.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.453+0000 7f256dd9b640 1 --2- 172.21.15.76:0/760175264 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25701018f0 0x7f2570198080 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2564009d50 tx=0x7f256400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:56.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.453+0000 7f255f7fe640 1 -- 172.21.15.76:0/760175264 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564003b70 con 0x7f25701018f0 2024-03-18T14:52:56.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.453+0000 7f255f7fe640 1 -- 172.21.15.76:0/760175264 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2564003d10 con 0x7f25701018f0 2024-03-18T14:52:56.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.453+0000 7f2574a54640 1 -- 172.21.15.76:0/760175264 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f257019fa50 con 0x7f25701018f0 2024-03-18T14:52:56.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.453+0000 7f255f7fe640 1 -- 172.21.15.76:0/760175264 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564010a20 con 0x7f25701018f0 2024-03-18T14:52:56.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.453+0000 7f2574a54640 1 -- 172.21.15.76:0/760175264 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f257019fff0 con 0x7f25701018f0 2024-03-18T14:52:56.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.454+0000 7f2574a54640 1 -- 172.21.15.76:0/760175264 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25701a0280 con 0x7f25701018f0 2024-03-18T14:52:56.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.455+0000 7f255f7fe640 1 -- 172.21.15.76:0/760175264 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2564010bc0 con 0x7f25701018f0 2024-03-18T14:52:56.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.456+0000 7f255f7fe640 1 --2- 172.21.15.76:0/760175264 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f25580a2880 0x7f25580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:56.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.456+0000 7f256e59c640 1 --2- 172.21.15.76:0/760175264 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f25580a2880 0x7f25580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:56.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.456+0000 7f255f7fe640 1 -- 172.21.15.76:0/760175264 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f25640cf120 con 0x7f25701018f0 2024-03-18T14:52:56.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.456+0000 7f256e59c640 1 --2- 172.21.15.76:0/760175264 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f25580a2880 0x7f25580a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2568000c00 tx=0x7f2568005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:56.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.459+0000 7f255f7fe640 1 -- 172.21.15.76:0/760175264 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f256409c280 con 0x7f25701018f0 2024-03-18T14:52:56.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:56 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1840717422' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:52:56.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:56 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/814280969' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:52:56.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:56 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1162777154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:52:56.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.674+0000 7f2574a54640 1 -- 172.21.15.76:0/760175264 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25701a0450 con 0x7f25701018f0 2024-03-18T14:52:56.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.674+0000 7f255f7fe640 1 -- 172.21.15.76:0/760175264 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11863 (secure 0 0 0) 0x7f256409bb80 con 0x7f25701018f0 2024-03-18T14:52:56.675 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:56.675 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/2801069369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":2801069369},{"type":"v1","addr":"172.21.15.76:6839","nonce":2801069369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2645229232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2645229232},{"type":"v1","addr":"172.21.15.76:6841","nonce":2645229232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T14:51:01.293009+0000","modified":"2024-03-18T14:51:07.032425+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24490,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6839/399810652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":399810652},{"type":"v1","addr":"172.21.15.174:6839","nonce":399810652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1685041985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1685041985},{"type":"v1","addr":"172.21.15.77:6839","nonce":1685041985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24463,24490,24524]},"id":1}]} 2024-03-18T14:52:56.675 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 23 2024-03-18T14:52:56.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.676+0000 7f2574a54640 1 -- 172.21.15.76:0/760175264 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f25580a2880 msgr2=0x7f25580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:56.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.676+0000 7f2574a54640 1 --2- 172.21.15.76:0/760175264 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f25580a2880 0x7f25580a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2568000c00 tx=0x7f2568005fd0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.676+0000 7f2574a54640 1 -- 172.21.15.76:0/760175264 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25701018f0 msgr2=0x7f2570198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:56.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.676+0000 7f2574a54640 1 --2- 172.21.15.76:0/760175264 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25701018f0 0x7f2570198080 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2564009d50 tx=0x7f256400c6a0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.676+0000 7f2574a54640 1 -- 172.21.15.76:0/760175264 shutdown_connections 2024-03-18T14:52:56.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.676+0000 7f2574a54640 1 --2- 172.21.15.76:0/760175264 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f25580a2880 0x7f25580a4d40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.676+0000 7f2574a54640 1 --2- 172.21.15.76:0/760175264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25701022b0 0x7f257019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.676+0000 7f2574a54640 1 --2- 172.21.15.76:0/760175264 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25701018f0 0x7f2570198080 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.676+0000 7f2574a54640 1 --2- 172.21.15.76:0/760175264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2570100700 0x7f2570197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.676+0000 7f2574a54640 1 -- 172.21.15.76:0/760175264 >> 172.21.15.76:0/760175264 conn(0x7f25700fbc70 msgr2=0x7f25700fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:56.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.676+0000 7f2574a54640 1 -- 172.21.15.76:0/760175264 shutdown_connections 2024-03-18T14:52:56.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.676+0000 7f2574a54640 1 -- 172.21.15.76:0/760175264 wait complete. 2024-03-18T14:52:56.688 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:52:56.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:56 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1840717422' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:52:56.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:56 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/814280969' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:52:56.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:56 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1162777154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:52:56.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:56 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1840717422' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:52:56.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:56 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/814280969' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:52:56.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:56 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1162777154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:52:56.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.788+0000 7f26add59640 1 -- 172.21.15.76:0/2103406072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26a8102a10 msgr2=0x7f26a8102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:56.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.788+0000 7f26add59640 1 --2- 172.21.15.76:0/2103406072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26a8102a10 0x7f26a8102e90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2690009900 tx=0x7f269002bc30 comp rx=0 tx=0).stop 2024-03-18T14:52:56.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.789+0000 7f26add59640 1 -- 172.21.15.76:0/2103406072 shutdown_connections 2024-03-18T14:52:56.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.789+0000 7f26add59640 1 --2- 172.21.15.76:0/2103406072 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f26a81033d0 0x7f26a8109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.789+0000 7f26add59640 1 --2- 172.21.15.76:0/2103406072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26a8102a10 0x7f26a8102e90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.789+0000 7f26add59640 1 --2- 172.21.15.76:0/2103406072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26a8101820 0x7f26a8101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.789+0000 7f26add59640 1 -- 172.21.15.76:0/2103406072 >> 172.21.15.76:0/2103406072 conn(0x7f26a80fcd90 msgr2=0x7f26a80ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.789+0000 7f26add59640 1 -- 172.21.15.76:0/2103406072 shutdown_connections 2024-03-18T14:52:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.789+0000 7f26add59640 1 -- 172.21.15.76:0/2103406072 wait complete. 2024-03-18T14:52:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.789+0000 7f26add59640 1 Processor -- start 2024-03-18T14:52:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.789+0000 7f26add59640 1 -- start start 2024-03-18T14:52:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.789+0000 7f26add59640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26a8101820 0x7f26a8197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.789+0000 7f26add59640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26a8102a10 0x7f26a81980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.789+0000 7f26add59640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f26a81033d0 0x7f26a819f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.789+0000 7f26add59640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f26a810d4a0 con 0x7f26a81033d0 2024-03-18T14:52:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.789+0000 7f26add59640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f26a810d320 con 0x7f26a8102a10 2024-03-18T14:52:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.789+0000 7f26add59640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f26a810d620 con 0x7f26a8101820 2024-03-18T14:52:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.790+0000 7f26a7fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f26a81033d0 0x7f26a819f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:56.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.790+0000 7f269effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26a8102a10 0x7f26a81980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:56.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.790+0000 7f26a7fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f26a81033d0 0x7f26a819f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51090/0 (socket says 172.21.15.76:51090) 2024-03-18T14:52:56.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.790+0000 7f26a7fff640 1 -- 172.21.15.76:0/3164818151 learned_addr learned my addr 172.21.15.76:0/3164818151 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:56.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.790+0000 7f26a77fe640 1 --2- 172.21.15.76:0/3164818151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26a8101820 0x7f26a8197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:56.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.790+0000 7f26a7fff640 1 -- 172.21.15.76:0/3164818151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26a8101820 msgr2=0x7f26a8197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:56.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.790+0000 7f26a7fff640 1 --2- 172.21.15.76:0/3164818151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26a8101820 0x7f26a8197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.790+0000 7f26a7fff640 1 -- 172.21.15.76:0/3164818151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26a8102a10 msgr2=0x7f26a81980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:56.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.790+0000 7f26a7fff640 1 --2- 172.21.15.76:0/3164818151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26a8102a10 0x7f26a81980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.790+0000 7f26a7fff640 1 -- 172.21.15.76:0/3164818151 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a819f790 con 0x7f26a81033d0 2024-03-18T14:52:56.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.790+0000 7f269effd640 1 --2- 172.21.15.76:0/3164818151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26a8102a10 0x7f26a81980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:56.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.790+0000 7f26a7fff640 1 --2- 172.21.15.76:0/3164818151 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f26a81033d0 0x7f26a819f120 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f26a000eb10 tx=0x7f26a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:56.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.790+0000 7f26a57fa640 1 -- 172.21.15.76:0/3164818151 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a0004b20 con 0x7f26a81033d0 2024-03-18T14:52:56.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.790+0000 7f26a57fa640 1 -- 172.21.15.76:0/3164818151 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26a0004cc0 con 0x7f26a81033d0 2024-03-18T14:52:56.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.790+0000 7f26add59640 1 -- 172.21.15.76:0/3164818151 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26a819fa80 con 0x7f26a81033d0 2024-03-18T14:52:56.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.790+0000 7f26a57fa640 1 -- 172.21.15.76:0/3164818151 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a0010760 con 0x7f26a81033d0 2024-03-18T14:52:56.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.790+0000 7f26add59640 1 -- 172.21.15.76:0/3164818151 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26a819ff50 con 0x7f26a81033d0 2024-03-18T14:52:56.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.791+0000 7f26add59640 1 -- 172.21.15.76:0/3164818151 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26a8108b40 con 0x7f26a81033d0 2024-03-18T14:52:56.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.792+0000 7f26a57fa640 1 -- 172.21.15.76:0/3164818151 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f26a0005020 con 0x7f26a81033d0 2024-03-18T14:52:56.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.793+0000 7f26a57fa640 1 --2- 172.21.15.76:0/3164818151 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f26880a2800 0x7f26880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:56.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.793+0000 7f26a77fe640 1 --2- 172.21.15.76:0/3164818151 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f26880a2800 0x7f26880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:56.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.793+0000 7f26a57fa640 1 -- 172.21.15.76:0/3164818151 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f26a00cefa0 con 0x7f26a81033d0 2024-03-18T14:52:56.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.793+0000 7f26a77fe640 1 --2- 172.21.15.76:0/3164818151 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f26880a2800 0x7f26880a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f26a8102870 tx=0x7f2698005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:56.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.797+0000 7f26a57fa640 1 -- 172.21.15.76:0/3164818151 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f26a009c070 con 0x7f26a81033d0 2024-03-18T14:52:56.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.973+0000 7f26add59640 1 -- 172.21.15.76:0/3164818151 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f26a80623e0 con 0x7f26a81033d0 2024-03-18T14:52:56.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.975+0000 7f26a57fa640 1 -- 172.21.15.76:0/3164818151 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22580 (secure 0 0 0) 0x7f26a009b970 con 0x7f26a81033d0 2024-03-18T14:52:56.976 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:52:56.976 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:52:07.361301+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:52:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.977+0000 7f26add59640 1 -- 172.21.15.76:0/3164818151 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f26880a2800 msgr2=0x7f26880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.977+0000 7f26add59640 1 --2- 172.21.15.76:0/3164818151 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f26880a2800 0x7f26880a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f26a8102870 tx=0x7f2698005f90 comp rx=0 tx=0).stop 2024-03-18T14:52:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.977+0000 7f26add59640 1 -- 172.21.15.76:0/3164818151 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f26a81033d0 msgr2=0x7f26a819f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.977+0000 7f26add59640 1 --2- 172.21.15.76:0/3164818151 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f26a81033d0 0x7f26a819f120 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f26a000eb10 tx=0x7f26a000c2d0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.977+0000 7f26add59640 1 -- 172.21.15.76:0/3164818151 shutdown_connections 2024-03-18T14:52:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.977+0000 7f26add59640 1 --2- 172.21.15.76:0/3164818151 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f26880a2800 0x7f26880a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.977+0000 7f26add59640 1 --2- 172.21.15.76:0/3164818151 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f26a81033d0 0x7f26a819f120 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.977+0000 7f26add59640 1 --2- 172.21.15.76:0/3164818151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26a8102a10 0x7f26a81980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.977+0000 7f26add59640 1 --2- 172.21.15.76:0/3164818151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26a8101820 0x7f26a8197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.977+0000 7f26add59640 1 -- 172.21.15.76:0/3164818151 >> 172.21.15.76:0/3164818151 conn(0x7f26a80fcd90 msgr2=0x7f26a8105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.977+0000 7f26add59640 1 -- 172.21.15.76:0/3164818151 shutdown_connections 2024-03-18T14:52:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:56.977+0000 7f26add59640 1 -- 172.21.15.76:0/3164818151 wait complete. 2024-03-18T14:52:56.989 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-18T14:51:01.293009+0000', 'modified': '2024-03-18T14:51:07.032425+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24524, 'mds_1': 24490, 'mds_2': 24463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24422': {'gid': 24422, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.77:6833/2726911960', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6832', 'nonce': 2726911960}, {'type': 'v1', 'addr': '172.21.15.77:6833', 'nonce': 2726911960}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24451': {'gid': 24451, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6835/603893267', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6834', 'nonce': 603893267}, {'type': 'v1', 'addr': '172.21.15.174:6835', 'nonce': 603893267}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.77:6835/3551464902', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 3551464902}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 3551464902}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.174:6837/239386414', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6836', 'nonce': 239386414}, {'type': 'v1', 'addr': '172.21.15.174:6837', 'nonce': 239386414}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24490': {'gid': 24490, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.174:6839/399810652', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6838', 'nonce': 399810652}, {'type': 'v1', 'addr': '172.21.15.174:6839', 'nonce': 399810652}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24524': {'gid': 24524, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6839/1685041985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6838', 'nonce': 1685041985}, {'type': 'v1', 'addr': '172.21.15.77:6839', 'nonce': 1685041985}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24524, 'qdb_cluster': [24463, 24490, 24524]} 2024-03-18T14:52:56.989 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-18T14:52:56.990 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T14:52:57.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.091+0000 7fe247b1b640 1 -- 172.21.15.76:0/4210308394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe240102290 msgr2=0x7fe240108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.091+0000 7fe247b1b640 1 --2- 172.21.15.76:0/4210308394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe240102290 0x7fe240108b20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe238009be0 tx=0x7fe23802bef0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe247b1b640 1 -- 172.21.15.76:0/4210308394 shutdown_connections 2024-03-18T14:52:57.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe247b1b640 1 --2- 172.21.15.76:0/4210308394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe2401018d0 0x7fe240101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe247b1b640 1 --2- 172.21.15.76:0/4210308394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe240102290 0x7fe240108b20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe247b1b640 1 --2- 172.21.15.76:0/4210308394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2401006e0 0x7fe240100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe247b1b640 1 -- 172.21.15.76:0/4210308394 >> 172.21.15.76:0/4210308394 conn(0x7fe2400fbc70 msgr2=0x7fe2400fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe247b1b640 1 -- 172.21.15.76:0/4210308394 shutdown_connections 2024-03-18T14:52:57.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe247b1b640 1 -- 172.21.15.76:0/4210308394 wait complete. 2024-03-18T14:52:57.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe247b1b640 1 Processor -- start 2024-03-18T14:52:57.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe247b1b640 1 -- start start 2024-03-18T14:52:57.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe247b1b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2401006e0 0x7fe240197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe247b1b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2401018d0 0x7fe240198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe247b1b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe240102290 0x7fe24019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe247b1b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe24010d480 con 0x7fe2401018d0 2024-03-18T14:52:57.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe247b1b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe24010d300 con 0x7fe2401006e0 2024-03-18T14:52:57.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe247b1b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe24010d600 con 0x7fe240102290 2024-03-18T14:52:57.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe2450b6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2401018d0 0x7fe240198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe2450b6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2401018d0 0x7fe240198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51106/0 (socket says 172.21.15.76:51106) 2024-03-18T14:52:57.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe2450b6640 1 -- 172.21.15.76:0/1216338202 learned_addr learned my addr 172.21.15.76:0/1216338202 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:57.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.092+0000 7fe2458b7640 1 --2- 172.21.15.76:0/1216338202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2401006e0 0x7fe240197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.093+0000 7fe2460b8640 1 --2- 172.21.15.76:0/1216338202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe240102290 0x7fe24019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.093+0000 7fe2450b6640 1 -- 172.21.15.76:0/1216338202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe240102290 msgr2=0x7fe24019f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.093+0000 7fe2450b6640 1 --2- 172.21.15.76:0/1216338202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe240102290 0x7fe24019f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.093+0000 7fe2450b6640 1 -- 172.21.15.76:0/1216338202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2401006e0 msgr2=0x7fe240197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.093+0000 7fe2450b6640 1 --2- 172.21.15.76:0/1216338202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2401006e0 0x7fe240197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.093+0000 7fe2450b6640 1 -- 172.21.15.76:0/1216338202 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe24019f720 con 0x7fe2401018d0 2024-03-18T14:52:57.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.093+0000 7fe2458b7640 1 --2- 172.21.15.76:0/1216338202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2401006e0 0x7fe240197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:57.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.093+0000 7fe2450b6640 1 --2- 172.21.15.76:0/1216338202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2401018d0 0x7fe240198040 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe23000e9f0 tx=0x7fe23000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.093+0000 7fe236ffd640 1 -- 172.21.15.76:0/1216338202 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe23000cd80 con 0x7fe2401018d0 2024-03-18T14:52:57.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.093+0000 7fe236ffd640 1 -- 172.21.15.76:0/1216338202 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe230005500 con 0x7fe2401018d0 2024-03-18T14:52:57.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.093+0000 7fe247b1b640 1 -- 172.21.15.76:0/1216338202 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe24019fa10 con 0x7fe2401018d0 2024-03-18T14:52:57.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.093+0000 7fe236ffd640 1 -- 172.21.15.76:0/1216338202 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe230010690 con 0x7fe2401018d0 2024-03-18T14:52:57.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.093+0000 7fe247b1b640 1 -- 172.21.15.76:0/1216338202 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe24019ff10 con 0x7fe2401018d0 2024-03-18T14:52:57.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.095+0000 7fe236ffd640 1 -- 172.21.15.76:0/1216338202 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe2300046c0 con 0x7fe2401018d0 2024-03-18T14:52:57.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.096+0000 7fe24a7e6740 1 Processor -- start 2024-03-18T14:52:57.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.096+0000 7fe236ffd640 1 --2- 172.21.15.76:0/1216338202 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe22c0a2800 0x7fe22c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.096+0000 7fe24a7e6740 1 -- start start 2024-03-18T14:52:57.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.096+0000 7fe24a7e6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324dbd0 0x55618324dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.096+0000 7fe2460b8640 1 --2- 172.21.15.76:0/1216338202 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe22c0a2800 0x7fe22c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.096+0000 7fe24a7e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55618324e5c0 0x5561832530c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.096+0000 7fe24a7e6740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556183253730 0x556183255bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.096+0000 7fe24a7e6740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5561831b1c80 con 0x556183253730 2024-03-18T14:52:57.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.096+0000 7fe24a7e6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55618315aaa0 con 0x55618324dbd0 2024-03-18T14:52:57.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.096+0000 7fe236ffd640 1 -- 172.21.15.76:0/1216338202 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fe2300cee20 con 0x7fe2401018d0 2024-03-18T14:52:57.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.096+0000 7fe24a7e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5561831add20 con 0x55618324e5c0 2024-03-18T14:52:57.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.096+0000 7fe2460b8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556183253730 0x556183255bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.096+0000 7fe2458b7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324dbd0 0x55618324dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.096+0000 7fe2460b8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556183253730 0x556183255bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51108/0 (socket says 172.21.15.76:51108) 2024-03-18T14:52:57.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.096+0000 7fe2460b8640 1 -- 172.21.15.76:0/3037891446 learned_addr learned my addr 172.21.15.76:0/3037891446 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:57.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.096+0000 7fe2450b6640 1 --2- 172.21.15.76:0/3037891446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55618324e5c0 0x5561832530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.097+0000 7fe2458b7640 1 -- 172.21.15.76:0/3037891446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55618324e5c0 msgr2=0x5561832530c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.097+0000 7fe2458b7640 1 --2- 172.21.15.76:0/3037891446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55618324e5c0 0x5561832530c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.097+0000 7fe2458b7640 1 -- 172.21.15.76:0/3037891446 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556183253730 msgr2=0x556183255bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.097+0000 7fe2458b7640 1 --2- 172.21.15.76:0/3037891446 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556183253730 0x556183255bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.097+0000 7fe2458b7640 1 -- 172.21.15.76:0/3037891446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55618317cff0 con 0x55618324dbd0 2024-03-18T14:52:57.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.097+0000 7fe2460b8640 1 --2- 172.21.15.76:0/1216338202 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe22c0a2800 0x7fe22c0a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe238009370 tx=0x7fe238033e60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.097+0000 7fe2458b7640 1 --2- 172.21.15.76:0/3037891446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324dbd0 0x55618324dff0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe240101730 tx=0x7fe23c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16b33884c5babe62 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.097+0000 7fe22bfff640 1 -- 172.21.15.76:0/3037891446 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe23c01e070 con 0x55618324dbd0 2024-03-18T14:52:57.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.097+0000 7fe22bfff640 1 -- 172.21.15.76:0/3037891446 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe23c00f960 con 0x55618324dbd0 2024-03-18T14:52:57.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.097+0000 7fe22bfff640 1 -- 172.21.15.76:0/3037891446 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe23c019af0 con 0x55618324dbd0 2024-03-18T14:52:57.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.098+0000 7fe24a7e6740 1 -- 172.21.15.76:0/3037891446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324dbd0 msgr2=0x55618324dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.098+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/3037891446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324dbd0 0x55618324dff0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe240101730 tx=0x7fe23c0118d0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.098+0000 7fe24a7e6740 1 -- 172.21.15.76:0/3037891446 shutdown_connections 2024-03-18T14:52:57.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.098+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/3037891446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55618324e5c0 0x5561832530c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.098+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/3037891446 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556183253730 0x556183255bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.098+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/3037891446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324dbd0 0x55618324dff0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.098+0000 7fe24a7e6740 1 -- 172.21.15.76:0/3037891446 >> 172.21.15.76:0/3037891446 conn(0x556183249140 msgr2=0x556183249540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.098+0000 7fe24a7e6740 1 -- 172.21.15.76:0/3037891446 shutdown_connections 2024-03-18T14:52:57.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.098+0000 7fe24a7e6740 1 -- 172.21.15.76:0/3037891446 wait complete. 2024-03-18T14:52:57.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.098+0000 7fe24a7e6740 1 -- start start 2024-03-18T14:52:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.098+0000 7fe24a7e6740 1 Processor -- start 2024-03-18T14:52:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.098+0000 7fe24a7e6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324dbd0 0x556183257de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.098+0000 7fe24a7e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55618324e5c0 0x556183258320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe24a7e6740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556183253730 0x55618325f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe24a7e6740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5561832566e0 con 0x556183253730 2024-03-18T14:52:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe24a7e6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55618316def0 con 0x55618324dbd0 2024-03-18T14:52:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe24a7e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556183256860 con 0x55618324e5c0 2024-03-18T14:52:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe24a7e6740 10 client.0 _ll_register_callbacks cb 0x556183168dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:52:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe2460b8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556183253730 0x55618325f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe2458b7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324dbd0 0x556183257de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe2460b8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556183253730 0x55618325f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51116/0 (socket says 172.21.15.76:51116) 2024-03-18T14:52:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe2460b8640 1 -- 172.21.15.76:0/1200718527 learned_addr learned my addr 172.21.15.76:0/1200718527 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe2450b6640 1 --2- 172.21.15.76:0/1200718527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55618324e5c0 0x556183258320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe2460b8640 1 -- 172.21.15.76:0/1200718527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55618324e5c0 msgr2=0x556183258320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe2460b8640 1 --2- 172.21.15.76:0/1200718527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55618324e5c0 0x556183258320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe2460b8640 1 -- 172.21.15.76:0/1200718527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324dbd0 msgr2=0x556183257de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe2460b8640 1 --2- 172.21.15.76:0/1200718527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324dbd0 0x556183257de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe2460b8640 1 -- 172.21.15.76:0/1200718527 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55618317aa30 con 0x556183253730 2024-03-18T14:52:57.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe2458b7640 1 --2- 172.21.15.76:0/1200718527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324dbd0 0x556183257de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:57.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe2460b8640 1 --2- 172.21.15.76:0/1200718527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556183253730 0x55618325f2f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe238006010 tx=0x7fe23802bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03fca697706eeea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe2297fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:52:57.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe2297fa640 1 -- 172.21.15.76:0/1200718527 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe238051070 con 0x556183253730 2024-03-18T14:52:57.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe2297fa640 1 -- 172.21.15.76:0/1200718527 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fe238033740 con 0x556183253730 2024-03-18T14:52:57.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe24a7e6740 1 -- 172.21.15.76:0/1200718527 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe23c00f960 con 0x556183253730 2024-03-18T14:52:57.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe2297fa640 1 -- 172.21.15.76:0/1200718527 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe23805d050 con 0x556183253730 2024-03-18T14:52:57.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe24a7e6740 10 client.15207 fetch_fsmap learned FSMap version 23 2024-03-18T14:52:57.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.099+0000 7fe24a7e6740 1 -- 172.21.15.76:0/1200718527 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556182ec62d0 con 0x556183253730 2024-03-18T14:52:57.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.100+0000 7fe2297fa640 1 -- 172.21.15.76:0/1200718527 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe23c00f960 con 0x556183253730 2024-03-18T14:52:57.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.100+0000 7fe2297fa640 1 -- 172.21.15.76:0/1200718527 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe238034840 con 0x556183253730 2024-03-18T14:52:57.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.100+0000 7fe24a7e6740 10 client.15207 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T14:52:57.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.100+0000 7fe24a7e6740 10 client.15207 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T14:52:57.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.100+0000 7fe24a7e6740 20 client.15207 populate_metadata read hostname 'smithi076' 2024-03-18T14:52:57.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.100+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/1200718527 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x556183264870 0x556183265ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.100+0000 7fe24a7e6740 4 client.15207 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:52:57.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.100+0000 7fe24a7e6740 1 -- 172.21.15.76:0/1200718527 --> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561831bf1c0 con 0x556183264870 2024-03-18T14:52:57.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.100+0000 7fe2458b7640 1 --2- 172.21.15.76:0/1200718527 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x556183264870 0x556183265ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.100+0000 7fe2458b7640 1 --2- 172.21.15.76:0/1200718527 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x556183264870 0x556183265ce0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4aa34e55244b76ba server_cookie=3895aee4f3c5813 in_seq=0 out_seq=0 2024-03-18T14:52:57.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.100+0000 7fe2297fa640 10 client.15207 ms_handle_connect on v2:172.21.15.174:6836/239386414 2024-03-18T14:52:57.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.101+0000 7fe2297fa640 1 -- 172.21.15.76:0/1200718527 <== mds.2 v2:172.21.15.174:6836/239386414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5561831bf1c0 con 0x556183264870 2024-03-18T14:52:57.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.101+0000 7fe2297fa640 10 client.15207 handle_command_reply: tid=0 2024-03-18T14:52:57.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.101+0000 7fe24a7e6740 1 client.15207 shutdown 2024-03-18T14:52:57.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.101+0000 7fe24a7e6740 1 -- 172.21.15.76:0/1200718527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556183253730 msgr2=0x55618325f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.101+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/1200718527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556183253730 0x55618325f2f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe238006010 tx=0x7fe23802bef0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.101+0000 7fe24a7e6740 1 -- 172.21.15.76:0/1200718527 shutdown_connections 2024-03-18T14:52:57.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.101+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/1200718527 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x556183264870 0x556183265ce0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.101+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/1200718527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55618324e5c0 0x556183258320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.101+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/1200718527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556183253730 0x55618325f2f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.101+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/1200718527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324dbd0 0x556183257de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.101+0000 7fe2297fa640 0 client.15207 ms_handle_reset on v2:172.21.15.174:6836/239386414 2024-03-18T14:52:57.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.101+0000 7fe24a7e6740 1 -- 172.21.15.76:0/1200718527 >> 172.21.15.76:0/1200718527 conn(0x556183249140 msgr2=0x556183248380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.101+0000 7fe24a7e6740 1 -- 172.21.15.76:0/1200718527 shutdown_connections 2024-03-18T14:52:57.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.101+0000 7fe24a7e6740 1 -- 172.21.15.76:0/1200718527 wait complete. 2024-03-18T14:52:57.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.101+0000 7fe24a7e6740 20 client.15207 trim_cache size 0 max 16384 2024-03-18T14:52:57.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.119+0000 7fe24a7e6740 1 Processor -- start 2024-03-18T14:52:57.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.119+0000 7fe24a7e6740 1 -- start start 2024-03-18T14:52:57.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.119+0000 7fe24a7e6740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55618324dbd0 0x556183247af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.119+0000 7fe24a7e6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324e5c0 0x55618325f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.119+0000 7fe24a7e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556183253730 0x556183248030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.119+0000 7fe24a7e6740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x556183264020 con 0x55618324dbd0 2024-03-18T14:52:57.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.119+0000 7fe24a7e6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5561832646f0 con 0x55618324e5c0 2024-03-18T14:52:57.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.119+0000 7fe24a7e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556183264dc0 con 0x556183253730 2024-03-18T14:52:57.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.119+0000 7fe2458b7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55618324dbd0 0x556183247af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.119+0000 7fe2458b7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55618324dbd0 0x556183247af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51118/0 (socket says 172.21.15.76:51118) 2024-03-18T14:52:57.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.119+0000 7fe2458b7640 1 -- 172.21.15.76:0/3514381732 learned_addr learned my addr 172.21.15.76:0/3514381732 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:57.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.119+0000 7fe2460b8640 1 --2- 172.21.15.76:0/3514381732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556183253730 0x556183248030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.119+0000 7fe2450b6640 1 --2- 172.21.15.76:0/3514381732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324e5c0 0x55618325f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe2458b7640 1 -- 172.21.15.76:0/3514381732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556183253730 msgr2=0x556183248030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe2458b7640 1 --2- 172.21.15.76:0/3514381732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556183253730 0x556183248030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe2458b7640 1 -- 172.21.15.76:0/3514381732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324e5c0 msgr2=0x55618325f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe2458b7640 1 --2- 172.21.15.76:0/3514381732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324e5c0 0x55618325f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe2458b7640 1 -- 172.21.15.76:0/3514381732 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556182f906b0 con 0x55618324dbd0 2024-03-18T14:52:57.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe2460b8640 1 --2- 172.21.15.76:0/3514381732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556183253730 0x556183248030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:52:57.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe2450b6640 1 --2- 172.21.15.76:0/3514381732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324e5c0 0x55618325f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:52:57.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe2458b7640 1 --2- 172.21.15.76:0/3514381732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55618324dbd0 0x556183247af0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009f50 tx=0x7fe23c011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c470fede9b3c03b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe22b7fe640 1 -- 172.21.15.76:0/3514381732 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe23c02a030 con 0x55618324dbd0 2024-03-18T14:52:57.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe22b7fe640 1 -- 172.21.15.76:0/3514381732 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe23c005040 con 0x55618324dbd0 2024-03-18T14:52:57.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe22b7fe640 1 -- 172.21.15.76:0/3514381732 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe23c01e040 con 0x55618324dbd0 2024-03-18T14:52:57.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe24a7e6740 1 -- 172.21.15.76:0/3514381732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55618324dbd0 msgr2=0x556183247af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/3514381732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55618324dbd0 0x556183247af0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009f50 tx=0x7fe23c011f90 comp rx=0 tx=0).stop 2024-03-18T14:52:57.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe24a7e6740 1 -- 172.21.15.76:0/3514381732 shutdown_connections 2024-03-18T14:52:57.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/3514381732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556183253730 0x556183248030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/3514381732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55618324e5c0 0x55618325f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/3514381732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55618324dbd0 0x556183247af0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe24a7e6740 1 -- 172.21.15.76:0/3514381732 >> 172.21.15.76:0/3514381732 conn(0x556183249140 msgr2=0x556183252de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.120+0000 7fe24a7e6740 1 -- 172.21.15.76:0/3514381732 shutdown_connections 2024-03-18T14:52:57.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe24a7e6740 1 -- 172.21.15.76:0/3514381732 wait complete. 2024-03-18T14:52:57.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe24a7e6740 1 -- start start 2024-03-18T14:52:57.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe24a7e6740 1 Processor -- start 2024-03-18T14:52:57.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe24a7e6740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55618324dbd0 0x55618325d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe24a7e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55618324e5c0 0x5561832c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe24a7e6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556183253730 0x5561832c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe24a7e6740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x556183265b60 con 0x55618324dbd0 2024-03-18T14:52:57.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe24a7e6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556183265490 con 0x556183253730 2024-03-18T14:52:57.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe24a7e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556183266230 con 0x55618324e5c0 2024-03-18T14:52:57.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe24a7e6740 10 client.0 _ll_register_callbacks cb 0x7fe238009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:52:57.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe2458b7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55618324dbd0 0x55618325d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe2458b7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55618324dbd0 0x55618325d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51124/0 (socket says 172.21.15.76:51124) 2024-03-18T14:52:57.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe2458b7640 1 -- 172.21.15.76:0/1792104655 learned_addr learned my addr 172.21.15.76:0/1792104655 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:57.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe2460b8640 1 --2- 172.21.15.76:0/1792104655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556183253730 0x5561832c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe2450b6640 1 --2- 172.21.15.76:0/1792104655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55618324e5c0 0x5561832c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe2458b7640 1 -- 172.21.15.76:0/1792104655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55618324e5c0 msgr2=0x5561832c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe2458b7640 1 --2- 172.21.15.76:0/1792104655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55618324e5c0 0x5561832c65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe2458b7640 1 -- 172.21.15.76:0/1792104655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556183253730 msgr2=0x5561832c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe2458b7640 1 --2- 172.21.15.76:0/1792104655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556183253730 0x5561832c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe2458b7640 1 -- 172.21.15.76:0/1792104655 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561832c9050 con 0x55618324dbd0 2024-03-18T14:52:57.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.121+0000 7fe2450b6640 1 --2- 172.21.15.76:0/1792104655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55618324e5c0 0x5561832c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:52:57.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.122+0000 7fe2458b7640 1 --2- 172.21.15.76:0/1792104655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55618324dbd0 0x55618325d200 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe23c013010 tx=0x7fe23c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3782a417e5c00a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.122+0000 7fe2297fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:52:57.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.122+0000 7fe2297fa640 1 -- 172.21.15.76:0/1792104655 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe23c01e040 con 0x55618324dbd0 2024-03-18T14:52:57.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.122+0000 7fe2297fa640 1 -- 172.21.15.76:0/1792104655 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fe23c00f800 con 0x55618324dbd0 2024-03-18T14:52:57.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.122+0000 7fe24a7e6740 1 -- 172.21.15.76:0/1792104655 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe23c005040 con 0x55618324dbd0 2024-03-18T14:52:57.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.122+0000 7fe2297fa640 1 -- 172.21.15.76:0/1792104655 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe23c02a410 con 0x55618324dbd0 2024-03-18T14:52:57.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.122+0000 7fe24a7e6740 10 client.15213 fetch_fsmap learned FSMap version 23 2024-03-18T14:52:57.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.122+0000 7fe24a7e6740 1 -- 172.21.15.76:0/1792104655 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5561832ca2f0 con 0x55618324dbd0 2024-03-18T14:52:57.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.122+0000 7fe2297fa640 1 -- 172.21.15.76:0/1792104655 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe23c005040 con 0x55618324dbd0 2024-03-18T14:52:57.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.122+0000 7fe2297fa640 1 -- 172.21.15.76:0/1792104655 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe23c004a80 con 0x55618324dbd0 2024-03-18T14:52:57.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.122+0000 7fe24a7e6740 10 client.15213 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T14:52:57.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.122+0000 7fe24a7e6740 10 client.15213 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T14:52:57.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.122+0000 7fe24a7e6740 20 client.15213 populate_metadata read hostname 'smithi076' 2024-03-18T14:52:57.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.122+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/1792104655 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x5561832ce7a0 0x5561832cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.122+0000 7fe24a7e6740 4 client.15213 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T14:52:57.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.122+0000 7fe24a7e6740 1 -- 172.21.15.76:0/1792104655 --> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] -- command(tid 0: {"prefix": "status"}) -- 0x5561831b9510 con 0x5561832ce7a0 2024-03-18T14:52:57.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe2450b6640 1 --2- 172.21.15.76:0/1792104655 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x5561832ce7a0 0x5561832cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe2297fa640 10 client.15213 ms_handle_connect on v2:172.21.15.174:6836/239386414 2024-03-18T14:52:57.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe2450b6640 1 --2- 172.21.15.76:0/1792104655 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x5561832ce7a0 0x5561832cd4b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ec8540de8eba6090 server_cookie=bbafe97e455807c7 in_seq=0 out_seq=0 2024-03-18T14:52:57.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe2297fa640 1 -- 172.21.15.76:0/1792104655 <== mds.2 v2:172.21.15.174:6836/239386414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5561831b9510 con 0x5561832ce7a0 2024-03-18T14:52:57.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe2297fa640 10 client.15213 handle_command_reply: tid=0 2024-03-18T14:52:57.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe24a7e6740 1 client.15213 shutdown 2024-03-18T14:52:57.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe24a7e6740 1 -- 172.21.15.76:0/1792104655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55618324dbd0 msgr2=0x55618325d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/1792104655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55618324dbd0 0x55618325d200 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe23c013010 tx=0x7fe23c000bc0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe24a7e6740 1 -- 172.21.15.76:0/1792104655 shutdown_connections 2024-03-18T14:52:57.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/1792104655 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x5561832ce7a0 0x5561832cd4b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/1792104655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55618324e5c0 0x5561832c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/1792104655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556183253730 0x5561832c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe2297fa640 0 client.15213 ms_handle_reset on v2:172.21.15.174:6836/239386414 2024-03-18T14:52:57.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe24a7e6740 1 --2- 172.21.15.76:0/1792104655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55618324dbd0 0x55618325d200 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe24a7e6740 1 -- 172.21.15.76:0/1792104655 >> 172.21.15.76:0/1792104655 conn(0x556183249140 msgr2=0x55618324c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe24a7e6740 1 -- 172.21.15.76:0/1792104655 shutdown_connections 2024-03-18T14:52:57.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe24a7e6740 1 -- 172.21.15.76:0/1792104655 wait complete. 2024-03-18T14:52:57.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.123+0000 7fe24a7e6740 20 client.15213 trim_cache size 0 max 16384 2024-03-18T14:52:57.131 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T14:52:57.131 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "9cc83b72-e535-11ee-95c9-87774f69a715", 2024-03-18T14:52:57.131 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 2, 2024-03-18T14:52:57.131 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24463, 2024-03-18T14:52:57.131 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-03-18T14:52:57.132 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-03-18T14:52:57.132 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-03-18T14:52:57.132 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 111.100065427, 2024-03-18T14:52:57.132 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 23, 2024-03-18T14:52:57.132 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 88, 2024-03-18T14:52:57.132 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T14:52:57.132 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 134.01709319299999 2024-03-18T14:52:57.132 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T14:52:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.124+0000 7fe247b1b640 1 -- 172.21.15.76:0/1216338202 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe22c0a2800 msgr2=0x7fe22c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.124+0000 7fe247b1b640 1 --2- 172.21.15.76:0/1216338202 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe22c0a2800 0x7fe22c0a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe238009370 tx=0x7fe238033e60 comp rx=0 tx=0).stop 2024-03-18T14:52:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.124+0000 7fe247b1b640 1 -- 172.21.15.76:0/1216338202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2401018d0 msgr2=0x7fe240198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.124+0000 7fe247b1b640 1 --2- 172.21.15.76:0/1216338202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2401018d0 0x7fe240198040 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe23000e9f0 tx=0x7fe23000edf0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.124+0000 7fe247b1b640 1 -- 172.21.15.76:0/1216338202 shutdown_connections 2024-03-18T14:52:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.124+0000 7fe247b1b640 1 --2- 172.21.15.76:0/1216338202 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fe22c0a2800 0x7fe22c0a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.124+0000 7fe247b1b640 1 --2- 172.21.15.76:0/1216338202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe240102290 0x7fe24019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.124+0000 7fe247b1b640 1 --2- 172.21.15.76:0/1216338202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2401018d0 0x7fe240198040 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.124+0000 7fe247b1b640 1 --2- 172.21.15.76:0/1216338202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2401006e0 0x7fe240197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.124+0000 7fe247b1b640 1 -- 172.21.15.76:0/1216338202 >> 172.21.15.76:0/1216338202 conn(0x7fe2400fbc70 msgr2=0x7fe240104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.124+0000 7fe247b1b640 1 -- 172.21.15.76:0/1216338202 shutdown_connections 2024-03-18T14:52:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.124+0000 7fe247b1b640 1 -- 172.21.15.76:0/1216338202 wait complete. 2024-03-18T14:52:57.135 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-18T14:52:57.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.237+0000 7fc0070f8640 1 -- 172.21.15.76:0/3374799323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc000102260 msgr2=0x7fc000104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.237+0000 7fc0070f8640 1 --2- 172.21.15.76:0/3374799323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc000102260 0x7fc000104650 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbff4009860 tx=0x7fbff402bb70 comp rx=0 tx=0).stop 2024-03-18T14:52:57.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.237+0000 7fc0070f8640 1 -- 172.21.15.76:0/3374799323 shutdown_connections 2024-03-18T14:52:57.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.237+0000 7fc0070f8640 1 --2- 172.21.15.76:0/3374799323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0001074c0 0x7fc000109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.237+0000 7fc0070f8640 1 --2- 172.21.15.76:0/3374799323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc000104b90 0x7fc000106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.237+0000 7fc0070f8640 1 --2- 172.21.15.76:0/3374799323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc000102260 0x7fc000104650 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.237+0000 7fc0070f8640 1 -- 172.21.15.76:0/3374799323 >> 172.21.15.76:0/3374799323 conn(0x7fc0000fbc50 msgr2=0x7fc0000fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.237+0000 7fc0070f8640 1 -- 172.21.15.76:0/3374799323 shutdown_connections 2024-03-18T14:52:57.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.237+0000 7fc0070f8640 1 -- 172.21.15.76:0/3374799323 wait complete. 2024-03-18T14:52:57.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc0070f8640 1 Processor -- start 2024-03-18T14:52:57.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc0070f8640 1 -- start start 2024-03-18T14:52:57.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc0070f8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc000102260 0x7fc000197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc0070f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc000104b90 0x7fc000198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc0070f8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0001074c0 0x7fc00019f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc0070f8640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc00010d480 con 0x7fc000102260 2024-03-18T14:52:57.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc0070f8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc00010d300 con 0x7fc000104b90 2024-03-18T14:52:57.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc0070f8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc00010d600 con 0x7fc0001074c0 2024-03-18T14:52:57.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc004e94640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc000102260 0x7fc000197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc004e94640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc000102260 0x7fc000197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51148/0 (socket says 172.21.15.76:51148) 2024-03-18T14:52:57.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc004e94640 1 -- 172.21.15.76:0/1046547100 learned_addr learned my addr 172.21.15.76:0/1046547100 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:57.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc005695640 1 --2- 172.21.15.76:0/1046547100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0001074c0 0x7fc00019f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fbffffff640 1 --2- 172.21.15.76:0/1046547100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc000104b90 0x7fc000198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc004e94640 1 -- 172.21.15.76:0/1046547100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0001074c0 msgr2=0x7fc00019f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc004e94640 1 --2- 172.21.15.76:0/1046547100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0001074c0 0x7fc00019f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc004e94640 1 -- 172.21.15.76:0/1046547100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc000104b90 msgr2=0x7fc000198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc004e94640 1 --2- 172.21.15.76:0/1046547100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc000104b90 0x7fc000198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc004e94640 1 -- 172.21.15.76:0/1046547100 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00019f770 con 0x7fc000102260 2024-03-18T14:52:57.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.238+0000 7fc004e94640 1 --2- 172.21.15.76:0/1046547100 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc000102260 0x7fc000197b50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbff4005940 tx=0x7fbff4032470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.239+0000 7fbffdffb640 1 -- 172.21.15.76:0/1046547100 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff4041020 con 0x7fc000102260 2024-03-18T14:52:57.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.239+0000 7fbffdffb640 1 -- 172.21.15.76:0/1046547100 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbff402c9f0 con 0x7fc000102260 2024-03-18T14:52:57.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.239+0000 7fc0070f8640 1 -- 172.21.15.76:0/1046547100 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc00019fa00 con 0x7fc000102260 2024-03-18T14:52:57.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.239+0000 7fbffdffb640 1 -- 172.21.15.76:0/1046547100 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff4039040 con 0x7fc000102260 2024-03-18T14:52:57.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.239+0000 7fc0070f8640 1 -- 172.21.15.76:0/1046547100 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc00019fef0 con 0x7fc000102260 2024-03-18T14:52:57.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.241+0000 7fbffdffb640 1 -- 172.21.15.76:0/1046547100 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbff4044020 con 0x7fc000102260 2024-03-18T14:52:57.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.241+0000 7fc009dc3740 1 Processor -- start 2024-03-18T14:52:57.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.241+0000 7fc009dc3740 1 -- start start 2024-03-18T14:52:57.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.241+0000 7fc009dc3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3abd0 0x563674a3aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.241+0000 7fc009dc3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a3b5c0 0x563674a400c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.241+0000 7fc009dc3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a40730 0x563674a42bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.241+0000 7fc009dc3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56367499ec80 con 0x563674a3b5c0 2024-03-18T14:52:57.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.241+0000 7fc009dc3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563674947aa0 con 0x563674a3abd0 2024-03-18T14:52:57.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.241+0000 7fc009dc3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56367499ad20 con 0x563674a40730 2024-03-18T14:52:57.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.241+0000 7fbffdffb640 1 --2- 172.21.15.76:0/1046547100 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fbff00a2800 0x7fbff00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.241+0000 7fc004e94640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a3b5c0 0x563674a400c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fc005695640 1 --2- 172.21.15.76:0/1046547100 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fbff00a2800 0x7fbff00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fc004e94640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a3b5c0 0x563674a400c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51156/0 (socket says 172.21.15.76:51156) 2024-03-18T14:52:57.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fbffffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a40730 0x563674a42bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fc004e94640 1 -- 172.21.15.76:0/3000894741 learned_addr learned my addr 172.21.15.76:0/3000894741 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:57.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fc005695640 1 --2- 172.21.15.76:0/3000894741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3abd0 0x563674a3aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fbffdffb640 1 -- 172.21.15.76:0/1046547100 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fbff40f03e0 con 0x7fc000102260 2024-03-18T14:52:57.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fc004e94640 1 -- 172.21.15.76:0/3000894741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a40730 msgr2=0x563674a42bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fc004e94640 1 --2- 172.21.15.76:0/3000894741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a40730 0x563674a42bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fc004e94640 1 -- 172.21.15.76:0/3000894741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3abd0 msgr2=0x563674a3aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fc004e94640 1 --2- 172.21.15.76:0/3000894741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3abd0 0x563674a3aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fc004e94640 1 -- 172.21.15.76:0/3000894741 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563674969ff0 con 0x563674a3b5c0 2024-03-18T14:52:57.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fbffffff640 1 --2- 172.21.15.76:0/3000894741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a40730 0x563674a42bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:57.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fc005695640 1 --2- 172.21.15.76:0/3000894741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3abd0 0x563674a3aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:57.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fc004e94640 1 --2- 172.21.15.76:0/3000894741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a3b5c0 0x563674a400c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fbff4034b10 tx=0x7fbff4005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934ef832fdd83a63 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fc005695640 1 --2- 172.21.15.76:0/1046547100 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fbff00a2800 0x7fbff00a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x563674a3be20 tx=0x7fbff8004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fbfeaffd640 1 -- 172.21.15.76:0/3000894741 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff40bd450 con 0x563674a3b5c0 2024-03-18T14:52:57.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fbfeaffd640 1 -- 172.21.15.76:0/3000894741 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbff40c54f0 con 0x563674a3b5c0 2024-03-18T14:52:57.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.242+0000 7fbfeaffd640 1 -- 172.21.15.76:0/3000894741 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff4032640 con 0x563674a3b5c0 2024-03-18T14:52:57.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.243+0000 7fc009dc3740 1 -- 172.21.15.76:0/3000894741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a3b5c0 msgr2=0x563674a400c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.243+0000 7fc009dc3740 1 --2- 172.21.15.76:0/3000894741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a3b5c0 0x563674a400c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fbff4034b10 tx=0x7fbff4005b20 comp rx=0 tx=0).stop 2024-03-18T14:52:57.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.243+0000 7fc009dc3740 1 -- 172.21.15.76:0/3000894741 shutdown_connections 2024-03-18T14:52:57.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.243+0000 7fc009dc3740 1 --2- 172.21.15.76:0/3000894741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a40730 0x563674a42bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.243+0000 7fc009dc3740 1 --2- 172.21.15.76:0/3000894741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a3b5c0 0x563674a400c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.243+0000 7fc009dc3740 1 --2- 172.21.15.76:0/3000894741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3abd0 0x563674a3aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.243+0000 7fc009dc3740 1 -- 172.21.15.76:0/3000894741 >> 172.21.15.76:0/3000894741 conn(0x563674a36140 msgr2=0x563674a36540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.243+0000 7fc009dc3740 1 -- 172.21.15.76:0/3000894741 shutdown_connections 2024-03-18T14:52:57.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.243+0000 7fc009dc3740 1 -- 172.21.15.76:0/3000894741 wait complete. 2024-03-18T14:52:57.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.243+0000 7fc009dc3740 1 -- start start 2024-03-18T14:52:57.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.243+0000 7fc009dc3740 1 Processor -- start 2024-03-18T14:52:57.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc009dc3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a3abd0 0x563674a44b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc009dc3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3b5c0 0x563674a45040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc009dc3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a40730 0x563674a4c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc009dc3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563674a434a0 con 0x563674a3abd0 2024-03-18T14:52:57.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc009dc3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56367495aef0 con 0x563674a3b5c0 2024-03-18T14:52:57.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc009dc3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563674a43620 con 0x563674a40730 2024-03-18T14:52:57.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc009dc3740 10 client.0 _ll_register_callbacks cb 0x563674955dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:52:57.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc004e94640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a3abd0 0x563674a44b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc004e94640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a3abd0 0x563674a44b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51158/0 (socket says 172.21.15.76:51158) 2024-03-18T14:52:57.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc004e94640 1 -- 172.21.15.76:0/3343633413 learned_addr learned my addr 172.21.15.76:0/3343633413 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:57.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc005695640 1 --2- 172.21.15.76:0/3343633413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a40730 0x563674a4c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fbffffff640 1 --2- 172.21.15.76:0/3343633413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3b5c0 0x563674a45040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc004e94640 1 -- 172.21.15.76:0/3343633413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a40730 msgr2=0x563674a4c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc004e94640 1 --2- 172.21.15.76:0/3343633413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a40730 0x563674a4c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc004e94640 1 -- 172.21.15.76:0/3343633413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3b5c0 msgr2=0x563674a45040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc004e94640 1 --2- 172.21.15.76:0/3343633413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3b5c0 0x563674a45040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc004e94640 1 -- 172.21.15.76:0/3343633413 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563674967a30 con 0x563674a3abd0 2024-03-18T14:52:57.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc004e94640 1 --2- 172.21.15.76:0/3343633413 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a3abd0 0x563674a44b00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbff40b9720 tx=0x7fbff4005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c987d5b0c3a8ec2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fbfdffff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:52:57.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fbfdffff640 1 -- 172.21.15.76:0/3343633413 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff40d9030 con 0x563674a3abd0 2024-03-18T14:52:57.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fbfdffff640 1 -- 172.21.15.76:0/3343633413 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fbff40d3040 con 0x563674a3abd0 2024-03-18T14:52:57.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc009dc3740 1 -- 172.21.15.76:0/3343633413 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbff40c54f0 con 0x563674a3abd0 2024-03-18T14:52:57.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fbfdffff640 1 -- 172.21.15.76:0/3343633413 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fbff40bcd50 con 0x563674a3abd0 2024-03-18T14:52:57.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc009dc3740 10 client.15225 fetch_fsmap learned FSMap version 23 2024-03-18T14:52:57.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.244+0000 7fc009dc3740 1 -- 172.21.15.76:0/3343633413 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5636746b32d0 con 0x563674a3abd0 2024-03-18T14:52:57.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.245+0000 7fbfdffff640 1 -- 172.21.15.76:0/3343633413 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbff40c54f0 con 0x563674a3abd0 2024-03-18T14:52:57.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.245+0000 7fbfdffff640 1 -- 172.21.15.76:0/3343633413 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fbff403c030 con 0x563674a3abd0 2024-03-18T14:52:57.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.245+0000 7fc009dc3740 10 client.15225 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T14:52:57.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.245+0000 7fc009dc3740 10 client.15225 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T14:52:57.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.245+0000 7fc009dc3740 20 client.15225 populate_metadata read hostname 'smithi076' 2024-03-18T14:52:57.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.245+0000 7fc009dc3740 1 --2- 172.21.15.76:0/3343633413 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x563674a511e0 0x563674a52650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.245+0000 7fc009dc3740 4 client.15225 mds_command: new command op to 24490 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:52:57.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.245+0000 7fc009dc3740 1 -- 172.21.15.76:0/3343633413 --> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636749ac1c0 con 0x563674a511e0 2024-03-18T14:52:57.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.245+0000 7fbffffff640 1 --2- 172.21.15.76:0/3343633413 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x563674a511e0 0x563674a52650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.245+0000 7fbfdffff640 10 client.15225 ms_handle_connect on v2:172.21.15.174:6838/399810652 2024-03-18T14:52:57.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.245+0000 7fbffffff640 1 --2- 172.21.15.76:0/3343633413 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x563674a511e0 0x563674a52650 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e93fe70d1fb83226 server_cookie=41f723e508cfd975 in_seq=0 out_seq=0 2024-03-18T14:52:57.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.246+0000 7fbfdffff640 1 -- 172.21.15.76:0/3343633413 <== mds.1 v2:172.21.15.174:6838/399810652 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5636749ac1c0 con 0x563674a511e0 2024-03-18T14:52:57.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.246+0000 7fbfdffff640 10 client.15225 handle_command_reply: tid=0 2024-03-18T14:52:57.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.246+0000 7fc009dc3740 1 client.15225 shutdown 2024-03-18T14:52:57.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.246+0000 7fc009dc3740 1 -- 172.21.15.76:0/3343633413 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a3abd0 msgr2=0x563674a44b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.246+0000 7fc009dc3740 1 --2- 172.21.15.76:0/3343633413 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a3abd0 0x563674a44b00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbff40b9720 tx=0x7fbff4005750 comp rx=0 tx=0).stop 2024-03-18T14:52:57.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.246+0000 7fc009dc3740 1 -- 172.21.15.76:0/3343633413 shutdown_connections 2024-03-18T14:52:57.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.246+0000 7fc009dc3740 1 --2- 172.21.15.76:0/3343633413 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x563674a511e0 0x563674a52650 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.246+0000 7fc009dc3740 1 --2- 172.21.15.76:0/3343633413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a40730 0x563674a4c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.246+0000 7fc009dc3740 1 --2- 172.21.15.76:0/3343633413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3b5c0 0x563674a45040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.246+0000 7fbfdffff640 0 client.15225 ms_handle_reset on v2:172.21.15.174:6838/399810652 2024-03-18T14:52:57.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.246+0000 7fc009dc3740 1 --2- 172.21.15.76:0/3343633413 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a3abd0 0x563674a44b00 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.246+0000 7fc009dc3740 1 -- 172.21.15.76:0/3343633413 >> 172.21.15.76:0/3343633413 conn(0x563674a36140 msgr2=0x563674a35380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.246+0000 7fc009dc3740 1 -- 172.21.15.76:0/3343633413 shutdown_connections 2024-03-18T14:52:57.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.246+0000 7fc009dc3740 1 -- 172.21.15.76:0/3343633413 wait complete. 2024-03-18T14:52:57.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.246+0000 7fc009dc3740 20 client.15225 trim_cache size 0 max 16384 2024-03-18T14:52:57.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.264+0000 7fc009dc3740 1 Processor -- start 2024-03-18T14:52:57.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.264+0000 7fc009dc3740 1 -- start start 2024-03-18T14:52:57.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.264+0000 7fc009dc3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a3abd0 0x563674a36520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.264+0000 7fc009dc3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3b5c0 0x563674a34af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.264+0000 7fc009dc3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a40730 0x563674a35030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.264+0000 7fc009dc3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563674a50990 con 0x563674a40730 2024-03-18T14:52:57.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.264+0000 7fc009dc3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563674a51060 con 0x563674a3b5c0 2024-03-18T14:52:57.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.264+0000 7fc009dc3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563674a51730 con 0x563674a3abd0 2024-03-18T14:52:57.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.264+0000 7fc005695640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a40730 0x563674a35030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.264+0000 7fc005695640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a40730 0x563674a35030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51164/0 (socket says 172.21.15.76:51164) 2024-03-18T14:52:57.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.264+0000 7fbffffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3b5c0 0x563674a34af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.264+0000 7fc005695640 1 -- 172.21.15.76:0/520246056 learned_addr learned my addr 172.21.15.76:0/520246056 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:57.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.264+0000 7fc004e94640 1 --2- 172.21.15.76:0/520246056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a3abd0 0x563674a36520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.265+0000 7fc005695640 1 -- 172.21.15.76:0/520246056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a3abd0 msgr2=0x563674a36520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.265+0000 7fc005695640 1 --2- 172.21.15.76:0/520246056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a3abd0 0x563674a36520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.265+0000 7fc005695640 1 -- 172.21.15.76:0/520246056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3b5c0 msgr2=0x563674a34af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.265+0000 7fc005695640 1 --2- 172.21.15.76:0/520246056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3b5c0 0x563674a34af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.265+0000 7fc005695640 1 -- 172.21.15.76:0/520246056 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56367477d6b0 con 0x563674a40730 2024-03-18T14:52:57.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.265+0000 7fbffffff640 1 --2- 172.21.15.76:0/520246056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3b5c0 0x563674a34af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:57.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.265+0000 7fc005695640 1 --2- 172.21.15.76:0/520246056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a40730 0x563674a35030 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fbff8011f90 tx=0x7fbff800e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35d23a71a2adb3d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.265+0000 7fbfea7fc640 1 -- 172.21.15.76:0/520246056 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8019040 con 0x563674a40730 2024-03-18T14:52:57.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.265+0000 7fbfea7fc640 1 -- 172.21.15.76:0/520246056 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbff800fcc0 con 0x563674a40730 2024-03-18T14:52:57.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.265+0000 7fbfea7fc640 1 -- 172.21.15.76:0/520246056 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8022e00 con 0x563674a40730 2024-03-18T14:52:57.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.265+0000 7fc009dc3740 1 -- 172.21.15.76:0/520246056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a40730 msgr2=0x563674a35030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.265+0000 7fc009dc3740 1 --2- 172.21.15.76:0/520246056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a40730 0x563674a35030 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fbff8011f90 tx=0x7fbff800e000 comp rx=0 tx=0).stop 2024-03-18T14:52:57.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc009dc3740 1 -- 172.21.15.76:0/520246056 shutdown_connections 2024-03-18T14:52:57.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc009dc3740 1 --2- 172.21.15.76:0/520246056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a40730 0x563674a35030 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc009dc3740 1 --2- 172.21.15.76:0/520246056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3b5c0 0x563674a34af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc009dc3740 1 --2- 172.21.15.76:0/520246056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a3abd0 0x563674a36520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc009dc3740 1 -- 172.21.15.76:0/520246056 >> 172.21.15.76:0/520246056 conn(0x563674a36140 msgr2=0x563674a3fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc009dc3740 1 -- 172.21.15.76:0/520246056 shutdown_connections 2024-03-18T14:52:57.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc009dc3740 1 -- 172.21.15.76:0/520246056 wait complete. 2024-03-18T14:52:57.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc009dc3740 1 -- start start 2024-03-18T14:52:57.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc009dc3740 1 Processor -- start 2024-03-18T14:52:57.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc009dc3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a3abd0 0x563674a474b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc009dc3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3b5c0 0x563674a479f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc009dc3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a40730 0x563674a4aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc009dc3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563674a524d0 con 0x563674a40730 2024-03-18T14:52:57.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc009dc3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563674a51e00 con 0x563674a3b5c0 2024-03-18T14:52:57.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc009dc3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563674a52ba0 con 0x563674a3abd0 2024-03-18T14:52:57.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc009dc3740 10 client.0 _ll_register_callbacks cb 0x7fbff8009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:52:57.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc005695640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a40730 0x563674a4aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc005695640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a40730 0x563674a4aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51180/0 (socket says 172.21.15.76:51180) 2024-03-18T14:52:57.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc005695640 1 -- 172.21.15.76:0/4266080837 learned_addr learned my addr 172.21.15.76:0/4266080837 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:57.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fc004e94640 1 --2- 172.21.15.76:0/4266080837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a3abd0 0x563674a474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.266+0000 7fbffffff640 1 --2- 172.21.15.76:0/4266080837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3b5c0 0x563674a479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fc005695640 1 -- 172.21.15.76:0/4266080837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a3abd0 msgr2=0x563674a474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fc005695640 1 --2- 172.21.15.76:0/4266080837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a3abd0 0x563674a474b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fc005695640 1 -- 172.21.15.76:0/4266080837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3b5c0 msgr2=0x563674a479f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fc005695640 1 --2- 172.21.15.76:0/4266080837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3b5c0 0x563674a479f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fc005695640 1 -- 172.21.15.76:0/4266080837 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563674a4b000 con 0x563674a40730 2024-03-18T14:52:57.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fc005695640 1 --2- 172.21.15.76:0/4266080837 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a40730 0x563674a4aa10 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fbff8003f80 tx=0x7fbff8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=427b0913ad36e4ed server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fbfdffff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:52:57.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fbfdffff640 1 -- 172.21.15.76:0/4266080837 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff803a060 con 0x563674a40730 2024-03-18T14:52:57.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fbfdffff640 1 -- 172.21.15.76:0/4266080837 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fbff8004b20 con 0x563674a40730 2024-03-18T14:52:57.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fc009dc3740 1 -- 172.21.15.76:0/4266080837 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbff800fcc0 con 0x563674a40730 2024-03-18T14:52:57.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fbfdffff640 1 -- 172.21.15.76:0/4266080837 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fbff8019040 con 0x563674a40730 2024-03-18T14:52:57.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fc009dc3740 10 client.15231 fetch_fsmap learned FSMap version 23 2024-03-18T14:52:57.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fc009dc3740 1 -- 172.21.15.76:0/4266080837 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x563674ab7890 con 0x563674a40730 2024-03-18T14:52:57.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fbfdffff640 1 -- 172.21.15.76:0/4266080837 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbff800fcc0 con 0x563674a40730 2024-03-18T14:52:57.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fbfdffff640 1 -- 172.21.15.76:0/4266080837 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fbff801f040 con 0x563674a40730 2024-03-18T14:52:57.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fc009dc3740 10 client.15231 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T14:52:57.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fc009dc3740 10 client.15231 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T14:52:57.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fc009dc3740 20 client.15231 populate_metadata read hostname 'smithi076' 2024-03-18T14:52:57.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fc009dc3740 1 --2- 172.21.15.76:0/4266080837 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x563674abc2f0 0x563674abb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fc009dc3740 4 client.15231 mds_command: new command op to 24490 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T14:52:57.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.267+0000 7fc009dc3740 1 -- 172.21.15.76:0/4266080837 --> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] -- command(tid 0: {"prefix": "status"}) -- 0x5636749a6510 con 0x563674abc2f0 2024-03-18T14:52:57.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.268+0000 7fc004e94640 1 --2- 172.21.15.76:0/4266080837 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x563674abc2f0 0x563674abb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.268+0000 7fbfdffff640 10 client.15231 ms_handle_connect on v2:172.21.15.174:6838/399810652 2024-03-18T14:52:57.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.268+0000 7fc004e94640 1 --2- 172.21.15.76:0/4266080837 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x563674abc2f0 0x563674abb000 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ea6a4724a52400a3 server_cookie=7295a622b3476b40 in_seq=0 out_seq=0 2024-03-18T14:52:57.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.268+0000 7fbfdffff640 1 -- 172.21.15.76:0/4266080837 <== mds.1 v2:172.21.15.174:6838/399810652 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5636749a6510 con 0x563674abc2f0 2024-03-18T14:52:57.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.268+0000 7fbfdffff640 10 client.15231 handle_command_reply: tid=0 2024-03-18T14:52:57.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.268+0000 7fc009dc3740 1 client.15231 shutdown 2024-03-18T14:52:57.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.268+0000 7fc009dc3740 1 -- 172.21.15.76:0/4266080837 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a40730 msgr2=0x563674a4aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.268+0000 7fc009dc3740 1 --2- 172.21.15.76:0/4266080837 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a40730 0x563674a4aa10 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fbff8003f80 tx=0x7fbff8004a40 comp rx=0 tx=0).stop 2024-03-18T14:52:57.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.268+0000 7fc009dc3740 1 -- 172.21.15.76:0/4266080837 shutdown_connections 2024-03-18T14:52:57.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.268+0000 7fc009dc3740 1 --2- 172.21.15.76:0/4266080837 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x563674abc2f0 0x563674abb000 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.268+0000 7fc009dc3740 1 --2- 172.21.15.76:0/4266080837 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563674a40730 0x563674a4aa10 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.268+0000 7fc009dc3740 1 --2- 172.21.15.76:0/4266080837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563674a3b5c0 0x563674a479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.268+0000 7fbfdffff640 0 client.15231 ms_handle_reset on v2:172.21.15.174:6838/399810652 2024-03-18T14:52:57.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.268+0000 7fc009dc3740 1 --2- 172.21.15.76:0/4266080837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563674a3abd0 0x563674a474b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.268+0000 7fc009dc3740 1 -- 172.21.15.76:0/4266080837 >> 172.21.15.76:0/4266080837 conn(0x563674a36140 msgr2=0x563674a3f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.268+0000 7fc009dc3740 1 -- 172.21.15.76:0/4266080837 shutdown_connections 2024-03-18T14:52:57.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.268+0000 7fc009dc3740 1 -- 172.21.15.76:0/4266080837 wait complete. 2024-03-18T14:52:57.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.269+0000 7fc009dc3740 20 client.15231 trim_cache size 0 max 16384 2024-03-18T14:52:57.276 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T14:52:57.276 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "9cc83b72-e535-11ee-95c9-87774f69a715", 2024-03-18T14:52:57.276 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 1, 2024-03-18T14:52:57.276 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24490, 2024-03-18T14:52:57.276 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-03-18T14:52:57.276 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-03-18T14:52:57.276 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-03-18T14:52:57.276 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 112.251696929, 2024-03-18T14:52:57.276 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 23, 2024-03-18T14:52:57.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 88, 2024-03-18T14:52:57.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T14:52:57.277 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 128.54264622400001 2024-03-18T14:52:57.277 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T14:52:57.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.269+0000 7fc0070f8640 1 -- 172.21.15.76:0/1046547100 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fbff00a2800 msgr2=0x7fbff00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.269+0000 7fc0070f8640 1 --2- 172.21.15.76:0/1046547100 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fbff00a2800 0x7fbff00a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x563674a3be20 tx=0x7fbff8004ab0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.269+0000 7fc0070f8640 1 -- 172.21.15.76:0/1046547100 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc000102260 msgr2=0x7fc000197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.269+0000 7fc0070f8640 1 --2- 172.21.15.76:0/1046547100 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc000102260 0x7fc000197b50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbff4005940 tx=0x7fbff4032470 comp rx=0 tx=0).stop 2024-03-18T14:52:57.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.269+0000 7fc0070f8640 1 -- 172.21.15.76:0/1046547100 shutdown_connections 2024-03-18T14:52:57.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.269+0000 7fc0070f8640 1 --2- 172.21.15.76:0/1046547100 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fbff00a2800 0x7fbff00a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.269+0000 7fc0070f8640 1 --2- 172.21.15.76:0/1046547100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0001074c0 0x7fc00019f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.269+0000 7fc0070f8640 1 --2- 172.21.15.76:0/1046547100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc000104b90 0x7fc000198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.269+0000 7fc0070f8640 1 --2- 172.21.15.76:0/1046547100 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc000102260 0x7fc000197b50 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.269+0000 7fc0070f8640 1 -- 172.21.15.76:0/1046547100 >> 172.21.15.76:0/1046547100 conn(0x7fc0000fbc50 msgr2=0x7fc000102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.269+0000 7fc0070f8640 1 -- 172.21.15.76:0/1046547100 shutdown_connections 2024-03-18T14:52:57.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.269+0000 7fc0070f8640 1 -- 172.21.15.76:0/1046547100 wait complete. 2024-03-18T14:52:57.281 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T14:52:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.382+0000 7eff9b75b640 1 -- 172.21.15.76:0/4098950713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff941022a0 msgr2=0x7eff94108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.382+0000 7eff9b75b640 1 --2- 172.21.15.76:0/4098950713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff941022a0 0x7eff94108b30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7eff8c009ca0 tx=0x7eff8c02c0f0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.382+0000 7eff9b75b640 1 -- 172.21.15.76:0/4098950713 shutdown_connections 2024-03-18T14:52:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.382+0000 7eff9b75b640 1 --2- 172.21.15.76:0/4098950713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff941022a0 0x7eff94108b30 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.382+0000 7eff9b75b640 1 --2- 172.21.15.76:0/4098950713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff941018e0 0x7eff94101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.382+0000 7eff9b75b640 1 --2- 172.21.15.76:0/4098950713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff941006f0 0x7eff94100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.382+0000 7eff9b75b640 1 -- 172.21.15.76:0/4098950713 >> 172.21.15.76:0/4098950713 conn(0x7eff940fbc80 msgr2=0x7eff940fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.382+0000 7eff9b75b640 1 -- 172.21.15.76:0/4098950713 shutdown_connections 2024-03-18T14:52:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.382+0000 7eff9b75b640 1 -- 172.21.15.76:0/4098950713 wait complete. 2024-03-18T14:52:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff9b75b640 1 Processor -- start 2024-03-18T14:52:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff9b75b640 1 -- start start 2024-03-18T14:52:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff9b75b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff941006f0 0x7eff9406a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff9b75b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff941018e0 0x7eff9419c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff9b75b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff941022a0 0x7eff9419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff9b75b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7eff9410c370 con 0x7eff941018e0 2024-03-18T14:52:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff9b75b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff9410c1f0 con 0x7eff941006f0 2024-03-18T14:52:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff9b75b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7eff9410c4f0 con 0x7eff941022a0 2024-03-18T14:52:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff98cf6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff941018e0 0x7eff9419c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff98cf6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff941018e0 0x7eff9419c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51192/0 (socket says 172.21.15.76:51192) 2024-03-18T14:52:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff98cf6640 1 -- 172.21.15.76:0/2438478174 learned_addr learned my addr 172.21.15.76:0/2438478174 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff994f7640 1 --2- 172.21.15.76:0/2438478174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff941006f0 0x7eff9406a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff99cf8640 1 --2- 172.21.15.76:0/2438478174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff941022a0 0x7eff9419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff98cf6640 1 -- 172.21.15.76:0/2438478174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff941022a0 msgr2=0x7eff9419ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff98cf6640 1 --2- 172.21.15.76:0/2438478174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff941022a0 0x7eff9419ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff98cf6640 1 -- 172.21.15.76:0/2438478174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff941006f0 msgr2=0x7eff9406a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff98cf6640 1 --2- 172.21.15.76:0/2438478174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff941006f0 0x7eff9406a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff98cf6640 1 -- 172.21.15.76:0/2438478174 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9419f6f0 con 0x7eff941018e0 2024-03-18T14:52:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff994f7640 1 --2- 172.21.15.76:0/2438478174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff941006f0 0x7eff9406a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.383+0000 7eff98cf6640 1 --2- 172.21.15.76:0/2438478174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff941018e0 0x7eff9419c7f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7eff8400ea60 tx=0x7eff8400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.384+0000 7eff8a7fc640 1 -- 172.21.15.76:0/2438478174 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8400cde0 con 0x7eff941018e0 2024-03-18T14:52:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.384+0000 7eff8a7fc640 1 -- 172.21.15.76:0/2438478174 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff840035f0 con 0x7eff941018e0 2024-03-18T14:52:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.384+0000 7eff9b75b640 1 -- 172.21.15.76:0/2438478174 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff9419f9e0 con 0x7eff941018e0 2024-03-18T14:52:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.384+0000 7eff8a7fc640 1 -- 172.21.15.76:0/2438478174 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff84010670 con 0x7eff941018e0 2024-03-18T14:52:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.384+0000 7eff9b75b640 1 -- 172.21.15.76:0/2438478174 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff9419ff30 con 0x7eff941018e0 2024-03-18T14:52:57.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.386+0000 7eff8a7fc640 1 -- 172.21.15.76:0/2438478174 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7eff840037a0 con 0x7eff941018e0 2024-03-18T14:52:57.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.386+0000 7eff9e424740 1 Processor -- start 2024-03-18T14:52:57.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.386+0000 7eff9e424740 1 -- start start 2024-03-18T14:52:57.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.386+0000 7eff9e424740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229bbd0 0x56191229bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.386+0000 7eff9e424740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229c5c0 0x5619122a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.386+0000 7eff9e424740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5619122a1730 0x5619122a3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.386+0000 7eff9e424740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5619121ffc80 con 0x56191229bbd0 2024-03-18T14:52:57.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.386+0000 7eff9e424740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5619121a8aa0 con 0x56191229c5c0 2024-03-18T14:52:57.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.386+0000 7eff9e424740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5619121fbd20 con 0x5619122a1730 2024-03-18T14:52:57.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.386+0000 7eff8a7fc640 1 --2- 172.21.15.76:0/2438478174 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7eff800a2800 0x7eff800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.386+0000 7eff99cf8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229bbd0 0x56191229bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.386+0000 7eff99cf8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229bbd0 0x56191229bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51194/0 (socket says 172.21.15.76:51194) 2024-03-18T14:52:57.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.386+0000 7eff99cf8640 1 -- 172.21.15.76:0/1432926925 learned_addr learned my addr 172.21.15.76:0/1432926925 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:57.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.386+0000 7eff994f7640 1 --2- 172.21.15.76:0/1432926925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229c5c0 0x5619122a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.386+0000 7eff98cf6640 1 --2- 172.21.15.76:0/1432926925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5619122a1730 0x5619122a3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.386+0000 7eff99cf8640 1 --2- 172.21.15.76:0/2438478174 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7eff800a2800 0x7eff800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.387+0000 7eff8a7fc640 1 -- 172.21.15.76:0/2438478174 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7eff840ced20 con 0x7eff941018e0 2024-03-18T14:52:57.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.387+0000 7eff994f7640 1 -- 172.21.15.76:0/1432926925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5619122a1730 msgr2=0x5619122a3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.387+0000 7eff994f7640 1 --2- 172.21.15.76:0/1432926925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5619122a1730 0x5619122a3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.387+0000 7eff994f7640 1 -- 172.21.15.76:0/1432926925 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229bbd0 msgr2=0x56191229bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.387+0000 7eff994f7640 1 --2- 172.21.15.76:0/1432926925 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229bbd0 0x56191229bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.387+0000 7eff994f7640 1 -- 172.21.15.76:0/1432926925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619121caff0 con 0x56191229c5c0 2024-03-18T14:52:57.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.387+0000 7eff994f7640 1 --2- 172.21.15.76:0/1432926925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229c5c0 0x5619122a10c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7eff90009fa0 tx=0x7eff900118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85e7465f8ddf8b00 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.387+0000 7eff99cf8640 1 --2- 172.21.15.76:0/2438478174 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7eff800a2800 0x7eff800a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x56191229ce20 tx=0x7eff8c032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.387+0000 7eff7f7fe640 1 -- 172.21.15.76:0/1432926925 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff90004030 con 0x56191229c5c0 2024-03-18T14:52:57.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.387+0000 7eff7f7fe640 1 -- 172.21.15.76:0/1432926925 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff900041d0 con 0x56191229c5c0 2024-03-18T14:52:57.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.387+0000 7eff7f7fe640 1 -- 172.21.15.76:0/1432926925 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff9000f910 con 0x56191229c5c0 2024-03-18T14:52:57.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.388+0000 7eff9e424740 1 -- 172.21.15.76:0/1432926925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229c5c0 msgr2=0x5619122a10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.388+0000 7eff9e424740 1 --2- 172.21.15.76:0/1432926925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229c5c0 0x5619122a10c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7eff90009fa0 tx=0x7eff900118d0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.388+0000 7eff9e424740 1 -- 172.21.15.76:0/1432926925 shutdown_connections 2024-03-18T14:52:57.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.388+0000 7eff9e424740 1 --2- 172.21.15.76:0/1432926925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5619122a1730 0x5619122a3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.388+0000 7eff9e424740 1 --2- 172.21.15.76:0/1432926925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229c5c0 0x5619122a10c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.388+0000 7eff9e424740 1 --2- 172.21.15.76:0/1432926925 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229bbd0 0x56191229bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.388+0000 7eff9e424740 1 -- 172.21.15.76:0/1432926925 >> 172.21.15.76:0/1432926925 conn(0x561912297140 msgr2=0x561912297540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.388+0000 7eff9e424740 1 -- 172.21.15.76:0/1432926925 shutdown_connections 2024-03-18T14:52:57.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.388+0000 7eff9e424740 1 -- 172.21.15.76:0/1432926925 wait complete. 2024-03-18T14:52:57.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.388+0000 7eff9e424740 1 -- start start 2024-03-18T14:52:57.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.388+0000 7eff9e424740 1 Processor -- start 2024-03-18T14:52:57.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff9e424740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56191229bbd0 0x5619122a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff9e424740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229c5c0 0x5619122a6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff9e424740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619122a1730 0x5619122ad390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff9e424740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5619122a46e0 con 0x5619122a1730 2024-03-18T14:52:57.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff9e424740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5619121bbef0 con 0x56191229c5c0 2024-03-18T14:52:57.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff9e424740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5619122a4860 con 0x56191229bbd0 2024-03-18T14:52:57.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff9e424740 10 client.0 _ll_register_callbacks cb 0x5619121b6dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:52:57.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff99cf8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619122a1730 0x5619122ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff99cf8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619122a1730 0x5619122ad390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51196/0 (socket says 172.21.15.76:51196) 2024-03-18T14:52:57.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff994f7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56191229bbd0 0x5619122a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff99cf8640 1 -- 172.21.15.76:0/4225734679 learned_addr learned my addr 172.21.15.76:0/4225734679 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:57.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff98cf6640 1 --2- 172.21.15.76:0/4225734679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229c5c0 0x5619122a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff99cf8640 1 -- 172.21.15.76:0/4225734679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56191229bbd0 msgr2=0x5619122a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff99cf8640 1 --2- 172.21.15.76:0/4225734679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56191229bbd0 0x5619122a5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff99cf8640 1 -- 172.21.15.76:0/4225734679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229c5c0 msgr2=0x5619122a6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff99cf8640 1 --2- 172.21.15.76:0/4225734679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229c5c0 0x5619122a6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff99cf8640 1 -- 172.21.15.76:0/4225734679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619121c8a30 con 0x5619122a1730 2024-03-18T14:52:57.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff994f7640 1 --2- 172.21.15.76:0/4225734679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56191229bbd0 0x5619122a5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:57.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff98cf6640 1 --2- 172.21.15.76:0/4225734679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229c5c0 0x5619122a6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:57.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff99cf8640 1 --2- 172.21.15.76:0/4225734679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619122a1730 0x5619122ad390 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7eff8c02c630 tx=0x7eff8c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=128d47a0a87e2a93 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff7cff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:52:57.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff7cff9640 1 -- 172.21.15.76:0/4225734679 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c054060 con 0x5619122a1730 2024-03-18T14:52:57.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff7cff9640 1 -- 172.21.15.76:0/4225734679 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7eff8c004040 con 0x5619122a1730 2024-03-18T14:52:57.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff9e424740 1 -- 172.21.15.76:0/4225734679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff900041d0 con 0x5619122a1730 2024-03-18T14:52:57.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff7cff9640 1 -- 172.21.15.76:0/4225734679 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7eff8c039040 con 0x5619122a1730 2024-03-18T14:52:57.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff9e424740 10 client.15243 fetch_fsmap learned FSMap version 23 2024-03-18T14:52:57.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.389+0000 7eff9e424740 1 -- 172.21.15.76:0/4225734679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561911f142d0 con 0x5619122a1730 2024-03-18T14:52:57.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.390+0000 7eff7cff9640 1 -- 172.21.15.76:0/4225734679 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff900041d0 con 0x5619122a1730 2024-03-18T14:52:57.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.390+0000 7eff7cff9640 1 -- 172.21.15.76:0/4225734679 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7eff8c04f500 con 0x5619122a1730 2024-03-18T14:52:57.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.390+0000 7eff9e424740 10 client.15243 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T14:52:57.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.390+0000 7eff9e424740 10 client.15243 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T14:52:57.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.390+0000 7eff9e424740 20 client.15243 populate_metadata read hostname 'smithi076' 2024-03-18T14:52:57.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.390+0000 7eff9e424740 1 --2- 172.21.15.76:0/4225734679 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5619122b2960 0x5619122b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.390+0000 7eff9e424740 4 client.15243 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:52:57.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.390+0000 7eff9e424740 1 -- 172.21.15.76:0/4225734679 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56191220d1c0 con 0x5619122b2960 2024-03-18T14:52:57.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.390+0000 7eff994f7640 1 --2- 172.21.15.76:0/4225734679 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5619122b2960 0x5619122b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.390+0000 7eff7cff9640 10 client.15243 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:52:57.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.390+0000 7eff994f7640 1 --2- 172.21.15.76:0/4225734679 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5619122b2960 0x5619122b3dd0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6430e6f5fc7b0d1 server_cookie=ee1b91a5ef83d25c in_seq=0 out_seq=0 2024-03-18T14:52:57.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.391+0000 7eff7cff9640 1 -- 172.21.15.76:0/4225734679 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56191220d1c0 con 0x5619122b2960 2024-03-18T14:52:57.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.391+0000 7eff7cff9640 10 client.15243 handle_command_reply: tid=0 2024-03-18T14:52:57.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.391+0000 7eff9e424740 1 client.15243 shutdown 2024-03-18T14:52:57.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.391+0000 7eff9e424740 1 -- 172.21.15.76:0/4225734679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619122a1730 msgr2=0x5619122ad390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.391+0000 7eff9e424740 1 --2- 172.21.15.76:0/4225734679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619122a1730 0x5619122ad390 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7eff8c02c630 tx=0x7eff8c0095c0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.391+0000 7eff9e424740 1 -- 172.21.15.76:0/4225734679 shutdown_connections 2024-03-18T14:52:57.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.391+0000 7eff9e424740 1 --2- 172.21.15.76:0/4225734679 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5619122b2960 0x5619122b3dd0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.391+0000 7eff9e424740 1 --2- 172.21.15.76:0/4225734679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619122a1730 0x5619122ad390 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.391+0000 7eff7cff9640 0 client.15243 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:52:57.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.391+0000 7eff9e424740 1 --2- 172.21.15.76:0/4225734679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229c5c0 0x5619122a6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.391+0000 7eff9e424740 1 --2- 172.21.15.76:0/4225734679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56191229bbd0 0x5619122a5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.391+0000 7eff9e424740 1 -- 172.21.15.76:0/4225734679 >> 172.21.15.76:0/4225734679 conn(0x561912297140 msgr2=0x561912296380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.391+0000 7eff9e424740 1 -- 172.21.15.76:0/4225734679 shutdown_connections 2024-03-18T14:52:57.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.391+0000 7eff9e424740 1 -- 172.21.15.76:0/4225734679 wait complete. 2024-03-18T14:52:57.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.391+0000 7eff9e424740 20 client.15243 trim_cache size 0 max 16384 2024-03-18T14:52:57.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.409+0000 7eff9e424740 1 Processor -- start 2024-03-18T14:52:57.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.409+0000 7eff9e424740 1 -- start start 2024-03-18T14:52:57.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.409+0000 7eff9e424740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229bbd0 0x5619122a4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.409+0000 7eff9e424740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229c5c0 0x5619122a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.409+0000 7eff9e424740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5619122a1730 0x5619122a5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.409+0000 7eff9e424740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5619122b2110 con 0x56191229c5c0 2024-03-18T14:52:57.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.409+0000 7eff9e424740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5619122b27e0 con 0x56191229bbd0 2024-03-18T14:52:57.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.409+0000 7eff9e424740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5619122b2eb0 con 0x5619122a1730 2024-03-18T14:52:57.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.410+0000 7eff98cf6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229c5c0 0x5619122a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.410+0000 7eff98cf6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229c5c0 0x5619122a51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51208/0 (socket says 172.21.15.76:51208) 2024-03-18T14:52:57.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.410+0000 7eff98cf6640 1 -- 172.21.15.76:0/1515032090 learned_addr learned my addr 172.21.15.76:0/1515032090 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:57.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.410+0000 7eff994f7640 1 --2- 172.21.15.76:0/1515032090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229bbd0 0x5619122a4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.410+0000 7eff99cf8640 1 --2- 172.21.15.76:0/1515032090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5619122a1730 0x5619122a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.410+0000 7eff98cf6640 1 -- 172.21.15.76:0/1515032090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5619122a1730 msgr2=0x5619122a5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.410+0000 7eff98cf6640 1 --2- 172.21.15.76:0/1515032090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5619122a1730 0x5619122a5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.410+0000 7eff98cf6640 1 -- 172.21.15.76:0/1515032090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229bbd0 msgr2=0x5619122a4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.410+0000 7eff98cf6640 1 --2- 172.21.15.76:0/1515032090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229bbd0 0x5619122a4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.410+0000 7eff98cf6640 1 -- 172.21.15.76:0/1515032090 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561911fde6b0 con 0x56191229c5c0 2024-03-18T14:52:57.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.410+0000 7eff994f7640 1 --2- 172.21.15.76:0/1515032090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229bbd0 0x5619122a4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:57.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.410+0000 7eff99cf8640 1 --2- 172.21.15.76:0/1515032090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5619122a1730 0x5619122a5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:52:57.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.410+0000 7eff98cf6640 1 --2- 172.21.15.76:0/1515032090 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229c5c0 0x5619122a51d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7eff840cc8a0 tx=0x7eff84007b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c497e6728ef9a37 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.410+0000 7eff7effd640 1 -- 172.21.15.76:0/1515032090 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff84099cb0 con 0x56191229c5c0 2024-03-18T14:52:57.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.410+0000 7eff7effd640 1 -- 172.21.15.76:0/1515032090 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff8409bdf0 con 0x56191229c5c0 2024-03-18T14:52:57.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.410+0000 7eff7effd640 1 -- 172.21.15.76:0/1515032090 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff840a2b90 con 0x56191229c5c0 2024-03-18T14:52:57.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.411+0000 7eff9e424740 1 -- 172.21.15.76:0/1515032090 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229c5c0 msgr2=0x5619122a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.411+0000 7eff9e424740 1 --2- 172.21.15.76:0/1515032090 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229c5c0 0x5619122a51d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7eff840cc8a0 tx=0x7eff84007b90 comp rx=0 tx=0).stop 2024-03-18T14:52:57.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.411+0000 7eff9e424740 1 -- 172.21.15.76:0/1515032090 shutdown_connections 2024-03-18T14:52:57.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.411+0000 7eff9e424740 1 --2- 172.21.15.76:0/1515032090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5619122a1730 0x5619122a5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.411+0000 7eff9e424740 1 --2- 172.21.15.76:0/1515032090 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229c5c0 0x5619122a51d0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.411+0000 7eff9e424740 1 --2- 172.21.15.76:0/1515032090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56191229bbd0 0x5619122a4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.411+0000 7eff9e424740 1 -- 172.21.15.76:0/1515032090 >> 172.21.15.76:0/1515032090 conn(0x561912297140 msgr2=0x5619122a5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.411+0000 7eff9e424740 1 -- 172.21.15.76:0/1515032090 shutdown_connections 2024-03-18T14:52:57.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.411+0000 7eff9e424740 1 -- 172.21.15.76:0/1515032090 wait complete. 2024-03-18T14:52:57.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.411+0000 7eff9e424740 1 -- start start 2024-03-18T14:52:57.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.411+0000 7eff9e424740 1 Processor -- start 2024-03-18T14:52:57.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff9e424740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229bbd0 0x56191229f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff9e424740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56191229c5c0 0x56191229f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff9e424740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619122a1730 0x56191231a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff9e424740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5619122b3c50 con 0x56191229bbd0 2024-03-18T14:52:57.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff9e424740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5619122b3580 con 0x5619122a1730 2024-03-18T14:52:57.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff9e424740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5619122b4320 con 0x56191229c5c0 2024-03-18T14:52:57.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff9e424740 10 client.0 _ll_register_callbacks cb 0x7eff90005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:52:57.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff994f7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229bbd0 0x56191229f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff994f7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229bbd0 0x56191229f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51222/0 (socket says 172.21.15.76:51222) 2024-03-18T14:52:57.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff994f7640 1 -- 172.21.15.76:0/2123977154 learned_addr learned my addr 172.21.15.76:0/2123977154 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:57.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff99cf8640 1 --2- 172.21.15.76:0/2123977154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619122a1730 0x56191231a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff98cf6640 1 --2- 172.21.15.76:0/2123977154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56191229c5c0 0x56191229f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff994f7640 1 -- 172.21.15.76:0/2123977154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56191229c5c0 msgr2=0x56191229f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff994f7640 1 --2- 172.21.15.76:0/2123977154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56191229c5c0 0x56191229f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff994f7640 1 -- 172.21.15.76:0/2123977154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619122a1730 msgr2=0x56191231a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:52:57.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff994f7640 1 --2- 172.21.15.76:0/2123977154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619122a1730 0x56191231a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff994f7640 1 -- 172.21.15.76:0/2123977154 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56191231af60 con 0x56191229bbd0 2024-03-18T14:52:57.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff994f7640 1 --2- 172.21.15.76:0/2123977154 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229bbd0 0x56191229f4a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7eff90009380 tx=0x7eff900096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4dd37d4bb9c115 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:57.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff7cff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:52:57.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff7cff9640 1 -- 172.21.15.76:0/2123977154 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff90003a40 con 0x56191229bbd0 2024-03-18T14:52:57.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff7cff9640 1 -- 172.21.15.76:0/2123977154 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7eff9002ba30 con 0x56191229bbd0 2024-03-18T14:52:57.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff9e424740 1 -- 172.21.15.76:0/2123977154 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff8409bdf0 con 0x56191229bbd0 2024-03-18T14:52:57.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff7cff9640 1 -- 172.21.15.76:0/2123977154 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7eff9002bc30 con 0x56191229bbd0 2024-03-18T14:52:57.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff9e424740 10 client.15249 fetch_fsmap learned FSMap version 23 2024-03-18T14:52:57.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.412+0000 7eff9e424740 1 -- 172.21.15.76:0/2123977154 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56191231c260 con 0x56191229bbd0 2024-03-18T14:52:57.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.413+0000 7eff7cff9640 1 -- 172.21.15.76:0/2123977154 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff8409bdf0 con 0x56191229bbd0 2024-03-18T14:52:57.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.413+0000 7eff7cff9640 1 -- 172.21.15.76:0/2123977154 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7eff9002b400 con 0x56191229bbd0 2024-03-18T14:52:57.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.413+0000 7eff9e424740 10 client.15249 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T14:52:57.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.413+0000 7eff9e424740 10 client.15249 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T14:52:57.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.413+0000 7eff9e424740 20 client.15249 populate_metadata read hostname 'smithi076' 2024-03-18T14:52:57.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.413+0000 7eff9e424740 1 --2- 172.21.15.76:0/2123977154 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561912320c80 0x56191231d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:57.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.413+0000 7eff9e424740 4 client.15249 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T14:52:57.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.413+0000 7eff9e424740 1 -- 172.21.15.76:0/2123977154 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "status"}) -- 0x561912207510 con 0x561912320c80 2024-03-18T14:52:57.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.413+0000 7eff98cf6640 1 --2- 172.21.15.76:0/2123977154 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561912320c80 0x56191231d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:57.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.413+0000 7eff98cf6640 1 --2- 172.21.15.76:0/2123977154 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561912320c80 0x56191231d790 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a480b12967f601c server_cookie=c08e94409f3534 in_seq=0 out_seq=0 2024-03-18T14:52:57.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.413+0000 7eff7cff9640 10 client.15249 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:52:57.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.413+0000 7eff7cff9640 1 -- 172.21.15.76:0/2123977154 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561912207510 con 0x561912320c80 2024-03-18T14:52:57.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.413+0000 7eff7cff9640 10 client.15249 handle_command_reply: tid=0 2024-03-18T14:52:57.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.413+0000 7eff9e424740 1 client.15249 shutdown 2024-03-18T14:52:57.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9e424740 1 -- 172.21.15.76:0/2123977154 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229bbd0 msgr2=0x56191229f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9e424740 1 --2- 172.21.15.76:0/2123977154 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229bbd0 0x56191229f4a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7eff90009380 tx=0x7eff900096a0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9e424740 1 -- 172.21.15.76:0/2123977154 shutdown_connections 2024-03-18T14:52:57.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9e424740 1 --2- 172.21.15.76:0/2123977154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56191229c5c0 0x56191229f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9e424740 1 --2- 172.21.15.76:0/2123977154 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561912320c80 0x56191231d790 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9e424740 1 --2- 172.21.15.76:0/2123977154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619122a1730 0x56191231a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff7cff9640 0 client.15249 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:52:57.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9e424740 1 --2- 172.21.15.76:0/2123977154 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56191229bbd0 0x56191229f4a0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9e424740 1 -- 172.21.15.76:0/2123977154 >> 172.21.15.76:0/2123977154 conn(0x561912297140 msgr2=0x56191229a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9e424740 1 -- 172.21.15.76:0/2123977154 shutdown_connections 2024-03-18T14:52:57.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9e424740 1 -- 172.21.15.76:0/2123977154 wait complete. 2024-03-18T14:52:57.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9e424740 20 client.15249 trim_cache size 0 max 16384 2024-03-18T14:52:57.421 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T14:52:57.421 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "9cc83b72-e535-11ee-95c9-87774f69a715", 2024-03-18T14:52:57.421 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 0, 2024-03-18T14:52:57.422 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24524, 2024-03-18T14:52:57.422 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-03-18T14:52:57.422 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-03-18T14:52:57.422 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-03-18T14:52:57.422 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 115.41711739599999, 2024-03-18T14:52:57.422 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 23, 2024-03-18T14:52:57.422 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 88, 2024-03-18T14:52:57.422 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T14:52:57.422 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 119.737918335 2024-03-18T14:52:57.422 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T14:52:57.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9b75b640 1 -- 172.21.15.76:0/2438478174 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7eff800a2800 msgr2=0x7eff800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9b75b640 1 --2- 172.21.15.76:0/2438478174 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7eff800a2800 0x7eff800a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x56191229ce20 tx=0x7eff8c032590 comp rx=0 tx=0).stop 2024-03-18T14:52:57.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9b75b640 1 -- 172.21.15.76:0/2438478174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff941018e0 msgr2=0x7eff9419c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:57.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9b75b640 1 --2- 172.21.15.76:0/2438478174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff941018e0 0x7eff9419c7f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7eff8400ea60 tx=0x7eff8400ef20 comp rx=0 tx=0).stop 2024-03-18T14:52:57.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9b75b640 1 -- 172.21.15.76:0/2438478174 shutdown_connections 2024-03-18T14:52:57.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9b75b640 1 --2- 172.21.15.76:0/2438478174 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7eff800a2800 0x7eff800a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9b75b640 1 --2- 172.21.15.76:0/2438478174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff941022a0 0x7eff9419ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9b75b640 1 --2- 172.21.15.76:0/2438478174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff941018e0 0x7eff9419c7f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9b75b640 1 --2- 172.21.15.76:0/2438478174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff941006f0 0x7eff9406a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:57.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9b75b640 1 -- 172.21.15.76:0/2438478174 >> 172.21.15.76:0/2438478174 conn(0x7eff940fbc80 msgr2=0x7eff94104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:57.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9b75b640 1 -- 172.21.15.76:0/2438478174 shutdown_connections 2024-03-18T14:52:57.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:57.414+0000 7eff9b75b640 1 -- 172.21.15.76:0/2438478174 wait complete. 2024-03-18T14:52:57.426 INFO:teuthology.run_tasks:Running task kclient... 2024-03-18T14:52:57.449 INFO:tasks.kclient:Mounting kernel clients... 2024-03-18T14:52:57.449 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-18T14:52:57.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:52:57.449 DEBUG:teuthology.orchestra.run.smithi076:> ip netns list 2024-03-18T14:52:57.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:52:57.465 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link delete ceph-brx 2024-03-18T14:52:57.533 INFO:teuthology.orchestra.run.smithi076.stderr:Cannot find device "ceph-brx" 2024-03-18T14:52:57.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:57 smithi076 ceph-mon[32205]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:52:57.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:57 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/760175264' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:52:57.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:57 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3164818151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:57.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:52:57.535 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-18T14:52:57.535 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T14:52:57.536 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T14:52:57.536 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-18T14:52:57.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:57 smithi174 ceph-mon[36787]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:52:57.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:57 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/760175264' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:52:57.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:57 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3164818151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:57.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:57 smithi077 ceph-mon[35865]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T14:52:57.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:57 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/760175264' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:52:57.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:57 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3164818151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:52:57.815 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T14:52:59.580 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:52:59 smithi076 ceph-mon[32205]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T14:52:59.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.679+0000 7fa65359a700 1 -- 172.21.15.76:0/3256383767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa64c101e50 msgr2=0x7fa64c105ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:59.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.679+0000 7fa65359a700 1 --2- 172.21.15.76:0/3256383767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa64c101e50 0x7fa64c105ea0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa63c0099c0 tx=0x7fa63c009cd0 comp rx=0 tx=0).stop 2024-03-18T14:52:59.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.679+0000 7fa65359a700 1 -- 172.21.15.76:0/3256383767 shutdown_connections 2024-03-18T14:52:59.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.679+0000 7fa65359a700 1 --2- 172.21.15.76:0/3256383767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa64c106580 0x7fa64c10e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:59.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.679+0000 7fa65359a700 1 --2- 172.21.15.76:0/3256383767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa64c1014a0 0x7fa64c101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:59.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.679+0000 7fa65359a700 1 --2- 172.21.15.76:0/3256383767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa64c101e50 0x7fa64c105ea0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:59.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.679+0000 7fa65359a700 1 -- 172.21.15.76:0/3256383767 >> 172.21.15.76:0/3256383767 conn(0x7fa64c078300 msgr2=0x7fa64c078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:59.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.679+0000 7fa65359a700 1 -- 172.21.15.76:0/3256383767 shutdown_connections 2024-03-18T14:52:59.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.679+0000 7fa65359a700 1 -- 172.21.15.76:0/3256383767 wait complete. 2024-03-18T14:52:59.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa65359a700 1 Processor -- start 2024-03-18T14:52:59.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa65359a700 1 -- start start 2024-03-18T14:52:59.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa65359a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa64c1014a0 0x7fa64c19c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:59.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa65359a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa64c101e50 0x7fa64c19cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:59.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa65359a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa64c106580 0x7fa64c1967d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:59.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa65359a700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa64c1104d0 con 0x7fa64c101e50 2024-03-18T14:52:59.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa65359a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa64c110350 con 0x7fa64c1014a0 2024-03-18T14:52:59.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa65359a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa64c110650 con 0x7fa64c106580 2024-03-18T14:52:59.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa650b35700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa64c101e50 0x7fa64c19cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:59.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa650b35700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa64c101e50 0x7fa64c19cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50192/0 (socket says 172.21.15.76:50192) 2024-03-18T14:52:59.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa650b35700 1 -- 172.21.15.76:0/4125485915 learned_addr learned my addr 172.21.15.76:0/4125485915 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:52:59.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa651336700 1 --2- 172.21.15.76:0/4125485915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa64c1014a0 0x7fa64c19c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:59.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa651b37700 1 --2- 172.21.15.76:0/4125485915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa64c106580 0x7fa64c1967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:59.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa650b35700 1 -- 172.21.15.76:0/4125485915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa64c106580 msgr2=0x7fa64c1967d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:59.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa650b35700 1 --2- 172.21.15.76:0/4125485915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa64c106580 0x7fa64c1967d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:59.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa650b35700 1 -- 172.21.15.76:0/4125485915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa64c1014a0 msgr2=0x7fa64c19c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:59.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa650b35700 1 --2- 172.21.15.76:0/4125485915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa64c1014a0 0x7fa64c19c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:59.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.680+0000 7fa650b35700 1 -- 172.21.15.76:0/4125485915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa64c196fb0 con 0x7fa64c101e50 2024-03-18T14:52:59.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.681+0000 7fa650b35700 1 --2- 172.21.15.76:0/4125485915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa64c101e50 0x7fa64c19cb90 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa63c0036c0 tx=0x7fa63c004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:59.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.681+0000 7fa6427fc700 1 -- 172.21.15.76:0/4125485915 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c022020 con 0x7fa64c101e50 2024-03-18T14:52:59.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.681+0000 7fa65359a700 1 -- 172.21.15.76:0/4125485915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa64c197240 con 0x7fa64c101e50 2024-03-18T14:52:59.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.681+0000 7fa6427fc700 1 -- 172.21.15.76:0/4125485915 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa63c015450 con 0x7fa64c101e50 2024-03-18T14:52:59.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.681+0000 7fa65359a700 1 -- 172.21.15.76:0/4125485915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa64c1a34b0 con 0x7fa64c101e50 2024-03-18T14:52:59.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.681+0000 7fa6427fc700 1 -- 172.21.15.76:0/4125485915 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c01a040 con 0x7fa64c101e50 2024-03-18T14:52:59.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.682+0000 7fa65359a700 1 -- 172.21.15.76:0/4125485915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa64c102980 con 0x7fa64c101e50 2024-03-18T14:52:59.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.683+0000 7fa6427fc700 1 -- 172.21.15.76:0/4125485915 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa63c025020 con 0x7fa64c101e50 2024-03-18T14:52:59.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.684+0000 7fa6427fc700 1 --2- 172.21.15.76:0/4125485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa6380a2890 0x7fa6380a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:52:59.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.684+0000 7fa651336700 1 --2- 172.21.15.76:0/4125485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa6380a2890 0x7fa6380a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:52:59.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.684+0000 7fa6427fc700 1 -- 172.21.15.76:0/4125485915 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fa63c0d06d0 con 0x7fa64c101e50 2024-03-18T14:52:59.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.684+0000 7fa651336700 1 --2- 172.21.15.76:0/4125485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa6380a2890 0x7fa6380a4d50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa648009c60 tx=0x7fa648009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:52:59.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.687+0000 7fa6427fc700 1 -- 172.21.15.76:0/4125485915 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa63c02f020 con 0x7fa64c101e50 2024-03-18T14:52:59.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:52:59 smithi077 ceph-mon[35865]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T14:52:59.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.933+0000 7fa65359a700 1 -- 172.21.15.76:0/4125485915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa64c197d00 con 0x7fa64c101e50 2024-03-18T14:52:59.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.933+0000 7fa6427fc700 1 -- 172.21.15.76:0/4125485915 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) ==== 53+0+73 (secure 0 0 0) 0x7fa63c09d830 con 0x7fa64c101e50 2024-03-18T14:52:59.934 INFO:teuthology.orchestra.run.smithi076.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T14:52:59.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.935+0000 7fa65359a700 1 -- 172.21.15.76:0/4125485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa6380a2890 msgr2=0x7fa6380a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:59.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.935+0000 7fa65359a700 1 --2- 172.21.15.76:0/4125485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa6380a2890 0x7fa6380a4d50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa648009c60 tx=0x7fa648009260 comp rx=0 tx=0).stop 2024-03-18T14:52:59.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.935+0000 7fa65359a700 1 -- 172.21.15.76:0/4125485915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa64c101e50 msgr2=0x7fa64c19cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:52:59.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.935+0000 7fa65359a700 1 --2- 172.21.15.76:0/4125485915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa64c101e50 0x7fa64c19cb90 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa63c0036c0 tx=0x7fa63c004410 comp rx=0 tx=0).stop 2024-03-18T14:52:59.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.935+0000 7fa65359a700 1 -- 172.21.15.76:0/4125485915 shutdown_connections 2024-03-18T14:52:59.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.935+0000 7fa65359a700 1 --2- 172.21.15.76:0/4125485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa6380a2890 0x7fa6380a4d50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:59.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.935+0000 7fa65359a700 1 --2- 172.21.15.76:0/4125485915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa64c106580 0x7fa64c1967d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:59.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.935+0000 7fa65359a700 1 --2- 172.21.15.76:0/4125485915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa64c1014a0 0x7fa64c19c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:59.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.935+0000 7fa65359a700 1 --2- 172.21.15.76:0/4125485915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa64c101e50 0x7fa64c19cb90 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:52:59.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.935+0000 7fa65359a700 1 -- 172.21.15.76:0/4125485915 >> 172.21.15.76:0/4125485915 conn(0x7fa64c078300 msgr2=0x7fa64c0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:52:59.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.935+0000 7fa65359a700 1 -- 172.21.15.76:0/4125485915 shutdown_connections 2024-03-18T14:52:59.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:52:59.935+0000 7fa65359a700 1 -- 172.21.15.76:0/4125485915 wait complete. 2024-03-18T14:52:59.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:52:59 smithi174 ceph-mon[36787]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T14:53:00.629 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:00 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/4125485915' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T14:53:00.655 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T14:53:00.655 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T14:53:00.655 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi076.front.sepia.ceph.com 2024-03-18T14:53:00.655 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T14:53:00.656 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T14:53:00.656 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T14:53:00.656 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T14:53:00.656 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T14:53:00.656 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:53:00.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:53:00.656 DEBUG:teuthology.orchestra.run.smithi076:> ip addr 2024-03-18T14:53:00.673 INFO:teuthology.orchestra.run.smithi076.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T14:53:00.673 INFO:teuthology.orchestra.run.smithi076.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T14:53:00.673 INFO:teuthology.orchestra.run.smithi076.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T14:53:00.673 INFO:teuthology.orchestra.run.smithi076.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:53:00.673 INFO:teuthology.orchestra.run.smithi076.stdout: inet6 ::1/128 scope host 2024-03-18T14:53:00.673 INFO:teuthology.orchestra.run.smithi076.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:53:00.673 INFO:teuthology.orchestra.run.smithi076.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:53:00.673 INFO:teuthology.orchestra.run.smithi076.stdout: link/ether 0c:c4:7a:d8:da:66 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:53:00.673 INFO:teuthology.orchestra.run.smithi076.stdout: altname enp7s0f0 2024-03-18T14:53:00.673 INFO:teuthology.orchestra.run.smithi076.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:53:00.673 INFO:teuthology.orchestra.run.smithi076.stdout: link/ether 0c:c4:7a:d8:da:67 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:53:00.673 INFO:teuthology.orchestra.run.smithi076.stdout: altname enp7s0f1 2024-03-18T14:53:00.674 INFO:teuthology.orchestra.run.smithi076.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:53:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: link/ether 0c:c4:7a:8f:47:e0 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:53:00.674 INFO:teuthology.orchestra.run.smithi076.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T14:53:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: link/ether 0c:c4:7a:8f:47:e1 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:53:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: inet 172.21.15.76/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-18T14:53:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: valid_lft 41988sec preferred_lft 41988sec 2024-03-18T14:53:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: inet6 fe80::ec4:7aff:fe8f:47e1/64 scope link 2024-03-18T14:53:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:53:00.674 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T14:53:00.675 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:53:00.675 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-03-18T14:53:00.675 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link add name ceph-brx type bridge 2024-03-18T14:53:00.675 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip addr flush dev ceph-brx 2024-03-18T14:53:00.675 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link set ceph-brx up 2024-03-18T14:53:00.675 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T14:53:00.675 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-03-18T14:53:00.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:00 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/4125485915' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T14:53:00.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:53:00.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:53:00.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:53:00.882 DEBUG:teuthology.orchestra.run.smithi076:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T14:53:00.909 INFO:teuthology.orchestra.run.smithi076.stdout:1 2024-03-18T14:53:00.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:53:00.910 DEBUG:teuthology.orchestra.run.smithi076:> route 2024-03-18T14:53:00.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:00 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/4125485915' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T14:53:00.984 INFO:teuthology.orchestra.run.smithi076.stdout:Kernel IP routing table 2024-03-18T14:53:00.984 INFO:teuthology.orchestra.run.smithi076.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T14:53:00.984 INFO:teuthology.orchestra.run.smithi076.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-18T14:53:00.985 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-18T14:53:00.985 INFO:teuthology.orchestra.run.smithi076.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T14:53:00.985 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:53:00.985 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-03-18T14:53:00.985 DEBUG:teuthology.orchestra.run.smithi076:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-18T14:53:00.985 DEBUG:teuthology.orchestra.run.smithi076:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-18T14:53:00.985 DEBUG:teuthology.orchestra.run.smithi076:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-18T14:53:00.985 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-03-18T14:53:01.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:53:01.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:53:01.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:53:01.296 DEBUG:teuthology.orchestra.run.smithi076:> ip netns list 2024-03-18T14:53:01.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:53:01.311 DEBUG:teuthology.orchestra.run.smithi076:> ip netns list-id 2024-03-18T14:53:01.368 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:53:01.368 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-03-18T14:53:01.368 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:53:01.368 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-18T14:53:01.368 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-03-18T14:53:01.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:53:01.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:53:01.491 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-18T14:53:01.491 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:53:01.491 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-03-18T14:53:01.491 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-18T14:53:01.491 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T14:53:01.491 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-18T14:53:01.491 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-18T14:53:01.491 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-18T14:53:01.491 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-03-18T14:53:01.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:53:01.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:53:01.632 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:53:01.632 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-03-18T14:53:01.632 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link set brx.0 up 2024-03-18T14:53:01.633 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T14:53:01.633 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-03-18T14:53:01.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:53:01.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:53:01.715 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:53:01.715 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T14:53:01.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:01 smithi077 ceph-mon[35865]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:53:01.729 INFO:teuthology.orchestra.run.smithi076.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:53:01.729 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:53:01.730 DEBUG:teuthology.orchestra.run.smithi076:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T14:53:01.784 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-18T14:53:01.784 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-18T14:53:01.784 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a 2024-03-18T14:53:01.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:53:01.784 DEBUG:teuthology.orchestra.run.smithi076:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-18T14:53:01.848 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:01 smithi076 ceph-mon[32205]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:53:01.869 INFO:teuthology.orchestra.run.smithi076.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-18T14:53:01.869 INFO:teuthology.orchestra.run.smithi076.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-03-18T14:53:01.869 INFO:teuthology.orchestra.run.smithi076.stdout:invalid new device string format 2024-03-18T14:53:01.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:01 smithi174 ceph-mon[36787]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T14:53:01.982 INFO:teuthology.orchestra.run.smithi076.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-18T14:53:01.983 INFO:teuthology.orchestra.run.smithi076.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-03-18T14:53:01.983 INFO:teuthology.orchestra.run.smithi076.stdout:invalid new device string format 2024-03-18T14:53:01.983 INFO:teuthology.orchestra.run.smithi076.stdout:mount.ceph: resolved to: "172.21.15.76:6789,172.21.15.77:6789,172.21.15.174:6789" 2024-03-18T14:53:01.983 INFO:teuthology.orchestra.run.smithi076.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.76:6789,172.21.15.77:6789,172.21.15.174:6789:/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a 2024-03-18T14:53:01.983 INFO:teuthology.orchestra.run.smithi076.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,key=0,fsid=9cc83b72-e535-11ee-95c9-87774f69a715" will pass to kernel 2024-03-18T14:53:01.984 INFO:teuthology.orchestra.run.smithi076.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-18T14:53:01.984 INFO:teuthology.orchestra.run.smithi076.stdout: You just mounted a file system that supports labels which does not 2024-03-18T14:53:01.984 INFO:teuthology.orchestra.run.smithi076.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-18T14:53:01.984 INFO:teuthology.orchestra.run.smithi076.stdout: applications will generate AVC messages and not be allowed access to 2024-03-18T14:53:01.984 INFO:teuthology.orchestra.run.smithi076.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-18T14:53:01.986 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-18T14:53:01.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:53:01.986 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T14:53:02.057 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T14:53:02.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T14:53:02.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:53:02.153 INFO:teuthology.orchestra.run.smithi076.stdout:client25120 2024-03-18T14:53:02.153 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T14:53:02.153 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:53:02.170 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T14:53:02.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T14:53:02.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:53:02.266 INFO:teuthology.orchestra.run.smithi076.stdout:client25120 2024-03-18T14:53:02.266 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T14:53:02.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T14:53:02.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:53:02.321 INFO:teuthology.orchestra.run.smithi076.stdout:client25120 2024-03-18T14:53:02.322 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/9cc83b72-e535-11ee-95c9-87774f69a715.client25120/status') 2024-03-18T14:53:02.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:53:02.382 INFO:teuthology.orchestra.run.smithi076.stdout:instance: client.25120 (3)172.21.15.76:0/1987132574 2024-03-18T14:53:02.382 INFO:teuthology.orchestra.run.smithi076.stdout:blocklisted: false 2024-03-18T14:53:02.382 INFO:teuthology.orchestra.run.smithi076.stderr:0+1 records in 2024-03-18T14:53:02.382 INFO:teuthology.orchestra.run.smithi076.stderr:0+1 records out 2024-03-18T14:53:02.382 INFO:teuthology.orchestra.run.smithi076.stderr:71 bytes copied, 2.1276e-05 s, 3.3 MB/s 2024-03-18T14:53:02.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:53:02.389 INFO:teuthology.run_tasks:Running task exec... 2024-03-18T14:53:02.399 INFO:teuthology.task.exec:Executing custom commands... 2024-03-18T14:53:02.399 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi076.front.sepia.ceph.com 2024-03-18T14:53:02.399 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-18T14:53:02.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.503+0000 7fcfde19f640 1 -- 172.21.15.76:0/2554432652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfd8101200 msgr2=0x7fcfd8101680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:02.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.503+0000 7fcfde19f640 1 --2- 172.21.15.76:0/2554432652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfd8101200 0x7fcfd8101680 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0009900 tx=0x7fcfd002bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:02.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.503+0000 7fcfde19f640 1 -- 172.21.15.76:0/2554432652 shutdown_connections 2024-03-18T14:53:02.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.503+0000 7fcfde19f640 1 --2- 172.21.15.76:0/2554432652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfd8100860 0x7fcfd8100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:02.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.503+0000 7fcfde19f640 1 --2- 172.21.15.76:0/2554432652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfd8101200 0x7fcfd8101680 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:02.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.503+0000 7fcfde19f640 1 --2- 172.21.15.76:0/2554432652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfd80fff00 0x7fcfd8100320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:02.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.503+0000 7fcfde19f640 1 -- 172.21.15.76:0/2554432652 >> 172.21.15.76:0/2554432652 conn(0x7fcfd80fbac0 msgr2=0x7fcfd80fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:02.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.503+0000 7fcfde19f640 1 -- 172.21.15.76:0/2554432652 shutdown_connections 2024-03-18T14:53:02.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.503+0000 7fcfde19f640 1 -- 172.21.15.76:0/2554432652 wait complete. 2024-03-18T14:53:02.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfde19f640 1 Processor -- start 2024-03-18T14:53:02.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfde19f640 1 -- start start 2024-03-18T14:53:02.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfde19f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfd80fff00 0x7fcfd8199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:02.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfde19f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfd8100860 0x7fcfd819a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:02.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfde19f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfd8101200 0x7fcfd81a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:02.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfde19f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd810f4a0 con 0x7fcfd8100860 2024-03-18T14:53:02.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfde19f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd810f320 con 0x7fcfd8101200 2024-03-18T14:53:02.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfde19f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd810f620 con 0x7fcfd80fff00 2024-03-18T14:53:02.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfdc99c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfd8100860 0x7fcfd819a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:02.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfdc99c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfd8100860 0x7fcfd819a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50208/0 (socket says 172.21.15.76:50208) 2024-03-18T14:53:02.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfdc99c640 1 -- 172.21.15.76:0/3807826142 learned_addr learned my addr 172.21.15.76:0/3807826142 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:02.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfdd19d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfd80fff00 0x7fcfd8199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:02.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfdd99e640 1 --2- 172.21.15.76:0/3807826142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfd8101200 0x7fcfd81a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:02.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfdc99c640 1 -- 172.21.15.76:0/3807826142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfd80fff00 msgr2=0x7fcfd8199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:02.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfdc99c640 1 --2- 172.21.15.76:0/3807826142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfd80fff00 0x7fcfd8199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:02.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfdc99c640 1 -- 172.21.15.76:0/3807826142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfd8101200 msgr2=0x7fcfd81a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:02.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfdc99c640 1 --2- 172.21.15.76:0/3807826142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfd8101200 0x7fcfd81a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:02.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfdc99c640 1 -- 172.21.15.76:0/3807826142 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd81a1700 con 0x7fcfd8100860 2024-03-18T14:53:02.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.504+0000 7fcfdc99c640 1 --2- 172.21.15.76:0/3807826142 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfd8100860 0x7fcfd819a020 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00e9f0 tx=0x7fcfcc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:02.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.505+0000 7fcfd67fc640 1 -- 172.21.15.76:0/3807826142 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc00cd80 con 0x7fcfd8100860 2024-03-18T14:53:02.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.505+0000 7fcfd67fc640 1 -- 172.21.15.76:0/3807826142 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfcc005500 con 0x7fcfd8100860 2024-03-18T14:53:02.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.505+0000 7fcfd67fc640 1 -- 172.21.15.76:0/3807826142 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc010690 con 0x7fcfd8100860 2024-03-18T14:53:02.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.505+0000 7fcfde19f640 1 -- 172.21.15.76:0/3807826142 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd81a19f0 con 0x7fcfd8100860 2024-03-18T14:53:02.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.505+0000 7fcfde19f640 1 -- 172.21.15.76:0/3807826142 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd81a1ef0 con 0x7fcfd8100860 2024-03-18T14:53:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.506+0000 7fcfde19f640 1 -- 172.21.15.76:0/3807826142 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd810ade0 con 0x7fcfd8100860 2024-03-18T14:53:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.507+0000 7fcfd67fc640 1 -- 172.21.15.76:0/3807826142 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcfcc0046c0 con 0x7fcfd8100860 2024-03-18T14:53:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.507+0000 7fcfd67fc640 1 --2- 172.21.15.76:0/3807826142 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fcfc00a27f0 0x7fcfc00a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.508+0000 7fcfdd19d640 1 --2- 172.21.15.76:0/3807826142 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fcfc00a27f0 0x7fcfc00a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.508+0000 7fcfd67fc640 1 -- 172.21.15.76:0/3807826142 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fcfcc0cee10 con 0x7fcfd8100860 2024-03-18T14:53:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.508+0000 7fcfdd19d640 1 --2- 172.21.15.76:0/3807826142 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fcfc00a27f0 0x7fcfc00a4cb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8005fd0 tx=0x7fcfc8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:02.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.513+0000 7fcfd67fc640 1 -- 172.21.15.76:0/3807826142 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcfcc09bee0 con 0x7fcfd8100860 2024-03-18T14:53:02.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:02 smithi076 ceph-mon[32205]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-18T14:53:02.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:02.726+0000 7fcfde19f640 1 -- 172.21.15.76:0/3807826142 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7fcfd802d1b0 con 0x7fcfd8100860 2024-03-18T14:53:02.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:02 smithi174 ceph-mon[36787]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-18T14:53:02.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:02 smithi077 ceph-mon[35865]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-18T14:53:03.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.531+0000 7fcfd67fc640 1 -- 172.21.15.76:0/3807826142 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v24) ==== 117+0+0 (secure 0 0 0) 0x7fcfcc09b7e0 con 0x7fcfd8100860 2024-03-18T14:53:03.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.534+0000 7fcfde19f640 1 -- 172.21.15.76:0/3807826142 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fcfc00a27f0 msgr2=0x7fcfc00a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:03.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.534+0000 7fcfde19f640 1 --2- 172.21.15.76:0/3807826142 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fcfc00a27f0 0x7fcfc00a4cb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8005fd0 tx=0x7fcfc8009500 comp rx=0 tx=0).stop 2024-03-18T14:53:03.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.534+0000 7fcfde19f640 1 -- 172.21.15.76:0/3807826142 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfd8100860 msgr2=0x7fcfd819a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:03.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.534+0000 7fcfde19f640 1 --2- 172.21.15.76:0/3807826142 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfd8100860 0x7fcfd819a020 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00e9f0 tx=0x7fcfcc00edf0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.534+0000 7fcfde19f640 1 -- 172.21.15.76:0/3807826142 shutdown_connections 2024-03-18T14:53:03.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.534+0000 7fcfde19f640 1 --2- 172.21.15.76:0/3807826142 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fcfc00a27f0 0x7fcfc00a4cb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.534+0000 7fcfde19f640 1 --2- 172.21.15.76:0/3807826142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfd8101200 0x7fcfd81a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.534+0000 7fcfde19f640 1 --2- 172.21.15.76:0/3807826142 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfd8100860 0x7fcfd819a020 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.534+0000 7fcfde19f640 1 --2- 172.21.15.76:0/3807826142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfd80fff00 0x7fcfd8199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.534+0000 7fcfde19f640 1 -- 172.21.15.76:0/3807826142 >> 172.21.15.76:0/3807826142 conn(0x7fcfd80fbac0 msgr2=0x7fcfd80fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:03.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.535+0000 7fcfde19f640 1 -- 172.21.15.76:0/3807826142 shutdown_connections 2024-03-18T14:53:03.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.535+0000 7fcfde19f640 1 -- 172.21.15.76:0/3807826142 wait complete. 2024-03-18T14:53:03.549 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-18T14:53:03.558 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-18T14:53:03.578 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:53:03.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.687+0000 7fed11cc0640 1 -- 172.21.15.76:0/844039344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed0c100720 msgr2=0x7fed0c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:03.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.687+0000 7fed11cc0640 1 --2- 172.21.15.76:0/844039344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed0c100720 0x7fed0c100b40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009860 tx=0x7fecfc02bb70 comp rx=0 tx=0).stop 2024-03-18T14:53:03.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.687+0000 7fed11cc0640 1 -- 172.21.15.76:0/844039344 shutdown_connections 2024-03-18T14:53:03.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.687+0000 7fed11cc0640 1 --2- 172.21.15.76:0/844039344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed0c1022d0 0x7fed0c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.687+0000 7fed11cc0640 1 --2- 172.21.15.76:0/844039344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed0c101910 0x7fed0c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.687+0000 7fed11cc0640 1 --2- 172.21.15.76:0/844039344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed0c100720 0x7fed0c100b40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.687+0000 7fed11cc0640 1 -- 172.21.15.76:0/844039344 >> 172.21.15.76:0/844039344 conn(0x7fed0c0fbc90 msgr2=0x7fed0c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:03.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.687+0000 7fed11cc0640 1 -- 172.21.15.76:0/844039344 shutdown_connections 2024-03-18T14:53:03.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.687+0000 7fed11cc0640 1 -- 172.21.15.76:0/844039344 wait complete. 2024-03-18T14:53:03.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.688+0000 7fed11cc0640 1 Processor -- start 2024-03-18T14:53:03.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.688+0000 7fed11cc0640 1 -- start start 2024-03-18T14:53:03.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.688+0000 7fed11cc0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed0c100720 0x7fed0c06a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:03.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.688+0000 7fed11cc0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed0c101910 0x7fed0c19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:03.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.688+0000 7fed11cc0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed0c1022d0 0x7fed0c19eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:03.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.688+0000 7fed11cc0640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c10c370 con 0x7fed0c101910 2024-03-18T14:53:03.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.688+0000 7fed11cc0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c10c1f0 con 0x7fed0c1022d0 2024-03-18T14:53:03.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.688+0000 7fed11cc0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c10c4f0 con 0x7fed0c100720 2024-03-18T14:53:03.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.688+0000 7fed0affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed0c101910 0x7fed0c19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:03.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.688+0000 7fed0affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed0c101910 0x7fed0c19c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50234/0 (socket says 172.21.15.76:50234) 2024-03-18T14:53:03.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.688+0000 7fed0affd640 1 -- 172.21.15.76:0/4203041775 learned_addr learned my addr 172.21.15.76:0/4203041775 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:03.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.688+0000 7fed0b7fe640 1 --2- 172.21.15.76:0/4203041775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed0c100720 0x7fed0c06a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:03.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.688+0000 7fed0bfff640 1 --2- 172.21.15.76:0/4203041775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed0c1022d0 0x7fed0c19eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:03.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.689+0000 7fed0affd640 1 -- 172.21.15.76:0/4203041775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed0c100720 msgr2=0x7fed0c06a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:03.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.689+0000 7fed0affd640 1 --2- 172.21.15.76:0/4203041775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed0c100720 0x7fed0c06a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.689+0000 7fed0affd640 1 -- 172.21.15.76:0/4203041775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed0c1022d0 msgr2=0x7fed0c19eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:03.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.689+0000 7fed0affd640 1 --2- 172.21.15.76:0/4203041775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed0c1022d0 0x7fed0c19eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.689+0000 7fed0affd640 1 -- 172.21.15.76:0/4203041775 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed0c19f7c0 con 0x7fed0c101910 2024-03-18T14:53:03.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.689+0000 7fed0bfff640 1 --2- 172.21.15.76:0/4203041775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed0c1022d0 0x7fed0c19eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:03.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.689+0000 7fed0b7fe640 1 --2- 172.21.15.76:0/4203041775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed0c100720 0x7fed0c06a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:03.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.689+0000 7fed0affd640 1 --2- 172.21.15.76:0/4203041775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed0c101910 0x7fed0c19c890 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fed0000e9c0 tx=0x7fed0000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:03.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.689+0000 7fed08ff9640 1 -- 172.21.15.76:0/4203041775 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0000cde0 con 0x7fed0c101910 2024-03-18T14:53:03.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.689+0000 7fed11cc0640 1 -- 172.21.15.76:0/4203041775 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed0c19fa50 con 0x7fed0c101910 2024-03-18T14:53:03.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.689+0000 7fed08ff9640 1 -- 172.21.15.76:0/4203041775 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed000035f0 con 0x7fed0c101910 2024-03-18T14:53:03.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.689+0000 7fed08ff9640 1 -- 172.21.15.76:0/4203041775 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed00010650 con 0x7fed0c101910 2024-03-18T14:53:03.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.689+0000 7fed11cc0640 1 -- 172.21.15.76:0/4203041775 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed0c19fff0 con 0x7fed0c101910 2024-03-18T14:53:03.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.690+0000 7fed11cc0640 1 -- 172.21.15.76:0/4203041775 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed0c107a40 con 0x7fed0c101910 2024-03-18T14:53:03.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.691+0000 7fed08ff9640 1 -- 172.21.15.76:0/4203041775 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fed000037a0 con 0x7fed0c101910 2024-03-18T14:53:03.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.691+0000 7fed08ff9640 1 --2- 172.21.15.76:0/4203041775 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fecf40a2800 0x7fecf40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:03.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.692+0000 7fed0b7fe640 1 --2- 172.21.15.76:0/4203041775 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fecf40a2800 0x7fecf40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:03.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.692+0000 7fed08ff9640 1 -- 172.21.15.76:0/4203041775 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fed000ced30 con 0x7fed0c101910 2024-03-18T14:53:03.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.692+0000 7fed0b7fe640 1 --2- 172.21.15.76:0/4203041775 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fecf40a2800 0x7fecf40a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009830 tx=0x7fecfc02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:03.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.696+0000 7fed08ff9640 1 -- 172.21.15.76:0/4203041775 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fed0009be00 con 0x7fed0c101910 2024-03-18T14:53:03.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.871+0000 7fed11cc0640 1 -- 172.21.15.76:0/4203041775 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fed0c0623e0 con 0x7fed0c101910 2024-03-18T14:53:03.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.872+0000 7fed08ff9640 1 -- 172.21.15.76:0/4203041775 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22580 (secure 0 0 0) 0x7fed0009b700 con 0x7fed0c101910 2024-03-18T14:53:03.873 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:53:03.873 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:52:07.361301+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:53:03.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.874+0000 7fed11cc0640 1 -- 172.21.15.76:0/4203041775 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fecf40a2800 msgr2=0x7fecf40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:03.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.874+0000 7fed11cc0640 1 --2- 172.21.15.76:0/4203041775 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fecf40a2800 0x7fecf40a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009830 tx=0x7fecfc02e040 comp rx=0 tx=0).stop 2024-03-18T14:53:03.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.874+0000 7fed11cc0640 1 -- 172.21.15.76:0/4203041775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed0c101910 msgr2=0x7fed0c19c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:03.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.874+0000 7fed11cc0640 1 --2- 172.21.15.76:0/4203041775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed0c101910 0x7fed0c19c890 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fed0000e9c0 tx=0x7fed0000ee80 comp rx=0 tx=0).stop 2024-03-18T14:53:03.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.875+0000 7fed11cc0640 1 -- 172.21.15.76:0/4203041775 shutdown_connections 2024-03-18T14:53:03.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.875+0000 7fed11cc0640 1 --2- 172.21.15.76:0/4203041775 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fecf40a2800 0x7fecf40a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.875+0000 7fed11cc0640 1 --2- 172.21.15.76:0/4203041775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed0c1022d0 0x7fed0c19eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.875+0000 7fed11cc0640 1 --2- 172.21.15.76:0/4203041775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed0c101910 0x7fed0c19c890 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.875+0000 7fed11cc0640 1 --2- 172.21.15.76:0/4203041775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed0c100720 0x7fed0c06a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.875+0000 7fed11cc0640 1 -- 172.21.15.76:0/4203041775 >> 172.21.15.76:0/4203041775 conn(0x7fed0c0fbc90 msgr2=0x7fed0c104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:03.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.875+0000 7fed11cc0640 1 -- 172.21.15.76:0/4203041775 shutdown_connections 2024-03-18T14:53:03.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.875+0000 7fed11cc0640 1 -- 172.21.15.76:0/4203041775 wait complete. 2024-03-18T14:53:03.885 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:03 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3807826142' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T14:53:03.887 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:46:26.588384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:50:59.606852+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:51:00.465325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:53:03.887 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:53:03.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:03 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3807826142' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T14:53:03.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:03 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3807826142' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T14:53:03.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.990+0000 7f1f9807e640 1 -- 172.21.15.76:0/774994272 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f901022b0 msgr2=0x7f1f90108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:03.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.990+0000 7f1f9807e640 1 --2- 172.21.15.76:0/774994272 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f901022b0 0x7f1f90108b40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1f88009ca0 tx=0x7f1f8802c0f0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f9807e640 1 -- 172.21.15.76:0/774994272 shutdown_connections 2024-03-18T14:53:03.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f9807e640 1 --2- 172.21.15.76:0/774994272 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f901018f0 0x7f1f90101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f9807e640 1 --2- 172.21.15.76:0/774994272 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f901022b0 0x7f1f90108b40 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f9807e640 1 --2- 172.21.15.76:0/774994272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f90100700 0x7f1f90100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f9807e640 1 -- 172.21.15.76:0/774994272 >> 172.21.15.76:0/774994272 conn(0x7f1f900fbc90 msgr2=0x7f1f900fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:03.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f9807e640 1 -- 172.21.15.76:0/774994272 shutdown_connections 2024-03-18T14:53:03.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f9807e640 1 -- 172.21.15.76:0/774994272 wait complete. 2024-03-18T14:53:03.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f9807e640 1 Processor -- start 2024-03-18T14:53:03.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f9807e640 1 -- start start 2024-03-18T14:53:03.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f9807e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f90100700 0x7f1f90197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:03.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f9807e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f901018f0 0x7f1f90198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:03.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f9807e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f901022b0 0x7f1f9019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:03.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f9807e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1f9010c380 con 0x7f1f90100700 2024-03-18T14:53:03.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f9807e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1f9010c200 con 0x7f1f901022b0 2024-03-18T14:53:03.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f9807e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1f9010c500 con 0x7f1f901018f0 2024-03-18T14:53:03.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f95e1a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f90100700 0x7f1f90197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:03.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f9661b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f901022b0 0x7f1f9019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:03.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f95e1a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f90100700 0x7f1f90197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50258/0 (socket says 172.21.15.76:50258) 2024-03-18T14:53:03.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.991+0000 7f1f95e1a640 1 -- 172.21.15.76:0/3899483721 learned_addr learned my addr 172.21.15.76:0/3899483721 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:03.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.992+0000 7f1f95619640 1 --2- 172.21.15.76:0/3899483721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f901018f0 0x7f1f90198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:03.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.992+0000 7f1f95e1a640 1 -- 172.21.15.76:0/3899483721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f901018f0 msgr2=0x7f1f90198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:03.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.992+0000 7f1f95e1a640 1 --2- 172.21.15.76:0/3899483721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f901018f0 0x7f1f90198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.992+0000 7f1f95e1a640 1 -- 172.21.15.76:0/3899483721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f901022b0 msgr2=0x7f1f9019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:03.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.992+0000 7f1f95e1a640 1 --2- 172.21.15.76:0/3899483721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f901022b0 0x7f1f9019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:03.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.992+0000 7f1f95e1a640 1 -- 172.21.15.76:0/3899483721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f9019f6e0 con 0x7f1f90100700 2024-03-18T14:53:03.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.992+0000 7f1f9661b640 1 --2- 172.21.15.76:0/3899483721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f901022b0 0x7f1f9019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:03.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.992+0000 7f1f95e1a640 1 --2- 172.21.15.76:0/3899483721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f90100700 0x7f1f90197ac0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c00e9c0 tx=0x7f1f8c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:03.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.992+0000 7f1f86ffd640 1 -- 172.21.15.76:0/3899483721 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c00cd80 con 0x7f1f90100700 2024-03-18T14:53:03.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.992+0000 7f1f86ffd640 1 -- 172.21.15.76:0/3899483721 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f8c0035f0 con 0x7f1f90100700 2024-03-18T14:53:03.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.992+0000 7f1f9807e640 1 -- 172.21.15.76:0/3899483721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f9019f9d0 con 0x7f1f90100700 2024-03-18T14:53:03.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.992+0000 7f1f86ffd640 1 -- 172.21.15.76:0/3899483721 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c010690 con 0x7f1f90100700 2024-03-18T14:53:03.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.992+0000 7f1f9807e640 1 -- 172.21.15.76:0/3899483721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f9019ff20 con 0x7f1f90100700 2024-03-18T14:53:03.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.993+0000 7f1f9807e640 1 -- 172.21.15.76:0/3899483721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f901a0150 con 0x7f1f90100700 2024-03-18T14:53:03.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.994+0000 7f1f86ffd640 1 -- 172.21.15.76:0/3899483721 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1f8c0037a0 con 0x7f1f90100700 2024-03-18T14:53:03.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.995+0000 7f1f86ffd640 1 --2- 172.21.15.76:0/3899483721 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1f7c0a2800 0x7f1f7c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:03.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.995+0000 7f1f95619640 1 --2- 172.21.15.76:0/3899483721 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1f7c0a2800 0x7f1f7c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:03.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.995+0000 7f1f86ffd640 1 -- 172.21.15.76:0/3899483721 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f1f8c0cede0 con 0x7f1f90100700 2024-03-18T14:53:03.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.995+0000 7f1f95619640 1 --2- 172.21.15.76:0/3899483721 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1f7c0a2800 0x7f1f7c0a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1f90198fd0 tx=0x7f1f80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:03.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:03.999+0000 7f1f86ffd640 1 -- 172.21.15.76:0/3899483721 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1f8c09beb0 con 0x7f1f90100700 2024-03-18T14:53:04.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.175+0000 7f1f9807e640 1 -- 172.21.15.76:0/3899483721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1f901a0320 con 0x7f1f90100700 2024-03-18T14:53:04.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.176+0000 7f1f86ffd640 1 -- 172.21.15.76:0/3899483721 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f1f8c09b7b0 con 0x7f1f90100700 2024-03-18T14:53:04.176 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-18T14:53:04.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.178+0000 7f1f9807e640 1 -- 172.21.15.76:0/3899483721 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1f7c0a2800 msgr2=0x7f1f7c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:04.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.178+0000 7f1f9807e640 1 --2- 172.21.15.76:0/3899483721 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1f7c0a2800 0x7f1f7c0a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1f90198fd0 tx=0x7f1f80005f90 comp rx=0 tx=0).stop 2024-03-18T14:53:04.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.178+0000 7f1f9807e640 1 -- 172.21.15.76:0/3899483721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f90100700 msgr2=0x7f1f90197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:04.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.178+0000 7f1f9807e640 1 --2- 172.21.15.76:0/3899483721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f90100700 0x7f1f90197ac0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c00e9c0 tx=0x7f1f8c00ee80 comp rx=0 tx=0).stop 2024-03-18T14:53:04.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.178+0000 7f1f9807e640 1 -- 172.21.15.76:0/3899483721 shutdown_connections 2024-03-18T14:53:04.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.178+0000 7f1f9807e640 1 --2- 172.21.15.76:0/3899483721 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f1f7c0a2800 0x7f1f7c0a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.178+0000 7f1f9807e640 1 --2- 172.21.15.76:0/3899483721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f901018f0 0x7f1f90198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.178+0000 7f1f9807e640 1 --2- 172.21.15.76:0/3899483721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f901022b0 0x7f1f9019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.178+0000 7f1f9807e640 1 --2- 172.21.15.76:0/3899483721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f90100700 0x7f1f90197ac0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.178+0000 7f1f9807e640 1 -- 172.21.15.76:0/3899483721 >> 172.21.15.76:0/3899483721 conn(0x7f1f900fbc90 msgr2=0x7f1f900fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:04.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.178+0000 7f1f9807e640 1 -- 172.21.15.76:0/3899483721 shutdown_connections 2024-03-18T14:53:04.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.178+0000 7f1f9807e640 1 -- 172.21.15.76:0/3899483721 wait complete. 2024-03-18T14:53:04.190 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:53:04.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.293+0000 7f3c98e59640 1 -- 172.21.15.76:0/637469904 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c941018d0 msgr2=0x7f3c94101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:04.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.293+0000 7f3c98e59640 1 --2- 172.21.15.76:0/637469904 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c941018d0 0x7f3c94101d50 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3c84009900 tx=0x7f3c8402bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:04.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.294+0000 7f3c98e59640 1 -- 172.21.15.76:0/637469904 shutdown_connections 2024-03-18T14:53:04.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.294+0000 7f3c98e59640 1 --2- 172.21.15.76:0/637469904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c94102290 0x7f3c94108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.294+0000 7f3c98e59640 1 --2- 172.21.15.76:0/637469904 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c941018d0 0x7f3c94101d50 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.294+0000 7f3c98e59640 1 --2- 172.21.15.76:0/637469904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c941006e0 0x7f3c94100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.294+0000 7f3c98e59640 1 -- 172.21.15.76:0/637469904 >> 172.21.15.76:0/637469904 conn(0x7f3c940fbc70 msgr2=0x7f3c940fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:04.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.294+0000 7f3c98e59640 1 -- 172.21.15.76:0/637469904 shutdown_connections 2024-03-18T14:53:04.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.294+0000 7f3c98e59640 1 -- 172.21.15.76:0/637469904 wait complete. 2024-03-18T14:53:04.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.294+0000 7f3c98e59640 1 Processor -- start 2024-03-18T14:53:04.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.294+0000 7f3c98e59640 1 -- start start 2024-03-18T14:53:04.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.294+0000 7f3c98e59640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c941006e0 0x7f3c940694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:04.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.294+0000 7f3c98e59640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c941018d0 0x7f3c9419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:04.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c98e59640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c94102290 0x7f3c9419eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:04.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c98e59640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9410c340 con 0x7f3c941018d0 2024-03-18T14:53:04.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c98e59640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9410c1c0 con 0x7f3c941006e0 2024-03-18T14:53:04.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c98e59640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9410c4c0 con 0x7f3c94102290 2024-03-18T14:53:04.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c91d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c941018d0 0x7f3c9419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:04.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c9259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c941006e0 0x7f3c940694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:04.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c9259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c941006e0 0x7f3c940694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:46828/0 (socket says 172.21.15.76:46828) 2024-03-18T14:53:04.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c9259c640 1 -- 172.21.15.76:0/813225674 learned_addr learned my addr 172.21.15.76:0/813225674 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:04.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c91d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c941018d0 0x7f3c9419c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50282/0 (socket says 172.21.15.76:50282) 2024-03-18T14:53:04.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c92d9d640 1 --2- 172.21.15.76:0/813225674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c94102290 0x7f3c9419eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:04.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c9259c640 1 -- 172.21.15.76:0/813225674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c94102290 msgr2=0x7f3c9419eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:04.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c9259c640 1 --2- 172.21.15.76:0/813225674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c94102290 0x7f3c9419eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c9259c640 1 -- 172.21.15.76:0/813225674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c941018d0 msgr2=0x7f3c9419c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:04.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c9259c640 1 --2- 172.21.15.76:0/813225674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c941018d0 0x7f3c9419c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c9259c640 1 -- 172.21.15.76:0/813225674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c9419f7b0 con 0x7f3c941006e0 2024-03-18T14:53:04.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c9259c640 1 --2- 172.21.15.76:0/813225674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c941006e0 0x7f3c940694b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800e9c0 tx=0x7f3c8800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:04.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c837fe640 1 -- 172.21.15.76:0/813225674 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8800cd80 con 0x7f3c941006e0 2024-03-18T14:53:04.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c837fe640 1 -- 172.21.15.76:0/813225674 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c880035f0 con 0x7f3c941006e0 2024-03-18T14:53:04.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c837fe640 1 -- 172.21.15.76:0/813225674 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c88010690 con 0x7f3c941006e0 2024-03-18T14:53:04.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c98e59640 1 -- 172.21.15.76:0/813225674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c9419fa40 con 0x7f3c941006e0 2024-03-18T14:53:04.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.295+0000 7f3c98e59640 1 -- 172.21.15.76:0/813225674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c9419ff10 con 0x7f3c941006e0 2024-03-18T14:53:04.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.296+0000 7f3c98e59640 1 -- 172.21.15.76:0/813225674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c94107ad0 con 0x7f3c941006e0 2024-03-18T14:53:04.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.297+0000 7f3c837fe640 1 -- 172.21.15.76:0/813225674 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3c880037a0 con 0x7f3c941006e0 2024-03-18T14:53:04.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.298+0000 7f3c837fe640 1 --2- 172.21.15.76:0/813225674 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3c7c0a2800 0x7f3c7c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:04.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.298+0000 7f3c91d9b640 1 --2- 172.21.15.76:0/813225674 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3c7c0a2800 0x7f3c7c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:04.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.298+0000 7f3c837fe640 1 -- 172.21.15.76:0/813225674 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f3c880cede0 con 0x7f3c941006e0 2024-03-18T14:53:04.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.298+0000 7f3c91d9b640 1 --2- 172.21.15.76:0/813225674 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3c7c0a2800 0x7f3c7c0a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3c8402c7b0 tx=0x7f3c84005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:04.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.302+0000 7f3c837fe640 1 -- 172.21.15.76:0/813225674 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3c8809beb0 con 0x7f3c941006e0 2024-03-18T14:53:04.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.478+0000 7f3c98e59640 1 -- 172.21.15.76:0/813225674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3c94062480 con 0x7f3c941006e0 2024-03-18T14:53:04.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.478+0000 7f3c837fe640 1 -- 172.21.15.76:0/813225674 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f3c8809b7b0 con 0x7f3c941006e0 2024-03-18T14:53:04.479 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-18T14:53:04.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.480+0000 7f3c98e59640 1 -- 172.21.15.76:0/813225674 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3c7c0a2800 msgr2=0x7f3c7c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:04.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.480+0000 7f3c98e59640 1 --2- 172.21.15.76:0/813225674 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3c7c0a2800 0x7f3c7c0a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3c8402c7b0 tx=0x7f3c84005fd0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.480+0000 7f3c98e59640 1 -- 172.21.15.76:0/813225674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c941006e0 msgr2=0x7f3c940694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:04.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.480+0000 7f3c98e59640 1 --2- 172.21.15.76:0/813225674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c941006e0 0x7f3c940694b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800e9c0 tx=0x7f3c8800ee80 comp rx=0 tx=0).stop 2024-03-18T14:53:04.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.480+0000 7f3c98e59640 1 -- 172.21.15.76:0/813225674 shutdown_connections 2024-03-18T14:53:04.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.480+0000 7f3c98e59640 1 --2- 172.21.15.76:0/813225674 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f3c7c0a2800 0x7f3c7c0a4cc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.480+0000 7f3c98e59640 1 --2- 172.21.15.76:0/813225674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c94102290 0x7f3c9419eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.480+0000 7f3c98e59640 1 --2- 172.21.15.76:0/813225674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c941018d0 0x7f3c9419c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.480+0000 7f3c98e59640 1 --2- 172.21.15.76:0/813225674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c941006e0 0x7f3c940694b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.480+0000 7f3c98e59640 1 -- 172.21.15.76:0/813225674 >> 172.21.15.76:0/813225674 conn(0x7f3c940fbc70 msgr2=0x7f3c94104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:04.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.480+0000 7f3c98e59640 1 -- 172.21.15.76:0/813225674 shutdown_connections 2024-03-18T14:53:04.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.480+0000 7f3c98e59640 1 -- 172.21.15.76:0/813225674 wait complete. 2024-03-18T14:53:04.492 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:53:04.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.597+0000 7f1570cac640 1 -- 172.21.15.76:0/654047384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f156c1017f0 msgr2=0x7f156c101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:04.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.597+0000 7f1570cac640 1 --2- 172.21.15.76:0/654047384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f156c1017f0 0x7f156c101c10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f155c009900 tx=0x7f155c02bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:04.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.597+0000 7f1570cac640 1 -- 172.21.15.76:0/654047384 shutdown_connections 2024-03-18T14:53:04.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.597+0000 7f1570cac640 1 --2- 172.21.15.76:0/654047384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f156c1033a0 0x7f156c109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.597+0000 7f1570cac640 1 --2- 172.21.15.76:0/654047384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f156c1029e0 0x7f156c102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.597+0000 7f1570cac640 1 --2- 172.21.15.76:0/654047384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f156c1017f0 0x7f156c101c10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.597+0000 7f1570cac640 1 -- 172.21.15.76:0/654047384 >> 172.21.15.76:0/654047384 conn(0x7f156c0fcd80 msgr2=0x7f156c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:04.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.598+0000 7f1570cac640 1 -- 172.21.15.76:0/654047384 shutdown_connections 2024-03-18T14:53:04.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.598+0000 7f1570cac640 1 -- 172.21.15.76:0/654047384 wait complete. 2024-03-18T14:53:04.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.598+0000 7f1570cac640 1 Processor -- start 2024-03-18T14:53:04.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.598+0000 7f1570cac640 1 -- start start 2024-03-18T14:53:04.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.598+0000 7f1570cac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f156c1017f0 0x7f156c075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:04.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.598+0000 7f1570cac640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f156c1029e0 0x7f156c073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:04.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.598+0000 7f1570cac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f156c1033a0 0x7f156c074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:04.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.598+0000 7f1570cac640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f156c10d440 con 0x7f156c1029e0 2024-03-18T14:53:04.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.598+0000 7f1570cac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f156c10d2c0 con 0x7f156c1017f0 2024-03-18T14:53:04.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.598+0000 7f1570cac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f156c10d5c0 con 0x7f156c1033a0 2024-03-18T14:53:04.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.599+0000 7f1569d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f156c1029e0 0x7f156c073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:04.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.599+0000 7f1569d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f156c1029e0 0x7f156c073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50304/0 (socket says 172.21.15.76:50304) 2024-03-18T14:53:04.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.599+0000 7f156ad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f156c1033a0 0x7f156c074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:04.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.599+0000 7f1569d9b640 1 -- 172.21.15.76:0/3832198944 learned_addr learned my addr 172.21.15.76:0/3832198944 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:04.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.599+0000 7f156a59c640 1 --2- 172.21.15.76:0/3832198944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f156c1017f0 0x7f156c075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:04.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.599+0000 7f1569d9b640 1 -- 172.21.15.76:0/3832198944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f156c1033a0 msgr2=0x7f156c074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:04.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.599+0000 7f1569d9b640 1 --2- 172.21.15.76:0/3832198944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f156c1033a0 0x7f156c074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.599+0000 7f1569d9b640 1 -- 172.21.15.76:0/3832198944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f156c1017f0 msgr2=0x7f156c075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:04.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.599+0000 7f1569d9b640 1 --2- 172.21.15.76:0/3832198944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f156c1017f0 0x7f156c075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.599+0000 7f1569d9b640 1 -- 172.21.15.76:0/3832198944 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156c0749a0 con 0x7f156c1029e0 2024-03-18T14:53:04.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.599+0000 7f156ad9d640 1 --2- 172.21.15.76:0/3832198944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f156c1033a0 0x7f156c074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:04.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.599+0000 7f1569d9b640 1 --2- 172.21.15.76:0/3832198944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f156c1029e0 0x7f156c073bd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f156000e9f0 tx=0x7f156000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:04.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.599+0000 7f155b7fe640 1 -- 172.21.15.76:0/3832198944 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156000ccc0 con 0x7f156c1029e0 2024-03-18T14:53:04.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.599+0000 7f1570cac640 1 -- 172.21.15.76:0/3832198944 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f156c1a1cb0 con 0x7f156c1029e0 2024-03-18T14:53:04.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.599+0000 7f155b7fe640 1 -- 172.21.15.76:0/3832198944 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1560010910 con 0x7f156c1029e0 2024-03-18T14:53:04.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.599+0000 7f155b7fe640 1 -- 172.21.15.76:0/3832198944 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15600188a0 con 0x7f156c1029e0 2024-03-18T14:53:04.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.599+0000 7f1570cac640 1 -- 172.21.15.76:0/3832198944 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f156c1a21a0 con 0x7f156c1029e0 2024-03-18T14:53:04.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.600+0000 7f1570cac640 1 -- 172.21.15.76:0/3832198944 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f156c108bb0 con 0x7f156c1029e0 2024-03-18T14:53:04.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.601+0000 7f155b7fe640 1 -- 172.21.15.76:0/3832198944 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1560010ac0 con 0x7f156c1029e0 2024-03-18T14:53:04.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.602+0000 7f155b7fe640 1 --2- 172.21.15.76:0/3832198944 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f15540a2800 0x7f15540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:04.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.602+0000 7f156a59c640 1 --2- 172.21.15.76:0/3832198944 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f15540a2800 0x7f15540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:04.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.602+0000 7f155b7fe640 1 -- 172.21.15.76:0/3832198944 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f15600ceed0 con 0x7f156c1029e0 2024-03-18T14:53:04.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.602+0000 7f156a59c640 1 --2- 172.21.15.76:0/3832198944 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f15540a2800 0x7f15540a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f155c000c00 tx=0x7f155c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:04.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.606+0000 7f155b7fe640 1 -- 172.21.15.76:0/3832198944 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f156009bf40 con 0x7f156c1029e0 2024-03-18T14:53:04.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.781+0000 7f1570cac640 1 -- 172.21.15.76:0/3832198944 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f156c0623e0 con 0x7f156c1029e0 2024-03-18T14:53:04.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.781+0000 7f155b7fe640 1 -- 172.21.15.76:0/3832198944 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f156009b840 con 0x7f156c1029e0 2024-03-18T14:53:04.782 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-18T14:53:04.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.783+0000 7f1570cac640 1 -- 172.21.15.76:0/3832198944 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f15540a2800 msgr2=0x7f15540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:04.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.783+0000 7f1570cac640 1 --2- 172.21.15.76:0/3832198944 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f15540a2800 0x7f15540a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f155c000c00 tx=0x7f155c005fd0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.783+0000 7f1570cac640 1 -- 172.21.15.76:0/3832198944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f156c1029e0 msgr2=0x7f156c073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:04.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.783+0000 7f1570cac640 1 --2- 172.21.15.76:0/3832198944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f156c1029e0 0x7f156c073bd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f156000e9f0 tx=0x7f156000edf0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.783+0000 7f1570cac640 1 -- 172.21.15.76:0/3832198944 shutdown_connections 2024-03-18T14:53:04.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.783+0000 7f1570cac640 1 --2- 172.21.15.76:0/3832198944 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f15540a2800 0x7f15540a4cc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.783+0000 7f1570cac640 1 --2- 172.21.15.76:0/3832198944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f156c1033a0 0x7f156c074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.783+0000 7f1570cac640 1 --2- 172.21.15.76:0/3832198944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f156c1029e0 0x7f156c073bd0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.783+0000 7f1570cac640 1 --2- 172.21.15.76:0/3832198944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f156c1017f0 0x7f156c075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.783+0000 7f1570cac640 1 -- 172.21.15.76:0/3832198944 >> 172.21.15.76:0/3832198944 conn(0x7f156c0fcd80 msgr2=0x7f156c105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:04.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.783+0000 7f1570cac640 1 -- 172.21.15.76:0/3832198944 shutdown_connections 2024-03-18T14:53:04.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.784+0000 7f1570cac640 1 -- 172.21.15.76:0/3832198944 wait complete. 2024-03-18T14:53:04.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:04 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3807826142' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-18T14:53:04.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:04 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:53:04.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:04 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/4203041775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:53:04.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:04 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3899483721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:53:04.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:04 smithi076 ceph-mon[32205]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-18T14:53:04.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:04 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/813225674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:53:04.796 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:53:04.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.897+0000 7fc3cdaf1640 1 -- 172.21.15.76:0/1041737643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3c81022d0 msgr2=0x7fc3c8108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:04.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.897+0000 7fc3cdaf1640 1 --2- 172.21.15.76:0/1041737643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3c81022d0 0x7fc3c8108b60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009ca0 tx=0x7fc3c002c0f0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.897+0000 7fc3cdaf1640 1 -- 172.21.15.76:0/1041737643 shutdown_connections 2024-03-18T14:53:04.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.897+0000 7fc3cdaf1640 1 --2- 172.21.15.76:0/1041737643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3c81022d0 0x7fc3c8108b60 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.897+0000 7fc3cdaf1640 1 --2- 172.21.15.76:0/1041737643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3c8101910 0x7fc3c8101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.897+0000 7fc3cdaf1640 1 --2- 172.21.15.76:0/1041737643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3c8100720 0x7fc3c8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.897+0000 7fc3cdaf1640 1 -- 172.21.15.76:0/1041737643 >> 172.21.15.76:0/1041737643 conn(0x7fc3c80fbc90 msgr2=0x7fc3c80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:04.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.897+0000 7fc3cdaf1640 1 -- 172.21.15.76:0/1041737643 shutdown_connections 2024-03-18T14:53:04.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.897+0000 7fc3cdaf1640 1 -- 172.21.15.76:0/1041737643 wait complete. 2024-03-18T14:53:04.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.898+0000 7fc3cdaf1640 1 Processor -- start 2024-03-18T14:53:04.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.898+0000 7fc3cdaf1640 1 -- start start 2024-03-18T14:53:04.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.898+0000 7fc3cdaf1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3c8100720 0x7fc3c8196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:04.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.898+0000 7fc3cdaf1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3c8101910 0x7fc3c8196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:04.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.898+0000 7fc3cdaf1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3c81022d0 0x7fc3c819dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:04.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.898+0000 7fc3cdaf1640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c810c380 con 0x7fc3c8101910 2024-03-18T14:53:04.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.898+0000 7fc3cdaf1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c810c200 con 0x7fc3c8100720 2024-03-18T14:53:04.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.898+0000 7fc3cdaf1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c810c500 con 0x7fc3c81022d0 2024-03-18T14:53:04.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.898+0000 7fc3c6ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3c8101910 0x7fc3c8196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:04.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.898+0000 7fc3c6ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3c8101910 0x7fc3c8196f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50324/0 (socket says 172.21.15.76:50324) 2024-03-18T14:53:04.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.898+0000 7fc3c6ffd640 1 -- 172.21.15.76:0/1773083363 learned_addr learned my addr 172.21.15.76:0/1773083363 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:04.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.898+0000 7fc3c7fff640 1 --2- 172.21.15.76:0/1773083363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3c81022d0 0x7fc3c819dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:04.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.898+0000 7fc3c77fe640 1 --2- 172.21.15.76:0/1773083363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3c8100720 0x7fc3c8196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:04.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.899+0000 7fc3c6ffd640 1 -- 172.21.15.76:0/1773083363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3c81022d0 msgr2=0x7fc3c819dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:04.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.899+0000 7fc3c6ffd640 1 --2- 172.21.15.76:0/1773083363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3c81022d0 0x7fc3c819dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.899+0000 7fc3c6ffd640 1 -- 172.21.15.76:0/1773083363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3c8100720 msgr2=0x7fc3c8196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:04.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.899+0000 7fc3c6ffd640 1 --2- 172.21.15.76:0/1773083363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3c8100720 0x7fc3c8196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:04.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.899+0000 7fc3c6ffd640 1 -- 172.21.15.76:0/1773083363 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3c819e650 con 0x7fc3c8101910 2024-03-18T14:53:04.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.899+0000 7fc3c6ffd640 1 --2- 172.21.15.76:0/1773083363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3c8101910 0x7fc3c8196f70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00e9f0 tx=0x7fc3bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:04.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.899+0000 7fc3c4ff9640 1 -- 172.21.15.76:0/1773083363 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc00cd80 con 0x7fc3c8101910 2024-03-18T14:53:04.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.899+0000 7fc3cdaf1640 1 -- 172.21.15.76:0/1773083363 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3c819e940 con 0x7fc3c8101910 2024-03-18T14:53:04.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.899+0000 7fc3c4ff9640 1 -- 172.21.15.76:0/1773083363 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3bc010910 con 0x7fc3c8101910 2024-03-18T14:53:04.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.899+0000 7fc3c4ff9640 1 -- 172.21.15.76:0/1773083363 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc0188f0 con 0x7fc3c8101910 2024-03-18T14:53:04.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.899+0000 7fc3cdaf1640 1 -- 172.21.15.76:0/1773083363 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3c819ee90 con 0x7fc3c8101910 2024-03-18T14:53:04.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.900+0000 7fc3cdaf1640 1 -- 172.21.15.76:0/1773083363 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3c8068a20 con 0x7fc3c8101910 2024-03-18T14:53:04.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.901+0000 7fc3c4ff9640 1 -- 172.21.15.76:0/1773083363 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc3bc010ac0 con 0x7fc3c8101910 2024-03-18T14:53:04.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.902+0000 7fc3c4ff9640 1 --2- 172.21.15.76:0/1773083363 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc3b00a2800 0x7fc3b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:04.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.902+0000 7fc3c77fe640 1 --2- 172.21.15.76:0/1773083363 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc3b00a2800 0x7fc3b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:04.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.902+0000 7fc3c4ff9640 1 -- 172.21.15.76:0/1773083363 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc3bc0cf1a0 con 0x7fc3c8101910 2024-03-18T14:53:04.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.902+0000 7fc3c77fe640 1 --2- 172.21.15.76:0/1773083363 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc3b00a2800 0x7fc3b00a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc3b80095f0 tx=0x7fc3b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:04.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:04.905+0000 7fc3c4ff9640 1 -- 172.21.15.76:0/1773083363 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc3bc09bc20 con 0x7fc3c8101910 2024-03-18T14:53:04.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:04 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3807826142' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-18T14:53:04.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:04 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:53:04.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:04 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/4203041775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:53:04.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:04 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3899483721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:53:04.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:04 smithi174 ceph-mon[36787]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-18T14:53:04.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:04 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/813225674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:53:04.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:04 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3807826142' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-18T14:53:04.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:04 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:53:04.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:04 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/4203041775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:53:04.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:04 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3899483721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:53:04.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:04 smithi077 ceph-mon[35865]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-18T14:53:04.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:04 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/813225674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:53:05.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.083+0000 7fc3cdaf1640 1 -- 172.21.15.76:0/1773083363 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3c80623e0 con 0x7fc3c8101910 2024-03-18T14:53:05.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.085+0000 7fc3c4ff9640 1 -- 172.21.15.76:0/1773083363 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22580 (secure 0 0 0) 0x7fc3bc086090 con 0x7fc3c8101910 2024-03-18T14:53:05.086 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:53:05.086 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:52:07.361301+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:53:05.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.087+0000 7fc3cdaf1640 1 -- 172.21.15.76:0/1773083363 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc3b00a2800 msgr2=0x7fc3b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:05.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.087+0000 7fc3cdaf1640 1 --2- 172.21.15.76:0/1773083363 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc3b00a2800 0x7fc3b00a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc3b80095f0 tx=0x7fc3b8005f90 comp rx=0 tx=0).stop 2024-03-18T14:53:05.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.087+0000 7fc3cdaf1640 1 -- 172.21.15.76:0/1773083363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3c8101910 msgr2=0x7fc3c8196f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:05.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.087+0000 7fc3cdaf1640 1 --2- 172.21.15.76:0/1773083363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3c8101910 0x7fc3c8196f70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00e9f0 tx=0x7fc3bc00eeb0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.087+0000 7fc3cdaf1640 1 -- 172.21.15.76:0/1773083363 shutdown_connections 2024-03-18T14:53:05.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.087+0000 7fc3cdaf1640 1 --2- 172.21.15.76:0/1773083363 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fc3b00a2800 0x7fc3b00a4cc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.087+0000 7fc3cdaf1640 1 --2- 172.21.15.76:0/1773083363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3c81022d0 0x7fc3c819dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.087+0000 7fc3cdaf1640 1 --2- 172.21.15.76:0/1773083363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3c8101910 0x7fc3c8196f70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.087+0000 7fc3cdaf1640 1 --2- 172.21.15.76:0/1773083363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3c8100720 0x7fc3c8196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.087+0000 7fc3cdaf1640 1 -- 172.21.15.76:0/1773083363 >> 172.21.15.76:0/1773083363 conn(0x7fc3c80fbc90 msgr2=0x7fc3c8104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:05.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.087+0000 7fc3cdaf1640 1 -- 172.21.15.76:0/1773083363 shutdown_connections 2024-03-18T14:53:05.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.087+0000 7fc3cdaf1640 1 -- 172.21.15.76:0/1773083363 wait complete. 2024-03-18T14:53:05.099 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:46:26.588384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:50:59.606852+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:51:00.465325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:53:05.099 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:53:05.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.204+0000 7f182b3a9640 1 -- 172.21.15.76:0/2577896473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18241033a0 msgr2=0x7f1824109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:05.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.204+0000 7f182b3a9640 1 --2- 172.21.15.76:0/2577896473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18241033a0 0x7f1824109c30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f181c009ca0 tx=0x7f181c02c0f0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.204+0000 7f182b3a9640 1 -- 172.21.15.76:0/2577896473 shutdown_connections 2024-03-18T14:53:05.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.204+0000 7f182b3a9640 1 --2- 172.21.15.76:0/2577896473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18241029e0 0x7f1824102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.204+0000 7f182b3a9640 1 --2- 172.21.15.76:0/2577896473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18241033a0 0x7f1824109c30 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.204+0000 7f182b3a9640 1 --2- 172.21.15.76:0/2577896473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18241017f0 0x7f1824101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.204+0000 7f182b3a9640 1 -- 172.21.15.76:0/2577896473 >> 172.21.15.76:0/2577896473 conn(0x7f18240fcd80 msgr2=0x7f18240ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:05.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.204+0000 7f182b3a9640 1 -- 172.21.15.76:0/2577896473 shutdown_connections 2024-03-18T14:53:05.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.204+0000 7f182b3a9640 1 -- 172.21.15.76:0/2577896473 wait complete. 2024-03-18T14:53:05.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.204+0000 7f182b3a9640 1 Processor -- start 2024-03-18T14:53:05.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.204+0000 7f182b3a9640 1 -- start start 2024-03-18T14:53:05.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.204+0000 7f182b3a9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18241017f0 0x7f1824075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:05.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.204+0000 7f182b3a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18241029e0 0x7f1824073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:05.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.204+0000 7f182b3a9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18241033a0 0x7f1824074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:05.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.204+0000 7f182b3a9640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f182410d450 con 0x7f18241033a0 2024-03-18T14:53:05.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.204+0000 7f182b3a9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f182410d2d0 con 0x7f18241029e0 2024-03-18T14:53:05.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.204+0000 7f182b3a9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f182410d5d0 con 0x7f18241017f0 2024-03-18T14:53:05.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.205+0000 7f1829946640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18241033a0 0x7f1824074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:05.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.205+0000 7f1829145640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18241017f0 0x7f1824075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:05.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.205+0000 7f1829946640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18241033a0 0x7f1824074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50346/0 (socket says 172.21.15.76:50346) 2024-03-18T14:53:05.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.205+0000 7f1829946640 1 -- 172.21.15.76:0/3720568988 learned_addr learned my addr 172.21.15.76:0/3720568988 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:05.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.205+0000 7f1828944640 1 --2- 172.21.15.76:0/3720568988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18241029e0 0x7f1824073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:05.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.205+0000 7f1829946640 1 -- 172.21.15.76:0/3720568988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18241017f0 msgr2=0x7f1824075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:05.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.205+0000 7f1829946640 1 --2- 172.21.15.76:0/3720568988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18241017f0 0x7f1824075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.205+0000 7f1829946640 1 -- 172.21.15.76:0/3720568988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18241029e0 msgr2=0x7f1824073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:05.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.205+0000 7f1829946640 1 --2- 172.21.15.76:0/3720568988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18241029e0 0x7f1824073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.205+0000 7f1829946640 1 -- 172.21.15.76:0/3720568988 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18240749a0 con 0x7f18241033a0 2024-03-18T14:53:05.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.205+0000 7f1829946640 1 --2- 172.21.15.76:0/3720568988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18241033a0 0x7f1824074110 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f181c037040 tx=0x7f181c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:05.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.205+0000 7f181a7fc640 1 -- 172.21.15.76:0/3720568988 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c041020 con 0x7f18241033a0 2024-03-18T14:53:05.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.205+0000 7f181a7fc640 1 -- 172.21.15.76:0/3720568988 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f181c034cd0 con 0x7f18241033a0 2024-03-18T14:53:05.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.205+0000 7f181a7fc640 1 -- 172.21.15.76:0/3720568988 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c039040 con 0x7f18241033a0 2024-03-18T14:53:05.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.205+0000 7f182b3a9640 1 -- 172.21.15.76:0/3720568988 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18241a1b50 con 0x7f18241033a0 2024-03-18T14:53:05.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.205+0000 7f182b3a9640 1 -- 172.21.15.76:0/3720568988 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18241a20a0 con 0x7f18241033a0 2024-03-18T14:53:05.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.206+0000 7f182b3a9640 1 -- 172.21.15.76:0/3720568988 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1824108b90 con 0x7f18241033a0 2024-03-18T14:53:05.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.207+0000 7f181a7fc640 1 -- 172.21.15.76:0/3720568988 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f181c044020 con 0x7f18241033a0 2024-03-18T14:53:05.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.208+0000 7f181a7fc640 1 --2- 172.21.15.76:0/3720568988 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f18100a2800 0x7f18100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:05.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.208+0000 7f1829145640 1 --2- 172.21.15.76:0/3720568988 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f18100a2800 0x7f18100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:05.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.208+0000 7f181a7fc640 1 -- 172.21.15.76:0/3720568988 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f181c0f0ca0 con 0x7f18241033a0 2024-03-18T14:53:05.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.209+0000 7f1829145640 1 --2- 172.21.15.76:0/3720568988 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f18100a2800 0x7f18100a4cc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1824102840 tx=0x7f1814005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:05.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.212+0000 7f181a7fc640 1 -- 172.21.15.76:0/3720568988 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f181c04b020 con 0x7f18241033a0 2024-03-18T14:53:05.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.388+0000 7f182b3a9640 1 -- 172.21.15.76:0/3720568988 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f182402d1b0 con 0x7f18241033a0 2024-03-18T14:53:05.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.388+0000 7f181a7fc640 1 -- 172.21.15.76:0/3720568988 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f181c0bdda0 con 0x7f18241033a0 2024-03-18T14:53:05.389 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-18T14:53:05.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.390+0000 7f182b3a9640 1 -- 172.21.15.76:0/3720568988 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f18100a2800 msgr2=0x7f18100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:05.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.390+0000 7f182b3a9640 1 --2- 172.21.15.76:0/3720568988 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f18100a2800 0x7f18100a4cc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1824102840 tx=0x7f1814005f90 comp rx=0 tx=0).stop 2024-03-18T14:53:05.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.390+0000 7f182b3a9640 1 -- 172.21.15.76:0/3720568988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18241033a0 msgr2=0x7f1824074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:05.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.390+0000 7f182b3a9640 1 --2- 172.21.15.76:0/3720568988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18241033a0 0x7f1824074110 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f181c037040 tx=0x7f181c032f40 comp rx=0 tx=0).stop 2024-03-18T14:53:05.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.390+0000 7f182b3a9640 1 -- 172.21.15.76:0/3720568988 shutdown_connections 2024-03-18T14:53:05.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.390+0000 7f182b3a9640 1 --2- 172.21.15.76:0/3720568988 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f18100a2800 0x7f18100a4cc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.390+0000 7f182b3a9640 1 --2- 172.21.15.76:0/3720568988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18241033a0 0x7f1824074110 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.390+0000 7f182b3a9640 1 --2- 172.21.15.76:0/3720568988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18241029e0 0x7f1824073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.390+0000 7f182b3a9640 1 --2- 172.21.15.76:0/3720568988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18241017f0 0x7f1824075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.390+0000 7f182b3a9640 1 -- 172.21.15.76:0/3720568988 >> 172.21.15.76:0/3720568988 conn(0x7f18240fcd80 msgr2=0x7f1824105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:05.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.390+0000 7f182b3a9640 1 -- 172.21.15.76:0/3720568988 shutdown_connections 2024-03-18T14:53:05.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.390+0000 7f182b3a9640 1 -- 172.21.15.76:0/3720568988 wait complete. 2024-03-18T14:53:05.402 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:53:05.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.506+0000 7f87411f4640 1 -- 172.21.15.76:0/290489472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f873c104c20 msgr2=0x7f873c107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:05.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.506+0000 7f87411f4640 1 --2- 172.21.15.76:0/290489472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f873c104c20 0x7f873c107010 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f8724009900 tx=0x7f872402bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:05.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.507+0000 7f87411f4640 1 -- 172.21.15.76:0/290489472 shutdown_connections 2024-03-18T14:53:05.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.507+0000 7f87411f4640 1 --2- 172.21.15.76:0/290489472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f873c107550 0x7f873c109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.507+0000 7f87411f4640 1 --2- 172.21.15.76:0/290489472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f873c104c20 0x7f873c107010 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.507+0000 7f87411f4640 1 --2- 172.21.15.76:0/290489472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f873c1022f0 0x7f873c1046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.507+0000 7f87411f4640 1 -- 172.21.15.76:0/290489472 >> 172.21.15.76:0/290489472 conn(0x7f873c0fbc80 msgr2=0x7f873c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:05.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.507+0000 7f87411f4640 1 -- 172.21.15.76:0/290489472 shutdown_connections 2024-03-18T14:53:05.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.507+0000 7f87411f4640 1 -- 172.21.15.76:0/290489472 wait complete. 2024-03-18T14:53:05.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.507+0000 7f87411f4640 1 Processor -- start 2024-03-18T14:53:05.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.507+0000 7f87411f4640 1 -- start start 2024-03-18T14:53:05.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.507+0000 7f87411f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f873c1022f0 0x7f873c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:05.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.507+0000 7f87411f4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f873c104c20 0x7f873c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:05.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.507+0000 7f87411f4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f873c107550 0x7f873c19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:05.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f87411f4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f873c10d440 con 0x7f873c107550 2024-03-18T14:53:05.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f87411f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f873c10d2c0 con 0x7f873c1022f0 2024-03-18T14:53:05.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f87411f4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f873c10d5c0 con 0x7f873c104c20 2024-03-18T14:53:05.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f873b59e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f873c107550 0x7f873c19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:05.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f873ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f873c1022f0 0x7f873c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:05.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f8733fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f873c104c20 0x7f873c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:05.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f873ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f873c1022f0 0x7f873c197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:46902/0 (socket says 172.21.15.76:46902) 2024-03-18T14:53:05.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f873ad9d640 1 -- 172.21.15.76:0/4213346704 learned_addr learned my addr 172.21.15.76:0/4213346704 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:05.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f873b59e640 1 -- 172.21.15.76:0/4213346704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f873c104c20 msgr2=0x7f873c198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:05.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f873b59e640 1 --2- 172.21.15.76:0/4213346704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f873c104c20 0x7f873c198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f873b59e640 1 -- 172.21.15.76:0/4213346704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f873c1022f0 msgr2=0x7f873c197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:05.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f873b59e640 1 --2- 172.21.15.76:0/4213346704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f873c1022f0 0x7f873c197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f873b59e640 1 -- 172.21.15.76:0/4213346704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f873c19f770 con 0x7f873c107550 2024-03-18T14:53:05.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f873b59e640 1 --2- 172.21.15.76:0/4213346704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f873c107550 0x7f873c19f100 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8734009fd0 tx=0x7f873400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:05.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f8738d99640 1 -- 172.21.15.76:0/4213346704 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873400ce20 con 0x7f873c107550 2024-03-18T14:53:05.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f8738d99640 1 -- 172.21.15.76:0/4213346704 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8734005020 con 0x7f873c107550 2024-03-18T14:53:05.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f87411f4640 1 -- 172.21.15.76:0/4213346704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f873c19fa60 con 0x7f873c107550 2024-03-18T14:53:05.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f8738d99640 1 -- 172.21.15.76:0/4213346704 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8734010720 con 0x7f873c107550 2024-03-18T14:53:05.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.508+0000 7f87411f4640 1 -- 172.21.15.76:0/4213346704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f873c19ff30 con 0x7f873c107550 2024-03-18T14:53:05.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.509+0000 7f87411f4640 1 -- 172.21.15.76:0/4213346704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f873c0fd770 con 0x7f873c107550 2024-03-18T14:53:05.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.510+0000 7f8738d99640 1 -- 172.21.15.76:0/4213346704 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8734005540 con 0x7f873c107550 2024-03-18T14:53:05.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.511+0000 7f8738d99640 1 --2- 172.21.15.76:0/4213346704 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f87280a2800 0x7f87280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:05.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.511+0000 7f873ad9d640 1 --2- 172.21.15.76:0/4213346704 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f87280a2800 0x7f87280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:05.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.511+0000 7f8738d99640 1 -- 172.21.15.76:0/4213346704 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f87340cf110 con 0x7f873c107550 2024-03-18T14:53:05.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.511+0000 7f873ad9d640 1 --2- 172.21.15.76:0/4213346704 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f87280a2800 0x7f87280a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f872c0095f0 tx=0x7f872c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:05.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.515+0000 7f8738d99640 1 -- 172.21.15.76:0/4213346704 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f873409c1e0 con 0x7f873c107550 2024-03-18T14:53:05.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:05 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3832198944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:53:05.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:05 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1773083363' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:53:05.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:05 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3720568988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:53:05.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.694+0000 7f87411f4640 1 -- 172.21.15.76:0/4213346704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f873c0623e0 con 0x7f873c107550 2024-03-18T14:53:05.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.694+0000 7f8738d99640 1 -- 172.21.15.76:0/4213346704 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f873409bae0 con 0x7f873c107550 2024-03-18T14:53:05.695 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-18T14:53:05.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.696+0000 7f87411f4640 1 -- 172.21.15.76:0/4213346704 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f87280a2800 msgr2=0x7f87280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:05.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.696+0000 7f87411f4640 1 --2- 172.21.15.76:0/4213346704 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f87280a2800 0x7f87280a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f872c0095f0 tx=0x7f872c005f90 comp rx=0 tx=0).stop 2024-03-18T14:53:05.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.696+0000 7f87411f4640 1 -- 172.21.15.76:0/4213346704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f873c107550 msgr2=0x7f873c19f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:05.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.696+0000 7f87411f4640 1 --2- 172.21.15.76:0/4213346704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f873c107550 0x7f873c19f100 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8734009fd0 tx=0x7f873400ef10 comp rx=0 tx=0).stop 2024-03-18T14:53:05.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.696+0000 7f87411f4640 1 -- 172.21.15.76:0/4213346704 shutdown_connections 2024-03-18T14:53:05.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.696+0000 7f87411f4640 1 --2- 172.21.15.76:0/4213346704 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f87280a2800 0x7f87280a4cc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.696+0000 7f87411f4640 1 --2- 172.21.15.76:0/4213346704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f873c104c20 0x7f873c198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.696+0000 7f87411f4640 1 --2- 172.21.15.76:0/4213346704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f873c107550 0x7f873c19f100 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.696+0000 7f87411f4640 1 --2- 172.21.15.76:0/4213346704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f873c1022f0 0x7f873c197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.696+0000 7f87411f4640 1 -- 172.21.15.76:0/4213346704 >> 172.21.15.76:0/4213346704 conn(0x7f873c0fbc80 msgr2=0x7f873c102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:05.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.696+0000 7f87411f4640 1 -- 172.21.15.76:0/4213346704 shutdown_connections 2024-03-18T14:53:05.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.696+0000 7f87411f4640 1 -- 172.21.15.76:0/4213346704 wait complete. 2024-03-18T14:53:05.708 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:53:05.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.812+0000 7f59882e8640 1 -- 172.21.15.76:0/2756899071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59801006f0 msgr2=0x7f5980100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:05.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.812+0000 7f59882e8640 1 --2- 172.21.15.76:0/2756899071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59801006f0 0x7f5980100b10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f597c009900 tx=0x7f597c02bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:05.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.812+0000 7f59882e8640 1 -- 172.21.15.76:0/2756899071 shutdown_connections 2024-03-18T14:53:05.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.812+0000 7f59882e8640 1 --2- 172.21.15.76:0/2756899071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59801018e0 0x7f5980101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.812+0000 7f59882e8640 1 --2- 172.21.15.76:0/2756899071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59801022a0 0x7f5980108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.812+0000 7f59882e8640 1 --2- 172.21.15.76:0/2756899071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59801006f0 0x7f5980100b10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.812+0000 7f59882e8640 1 -- 172.21.15.76:0/2756899071 >> 172.21.15.76:0/2756899071 conn(0x7f59800fbc80 msgr2=0x7f59800fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:05.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.812+0000 7f59882e8640 1 -- 172.21.15.76:0/2756899071 shutdown_connections 2024-03-18T14:53:05.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.812+0000 7f59882e8640 1 -- 172.21.15.76:0/2756899071 wait complete. 2024-03-18T14:53:05.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f59882e8640 1 Processor -- start 2024-03-18T14:53:05.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f59882e8640 1 -- start start 2024-03-18T14:53:05.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f59882e8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59801006f0 0x7f5980197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:05.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f59882e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59801018e0 0x7f5980198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:05.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f59882e8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59801022a0 0x7f598019f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:05.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f59882e8640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f598010d490 con 0x7f59801006f0 2024-03-18T14:53:05.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f59882e8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f598010d310 con 0x7f59801018e0 2024-03-18T14:53:05.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f59882e8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f598010d610 con 0x7f59801022a0 2024-03-18T14:53:05.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f5986084640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59801006f0 0x7f5980197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:05.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f5986084640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59801006f0 0x7f5980197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50368/0 (socket says 172.21.15.76:50368) 2024-03-18T14:53:05.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f5986084640 1 -- 172.21.15.76:0/3297089238 learned_addr learned my addr 172.21.15.76:0/3297089238 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:05.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f5985883640 1 --2- 172.21.15.76:0/3297089238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59801018e0 0x7f5980198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:05.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f5986885640 1 --2- 172.21.15.76:0/3297089238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59801022a0 0x7f598019f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:05.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f5986084640 1 -- 172.21.15.76:0/3297089238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59801022a0 msgr2=0x7f598019f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:05.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f5986084640 1 --2- 172.21.15.76:0/3297089238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59801022a0 0x7f598019f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f5986084640 1 -- 172.21.15.76:0/3297089238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59801018e0 msgr2=0x7f5980198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:05.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f5986084640 1 --2- 172.21.15.76:0/3297089238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59801018e0 0x7f5980198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f5986084640 1 -- 172.21.15.76:0/3297089238 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f598019f770 con 0x7f59801006f0 2024-03-18T14:53:05.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.813+0000 7f5986885640 1 --2- 172.21.15.76:0/3297089238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59801022a0 0x7f598019f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:53:05.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.814+0000 7f5986084640 1 --2- 172.21.15.76:0/3297089238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59801006f0 0x7f5980197b50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f597c000c00 tx=0x7f597c02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:05.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.814+0000 7f59777fe640 1 -- 172.21.15.76:0/3297089238 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597c041070 con 0x7f59801006f0 2024-03-18T14:53:05.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.814+0000 7f59777fe640 1 -- 172.21.15.76:0/3297089238 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f597c004450 con 0x7f59801006f0 2024-03-18T14:53:05.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.814+0000 7f59777fe640 1 -- 172.21.15.76:0/3297089238 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597c034440 con 0x7f59801006f0 2024-03-18T14:53:05.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.814+0000 7f59882e8640 1 -- 172.21.15.76:0/3297089238 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f598019fa00 con 0x7f59801006f0 2024-03-18T14:53:05.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.814+0000 7f59882e8640 1 -- 172.21.15.76:0/3297089238 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f598019fea0 con 0x7f59801006f0 2024-03-18T14:53:05.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.815+0000 7f59882e8640 1 -- 172.21.15.76:0/3297089238 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5980107a80 con 0x7f59801006f0 2024-03-18T14:53:05.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.816+0000 7f59777fe640 1 -- 172.21.15.76:0/3297089238 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f597c044070 con 0x7f59801006f0 2024-03-18T14:53:05.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.817+0000 7f59777fe640 1 --2- 172.21.15.76:0/3297089238 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f596c0a2800 0x7f596c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:05.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.817+0000 7f5985883640 1 --2- 172.21.15.76:0/3297089238 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f596c0a2800 0x7f596c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:05.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.817+0000 7f59777fe640 1 -- 172.21.15.76:0/3297089238 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f597c0f07b0 con 0x7f59801006f0 2024-03-18T14:53:05.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.817+0000 7f5985883640 1 --2- 172.21.15.76:0/3297089238 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f596c0a2800 0x7f596c0a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f59700095f0 tx=0x7f5970005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:05.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.821+0000 7f59777fe640 1 -- 172.21.15.76:0/3297089238 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f597c0bd910 con 0x7f59801006f0 2024-03-18T14:53:05.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:05 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3832198944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:53:05.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:05 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1773083363' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:53:05.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:05 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3720568988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:53:05.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:05 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3832198944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:53:05.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:05 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1773083363' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:53:05.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:05 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3720568988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:53:05.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.993+0000 7f59882e8640 1 -- 172.21.15.76:0/3297089238 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f598002d420 con 0x7f59801006f0 2024-03-18T14:53:05.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.994+0000 7f59777fe640 1 -- 172.21.15.76:0/3297089238 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f597c0bd210 con 0x7f59801006f0 2024-03-18T14:53:05.994 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-18T14:53:05.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.995+0000 7f59882e8640 1 -- 172.21.15.76:0/3297089238 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f596c0a2800 msgr2=0x7f596c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:05.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.995+0000 7f59882e8640 1 --2- 172.21.15.76:0/3297089238 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f596c0a2800 0x7f596c0a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f59700095f0 tx=0x7f5970005f90 comp rx=0 tx=0).stop 2024-03-18T14:53:05.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.995+0000 7f59882e8640 1 -- 172.21.15.76:0/3297089238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59801006f0 msgr2=0x7f5980197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:05.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.995+0000 7f59882e8640 1 --2- 172.21.15.76:0/3297089238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59801006f0 0x7f5980197b50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f597c000c00 tx=0x7f597c02c640 comp rx=0 tx=0).stop 2024-03-18T14:53:05.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.996+0000 7f59882e8640 1 -- 172.21.15.76:0/3297089238 shutdown_connections 2024-03-18T14:53:05.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.996+0000 7f59882e8640 1 --2- 172.21.15.76:0/3297089238 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f596c0a2800 0x7f596c0a4cc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.996+0000 7f59882e8640 1 --2- 172.21.15.76:0/3297089238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59801022a0 0x7f598019f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.996+0000 7f59882e8640 1 --2- 172.21.15.76:0/3297089238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59801018e0 0x7f5980198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.996+0000 7f59882e8640 1 --2- 172.21.15.76:0/3297089238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59801006f0 0x7f5980197b50 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:05.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.996+0000 7f59882e8640 1 -- 172.21.15.76:0/3297089238 >> 172.21.15.76:0/3297089238 conn(0x7f59800fbc80 msgr2=0x7f5980104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:05.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.996+0000 7f59882e8640 1 -- 172.21.15.76:0/3297089238 shutdown_connections 2024-03-18T14:53:05.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:05.996+0000 7f59882e8640 1 -- 172.21.15.76:0/3297089238 wait complete. 2024-03-18T14:53:06.007 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-18T14:53:06.007 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:53:06.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.109+0000 7f6a03214640 1 -- 172.21.15.76:0/188699696 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc1018f0 msgr2=0x7f69fc101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:06.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.109+0000 7f6a03214640 1 --2- 172.21.15.76:0/188699696 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc1018f0 0x7f69fc101d70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f69f0009900 tx=0x7f69f002bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:06.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.109+0000 7f6a03214640 1 -- 172.21.15.76:0/188699696 shutdown_connections 2024-03-18T14:53:06.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.109+0000 7f6a03214640 1 --2- 172.21.15.76:0/188699696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc1022b0 0x7f69fc108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.109+0000 7f6a03214640 1 --2- 172.21.15.76:0/188699696 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc1018f0 0x7f69fc101d70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.109+0000 7f6a03214640 1 --2- 172.21.15.76:0/188699696 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69fc100700 0x7f69fc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.109+0000 7f6a03214640 1 -- 172.21.15.76:0/188699696 >> 172.21.15.76:0/188699696 conn(0x7f69fc0fbc70 msgr2=0x7f69fc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:06.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.109+0000 7f6a03214640 1 -- 172.21.15.76:0/188699696 shutdown_connections 2024-03-18T14:53:06.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.109+0000 7f6a03214640 1 -- 172.21.15.76:0/188699696 wait complete. 2024-03-18T14:53:06.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.110+0000 7f6a03214640 1 Processor -- start 2024-03-18T14:53:06.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.110+0000 7f6a03214640 1 -- start start 2024-03-18T14:53:06.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.110+0000 7f6a03214640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc100700 0x7f69fc0694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:06.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.110+0000 7f6a03214640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69fc1018f0 0x7f69fc19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:06.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.110+0000 7f6a03214640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc1022b0 0x7f69fc19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:06.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.110+0000 7f6a03214640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f69fc10c360 con 0x7f69fc1022b0 2024-03-18T14:53:06.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.110+0000 7f6a03214640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f69fc10c1e0 con 0x7f69fc100700 2024-03-18T14:53:06.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.110+0000 7f6a03214640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f69fc10c4e0 con 0x7f69fc1018f0 2024-03-18T14:53:06.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.110+0000 7f6a017b1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc1022b0 0x7f69fc19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:06.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.110+0000 7f6a00fb0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc100700 0x7f69fc0694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:06.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.110+0000 7f69fbfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69fc1018f0 0x7f69fc19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:06.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.110+0000 7f69fbfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69fc1018f0 0x7f69fc19c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:42516/0 (socket says 172.21.15.76:42516) 2024-03-18T14:53:06.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.110+0000 7f6a00fb0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc100700 0x7f69fc0694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:46930/0 (socket says 172.21.15.76:46930) 2024-03-18T14:53:06.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.110+0000 7f69fbfff640 1 -- 172.21.15.76:0/130036353 learned_addr learned my addr 172.21.15.76:0/130036353 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:06.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.110+0000 7f6a017b1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc1022b0 0x7f69fc19ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50390/0 (socket says 172.21.15.76:50390) 2024-03-18T14:53:06.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.111+0000 7f6a017b1640 1 -- 172.21.15.76:0/130036353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69fc1018f0 msgr2=0x7f69fc19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:06.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.111+0000 7f6a017b1640 1 --2- 172.21.15.76:0/130036353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69fc1018f0 0x7f69fc19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.111+0000 7f6a017b1640 1 -- 172.21.15.76:0/130036353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc100700 msgr2=0x7f69fc0694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:06.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.111+0000 7f6a017b1640 1 --2- 172.21.15.76:0/130036353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc100700 0x7f69fc0694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.111+0000 7f6a017b1640 1 -- 172.21.15.76:0/130036353 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69fc19f730 con 0x7f69fc1022b0 2024-03-18T14:53:06.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.111+0000 7f6a017b1640 1 --2- 172.21.15.76:0/130036353 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc1022b0 0x7f69fc19ee70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f69f4009fd0 tx=0x7f69f400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:06.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.111+0000 7f69f9ffb640 1 -- 172.21.15.76:0/130036353 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f400ce20 con 0x7f69fc1022b0 2024-03-18T14:53:06.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.111+0000 7f69f9ffb640 1 -- 172.21.15.76:0/130036353 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69f4005020 con 0x7f69fc1022b0 2024-03-18T14:53:06.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.111+0000 7f6a03214640 1 -- 172.21.15.76:0/130036353 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69fc19fa20 con 0x7f69fc1022b0 2024-03-18T14:53:06.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.111+0000 7f69f9ffb640 1 -- 172.21.15.76:0/130036353 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f4010720 con 0x7f69fc1022b0 2024-03-18T14:53:06.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.111+0000 7f6a03214640 1 -- 172.21.15.76:0/130036353 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69fc19fef0 con 0x7f69fc1022b0 2024-03-18T14:53:06.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.112+0000 7f6a03214640 1 -- 172.21.15.76:0/130036353 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69fc107a20 con 0x7f69fc1022b0 2024-03-18T14:53:06.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.113+0000 7f69f9ffb640 1 -- 172.21.15.76:0/130036353 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f69f4005540 con 0x7f69fc1022b0 2024-03-18T14:53:06.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.114+0000 7f69f9ffb640 1 --2- 172.21.15.76:0/130036353 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f69e40a2800 0x7f69e40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:06.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.114+0000 7f6a00fb0640 1 --2- 172.21.15.76:0/130036353 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f69e40a2800 0x7f69e40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:06.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.114+0000 7f69f9ffb640 1 -- 172.21.15.76:0/130036353 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f69f40cf110 con 0x7f69fc1022b0 2024-03-18T14:53:06.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.114+0000 7f6a00fb0640 1 --2- 172.21.15.76:0/130036353 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f69e40a2800 0x7f69e40a4cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f69fc101750 tx=0x7f69ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:06.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.118+0000 7f69f9ffb640 1 -- 172.21.15.76:0/130036353 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f69f409c1e0 con 0x7f69fc1022b0 2024-03-18T14:53:06.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.331+0000 7f6a03214640 1 -- 172.21.15.76:0/130036353 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69fc0623e0 con 0x7f69fc1022b0 2024-03-18T14:53:06.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.331+0000 7f69f9ffb640 1 -- 172.21.15.76:0/130036353 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11863 (secure 0 0 0) 0x7f69f409bae0 con 0x7f69fc1022b0 2024-03-18T14:53:06.332 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:53:06.332 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/2801069369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":2801069369},{"type":"v1","addr":"172.21.15.76:6839","nonce":2801069369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2645229232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2645229232},{"type":"v1","addr":"172.21.15.76:6841","nonce":2645229232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T14:51:01.293009+0000","modified":"2024-03-18T14:53:02.728776+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24490,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6839/399810652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":399810652},{"type":"v1","addr":"172.21.15.174:6839","nonce":399810652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1685041985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1685041985},{"type":"v1","addr":"172.21.15.77:6839","nonce":1685041985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524,24463,24490]},"id":1}]} 2024-03-18T14:53:06.333 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 24 2024-03-18T14:53:06.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.333+0000 7f6a03214640 1 -- 172.21.15.76:0/130036353 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f69e40a2800 msgr2=0x7f69e40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.333+0000 7f6a03214640 1 --2- 172.21.15.76:0/130036353 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f69e40a2800 0x7f69e40a4cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f69fc101750 tx=0x7f69ec005f90 comp rx=0 tx=0).stop 2024-03-18T14:53:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.333+0000 7f6a03214640 1 -- 172.21.15.76:0/130036353 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc1022b0 msgr2=0x7f69fc19ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.333+0000 7f6a03214640 1 --2- 172.21.15.76:0/130036353 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc1022b0 0x7f69fc19ee70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f69f4009fd0 tx=0x7f69f400ef10 comp rx=0 tx=0).stop 2024-03-18T14:53:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.334+0000 7f6a03214640 1 -- 172.21.15.76:0/130036353 shutdown_connections 2024-03-18T14:53:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.334+0000 7f6a03214640 1 --2- 172.21.15.76:0/130036353 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f69e40a2800 0x7f69e40a4cc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.334+0000 7f6a03214640 1 --2- 172.21.15.76:0/130036353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69fc1018f0 0x7f69fc19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.334+0000 7f6a03214640 1 --2- 172.21.15.76:0/130036353 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc1022b0 0x7f69fc19ee70 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.334+0000 7f6a03214640 1 --2- 172.21.15.76:0/130036353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc100700 0x7f69fc0694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.334+0000 7f6a03214640 1 -- 172.21.15.76:0/130036353 >> 172.21.15.76:0/130036353 conn(0x7f69fc0fbc70 msgr2=0x7f69fc104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.334+0000 7f6a03214640 1 -- 172.21.15.76:0/130036353 shutdown_connections 2024-03-18T14:53:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.334+0000 7f6a03214640 1 -- 172.21.15.76:0/130036353 wait complete. 2024-03-18T14:53:06.347 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-18T14:53:06.347 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-18T14:53:06.347 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:53:06.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.449+0000 7fd946910640 1 -- 172.21.15.76:0/3119567571 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd940102ac0 msgr2=0x7fd940102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:06.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.449+0000 7fd946910640 1 --2- 172.21.15.76:0/3119567571 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd940102ac0 0x7fd940102f40 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd934009930 tx=0x7fd93402bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:06.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.449+0000 7fd946910640 1 -- 172.21.15.76:0/3119567571 shutdown_connections 2024-03-18T14:53:06.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.449+0000 7fd946910640 1 --2- 172.21.15.76:0/3119567571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd940103480 0x7fd940109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.449+0000 7fd946910640 1 --2- 172.21.15.76:0/3119567571 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd940102ac0 0x7fd940102f40 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.449+0000 7fd946910640 1 --2- 172.21.15.76:0/3119567571 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9400ffaf0 0x7fd9400fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.449+0000 7fd946910640 1 -- 172.21.15.76:0/3119567571 >> 172.21.15.76:0/3119567571 conn(0x7fd9400fb420 msgr2=0x7fd9400fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:06.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.449+0000 7fd946910640 1 -- 172.21.15.76:0/3119567571 shutdown_connections 2024-03-18T14:53:06.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.449+0000 7fd946910640 1 -- 172.21.15.76:0/3119567571 wait complete. 2024-03-18T14:53:06.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd946910640 1 Processor -- start 2024-03-18T14:53:06.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd946910640 1 -- start start 2024-03-18T14:53:06.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd946910640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9400ffaf0 0x7fd940102430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:06.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd946910640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd940102ac0 0x7fd940100a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:06.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd946910640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd940103480 0x7fd940100fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:06.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd946910640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd94010d4b0 con 0x7fd940103480 2024-03-18T14:53:06.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd946910640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd94010d330 con 0x7fd9400ffaf0 2024-03-18T14:53:06.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd946910640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd94010d630 con 0x7fd940102ac0 2024-03-18T14:53:06.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd944ead640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd940103480 0x7fd940100fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:06.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd944ead640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd940103480 0x7fd940100fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50396/0 (socket says 172.21.15.76:50396) 2024-03-18T14:53:06.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd944ead640 1 -- 172.21.15.76:0/266200680 learned_addr learned my addr 172.21.15.76:0/266200680 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:06.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd93f7fe640 1 --2- 172.21.15.76:0/266200680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd940102ac0 0x7fd940100a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:06.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd93ffff640 1 --2- 172.21.15.76:0/266200680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9400ffaf0 0x7fd940102430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:06.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd944ead640 1 -- 172.21.15.76:0/266200680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd940102ac0 msgr2=0x7fd940100a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:06.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd944ead640 1 --2- 172.21.15.76:0/266200680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd940102ac0 0x7fd940100a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd944ead640 1 -- 172.21.15.76:0/266200680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9400ffaf0 msgr2=0x7fd940102430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:06.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd944ead640 1 --2- 172.21.15.76:0/266200680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9400ffaf0 0x7fd940102430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.450+0000 7fd944ead640 1 -- 172.21.15.76:0/266200680 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd940101840 con 0x7fd940103480 2024-03-18T14:53:06.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.451+0000 7fd944ead640 1 --2- 172.21.15.76:0/266200680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd940103480 0x7fd940100fb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd93800efd0 tx=0x7fd93800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:06.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.451+0000 7fd93d7fa640 1 -- 172.21.15.76:0/266200680 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938003b70 con 0x7fd940103480 2024-03-18T14:53:06.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.451+0000 7fd946910640 1 -- 172.21.15.76:0/266200680 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9401a1c80 con 0x7fd940103480 2024-03-18T14:53:06.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.451+0000 7fd93d7fa640 1 -- 172.21.15.76:0/266200680 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd938003d10 con 0x7fd940103480 2024-03-18T14:53:06.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.451+0000 7fd93d7fa640 1 -- 172.21.15.76:0/266200680 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9380109f0 con 0x7fd940103480 2024-03-18T14:53:06.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.451+0000 7fd946910640 1 -- 172.21.15.76:0/266200680 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9401a21a0 con 0x7fd940103480 2024-03-18T14:53:06.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.452+0000 7fd946910640 1 -- 172.21.15.76:0/266200680 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd940068a20 con 0x7fd940103480 2024-03-18T14:53:06.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.453+0000 7fd93d7fa640 1 -- 172.21.15.76:0/266200680 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd938010b90 con 0x7fd940103480 2024-03-18T14:53:06.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.453+0000 7fd93d7fa640 1 --2- 172.21.15.76:0/266200680 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd9280a2800 0x7fd9280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:06.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.454+0000 7fd93ffff640 1 --2- 172.21.15.76:0/266200680 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd9280a2800 0x7fd9280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:06.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.454+0000 7fd93d7fa640 1 -- 172.21.15.76:0/266200680 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fd9380cf260 con 0x7fd940103480 2024-03-18T14:53:06.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.454+0000 7fd93ffff640 1 --2- 172.21.15.76:0/266200680 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd9280a2800 0x7fd9280a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd9300095f0 tx=0x7fd930005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:06.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.457+0000 7fd93d7fa640 1 -- 172.21.15.76:0/266200680 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd93809c330 con 0x7fd940103480 2024-03-18T14:53:06.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.634+0000 7fd946910640 1 -- 172.21.15.76:0/266200680 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd940061ce0 con 0x7fd9280a2800 2024-03-18T14:53:06.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.641+0000 7fd93d7fa640 1 -- 172.21.15.76:0/266200680 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259063 (secure 0 0 0) 0x7fd940061ce0 con 0x7fd9280a2800 2024-03-18T14:53:06.644 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:53:06.646 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-03-18T14:53:06.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.644+0000 7fd946910640 1 -- 172.21.15.76:0/266200680 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd9280a2800 msgr2=0x7fd9280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:06.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.644+0000 7fd946910640 1 --2- 172.21.15.76:0/266200680 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd9280a2800 0x7fd9280a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd9300095f0 tx=0x7fd930005f90 comp rx=0 tx=0).stop 2024-03-18T14:53:06.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.644+0000 7fd946910640 1 -- 172.21.15.76:0/266200680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd940103480 msgr2=0x7fd940100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:06.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.644+0000 7fd946910640 1 --2- 172.21.15.76:0/266200680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd940103480 0x7fd940100fb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd93800efd0 tx=0x7fd93800c6a0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.644+0000 7fd946910640 1 -- 172.21.15.76:0/266200680 shutdown_connections 2024-03-18T14:53:06.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.644+0000 7fd946910640 1 --2- 172.21.15.76:0/266200680 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fd9280a2800 0x7fd9280a4cc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.644+0000 7fd946910640 1 --2- 172.21.15.76:0/266200680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd940102ac0 0x7fd940100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.644+0000 7fd946910640 1 --2- 172.21.15.76:0/266200680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd940103480 0x7fd940100fb0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.644+0000 7fd946910640 1 --2- 172.21.15.76:0/266200680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9400ffaf0 0x7fd940102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.644+0000 7fd946910640 1 -- 172.21.15.76:0/266200680 >> 172.21.15.76:0/266200680 conn(0x7fd9400fb420 msgr2=0x7fd940105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:06.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.644+0000 7fd946910640 1 -- 172.21.15.76:0/266200680 shutdown_connections 2024-03-18T14:53:06.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.644+0000 7fd946910640 1 -- 172.21.15.76:0/266200680 wait complete. 2024-03-18T14:53:06.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:06 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/4213346704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:53:06.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:06 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3297089238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:53:06.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:06 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/130036353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:53:06.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:06 smithi076 ceph-mon[32205]: pgmap v348: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T14:53:06.656 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-03-18T14:53:06.359535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654521,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":863,"num_read_kb":549,"num_write":350,"num_write_kb":3055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1239648,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":783,"ondisk_log_size":783,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369088,"kb_used_data":11616,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572736,"statfs":{"total":1151940427776,"available":1151562481664,"internally_reserved":0,"allocated":11894784,"data_stored":7470517,"data_compressed":120345,"data_compressed_allocated":155648,"data_compressed_original":1843200,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":36,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002927"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.371528+0000","last_change":"2024-03-18T14:51:01.365948+0000","last_active":"2024-03-18T14:52:36.371528+0000","last_peered":"2024-03-18T14:52:36.371528+0000","last_clean":"2024-03-18T14:52:36.371528+0000","last_became_active":"2024-03-18T14:51:01.364600+0000","last_became_peered":"2024-03-18T14:51:01.364600+0000","last_unstale":"2024-03-18T14:52:36.371528+0000","last_undegraded":"2024-03-18T14:52:36.371528+0000","last_fullsized":"2024-03-18T14:52:36.371528+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:14:45.198618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.359354+0000","last_change":"2024-03-18T14:51:01.360186+0000","last_active":"2024-03-18T14:52:36.359354+0000","last_peered":"2024-03-18T14:52:36.359354+0000","last_clean":"2024-03-18T14:52:36.359354+0000","last_became_active":"2024-03-18T14:51:01.359862+0000","last_became_peered":"2024-03-18T14:51:01.359862+0000","last_unstale":"2024-03-18T14:52:36.359354+0000","last_undegraded":"2024-03-18T14:52:36.359354+0000","last_fullsized":"2024-03-18T14:52:36.359354+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:46.206823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375396+0000","last_change":"2024-03-18T14:51:01.364635+0000","last_active":"2024-03-18T14:52:36.375396+0000","last_peered":"2024-03-18T14:52:36.375396+0000","last_clean":"2024-03-18T14:52:36.375396+0000","last_became_active":"2024-03-18T14:51:01.364320+0000","last_became_peered":"2024-03-18T14:51:01.364320+0000","last_unstale":"2024-03-18T14:52:36.375396+0000","last_undegraded":"2024-03-18T14:52:36.375396+0000","last_fullsized":"2024-03-18T14:52:36.375396+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:08:20.227639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.358624+0000","last_change":"2024-03-18T14:52:07.372090+0000","last_active":"2024-03-18T14:52:36.358624+0000","last_peered":"2024-03-18T14:52:36.358624+0000","last_clean":"2024-03-18T14:52:36.358624+0000","last_became_active":"2024-03-18T14:52:07.371775+0000","last_became_peered":"2024-03-18T14:52:07.371775+0000","last_unstale":"2024-03-18T14:52:36.358624+0000","last_undegraded":"2024-03-18T14:52:36.358624+0000","last_fullsized":"2024-03-18T14:52:36.358624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:48.128827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.356734+0000","last_change":"2024-03-18T14:51:01.356839+0000","last_active":"2024-03-18T14:52:36.356734+0000","last_peered":"2024-03-18T14:52:36.356734+0000","last_clean":"2024-03-18T14:52:36.356734+0000","last_became_active":"2024-03-18T14:51:01.356531+0000","last_became_peered":"2024-03-18T14:51:01.356531+0000","last_unstale":"2024-03-18T14:52:36.356734+0000","last_undegraded":"2024-03-18T14:52:36.356734+0000","last_fullsized":"2024-03-18T14:52:36.356734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:28.499573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.357939+0000","last_change":"2024-03-18T14:51:01.362912+0000","last_active":"2024-03-18T14:52:36.357939+0000","last_peered":"2024-03-18T14:52:36.357939+0000","last_clean":"2024-03-18T14:52:36.357939+0000","last_became_active":"2024-03-18T14:51:01.362590+0000","last_became_peered":"2024-03-18T14:51:01.362590+0000","last_unstale":"2024-03-18T14:52:36.357939+0000","last_undegraded":"2024-03-18T14:52:36.357939+0000","last_fullsized":"2024-03-18T14:52:36.357939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:49.199616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.370814+0000","last_change":"2024-03-18T14:51:01.358652+0000","last_active":"2024-03-18T14:52:36.370814+0000","last_peered":"2024-03-18T14:52:36.370814+0000","last_clean":"2024-03-18T14:52:36.370814+0000","last_became_active":"2024-03-18T14:51:01.358312+0000","last_became_peered":"2024-03-18T14:51:01.358312+0000","last_unstale":"2024-03-18T14:52:36.370814+0000","last_undegraded":"2024-03-18T14:52:36.370814+0000","last_fullsized":"2024-03-18T14:52:36.370814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:40.150240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.367332+0000","last_change":"2024-03-18T14:51:01.360082+0000","last_active":"2024-03-18T14:52:36.367332+0000","last_peered":"2024-03-18T14:52:36.367332+0000","last_clean":"2024-03-18T14:52:36.367332+0000","last_became_active":"2024-03-18T14:51:01.359769+0000","last_became_peered":"2024-03-18T14:51:01.359769+0000","last_unstale":"2024-03-18T14:52:36.367332+0000","last_undegraded":"2024-03-18T14:52:36.367332+0000","last_fullsized":"2024-03-18T14:52:36.367332+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:53:27.081546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.370348+0000","last_change":"2024-03-18T14:51:01.363947+0000","last_active":"2024-03-18T14:52:36.370348+0000","last_peered":"2024-03-18T14:52:36.370348+0000","last_clean":"2024-03-18T14:52:36.370348+0000","last_became_active":"2024-03-18T14:51:01.363106+0000","last_became_peered":"2024-03-18T14:51:01.363106+0000","last_unstale":"2024-03-18T14:52:36.370348+0000","last_undegraded":"2024-03-18T14:52:36.370348+0000","last_fullsized":"2024-03-18T14:52:36.370348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:24:50.478426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.363104+0000","last_change":"2024-03-18T14:51:01.357950+0000","last_active":"2024-03-18T14:52:36.363104+0000","last_peered":"2024-03-18T14:52:36.363104+0000","last_clean":"2024-03-18T14:52:36.363104+0000","last_became_active":"2024-03-18T14:51:01.357651+0000","last_became_peered":"2024-03-18T14:51:01.357651+0000","last_unstale":"2024-03-18T14:52:36.363104+0000","last_undegraded":"2024-03-18T14:52:36.363104+0000","last_fullsized":"2024-03-18T14:52:36.363104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:49:01.376008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375514+0000","last_change":"2024-03-18T14:51:01.360069+0000","last_active":"2024-03-18T14:52:36.375514+0000","last_peered":"2024-03-18T14:52:36.375514+0000","last_clean":"2024-03-18T14:52:36.375514+0000","last_became_active":"2024-03-18T14:51:01.358960+0000","last_became_peered":"2024-03-18T14:51:01.358960+0000","last_unstale":"2024-03-18T14:52:36.375514+0000","last_undegraded":"2024-03-18T14:52:36.375514+0000","last_fullsized":"2024-03-18T14:52:36.375514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:48:42.197329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.364663+0000","last_change":"2024-03-18T14:51:01.361200+0000","last_active":"2024-03-18T14:52:36.364663+0000","last_peered":"2024-03-18T14:52:36.364663+0000","last_clean":"2024-03-18T14:52:36.364663+0000","last_became_active":"2024-03-18T14:51:01.360879+0000","last_became_peered":"2024-03-18T14:51:01.360879+0000","last_unstale":"2024-03-18T14:52:36.364663+0000","last_undegraded":"2024-03-18T14:52:36.364663+0000","last_fullsized":"2024-03-18T14:52:36.364663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:05.335627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.364978+0000","last_change":"2024-03-18T14:51:01.352441+0000","last_active":"2024-03-18T14:52:36.364978+0000","last_peered":"2024-03-18T14:52:36.364978+0000","last_clean":"2024-03-18T14:52:36.364978+0000","last_became_active":"2024-03-18T14:51:01.352118+0000","last_became_peered":"2024-03-18T14:51:01.352118+0000","last_unstale":"2024-03-18T14:52:36.364978+0000","last_undegraded":"2024-03-18T14:52:36.364978+0000","last_fullsized":"2024-03-18T14:52:36.364978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:45:42.781439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.364844+0000","last_change":"2024-03-18T14:51:01.361257+0000","last_active":"2024-03-18T14:52:36.364844+0000","last_peered":"2024-03-18T14:52:36.364844+0000","last_clean":"2024-03-18T14:52:36.364844+0000","last_became_active":"2024-03-18T14:51:01.360898+0000","last_became_peered":"2024-03-18T14:51:01.360898+0000","last_unstale":"2024-03-18T14:52:36.364844+0000","last_undegraded":"2024-03-18T14:52:36.364844+0000","last_fullsized":"2024-03-18T14:52:36.364844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:38:26.200678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.367432+0000","last_change":"2024-03-18T14:52:07.374822+0000","last_active":"2024-03-18T14:52:36.367432+0000","last_peered":"2024-03-18T14:52:36.367432+0000","last_clean":"2024-03-18T14:52:36.367432+0000","last_became_active":"2024-03-18T14:52:07.374537+0000","last_became_peered":"2024-03-18T14:52:07.374537+0000","last_unstale":"2024-03-18T14:52:36.367432+0000","last_undegraded":"2024-03-18T14:52:36.367432+0000","last_fullsized":"2024-03-18T14:52:36.367432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:57:29.971366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.374959+0000","last_change":"2024-03-18T14:51:01.357223+0000","last_active":"2024-03-18T14:52:36.374959+0000","last_peered":"2024-03-18T14:52:36.374959+0000","last_clean":"2024-03-18T14:52:36.374959+0000","last_became_active":"2024-03-18T14:51:01.356934+0000","last_became_peered":"2024-03-18T14:51:01.356934+0000","last_unstale":"2024-03-18T14:52:36.374959+0000","last_undegraded":"2024-03-18T14:52:36.374959+0000","last_fullsized":"2024-03-18T14:52:36.374959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:28.683605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.372986+0000","last_change":"2024-03-18T14:51:01.366414+0000","last_active":"2024-03-18T14:52:36.372986+0000","last_peered":"2024-03-18T14:52:36.372986+0000","last_clean":"2024-03-18T14:52:36.372986+0000","last_became_active":"2024-03-18T14:51:01.365534+0000","last_became_peered":"2024-03-18T14:51:01.365534+0000","last_unstale":"2024-03-18T14:52:36.372986+0000","last_undegraded":"2024-03-18T14:52:36.372986+0000","last_fullsized":"2024-03-18T14:52:36.372986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:28:51.568662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.368646+0000","last_change":"2024-03-18T14:51:01.354084+0000","last_active":"2024-03-18T14:52:36.368646+0000","last_peered":"2024-03-18T14:52:36.368646+0000","last_clean":"2024-03-18T14:52:36.368646+0000","last_became_active":"2024-03-18T14:51:01.353599+0000","last_became_peered":"2024-03-18T14:51:01.353599+0000","last_unstale":"2024-03-18T14:52:36.368646+0000","last_undegraded":"2024-03-18T14:52:36.368646+0000","last_fullsized":"2024-03-18T14:52:36.368646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:40.502783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.366302+0000","last_change":"2024-03-18T14:51:01.353391+0000","last_active":"2024-03-18T14:52:36.366302+0000","last_peered":"2024-03-18T14:52:36.366302+0000","last_clean":"2024-03-18T14:52:36.366302+0000","last_became_active":"2024-03-18T14:51:01.353084+0000","last_became_peered":"2024-03-18T14:51:01.353084+0000","last_unstale":"2024-03-18T14:52:36.366302+0000","last_undegraded":"2024-03-18T14:52:36.366302+0000","last_fullsized":"2024-03-18T14:52:36.366302+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:18:02.774589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.363884+0000","last_change":"2024-03-18T14:51:00.942617+0000","last_active":"2024-03-18T14:52:07.363884+0000","last_peered":"2024-03-18T14:52:07.363884+0000","last_clean":"2024-03-18T14:52:07.363884+0000","last_became_active":"2024-03-18T14:51:00.942276+0000","last_became_peered":"2024-03-18T14:51:00.942276+0000","last_unstale":"2024-03-18T14:52:07.363884+0000","last_undegraded":"2024-03-18T14:52:07.363884+0000","last_fullsized":"2024-03-18T14:52:07.363884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:03:05.984066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.366394+0000","last_change":"2024-03-18T14:51:01.363887+0000","last_active":"2024-03-18T14:52:36.366394+0000","last_peered":"2024-03-18T14:52:36.366394+0000","last_clean":"2024-03-18T14:52:36.366394+0000","last_became_active":"2024-03-18T14:51:01.361359+0000","last_became_peered":"2024-03-18T14:51:01.361359+0000","last_unstale":"2024-03-18T14:52:36.366394+0000","last_undegraded":"2024-03-18T14:52:36.366394+0000","last_fullsized":"2024-03-18T14:52:36.366394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:22.996930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368943+0000","last_change":"2024-03-18T14:51:00.940392+0000","last_active":"2024-03-18T14:52:07.368943+0000","last_peered":"2024-03-18T14:52:07.368943+0000","last_clean":"2024-03-18T14:52:07.368943+0000","last_became_active":"2024-03-18T14:51:00.940055+0000","last_became_peered":"2024-03-18T14:51:00.940055+0000","last_unstale":"2024-03-18T14:52:07.368943+0000","last_undegraded":"2024-03-18T14:52:07.368943+0000","last_fullsized":"2024-03-18T14:52:07.368943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:57:28.523549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375190+0000","last_change":"2024-03-18T14:51:01.361458+0000","last_active":"2024-03-18T14:52:36.375190+0000","last_peered":"2024-03-18T14:52:36.375190+0000","last_clean":"2024-03-18T14:52:36.375190+0000","last_became_active":"2024-03-18T14:51:01.361105+0000","last_became_peered":"2024-03-18T14:51:01.361105+0000","last_unstale":"2024-03-18T14:52:36.375190+0000","last_undegraded":"2024-03-18T14:52:36.375190+0000","last_fullsized":"2024-03-18T14:52:36.375190+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:22:59.410080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.379071+0000","last_change":"2024-03-18T14:52:07.378642+0000","last_active":"2024-03-18T14:52:07.379071+0000","last_peered":"2024-03-18T14:52:07.379071+0000","last_clean":"2024-03-18T14:52:07.379071+0000","last_became_active":"2024-03-18T14:52:07.371947+0000","last_became_peered":"2024-03-18T14:52:07.371947+0000","last_unstale":"2024-03-18T14:52:07.379071+0000","last_undegraded":"2024-03-18T14:52:07.379071+0000","last_fullsized":"2024-03-18T14:52:07.379071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:46:00.898462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.371790+0000","last_change":"2024-03-18T14:51:01.366305+0000","last_active":"2024-03-18T14:52:36.371790+0000","last_peered":"2024-03-18T14:52:36.371790+0000","last_clean":"2024-03-18T14:52:36.371790+0000","last_became_active":"2024-03-18T14:51:01.364626+0000","last_became_peered":"2024-03-18T14:51:01.364626+0000","last_unstale":"2024-03-18T14:52:36.371790+0000","last_undegraded":"2024-03-18T14:52:36.371790+0000","last_fullsized":"2024-03-18T14:52:36.371790+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:06.212030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370835+0000","last_change":"2024-03-18T14:51:00.932514+0000","last_active":"2024-03-18T14:52:07.370835+0000","last_peered":"2024-03-18T14:52:07.370835+0000","last_clean":"2024-03-18T14:52:07.370835+0000","last_became_active":"2024-03-18T14:51:00.932183+0000","last_became_peered":"2024-03-18T14:51:00.932183+0000","last_unstale":"2024-03-18T14:52:07.370835+0000","last_undegraded":"2024-03-18T14:52:07.370835+0000","last_fullsized":"2024-03-18T14:52:07.370835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:36:37.230185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.372279+0000","last_change":"2024-03-18T14:51:01.358939+0000","last_active":"2024-03-18T14:52:36.372279+0000","last_peered":"2024-03-18T14:52:36.372279+0000","last_clean":"2024-03-18T14:52:36.372279+0000","last_became_active":"2024-03-18T14:51:01.358601+0000","last_became_peered":"2024-03-18T14:51:01.358601+0000","last_unstale":"2024-03-18T14:52:36.372279+0000","last_undegraded":"2024-03-18T14:52:36.372279+0000","last_fullsized":"2024-03-18T14:52:36.372279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:40:14.876143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":145,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:53:04.156527+0000","last_change":"2024-03-18T14:51:00.945116+0000","last_active":"2024-03-18T14:53:04.156527+0000","last_peered":"2024-03-18T14:53:04.156527+0000","last_clean":"2024-03-18T14:53:04.156527+0000","last_became_active":"2024-03-18T14:51:00.944791+0000","last_became_peered":"2024-03-18T14:51:00.944791+0000","last_unstale":"2024-03-18T14:53:04.156527+0000","last_undegraded":"2024-03-18T14:53:04.156527+0000","last_fullsized":"2024-03-18T14:53:04.156527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:06.637940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.357355+0000","last_change":"2024-03-18T14:51:01.358682+0000","last_active":"2024-03-18T14:52:36.357355+0000","last_peered":"2024-03-18T14:52:36.357355+0000","last_clean":"2024-03-18T14:52:36.357355+0000","last_became_active":"2024-03-18T14:51:01.358378+0000","last_became_peered":"2024-03-18T14:51:01.358378+0000","last_unstale":"2024-03-18T14:52:36.357355+0000","last_undegraded":"2024-03-18T14:52:36.357355+0000","last_fullsized":"2024-03-18T14:52:36.357355+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:42.697944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369158+0000","last_change":"2024-03-18T14:51:00.943976+0000","last_active":"2024-03-18T14:52:07.369158+0000","last_peered":"2024-03-18T14:52:07.369158+0000","last_clean":"2024-03-18T14:52:07.369158+0000","last_became_active":"2024-03-18T14:51:00.943614+0000","last_became_peered":"2024-03-18T14:51:00.943614+0000","last_unstale":"2024-03-18T14:52:07.369158+0000","last_undegraded":"2024-03-18T14:52:07.369158+0000","last_fullsized":"2024-03-18T14:52:07.369158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:07:00.744708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.365708+0000","last_change":"2024-03-18T14:51:01.365316+0000","last_active":"2024-03-18T14:52:36.365708+0000","last_peered":"2024-03-18T14:52:36.365708+0000","last_clean":"2024-03-18T14:52:36.365708+0000","last_became_active":"2024-03-18T14:51:01.365019+0000","last_became_peered":"2024-03-18T14:51:01.365019+0000","last_unstale":"2024-03-18T14:52:36.365708+0000","last_undegraded":"2024-03-18T14:52:36.365708+0000","last_fullsized":"2024-03-18T14:52:36.365708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:39:59.040907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481239+0000","last_change":"2024-03-18T14:51:00.936923+0000","last_active":"2024-03-18T14:52:07.481239+0000","last_peered":"2024-03-18T14:52:07.481239+0000","last_clean":"2024-03-18T14:52:07.481239+0000","last_became_active":"2024-03-18T14:51:00.936591+0000","last_became_peered":"2024-03-18T14:51:00.936591+0000","last_unstale":"2024-03-18T14:52:07.481239+0000","last_undegraded":"2024-03-18T14:52:07.481239+0000","last_fullsized":"2024-03-18T14:52:07.481239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:48:46.423033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.378104+0000","last_change":"2024-03-18T14:51:01.356827+0000","last_active":"2024-03-18T14:52:36.378104+0000","last_peered":"2024-03-18T14:52:36.378104+0000","last_clean":"2024-03-18T14:52:36.378104+0000","last_became_active":"2024-03-18T14:51:01.356148+0000","last_became_peered":"2024-03-18T14:51:01.356148+0000","last_unstale":"2024-03-18T14:52:36.378104+0000","last_undegraded":"2024-03-18T14:52:36.378104+0000","last_fullsized":"2024-03-18T14:52:36.378104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:18.477890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423674+0000","last_change":"2024-03-18T14:51:00.934024+0000","last_active":"2024-03-18T14:52:07.423674+0000","last_peered":"2024-03-18T14:52:07.423674+0000","last_clean":"2024-03-18T14:52:07.423674+0000","last_became_active":"2024-03-18T14:51:00.933739+0000","last_became_peered":"2024-03-18T14:51:00.933739+0000","last_unstale":"2024-03-18T14:52:07.423674+0000","last_undegraded":"2024-03-18T14:52:07.423674+0000","last_fullsized":"2024-03-18T14:52:07.423674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:03.967260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.362587+0000","last_change":"2024-03-18T14:51:01.356519+0000","last_active":"2024-03-18T14:52:36.362587+0000","last_peered":"2024-03-18T14:52:36.362587+0000","last_clean":"2024-03-18T14:52:36.362587+0000","last_became_active":"2024-03-18T14:51:01.356193+0000","last_became_peered":"2024-03-18T14:51:01.356193+0000","last_unstale":"2024-03-18T14:52:36.362587+0000","last_undegraded":"2024-03-18T14:52:36.362587+0000","last_fullsized":"2024-03-18T14:52:36.362587+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:34:59.633241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:53:01.153237+0000","last_change":"2024-03-18T14:51:00.940189+0000","last_active":"2024-03-18T14:53:01.153237+0000","last_peered":"2024-03-18T14:53:01.153237+0000","last_clean":"2024-03-18T14:53:01.153237+0000","last_became_active":"2024-03-18T14:51:00.939820+0000","last_became_peered":"2024-03-18T14:51:00.939820+0000","last_unstale":"2024-03-18T14:53:01.153237+0000","last_undegraded":"2024-03-18T14:53:01.153237+0000","last_fullsized":"2024-03-18T14:53:01.153237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:44:09.749747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.371264+0000","last_change":"2024-03-18T14:51:01.355756+0000","last_active":"2024-03-18T14:52:36.371264+0000","last_peered":"2024-03-18T14:52:36.371264+0000","last_clean":"2024-03-18T14:52:36.371264+0000","last_became_active":"2024-03-18T14:51:01.352871+0000","last_became_peered":"2024-03-18T14:51:01.352871+0000","last_unstale":"2024-03-18T14:52:36.371264+0000","last_undegraded":"2024-03-18T14:52:36.371264+0000","last_fullsized":"2024-03-18T14:52:36.371264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:20:26.533066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366850+0000","last_change":"2024-03-18T14:51:00.941908+0000","last_active":"2024-03-18T14:52:07.366850+0000","last_peered":"2024-03-18T14:52:07.366850+0000","last_clean":"2024-03-18T14:52:07.366850+0000","last_became_active":"2024-03-18T14:51:00.940978+0000","last_became_peered":"2024-03-18T14:51:00.940978+0000","last_unstale":"2024-03-18T14:52:07.366850+0000","last_undegraded":"2024-03-18T14:52:07.366850+0000","last_fullsized":"2024-03-18T14:52:07.366850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:52.705243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.366882+0000","last_change":"2024-03-18T14:51:01.363462+0000","last_active":"2024-03-18T14:52:36.366882+0000","last_peered":"2024-03-18T14:52:36.366882+0000","last_clean":"2024-03-18T14:52:36.366882+0000","last_became_active":"2024-03-18T14:51:01.362262+0000","last_became_peered":"2024-03-18T14:51:01.362262+0000","last_unstale":"2024-03-18T14:52:36.366882+0000","last_undegraded":"2024-03-18T14:52:36.366882+0000","last_fullsized":"2024-03-18T14:52:36.366882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:07:18.952876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.424170+0000","last_change":"2024-03-18T14:51:00.936050+0000","last_active":"2024-03-18T14:52:07.424170+0000","last_peered":"2024-03-18T14:52:07.424170+0000","last_clean":"2024-03-18T14:52:07.424170+0000","last_became_active":"2024-03-18T14:51:00.935676+0000","last_became_peered":"2024-03-18T14:51:00.935676+0000","last_unstale":"2024-03-18T14:52:07.424170+0000","last_undegraded":"2024-03-18T14:52:07.424170+0000","last_fullsized":"2024-03-18T14:52:07.424170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:29.423110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.374032+0000","last_change":"2024-03-18T14:51:01.364295+0000","last_active":"2024-03-18T14:52:36.374032+0000","last_peered":"2024-03-18T14:52:36.374032+0000","last_clean":"2024-03-18T14:52:36.374032+0000","last_became_active":"2024-03-18T14:51:01.363916+0000","last_became_peered":"2024-03-18T14:51:01.363916+0000","last_unstale":"2024-03-18T14:52:36.374032+0000","last_undegraded":"2024-03-18T14:52:36.374032+0000","last_fullsized":"2024-03-18T14:52:36.374032+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:21:17.905819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'31","reported_seq":305,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:53:02.055197+0000","last_change":"2024-03-18T14:52:07.427262+0000","last_active":"2024-03-18T14:53:02.055197+0000","last_peered":"2024-03-18T14:53:02.055197+0000","last_clean":"2024-03-18T14:53:02.055197+0000","last_became_active":"2024-03-18T14:52:07.421388+0000","last_became_peered":"2024-03-18T14:52:07.421388+0000","last_unstale":"2024-03-18T14:53:02.055197+0000","last_undegraded":"2024-03-18T14:53:02.055197+0000","last_fullsized":"2024-03-18T14:53:02.055197+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:28.159787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59469,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":233,"num_read_kb":174,"num_write":32,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":58912,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.376345+0000","last_change":"2024-03-18T14:51:01.362426+0000","last_active":"2024-03-18T14:52:36.376345+0000","last_peered":"2024-03-18T14:52:36.376345+0000","last_clean":"2024-03-18T14:52:36.376345+0000","last_became_active":"2024-03-18T14:51:01.362130+0000","last_became_peered":"2024-03-18T14:51:01.362130+0000","last_unstale":"2024-03-18T14:52:36.376345+0000","last_undegraded":"2024-03-18T14:52:36.376345+0000","last_fullsized":"2024-03-18T14:52:36.376345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:14.545997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368774+0000","last_change":"2024-03-18T14:51:00.934028+0000","last_active":"2024-03-18T14:52:07.368774+0000","last_peered":"2024-03-18T14:52:07.368774+0000","last_clean":"2024-03-18T14:52:07.368774+0000","last_became_active":"2024-03-18T14:51:00.933594+0000","last_became_peered":"2024-03-18T14:51:00.933594+0000","last_unstale":"2024-03-18T14:52:07.368774+0000","last_undegraded":"2024-03-18T14:52:07.368774+0000","last_fullsized":"2024-03-18T14:52:07.368774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:55:08.675533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.374723+0000","last_change":"2024-03-18T14:51:01.353544+0000","last_active":"2024-03-18T14:52:36.374723+0000","last_peered":"2024-03-18T14:52:36.374723+0000","last_clean":"2024-03-18T14:52:36.374723+0000","last_became_active":"2024-03-18T14:51:01.353241+0000","last_became_peered":"2024-03-18T14:51:01.353241+0000","last_unstale":"2024-03-18T14:52:36.374723+0000","last_undegraded":"2024-03-18T14:52:36.374723+0000","last_fullsized":"2024-03-18T14:52:36.374723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:45:27.617681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423792+0000","last_change":"2024-03-18T14:51:00.944492+0000","last_active":"2024-03-18T14:52:07.423792+0000","last_peered":"2024-03-18T14:52:07.423792+0000","last_clean":"2024-03-18T14:52:07.423792+0000","last_became_active":"2024-03-18T14:51:00.944165+0000","last_became_peered":"2024-03-18T14:51:00.944165+0000","last_unstale":"2024-03-18T14:52:07.423792+0000","last_undegraded":"2024-03-18T14:52:07.423792+0000","last_fullsized":"2024-03-18T14:52:07.423792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:47:05.968966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.371403+0000","last_change":"2024-03-18T14:51:01.356782+0000","last_active":"2024-03-18T14:52:36.371403+0000","last_peered":"2024-03-18T14:52:36.371403+0000","last_clean":"2024-03-18T14:52:36.371403+0000","last_became_active":"2024-03-18T14:51:01.356242+0000","last_became_peered":"2024-03-18T14:51:01.356242+0000","last_unstale":"2024-03-18T14:52:36.371403+0000","last_undegraded":"2024-03-18T14:52:36.371403+0000","last_fullsized":"2024-03-18T14:52:36.371403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:56:07.662545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'6","reported_seq":149,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:53:04.168246+0000","last_change":"2024-03-18T14:51:00.937873+0000","last_active":"2024-03-18T14:53:04.168246+0000","last_peered":"2024-03-18T14:53:04.168246+0000","last_clean":"2024-03-18T14:53:04.168246+0000","last_became_active":"2024-03-18T14:51:00.937562+0000","last_became_peered":"2024-03-18T14:51:00.937562+0000","last_unstale":"2024-03-18T14:53:04.168246+0000","last_undegraded":"2024-03-18T14:53:04.168246+0000","last_fullsized":"2024-03-18T14:53:04.168246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:36:44.342414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1428,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375613+0000","last_change":"2024-03-18T14:51:01.358681+0000","last_active":"2024-03-18T14:52:36.375613+0000","last_peered":"2024-03-18T14:52:36.375613+0000","last_clean":"2024-03-18T14:52:36.375613+0000","last_became_active":"2024-03-18T14:51:01.358374+0000","last_became_peered":"2024-03-18T14:51:01.358374+0000","last_unstale":"2024-03-18T14:52:36.375613+0000","last_undegraded":"2024-03-18T14:52:36.375613+0000","last_fullsized":"2024-03-18T14:52:36.375613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:43:47.804271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370451+0000","last_change":"2024-03-18T14:51:00.935639+0000","last_active":"2024-03-18T14:52:07.370451+0000","last_peered":"2024-03-18T14:52:07.370451+0000","last_clean":"2024-03-18T14:52:07.370451+0000","last_became_active":"2024-03-18T14:51:00.935334+0000","last_became_peered":"2024-03-18T14:51:00.935334+0000","last_unstale":"2024-03-18T14:52:07.370451+0000","last_undegraded":"2024-03-18T14:52:07.370451+0000","last_fullsized":"2024-03-18T14:52:07.370451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:46.966812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.370423+0000","last_change":"2024-03-18T14:51:01.364755+0000","last_active":"2024-03-18T14:52:36.370423+0000","last_peered":"2024-03-18T14:52:36.370423+0000","last_clean":"2024-03-18T14:52:36.370423+0000","last_became_active":"2024-03-18T14:51:01.364448+0000","last_became_peered":"2024-03-18T14:51:01.364448+0000","last_unstale":"2024-03-18T14:52:36.370423+0000","last_undegraded":"2024-03-18T14:52:36.370423+0000","last_fullsized":"2024-03-18T14:52:36.370423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:37:49.480787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.371021+0000","last_change":"2024-03-18T14:51:00.933271+0000","last_active":"2024-03-18T14:52:07.371021+0000","last_peered":"2024-03-18T14:52:07.371021+0000","last_clean":"2024-03-18T14:52:07.371021+0000","last_became_active":"2024-03-18T14:51:00.932928+0000","last_became_peered":"2024-03-18T14:51:00.932928+0000","last_unstale":"2024-03-18T14:52:07.371021+0000","last_undegraded":"2024-03-18T14:52:07.371021+0000","last_fullsized":"2024-03-18T14:52:07.371021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:11:55.281763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375943+0000","last_change":"2024-03-18T14:51:01.362911+0000","last_active":"2024-03-18T14:52:36.375943+0000","last_peered":"2024-03-18T14:52:36.375943+0000","last_clean":"2024-03-18T14:52:36.375943+0000","last_became_active":"2024-03-18T14:51:01.362598+0000","last_became_peered":"2024-03-18T14:51:01.362598+0000","last_unstale":"2024-03-18T14:52:36.375943+0000","last_undegraded":"2024-03-18T14:52:36.375943+0000","last_fullsized":"2024-03-18T14:52:36.375943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:22:58.627753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.363811+0000","last_change":"2024-03-18T14:51:00.935554+0000","last_active":"2024-03-18T14:52:07.363811+0000","last_peered":"2024-03-18T14:52:07.363811+0000","last_clean":"2024-03-18T14:52:07.363811+0000","last_became_active":"2024-03-18T14:51:00.935184+0000","last_became_peered":"2024-03-18T14:51:00.935184+0000","last_unstale":"2024-03-18T14:52:07.363811+0000","last_undegraded":"2024-03-18T14:52:07.363811+0000","last_fullsized":"2024-03-18T14:52:07.363811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:21:13.973570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.361967+0000","last_change":"2024-03-18T14:51:01.355990+0000","last_active":"2024-03-18T14:52:36.361967+0000","last_peered":"2024-03-18T14:52:36.361967+0000","last_clean":"2024-03-18T14:52:36.361967+0000","last_became_active":"2024-03-18T14:51:01.355692+0000","last_became_peered":"2024-03-18T14:51:01.355692+0000","last_unstale":"2024-03-18T14:52:36.361967+0000","last_undegraded":"2024-03-18T14:52:36.361967+0000","last_fullsized":"2024-03-18T14:52:36.361967+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:27:19.742684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366919+0000","last_change":"2024-03-18T14:51:00.943947+0000","last_active":"2024-03-18T14:52:07.366919+0000","last_peered":"2024-03-18T14:52:07.366919+0000","last_clean":"2024-03-18T14:52:07.366919+0000","last_became_active":"2024-03-18T14:51:00.943565+0000","last_became_peered":"2024-03-18T14:51:00.943565+0000","last_unstale":"2024-03-18T14:52:07.366919+0000","last_undegraded":"2024-03-18T14:52:07.366919+0000","last_fullsized":"2024-03-18T14:52:07.366919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:33:06.103148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.367260+0000","last_change":"2024-03-18T14:51:01.363221+0000","last_active":"2024-03-18T14:52:36.367260+0000","last_peered":"2024-03-18T14:52:36.367260+0000","last_clean":"2024-03-18T14:52:36.367260+0000","last_became_active":"2024-03-18T14:51:01.362848+0000","last_became_peered":"2024-03-18T14:51:01.362848+0000","last_unstale":"2024-03-18T14:52:36.367260+0000","last_undegraded":"2024-03-18T14:52:36.367260+0000","last_fullsized":"2024-03-18T14:52:36.367260+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:04.139365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.365329+0000","last_change":"2024-03-18T14:51:01.356237+0000","last_active":"2024-03-18T14:52:36.365329+0000","last_peered":"2024-03-18T14:52:36.365329+0000","last_clean":"2024-03-18T14:52:36.365329+0000","last_became_active":"2024-03-18T14:51:01.355922+0000","last_became_peered":"2024-03-18T14:51:01.355922+0000","last_unstale":"2024-03-18T14:52:36.365329+0000","last_undegraded":"2024-03-18T14:52:36.365329+0000","last_fullsized":"2024-03-18T14:52:36.365329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:04:50.491752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'6","reported_seq":145,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:53:05.168942+0000","last_change":"2024-03-18T14:51:00.936058+0000","last_active":"2024-03-18T14:53:05.168942+0000","last_peered":"2024-03-18T14:53:05.168942+0000","last_clean":"2024-03-18T14:53:05.168942+0000","last_became_active":"2024-03-18T14:51:00.935731+0000","last_became_peered":"2024-03-18T14:51:00.935731+0000","last_unstale":"2024-03-18T14:53:05.168942+0000","last_undegraded":"2024-03-18T14:53:05.168942+0000","last_fullsized":"2024-03-18T14:53:05.168942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:47:49.023943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.374426+0000","last_change":"2024-03-18T14:51:01.359742+0000","last_active":"2024-03-18T14:52:36.374426+0000","last_peered":"2024-03-18T14:52:36.374426+0000","last_clean":"2024-03-18T14:52:36.374426+0000","last_became_active":"2024-03-18T14:51:01.358507+0000","last_became_peered":"2024-03-18T14:51:01.358507+0000","last_unstale":"2024-03-18T14:52:36.374426+0000","last_undegraded":"2024-03-18T14:52:36.374426+0000","last_fullsized":"2024-03-18T14:52:36.374426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:17.129695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.419738+0000","last_change":"2024-03-18T14:51:00.934593+0000","last_active":"2024-03-18T14:52:07.419738+0000","last_peered":"2024-03-18T14:52:07.419738+0000","last_clean":"2024-03-18T14:52:07.419738+0000","last_became_active":"2024-03-18T14:51:00.934253+0000","last_became_peered":"2024-03-18T14:51:00.934253+0000","last_unstale":"2024-03-18T14:52:07.419738+0000","last_undegraded":"2024-03-18T14:52:07.419738+0000","last_fullsized":"2024-03-18T14:52:07.419738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:43:26.323162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":352,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366664+0000","last_change":"2024-03-18T14:49:51.158965+0000","last_active":"2024-03-18T14:52:07.366664+0000","last_peered":"2024-03-18T14:52:07.366664+0000","last_clean":"2024-03-18T14:52:07.366664+0000","last_became_active":"2024-03-18T14:49:51.158580+0000","last_became_peered":"2024-03-18T14:49:51.158580+0000","last_unstale":"2024-03-18T14:52:07.366664+0000","last_undegraded":"2024-03-18T14:52:07.366664+0000","last_fullsized":"2024-03-18T14:52:07.366664+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:46:27.210244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:46:27.210244+0000","last_clean_scrub_stamp":"2024-03-18T14:46:27.210244+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:37:08.104617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.359433+0000","last_change":"2024-03-18T14:51:01.359972+0000","last_active":"2024-03-18T14:52:36.359433+0000","last_peered":"2024-03-18T14:52:36.359433+0000","last_clean":"2024-03-18T14:52:36.359433+0000","last_became_active":"2024-03-18T14:51:01.359430+0000","last_became_peered":"2024-03-18T14:51:01.359430+0000","last_unstale":"2024-03-18T14:52:36.359433+0000","last_undegraded":"2024-03-18T14:52:36.359433+0000","last_fullsized":"2024-03-18T14:52:36.359433+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:00:14.099499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366772+0000","last_change":"2024-03-18T14:51:00.934196+0000","last_active":"2024-03-18T14:52:07.366772+0000","last_peered":"2024-03-18T14:52:07.366772+0000","last_clean":"2024-03-18T14:52:07.366772+0000","last_became_active":"2024-03-18T14:51:00.933875+0000","last_became_peered":"2024-03-18T14:51:00.933875+0000","last_unstale":"2024-03-18T14:52:07.366772+0000","last_undegraded":"2024-03-18T14:52:07.366772+0000","last_fullsized":"2024-03-18T14:52:07.366772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:56.797227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370852+0000","last_change":"2024-03-18T14:51:00.941862+0000","last_active":"2024-03-18T14:52:07.370852+0000","last_peered":"2024-03-18T14:52:07.370852+0000","last_clean":"2024-03-18T14:52:07.370852+0000","last_became_active":"2024-03-18T14:51:00.941536+0000","last_became_peered":"2024-03-18T14:51:00.941536+0000","last_unstale":"2024-03-18T14:52:07.370852+0000","last_undegraded":"2024-03-18T14:52:07.370852+0000","last_fullsized":"2024-03-18T14:52:07.370852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:56.364552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.359217+0000","last_change":"2024-03-18T14:51:01.352404+0000","last_active":"2024-03-18T14:52:36.359217+0000","last_peered":"2024-03-18T14:52:36.359217+0000","last_clean":"2024-03-18T14:52:36.359217+0000","last_became_active":"2024-03-18T14:51:01.352077+0000","last_became_peered":"2024-03-18T14:51:01.352077+0000","last_unstale":"2024-03-18T14:52:36.359217+0000","last_undegraded":"2024-03-18T14:52:36.359217+0000","last_fullsized":"2024-03-18T14:52:36.359217+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:30:38.050712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.364118+0000","last_change":"2024-03-18T14:51:00.935564+0000","last_active":"2024-03-18T14:52:07.364118+0000","last_peered":"2024-03-18T14:52:07.364118+0000","last_clean":"2024-03-18T14:52:07.364118+0000","last_became_active":"2024-03-18T14:51:00.935268+0000","last_became_peered":"2024-03-18T14:51:00.935268+0000","last_unstale":"2024-03-18T14:52:07.364118+0000","last_undegraded":"2024-03-18T14:52:07.364118+0000","last_fullsized":"2024-03-18T14:52:07.364118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:22:33.504184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.372551+0000","last_change":"2024-03-18T14:51:01.356096+0000","last_active":"2024-03-18T14:52:36.372551+0000","last_peered":"2024-03-18T14:52:36.372551+0000","last_clean":"2024-03-18T14:52:36.372551+0000","last_became_active":"2024-03-18T14:51:01.355688+0000","last_became_peered":"2024-03-18T14:51:01.355688+0000","last_unstale":"2024-03-18T14:52:36.372551+0000","last_undegraded":"2024-03-18T14:52:36.372551+0000","last_fullsized":"2024-03-18T14:52:36.372551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:04:38.678482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.419726+0000","last_change":"2024-03-18T14:51:00.942610+0000","last_active":"2024-03-18T14:52:07.419726+0000","last_peered":"2024-03-18T14:52:07.419726+0000","last_clean":"2024-03-18T14:52:07.419726+0000","last_became_active":"2024-03-18T14:51:00.942203+0000","last_became_peered":"2024-03-18T14:51:00.942203+0000","last_unstale":"2024-03-18T14:52:07.419726+0000","last_undegraded":"2024-03-18T14:52:07.419726+0000","last_fullsized":"2024-03-18T14:52:07.419726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:17:26.324282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.367722+0000","last_change":"2024-03-18T14:51:01.362931+0000","last_active":"2024-03-18T14:52:36.367722+0000","last_peered":"2024-03-18T14:52:36.367722+0000","last_clean":"2024-03-18T14:52:36.367722+0000","last_became_active":"2024-03-18T14:51:01.362624+0000","last_became_peered":"2024-03-18T14:51:01.362624+0000","last_unstale":"2024-03-18T14:52:36.367722+0000","last_undegraded":"2024-03-18T14:52:36.367722+0000","last_fullsized":"2024-03-18T14:52:36.367722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:40:52.512871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370722+0000","last_change":"2024-03-18T14:51:00.938971+0000","last_active":"2024-03-18T14:52:07.370722+0000","last_peered":"2024-03-18T14:52:07.370722+0000","last_clean":"2024-03-18T14:52:07.370722+0000","last_became_active":"2024-03-18T14:51:00.938645+0000","last_became_peered":"2024-03-18T14:51:00.938645+0000","last_unstale":"2024-03-18T14:52:07.370722+0000","last_undegraded":"2024-03-18T14:52:07.370722+0000","last_fullsized":"2024-03-18T14:52:07.370722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:18:54.999927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.334973+0000","last_change":"2024-03-18T14:51:01.359510+0000","last_active":"2024-03-18T14:52:36.334973+0000","last_peered":"2024-03-18T14:52:36.334973+0000","last_clean":"2024-03-18T14:52:36.334973+0000","last_became_active":"2024-03-18T14:51:01.359192+0000","last_became_peered":"2024-03-18T14:51:01.359192+0000","last_unstale":"2024-03-18T14:52:36.334973+0000","last_undegraded":"2024-03-18T14:52:36.334973+0000","last_fullsized":"2024-03-18T14:52:36.334973+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:11:52.942825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368576+0000","last_change":"2024-03-18T14:51:00.940194+0000","last_active":"2024-03-18T14:52:07.368576+0000","last_peered":"2024-03-18T14:52:07.368576+0000","last_clean":"2024-03-18T14:52:07.368576+0000","last_became_active":"2024-03-18T14:51:00.939861+0000","last_became_peered":"2024-03-18T14:51:00.939861+0000","last_unstale":"2024-03-18T14:52:07.368576+0000","last_undegraded":"2024-03-18T14:52:07.368576+0000","last_fullsized":"2024-03-18T14:52:07.368576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:06:30.314102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.367604+0000","last_change":"2024-03-18T14:51:01.355098+0000","last_active":"2024-03-18T14:52:36.367604+0000","last_peered":"2024-03-18T14:52:36.367604+0000","last_clean":"2024-03-18T14:52:36.367604+0000","last_became_active":"2024-03-18T14:51:01.354452+0000","last_became_peered":"2024-03-18T14:51:01.354452+0000","last_unstale":"2024-03-18T14:52:36.367604+0000","last_undegraded":"2024-03-18T14:52:36.367604+0000","last_fullsized":"2024-03-18T14:52:36.367604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:40:54.650294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.381529+0000","last_change":"2024-03-18T14:52:07.381159+0000","last_active":"2024-03-18T14:52:07.381529+0000","last_peered":"2024-03-18T14:52:07.381529+0000","last_clean":"2024-03-18T14:52:07.381529+0000","last_became_active":"2024-03-18T14:52:07.371943+0000","last_became_peered":"2024-03-18T14:52:07.371943+0000","last_unstale":"2024-03-18T14:52:07.381529+0000","last_undegraded":"2024-03-18T14:52:07.381529+0000","last_fullsized":"2024-03-18T14:52:07.381529+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:11.329239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.359825+0000","last_change":"2024-03-18T14:51:01.357920+0000","last_active":"2024-03-18T14:52:36.359825+0000","last_peered":"2024-03-18T14:52:36.359825+0000","last_clean":"2024-03-18T14:52:36.359825+0000","last_became_active":"2024-03-18T14:51:01.357565+0000","last_became_peered":"2024-03-18T14:51:01.357565+0000","last_unstale":"2024-03-18T14:52:36.359825+0000","last_undegraded":"2024-03-18T14:52:36.359825+0000","last_fullsized":"2024-03-18T14:52:36.359825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:39:57.880178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370714+0000","last_change":"2024-03-18T14:51:00.937887+0000","last_active":"2024-03-18T14:52:07.370714+0000","last_peered":"2024-03-18T14:52:07.370714+0000","last_clean":"2024-03-18T14:52:07.370714+0000","last_became_active":"2024-03-18T14:51:00.937588+0000","last_became_peered":"2024-03-18T14:51:00.937588+0000","last_unstale":"2024-03-18T14:52:07.370714+0000","last_undegraded":"2024-03-18T14:52:07.370714+0000","last_fullsized":"2024-03-18T14:52:07.370714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:43:37.558412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375612+0000","last_change":"2024-03-18T14:51:01.354175+0000","last_active":"2024-03-18T14:52:36.375612+0000","last_peered":"2024-03-18T14:52:36.375612+0000","last_clean":"2024-03-18T14:52:36.375612+0000","last_became_active":"2024-03-18T14:51:01.353839+0000","last_became_peered":"2024-03-18T14:51:01.353839+0000","last_unstale":"2024-03-18T14:52:36.375612+0000","last_undegraded":"2024-03-18T14:52:36.375612+0000","last_fullsized":"2024-03-18T14:52:36.375612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:21:52.372938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369167+0000","last_change":"2024-03-18T14:51:00.941600+0000","last_active":"2024-03-18T14:52:07.369167+0000","last_peered":"2024-03-18T14:52:07.369167+0000","last_clean":"2024-03-18T14:52:07.369167+0000","last_became_active":"2024-03-18T14:51:00.941291+0000","last_became_peered":"2024-03-18T14:51:00.941291+0000","last_unstale":"2024-03-18T14:52:07.369167+0000","last_undegraded":"2024-03-18T14:52:07.369167+0000","last_fullsized":"2024-03-18T14:52:07.369167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:04:11.142029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.376178+0000","last_change":"2024-03-18T14:51:01.359058+0000","last_active":"2024-03-18T14:52:36.376178+0000","last_peered":"2024-03-18T14:52:36.376178+0000","last_clean":"2024-03-18T14:52:36.376178+0000","last_became_active":"2024-03-18T14:51:01.358437+0000","last_became_peered":"2024-03-18T14:51:01.358437+0000","last_unstale":"2024-03-18T14:52:36.376178+0000","last_undegraded":"2024-03-18T14:52:36.376178+0000","last_fullsized":"2024-03-18T14:52:36.376178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:39:58.945080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.424033+0000","last_change":"2024-03-18T14:51:00.933647+0000","last_active":"2024-03-18T14:52:07.424033+0000","last_peered":"2024-03-18T14:52:07.424033+0000","last_clean":"2024-03-18T14:52:07.424033+0000","last_became_active":"2024-03-18T14:51:00.933320+0000","last_became_peered":"2024-03-18T14:51:00.933320+0000","last_unstale":"2024-03-18T14:52:07.424033+0000","last_undegraded":"2024-03-18T14:52:07.424033+0000","last_fullsized":"2024-03-18T14:52:07.424033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:16:46.498845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.370719+0000","last_change":"2024-03-18T14:51:01.359874+0000","last_active":"2024-03-18T14:52:36.370719+0000","last_peered":"2024-03-18T14:52:36.370719+0000","last_clean":"2024-03-18T14:52:36.370719+0000","last_became_active":"2024-03-18T14:51:01.359335+0000","last_became_peered":"2024-03-18T14:51:01.359335+0000","last_unstale":"2024-03-18T14:52:36.370719+0000","last_undegraded":"2024-03-18T14:52:36.370719+0000","last_fullsized":"2024-03-18T14:52:36.370719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:03.583402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481395+0000","last_change":"2024-03-18T14:51:00.935755+0000","last_active":"2024-03-18T14:52:07.481395+0000","last_peered":"2024-03-18T14:52:07.481395+0000","last_clean":"2024-03-18T14:52:07.481395+0000","last_became_active":"2024-03-18T14:51:00.935421+0000","last_became_peered":"2024-03-18T14:51:00.935421+0000","last_unstale":"2024-03-18T14:52:07.481395+0000","last_undegraded":"2024-03-18T14:52:07.481395+0000","last_fullsized":"2024-03-18T14:52:07.481395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:46:11.680972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.373520+0000","last_change":"2024-03-18T14:51:01.355605+0000","last_active":"2024-03-18T14:52:36.373520+0000","last_peered":"2024-03-18T14:52:36.373520+0000","last_clean":"2024-03-18T14:52:36.373520+0000","last_became_active":"2024-03-18T14:51:01.355295+0000","last_became_peered":"2024-03-18T14:51:01.355295+0000","last_unstale":"2024-03-18T14:52:36.373520+0000","last_undegraded":"2024-03-18T14:52:36.373520+0000","last_fullsized":"2024-03-18T14:52:36.373520+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:15:20.466314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.379370+0000","last_change":"2024-03-18T14:51:01.359704+0000","last_active":"2024-03-18T14:52:36.379370+0000","last_peered":"2024-03-18T14:52:36.379370+0000","last_clean":"2024-03-18T14:52:36.379370+0000","last_became_active":"2024-03-18T14:51:01.359367+0000","last_became_peered":"2024-03-18T14:51:01.359367+0000","last_unstale":"2024-03-18T14:52:36.379370+0000","last_undegraded":"2024-03-18T14:52:36.379370+0000","last_fullsized":"2024-03-18T14:52:36.379370+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:26.933993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.365339+0000","last_change":"2024-03-18T14:52:07.420506+0000","last_active":"2024-03-18T14:52:36.365339+0000","last_peered":"2024-03-18T14:52:36.365339+0000","last_clean":"2024-03-18T14:52:36.365339+0000","last_became_active":"2024-03-18T14:52:07.420254+0000","last_became_peered":"2024-03-18T14:52:07.420254+0000","last_unstale":"2024-03-18T14:52:36.365339+0000","last_undegraded":"2024-03-18T14:52:36.365339+0000","last_fullsized":"2024-03-18T14:52:36.365339+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:32:08.702462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.369110+0000","last_change":"2024-03-18T14:51:01.355678+0000","last_active":"2024-03-18T14:52:36.369110+0000","last_peered":"2024-03-18T14:52:36.369110+0000","last_clean":"2024-03-18T14:52:36.369110+0000","last_became_active":"2024-03-18T14:51:01.355197+0000","last_became_peered":"2024-03-18T14:51:01.355197+0000","last_unstale":"2024-03-18T14:52:36.369110+0000","last_undegraded":"2024-03-18T14:52:36.369110+0000","last_fullsized":"2024-03-18T14:52:36.369110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:34.194834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.366017+0000","last_change":"2024-03-18T14:51:01.363964+0000","last_active":"2024-03-18T14:52:36.366017+0000","last_peered":"2024-03-18T14:52:36.366017+0000","last_clean":"2024-03-18T14:52:36.366017+0000","last_became_active":"2024-03-18T14:51:01.363590+0000","last_became_peered":"2024-03-18T14:51:01.363590+0000","last_unstale":"2024-03-18T14:52:36.366017+0000","last_undegraded":"2024-03-18T14:52:36.366017+0000","last_fullsized":"2024-03-18T14:52:36.366017+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:59.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.366199+0000","last_change":"2024-03-18T14:51:01.361599+0000","last_active":"2024-03-18T14:52:36.366199+0000","last_peered":"2024-03-18T14:52:36.366199+0000","last_clean":"2024-03-18T14:52:36.366199+0000","last_became_active":"2024-03-18T14:51:01.361280+0000","last_became_peered":"2024-03-18T14:51:01.361280+0000","last_unstale":"2024-03-18T14:52:36.366199+0000","last_undegraded":"2024-03-18T14:52:36.366199+0000","last_fullsized":"2024-03-18T14:52:36.366199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:52:50.572257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.376675+0000","last_change":"2024-03-18T14:51:01.360495+0000","last_active":"2024-03-18T14:52:36.376675+0000","last_peered":"2024-03-18T14:52:36.376675+0000","last_clean":"2024-03-18T14:52:36.376675+0000","last_became_active":"2024-03-18T14:51:01.360146+0000","last_became_peered":"2024-03-18T14:51:01.360146+0000","last_unstale":"2024-03-18T14:52:36.376675+0000","last_undegraded":"2024-03-18T14:52:36.376675+0000","last_fullsized":"2024-03-18T14:52:36.376675+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:52.048137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.363452+0000","last_change":"2024-03-18T14:51:01.357650+0000","last_active":"2024-03-18T14:52:36.363452+0000","last_peered":"2024-03-18T14:52:36.363452+0000","last_clean":"2024-03-18T14:52:36.363452+0000","last_became_active":"2024-03-18T14:51:01.356265+0000","last_became_peered":"2024-03-18T14:51:01.356265+0000","last_unstale":"2024-03-18T14:52:36.363452+0000","last_undegraded":"2024-03-18T14:52:36.363452+0000","last_fullsized":"2024-03-18T14:52:36.363452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:49.328404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.357983+0000","last_change":"2024-03-18T14:51:01.363753+0000","last_active":"2024-03-18T14:52:36.357983+0000","last_peered":"2024-03-18T14:52:36.357983+0000","last_clean":"2024-03-18T14:52:36.357983+0000","last_became_active":"2024-03-18T14:51:01.363411+0000","last_became_peered":"2024-03-18T14:51:01.363411+0000","last_unstale":"2024-03-18T14:52:36.357983+0000","last_undegraded":"2024-03-18T14:52:36.357983+0000","last_fullsized":"2024-03-18T14:52:36.357983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:55.452209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.365639+0000","last_change":"2024-03-18T14:51:01.360723+0000","last_active":"2024-03-18T14:52:36.365639+0000","last_peered":"2024-03-18T14:52:36.365639+0000","last_clean":"2024-03-18T14:52:36.365639+0000","last_became_active":"2024-03-18T14:51:01.360130+0000","last_became_peered":"2024-03-18T14:51:01.360130+0000","last_unstale":"2024-03-18T14:52:36.365639+0000","last_undegraded":"2024-03-18T14:52:36.365639+0000","last_fullsized":"2024-03-18T14:52:36.365639+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:51.634011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.372926+0000","last_change":"2024-03-18T14:51:01.356462+0000","last_active":"2024-03-18T14:52:36.372926+0000","last_peered":"2024-03-18T14:52:36.372926+0000","last_clean":"2024-03-18T14:52:36.372926+0000","last_became_active":"2024-03-18T14:51:01.355395+0000","last_became_peered":"2024-03-18T14:51:01.355395+0000","last_unstale":"2024-03-18T14:52:36.372926+0000","last_undegraded":"2024-03-18T14:52:36.372926+0000","last_fullsized":"2024-03-18T14:52:36.372926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:37:15.326354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.376909+0000","last_change":"2024-03-18T14:51:01.358626+0000","last_active":"2024-03-18T14:52:36.376909+0000","last_peered":"2024-03-18T14:52:36.376909+0000","last_clean":"2024-03-18T14:52:36.376909+0000","last_became_active":"2024-03-18T14:51:01.358293+0000","last_became_peered":"2024-03-18T14:51:01.358293+0000","last_unstale":"2024-03-18T14:52:36.376909+0000","last_undegraded":"2024-03-18T14:52:36.376909+0000","last_fullsized":"2024-03-18T14:52:36.376909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:52:41.139885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375009+0000","last_change":"2024-03-18T14:51:01.355276+0000","last_active":"2024-03-18T14:52:36.375009+0000","last_peered":"2024-03-18T14:52:36.375009+0000","last_clean":"2024-03-18T14:52:36.375009+0000","last_became_active":"2024-03-18T14:51:01.354661+0000","last_became_peered":"2024-03-18T14:51:01.354661+0000","last_unstale":"2024-03-18T14:52:36.375009+0000","last_undegraded":"2024-03-18T14:52:36.375009+0000","last_fullsized":"2024-03-18T14:52:36.375009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:28.569055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.377238+0000","last_change":"2024-03-18T14:52:07.372297+0000","last_active":"2024-03-18T14:52:36.377238+0000","last_peered":"2024-03-18T14:52:36.377238+0000","last_clean":"2024-03-18T14:52:36.377238+0000","last_became_active":"2024-03-18T14:52:07.371918+0000","last_became_peered":"2024-03-18T14:52:07.371918+0000","last_unstale":"2024-03-18T14:52:36.377238+0000","last_undegraded":"2024-03-18T14:52:36.377238+0000","last_fullsized":"2024-03-18T14:52:36.377238+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:28:23.431373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280632,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63919,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":727,"num_read_kb":430,"num_write":167,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":58912,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":200385,"data_compressed":13263,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351014,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":459161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612650,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":1110643,"data_compressed":37553,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776176,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037813,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":459317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":1051696,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462975,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":458786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":518048,"data_compressed":1859,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":460057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":459929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":459097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888138,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":523846,"data_compressed":9545,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724612,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":1050540,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63783,"data_compressed":1859,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":65267,"data_compressed":9545,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":61887,"data_compressed":1859,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:53:06.661 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:53:06.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.797+0000 7ff09533b640 1 -- 172.21.15.76:0/1162698116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff090100260 msgr2=0x7ff090109e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:06.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.797+0000 7ff09533b640 1 --2- 172.21.15.76:0/1162698116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff090100260 0x7ff090109e10 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff088009ca0 tx=0x7ff08802c0f0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.797+0000 7ff09533b640 1 -- 172.21.15.76:0/1162698116 shutdown_connections 2024-03-18T14:53:06.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.797+0000 7ff09533b640 1 --2- 172.21.15.76:0/1162698116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff090100260 0x7ff090109e10 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.797+0000 7ff09533b640 1 --2- 172.21.15.76:0/1162698116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0900ff8a0 0x7ff0900ffd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.797+0000 7ff09533b640 1 --2- 172.21.15.76:0/1162698116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff090102e30 0x7ff090103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.797+0000 7ff09533b640 1 -- 172.21.15.76:0/1162698116 >> 172.21.15.76:0/1162698116 conn(0x7ff0900fb420 msgr2=0x7ff0900fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:06.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.797+0000 7ff09533b640 1 -- 172.21.15.76:0/1162698116 shutdown_connections 2024-03-18T14:53:06.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.797+0000 7ff09533b640 1 -- 172.21.15.76:0/1162698116 wait complete. 2024-03-18T14:53:06.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.798+0000 7ff09533b640 1 Processor -- start 2024-03-18T14:53:06.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.798+0000 7ff09533b640 1 -- start start 2024-03-18T14:53:06.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.798+0000 7ff09533b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0900ff8a0 0x7ff090197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:06.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.798+0000 7ff09533b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff090100260 0x7ff090198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:06.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.798+0000 7ff09533b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff090102e30 0x7ff09019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:06.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.798+0000 7ff09533b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff09010d4b0 con 0x7ff090102e30 2024-03-18T14:53:06.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.798+0000 7ff09533b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff09010d330 con 0x7ff0900ff8a0 2024-03-18T14:53:06.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.798+0000 7ff09533b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff09010d630 con 0x7ff090100260 2024-03-18T14:53:06.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.798+0000 7ff08f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff090102e30 0x7ff09019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:06.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.798+0000 7ff08f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff090102e30 0x7ff09019f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50408/0 (socket says 172.21.15.76:50408) 2024-03-18T14:53:06.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.798+0000 7ff08f7fe640 1 -- 172.21.15.76:0/2098657630 learned_addr learned my addr 172.21.15.76:0/2098657630 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:06.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.798+0000 7ff08effd640 1 --2- 172.21.15.76:0/2098657630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0900ff8a0 0x7ff090197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:06.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.798+0000 7ff08e7fc640 1 --2- 172.21.15.76:0/2098657630 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff090100260 0x7ff090198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:06.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.799+0000 7ff08f7fe640 1 -- 172.21.15.76:0/2098657630 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff090100260 msgr2=0x7ff090198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:06.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.799+0000 7ff08f7fe640 1 --2- 172.21.15.76:0/2098657630 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff090100260 0x7ff090198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.799+0000 7ff08f7fe640 1 -- 172.21.15.76:0/2098657630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0900ff8a0 msgr2=0x7ff090197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:06.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.799+0000 7ff08f7fe640 1 --2- 172.21.15.76:0/2098657630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0900ff8a0 0x7ff090197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.799+0000 7ff08f7fe640 1 -- 172.21.15.76:0/2098657630 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff09019f6e0 con 0x7ff090102e30 2024-03-18T14:53:06.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.799+0000 7ff08effd640 1 --2- 172.21.15.76:0/2098657630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0900ff8a0 0x7ff090197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:06.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.799+0000 7ff08f7fe640 1 --2- 172.21.15.76:0/2098657630 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff090102e30 0x7ff09019f070 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff088037040 tx=0x7ff088032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:06.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.799+0000 7ff077fff640 1 -- 172.21.15.76:0/2098657630 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088041020 con 0x7ff090102e30 2024-03-18T14:53:06.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.799+0000 7ff077fff640 1 -- 172.21.15.76:0/2098657630 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff088034cd0 con 0x7ff090102e30 2024-03-18T14:53:06.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.799+0000 7ff09533b640 1 -- 172.21.15.76:0/2098657630 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff09019f970 con 0x7ff090102e30 2024-03-18T14:53:06.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.799+0000 7ff077fff640 1 -- 172.21.15.76:0/2098657630 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088039040 con 0x7ff090102e30 2024-03-18T14:53:06.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.799+0000 7ff09533b640 1 -- 172.21.15.76:0/2098657630 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff09019fe60 con 0x7ff090102e30 2024-03-18T14:53:06.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.800+0000 7ff09533b640 1 -- 172.21.15.76:0/2098657630 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff090108cf0 con 0x7ff090102e30 2024-03-18T14:53:06.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.801+0000 7ff077fff640 1 -- 172.21.15.76:0/2098657630 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff088044020 con 0x7ff090102e30 2024-03-18T14:53:06.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.801+0000 7ff077fff640 1 --2- 172.21.15.76:0/2098657630 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff0780a2800 0x7ff0780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:06.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.802+0000 7ff08effd640 1 --2- 172.21.15.76:0/2098657630 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff0780a2800 0x7ff0780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:06.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.802+0000 7ff077fff640 1 -- 172.21.15.76:0/2098657630 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7ff0880f0920 con 0x7ff090102e30 2024-03-18T14:53:06.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.802+0000 7ff08effd640 1 --2- 172.21.15.76:0/2098657630 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff0780a2800 0x7ff0780a4cc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff090069b30 tx=0x7ff080005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:06.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.806+0000 7ff077fff640 1 -- 172.21.15.76:0/2098657630 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff08804b020 con 0x7ff090102e30 2024-03-18T14:53:06.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:06 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/4213346704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:53:06.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:06 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3297089238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:53:06.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:06 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/130036353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:53:06.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:06 smithi174 ceph-mon[36787]: pgmap v348: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T14:53:06.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:06 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/4213346704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:53:06.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:06 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3297089238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:53:06.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:06 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/130036353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:53:06.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:06 smithi077 ceph-mon[35865]: pgmap v348: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T14:53:06.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.981+0000 7ff09533b640 1 -- 172.21.15.76:0/2098657630 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff090061c20 con 0x7ff0780a2800 2024-03-18T14:53:06.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.988+0000 7ff077fff640 1 -- 172.21.15.76:0/2098657630 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259063 (secure 0 0 0) 0x7ff090061c20 con 0x7ff0780a2800 2024-03-18T14:53:06.991 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:53:06.993 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-03-18T14:53:06.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.991+0000 7ff09533b640 1 -- 172.21.15.76:0/2098657630 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff0780a2800 msgr2=0x7ff0780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:06.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.991+0000 7ff09533b640 1 --2- 172.21.15.76:0/2098657630 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff0780a2800 0x7ff0780a4cc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff090069b30 tx=0x7ff080005f90 comp rx=0 tx=0).stop 2024-03-18T14:53:06.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.991+0000 7ff09533b640 1 -- 172.21.15.76:0/2098657630 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff090102e30 msgr2=0x7ff09019f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:06.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.991+0000 7ff09533b640 1 --2- 172.21.15.76:0/2098657630 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff090102e30 0x7ff09019f070 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff088037040 tx=0x7ff088032f40 comp rx=0 tx=0).stop 2024-03-18T14:53:06.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.991+0000 7ff09533b640 1 -- 172.21.15.76:0/2098657630 shutdown_connections 2024-03-18T14:53:06.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.991+0000 7ff09533b640 1 --2- 172.21.15.76:0/2098657630 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff0780a2800 0x7ff0780a4cc0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.991+0000 7ff09533b640 1 --2- 172.21.15.76:0/2098657630 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff090100260 0x7ff090198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.991+0000 7ff09533b640 1 --2- 172.21.15.76:0/2098657630 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff090102e30 0x7ff09019f070 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.991+0000 7ff09533b640 1 --2- 172.21.15.76:0/2098657630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0900ff8a0 0x7ff090197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:06.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.991+0000 7ff09533b640 1 -- 172.21.15.76:0/2098657630 >> 172.21.15.76:0/2098657630 conn(0x7ff0900fb420 msgr2=0x7ff090105ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:06.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.992+0000 7ff09533b640 1 -- 172.21.15.76:0/2098657630 shutdown_connections 2024-03-18T14:53:06.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:06.992+0000 7ff09533b640 1 -- 172.21.15.76:0/2098657630 wait complete. 2024-03-18T14:53:07.003 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-03-18T14:53:06.359535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654521,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":863,"num_read_kb":549,"num_write":350,"num_write_kb":3055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1239648,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":783,"ondisk_log_size":783,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369088,"kb_used_data":11616,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572736,"statfs":{"total":1151940427776,"available":1151562481664,"internally_reserved":0,"allocated":11894784,"data_stored":7470517,"data_compressed":120345,"data_compressed_allocated":155648,"data_compressed_original":1843200,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":36,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002927"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.371528+0000","last_change":"2024-03-18T14:51:01.365948+0000","last_active":"2024-03-18T14:52:36.371528+0000","last_peered":"2024-03-18T14:52:36.371528+0000","last_clean":"2024-03-18T14:52:36.371528+0000","last_became_active":"2024-03-18T14:51:01.364600+0000","last_became_peered":"2024-03-18T14:51:01.364600+0000","last_unstale":"2024-03-18T14:52:36.371528+0000","last_undegraded":"2024-03-18T14:52:36.371528+0000","last_fullsized":"2024-03-18T14:52:36.371528+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:14:45.198618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.359354+0000","last_change":"2024-03-18T14:51:01.360186+0000","last_active":"2024-03-18T14:52:36.359354+0000","last_peered":"2024-03-18T14:52:36.359354+0000","last_clean":"2024-03-18T14:52:36.359354+0000","last_became_active":"2024-03-18T14:51:01.359862+0000","last_became_peered":"2024-03-18T14:51:01.359862+0000","last_unstale":"2024-03-18T14:52:36.359354+0000","last_undegraded":"2024-03-18T14:52:36.359354+0000","last_fullsized":"2024-03-18T14:52:36.359354+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:46.206823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375396+0000","last_change":"2024-03-18T14:51:01.364635+0000","last_active":"2024-03-18T14:52:36.375396+0000","last_peered":"2024-03-18T14:52:36.375396+0000","last_clean":"2024-03-18T14:52:36.375396+0000","last_became_active":"2024-03-18T14:51:01.364320+0000","last_became_peered":"2024-03-18T14:51:01.364320+0000","last_unstale":"2024-03-18T14:52:36.375396+0000","last_undegraded":"2024-03-18T14:52:36.375396+0000","last_fullsized":"2024-03-18T14:52:36.375396+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:08:20.227639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.358624+0000","last_change":"2024-03-18T14:52:07.372090+0000","last_active":"2024-03-18T14:52:36.358624+0000","last_peered":"2024-03-18T14:52:36.358624+0000","last_clean":"2024-03-18T14:52:36.358624+0000","last_became_active":"2024-03-18T14:52:07.371775+0000","last_became_peered":"2024-03-18T14:52:07.371775+0000","last_unstale":"2024-03-18T14:52:36.358624+0000","last_undegraded":"2024-03-18T14:52:36.358624+0000","last_fullsized":"2024-03-18T14:52:36.358624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:48.128827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.356734+0000","last_change":"2024-03-18T14:51:01.356839+0000","last_active":"2024-03-18T14:52:36.356734+0000","last_peered":"2024-03-18T14:52:36.356734+0000","last_clean":"2024-03-18T14:52:36.356734+0000","last_became_active":"2024-03-18T14:51:01.356531+0000","last_became_peered":"2024-03-18T14:51:01.356531+0000","last_unstale":"2024-03-18T14:52:36.356734+0000","last_undegraded":"2024-03-18T14:52:36.356734+0000","last_fullsized":"2024-03-18T14:52:36.356734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:28.499573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.357939+0000","last_change":"2024-03-18T14:51:01.362912+0000","last_active":"2024-03-18T14:52:36.357939+0000","last_peered":"2024-03-18T14:52:36.357939+0000","last_clean":"2024-03-18T14:52:36.357939+0000","last_became_active":"2024-03-18T14:51:01.362590+0000","last_became_peered":"2024-03-18T14:51:01.362590+0000","last_unstale":"2024-03-18T14:52:36.357939+0000","last_undegraded":"2024-03-18T14:52:36.357939+0000","last_fullsized":"2024-03-18T14:52:36.357939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:49.199616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.370814+0000","last_change":"2024-03-18T14:51:01.358652+0000","last_active":"2024-03-18T14:52:36.370814+0000","last_peered":"2024-03-18T14:52:36.370814+0000","last_clean":"2024-03-18T14:52:36.370814+0000","last_became_active":"2024-03-18T14:51:01.358312+0000","last_became_peered":"2024-03-18T14:51:01.358312+0000","last_unstale":"2024-03-18T14:52:36.370814+0000","last_undegraded":"2024-03-18T14:52:36.370814+0000","last_fullsized":"2024-03-18T14:52:36.370814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:40.150240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.367332+0000","last_change":"2024-03-18T14:51:01.360082+0000","last_active":"2024-03-18T14:52:36.367332+0000","last_peered":"2024-03-18T14:52:36.367332+0000","last_clean":"2024-03-18T14:52:36.367332+0000","last_became_active":"2024-03-18T14:51:01.359769+0000","last_became_peered":"2024-03-18T14:51:01.359769+0000","last_unstale":"2024-03-18T14:52:36.367332+0000","last_undegraded":"2024-03-18T14:52:36.367332+0000","last_fullsized":"2024-03-18T14:52:36.367332+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:53:27.081546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.370348+0000","last_change":"2024-03-18T14:51:01.363947+0000","last_active":"2024-03-18T14:52:36.370348+0000","last_peered":"2024-03-18T14:52:36.370348+0000","last_clean":"2024-03-18T14:52:36.370348+0000","last_became_active":"2024-03-18T14:51:01.363106+0000","last_became_peered":"2024-03-18T14:51:01.363106+0000","last_unstale":"2024-03-18T14:52:36.370348+0000","last_undegraded":"2024-03-18T14:52:36.370348+0000","last_fullsized":"2024-03-18T14:52:36.370348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:24:50.478426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.363104+0000","last_change":"2024-03-18T14:51:01.357950+0000","last_active":"2024-03-18T14:52:36.363104+0000","last_peered":"2024-03-18T14:52:36.363104+0000","last_clean":"2024-03-18T14:52:36.363104+0000","last_became_active":"2024-03-18T14:51:01.357651+0000","last_became_peered":"2024-03-18T14:51:01.357651+0000","last_unstale":"2024-03-18T14:52:36.363104+0000","last_undegraded":"2024-03-18T14:52:36.363104+0000","last_fullsized":"2024-03-18T14:52:36.363104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:49:01.376008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375514+0000","last_change":"2024-03-18T14:51:01.360069+0000","last_active":"2024-03-18T14:52:36.375514+0000","last_peered":"2024-03-18T14:52:36.375514+0000","last_clean":"2024-03-18T14:52:36.375514+0000","last_became_active":"2024-03-18T14:51:01.358960+0000","last_became_peered":"2024-03-18T14:51:01.358960+0000","last_unstale":"2024-03-18T14:52:36.375514+0000","last_undegraded":"2024-03-18T14:52:36.375514+0000","last_fullsized":"2024-03-18T14:52:36.375514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:48:42.197329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.364663+0000","last_change":"2024-03-18T14:51:01.361200+0000","last_active":"2024-03-18T14:52:36.364663+0000","last_peered":"2024-03-18T14:52:36.364663+0000","last_clean":"2024-03-18T14:52:36.364663+0000","last_became_active":"2024-03-18T14:51:01.360879+0000","last_became_peered":"2024-03-18T14:51:01.360879+0000","last_unstale":"2024-03-18T14:52:36.364663+0000","last_undegraded":"2024-03-18T14:52:36.364663+0000","last_fullsized":"2024-03-18T14:52:36.364663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:05.335627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.364978+0000","last_change":"2024-03-18T14:51:01.352441+0000","last_active":"2024-03-18T14:52:36.364978+0000","last_peered":"2024-03-18T14:52:36.364978+0000","last_clean":"2024-03-18T14:52:36.364978+0000","last_became_active":"2024-03-18T14:51:01.352118+0000","last_became_peered":"2024-03-18T14:51:01.352118+0000","last_unstale":"2024-03-18T14:52:36.364978+0000","last_undegraded":"2024-03-18T14:52:36.364978+0000","last_fullsized":"2024-03-18T14:52:36.364978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:45:42.781439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.364844+0000","last_change":"2024-03-18T14:51:01.361257+0000","last_active":"2024-03-18T14:52:36.364844+0000","last_peered":"2024-03-18T14:52:36.364844+0000","last_clean":"2024-03-18T14:52:36.364844+0000","last_became_active":"2024-03-18T14:51:01.360898+0000","last_became_peered":"2024-03-18T14:51:01.360898+0000","last_unstale":"2024-03-18T14:52:36.364844+0000","last_undegraded":"2024-03-18T14:52:36.364844+0000","last_fullsized":"2024-03-18T14:52:36.364844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:38:26.200678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.367432+0000","last_change":"2024-03-18T14:52:07.374822+0000","last_active":"2024-03-18T14:52:36.367432+0000","last_peered":"2024-03-18T14:52:36.367432+0000","last_clean":"2024-03-18T14:52:36.367432+0000","last_became_active":"2024-03-18T14:52:07.374537+0000","last_became_peered":"2024-03-18T14:52:07.374537+0000","last_unstale":"2024-03-18T14:52:36.367432+0000","last_undegraded":"2024-03-18T14:52:36.367432+0000","last_fullsized":"2024-03-18T14:52:36.367432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:57:29.971366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.374959+0000","last_change":"2024-03-18T14:51:01.357223+0000","last_active":"2024-03-18T14:52:36.374959+0000","last_peered":"2024-03-18T14:52:36.374959+0000","last_clean":"2024-03-18T14:52:36.374959+0000","last_became_active":"2024-03-18T14:51:01.356934+0000","last_became_peered":"2024-03-18T14:51:01.356934+0000","last_unstale":"2024-03-18T14:52:36.374959+0000","last_undegraded":"2024-03-18T14:52:36.374959+0000","last_fullsized":"2024-03-18T14:52:36.374959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:28.683605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.372986+0000","last_change":"2024-03-18T14:51:01.366414+0000","last_active":"2024-03-18T14:52:36.372986+0000","last_peered":"2024-03-18T14:52:36.372986+0000","last_clean":"2024-03-18T14:52:36.372986+0000","last_became_active":"2024-03-18T14:51:01.365534+0000","last_became_peered":"2024-03-18T14:51:01.365534+0000","last_unstale":"2024-03-18T14:52:36.372986+0000","last_undegraded":"2024-03-18T14:52:36.372986+0000","last_fullsized":"2024-03-18T14:52:36.372986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:28:51.568662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.368646+0000","last_change":"2024-03-18T14:51:01.354084+0000","last_active":"2024-03-18T14:52:36.368646+0000","last_peered":"2024-03-18T14:52:36.368646+0000","last_clean":"2024-03-18T14:52:36.368646+0000","last_became_active":"2024-03-18T14:51:01.353599+0000","last_became_peered":"2024-03-18T14:51:01.353599+0000","last_unstale":"2024-03-18T14:52:36.368646+0000","last_undegraded":"2024-03-18T14:52:36.368646+0000","last_fullsized":"2024-03-18T14:52:36.368646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:40.502783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.366302+0000","last_change":"2024-03-18T14:51:01.353391+0000","last_active":"2024-03-18T14:52:36.366302+0000","last_peered":"2024-03-18T14:52:36.366302+0000","last_clean":"2024-03-18T14:52:36.366302+0000","last_became_active":"2024-03-18T14:51:01.353084+0000","last_became_peered":"2024-03-18T14:51:01.353084+0000","last_unstale":"2024-03-18T14:52:36.366302+0000","last_undegraded":"2024-03-18T14:52:36.366302+0000","last_fullsized":"2024-03-18T14:52:36.366302+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:18:02.774589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.363884+0000","last_change":"2024-03-18T14:51:00.942617+0000","last_active":"2024-03-18T14:52:07.363884+0000","last_peered":"2024-03-18T14:52:07.363884+0000","last_clean":"2024-03-18T14:52:07.363884+0000","last_became_active":"2024-03-18T14:51:00.942276+0000","last_became_peered":"2024-03-18T14:51:00.942276+0000","last_unstale":"2024-03-18T14:52:07.363884+0000","last_undegraded":"2024-03-18T14:52:07.363884+0000","last_fullsized":"2024-03-18T14:52:07.363884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:03:05.984066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.366394+0000","last_change":"2024-03-18T14:51:01.363887+0000","last_active":"2024-03-18T14:52:36.366394+0000","last_peered":"2024-03-18T14:52:36.366394+0000","last_clean":"2024-03-18T14:52:36.366394+0000","last_became_active":"2024-03-18T14:51:01.361359+0000","last_became_peered":"2024-03-18T14:51:01.361359+0000","last_unstale":"2024-03-18T14:52:36.366394+0000","last_undegraded":"2024-03-18T14:52:36.366394+0000","last_fullsized":"2024-03-18T14:52:36.366394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:22.996930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368943+0000","last_change":"2024-03-18T14:51:00.940392+0000","last_active":"2024-03-18T14:52:07.368943+0000","last_peered":"2024-03-18T14:52:07.368943+0000","last_clean":"2024-03-18T14:52:07.368943+0000","last_became_active":"2024-03-18T14:51:00.940055+0000","last_became_peered":"2024-03-18T14:51:00.940055+0000","last_unstale":"2024-03-18T14:52:07.368943+0000","last_undegraded":"2024-03-18T14:52:07.368943+0000","last_fullsized":"2024-03-18T14:52:07.368943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:57:28.523549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375190+0000","last_change":"2024-03-18T14:51:01.361458+0000","last_active":"2024-03-18T14:52:36.375190+0000","last_peered":"2024-03-18T14:52:36.375190+0000","last_clean":"2024-03-18T14:52:36.375190+0000","last_became_active":"2024-03-18T14:51:01.361105+0000","last_became_peered":"2024-03-18T14:51:01.361105+0000","last_unstale":"2024-03-18T14:52:36.375190+0000","last_undegraded":"2024-03-18T14:52:36.375190+0000","last_fullsized":"2024-03-18T14:52:36.375190+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:22:59.410080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.379071+0000","last_change":"2024-03-18T14:52:07.378642+0000","last_active":"2024-03-18T14:52:07.379071+0000","last_peered":"2024-03-18T14:52:07.379071+0000","last_clean":"2024-03-18T14:52:07.379071+0000","last_became_active":"2024-03-18T14:52:07.371947+0000","last_became_peered":"2024-03-18T14:52:07.371947+0000","last_unstale":"2024-03-18T14:52:07.379071+0000","last_undegraded":"2024-03-18T14:52:07.379071+0000","last_fullsized":"2024-03-18T14:52:07.379071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:46:00.898462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.371790+0000","last_change":"2024-03-18T14:51:01.366305+0000","last_active":"2024-03-18T14:52:36.371790+0000","last_peered":"2024-03-18T14:52:36.371790+0000","last_clean":"2024-03-18T14:52:36.371790+0000","last_became_active":"2024-03-18T14:51:01.364626+0000","last_became_peered":"2024-03-18T14:51:01.364626+0000","last_unstale":"2024-03-18T14:52:36.371790+0000","last_undegraded":"2024-03-18T14:52:36.371790+0000","last_fullsized":"2024-03-18T14:52:36.371790+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:06.212030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370835+0000","last_change":"2024-03-18T14:51:00.932514+0000","last_active":"2024-03-18T14:52:07.370835+0000","last_peered":"2024-03-18T14:52:07.370835+0000","last_clean":"2024-03-18T14:52:07.370835+0000","last_became_active":"2024-03-18T14:51:00.932183+0000","last_became_peered":"2024-03-18T14:51:00.932183+0000","last_unstale":"2024-03-18T14:52:07.370835+0000","last_undegraded":"2024-03-18T14:52:07.370835+0000","last_fullsized":"2024-03-18T14:52:07.370835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:36:37.230185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.372279+0000","last_change":"2024-03-18T14:51:01.358939+0000","last_active":"2024-03-18T14:52:36.372279+0000","last_peered":"2024-03-18T14:52:36.372279+0000","last_clean":"2024-03-18T14:52:36.372279+0000","last_became_active":"2024-03-18T14:51:01.358601+0000","last_became_peered":"2024-03-18T14:51:01.358601+0000","last_unstale":"2024-03-18T14:52:36.372279+0000","last_undegraded":"2024-03-18T14:52:36.372279+0000","last_fullsized":"2024-03-18T14:52:36.372279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:40:14.876143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":145,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:53:04.156527+0000","last_change":"2024-03-18T14:51:00.945116+0000","last_active":"2024-03-18T14:53:04.156527+0000","last_peered":"2024-03-18T14:53:04.156527+0000","last_clean":"2024-03-18T14:53:04.156527+0000","last_became_active":"2024-03-18T14:51:00.944791+0000","last_became_peered":"2024-03-18T14:51:00.944791+0000","last_unstale":"2024-03-18T14:53:04.156527+0000","last_undegraded":"2024-03-18T14:53:04.156527+0000","last_fullsized":"2024-03-18T14:53:04.156527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:06.637940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.357355+0000","last_change":"2024-03-18T14:51:01.358682+0000","last_active":"2024-03-18T14:52:36.357355+0000","last_peered":"2024-03-18T14:52:36.357355+0000","last_clean":"2024-03-18T14:52:36.357355+0000","last_became_active":"2024-03-18T14:51:01.358378+0000","last_became_peered":"2024-03-18T14:51:01.358378+0000","last_unstale":"2024-03-18T14:52:36.357355+0000","last_undegraded":"2024-03-18T14:52:36.357355+0000","last_fullsized":"2024-03-18T14:52:36.357355+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:42.697944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369158+0000","last_change":"2024-03-18T14:51:00.943976+0000","last_active":"2024-03-18T14:52:07.369158+0000","last_peered":"2024-03-18T14:52:07.369158+0000","last_clean":"2024-03-18T14:52:07.369158+0000","last_became_active":"2024-03-18T14:51:00.943614+0000","last_became_peered":"2024-03-18T14:51:00.943614+0000","last_unstale":"2024-03-18T14:52:07.369158+0000","last_undegraded":"2024-03-18T14:52:07.369158+0000","last_fullsized":"2024-03-18T14:52:07.369158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:07:00.744708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.365708+0000","last_change":"2024-03-18T14:51:01.365316+0000","last_active":"2024-03-18T14:52:36.365708+0000","last_peered":"2024-03-18T14:52:36.365708+0000","last_clean":"2024-03-18T14:52:36.365708+0000","last_became_active":"2024-03-18T14:51:01.365019+0000","last_became_peered":"2024-03-18T14:51:01.365019+0000","last_unstale":"2024-03-18T14:52:36.365708+0000","last_undegraded":"2024-03-18T14:52:36.365708+0000","last_fullsized":"2024-03-18T14:52:36.365708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:39:59.040907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481239+0000","last_change":"2024-03-18T14:51:00.936923+0000","last_active":"2024-03-18T14:52:07.481239+0000","last_peered":"2024-03-18T14:52:07.481239+0000","last_clean":"2024-03-18T14:52:07.481239+0000","last_became_active":"2024-03-18T14:51:00.936591+0000","last_became_peered":"2024-03-18T14:51:00.936591+0000","last_unstale":"2024-03-18T14:52:07.481239+0000","last_undegraded":"2024-03-18T14:52:07.481239+0000","last_fullsized":"2024-03-18T14:52:07.481239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:48:46.423033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.378104+0000","last_change":"2024-03-18T14:51:01.356827+0000","last_active":"2024-03-18T14:52:36.378104+0000","last_peered":"2024-03-18T14:52:36.378104+0000","last_clean":"2024-03-18T14:52:36.378104+0000","last_became_active":"2024-03-18T14:51:01.356148+0000","last_became_peered":"2024-03-18T14:51:01.356148+0000","last_unstale":"2024-03-18T14:52:36.378104+0000","last_undegraded":"2024-03-18T14:52:36.378104+0000","last_fullsized":"2024-03-18T14:52:36.378104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:18.477890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423674+0000","last_change":"2024-03-18T14:51:00.934024+0000","last_active":"2024-03-18T14:52:07.423674+0000","last_peered":"2024-03-18T14:52:07.423674+0000","last_clean":"2024-03-18T14:52:07.423674+0000","last_became_active":"2024-03-18T14:51:00.933739+0000","last_became_peered":"2024-03-18T14:51:00.933739+0000","last_unstale":"2024-03-18T14:52:07.423674+0000","last_undegraded":"2024-03-18T14:52:07.423674+0000","last_fullsized":"2024-03-18T14:52:07.423674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:03.967260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.362587+0000","last_change":"2024-03-18T14:51:01.356519+0000","last_active":"2024-03-18T14:52:36.362587+0000","last_peered":"2024-03-18T14:52:36.362587+0000","last_clean":"2024-03-18T14:52:36.362587+0000","last_became_active":"2024-03-18T14:51:01.356193+0000","last_became_peered":"2024-03-18T14:51:01.356193+0000","last_unstale":"2024-03-18T14:52:36.362587+0000","last_undegraded":"2024-03-18T14:52:36.362587+0000","last_fullsized":"2024-03-18T14:52:36.362587+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:34:59.633241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:53:01.153237+0000","last_change":"2024-03-18T14:51:00.940189+0000","last_active":"2024-03-18T14:53:01.153237+0000","last_peered":"2024-03-18T14:53:01.153237+0000","last_clean":"2024-03-18T14:53:01.153237+0000","last_became_active":"2024-03-18T14:51:00.939820+0000","last_became_peered":"2024-03-18T14:51:00.939820+0000","last_unstale":"2024-03-18T14:53:01.153237+0000","last_undegraded":"2024-03-18T14:53:01.153237+0000","last_fullsized":"2024-03-18T14:53:01.153237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:44:09.749747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.371264+0000","last_change":"2024-03-18T14:51:01.355756+0000","last_active":"2024-03-18T14:52:36.371264+0000","last_peered":"2024-03-18T14:52:36.371264+0000","last_clean":"2024-03-18T14:52:36.371264+0000","last_became_active":"2024-03-18T14:51:01.352871+0000","last_became_peered":"2024-03-18T14:51:01.352871+0000","last_unstale":"2024-03-18T14:52:36.371264+0000","last_undegraded":"2024-03-18T14:52:36.371264+0000","last_fullsized":"2024-03-18T14:52:36.371264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:20:26.533066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366850+0000","last_change":"2024-03-18T14:51:00.941908+0000","last_active":"2024-03-18T14:52:07.366850+0000","last_peered":"2024-03-18T14:52:07.366850+0000","last_clean":"2024-03-18T14:52:07.366850+0000","last_became_active":"2024-03-18T14:51:00.940978+0000","last_became_peered":"2024-03-18T14:51:00.940978+0000","last_unstale":"2024-03-18T14:52:07.366850+0000","last_undegraded":"2024-03-18T14:52:07.366850+0000","last_fullsized":"2024-03-18T14:52:07.366850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:52.705243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.366882+0000","last_change":"2024-03-18T14:51:01.363462+0000","last_active":"2024-03-18T14:52:36.366882+0000","last_peered":"2024-03-18T14:52:36.366882+0000","last_clean":"2024-03-18T14:52:36.366882+0000","last_became_active":"2024-03-18T14:51:01.362262+0000","last_became_peered":"2024-03-18T14:51:01.362262+0000","last_unstale":"2024-03-18T14:52:36.366882+0000","last_undegraded":"2024-03-18T14:52:36.366882+0000","last_fullsized":"2024-03-18T14:52:36.366882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:07:18.952876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.424170+0000","last_change":"2024-03-18T14:51:00.936050+0000","last_active":"2024-03-18T14:52:07.424170+0000","last_peered":"2024-03-18T14:52:07.424170+0000","last_clean":"2024-03-18T14:52:07.424170+0000","last_became_active":"2024-03-18T14:51:00.935676+0000","last_became_peered":"2024-03-18T14:51:00.935676+0000","last_unstale":"2024-03-18T14:52:07.424170+0000","last_undegraded":"2024-03-18T14:52:07.424170+0000","last_fullsized":"2024-03-18T14:52:07.424170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:29.423110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.374032+0000","last_change":"2024-03-18T14:51:01.364295+0000","last_active":"2024-03-18T14:52:36.374032+0000","last_peered":"2024-03-18T14:52:36.374032+0000","last_clean":"2024-03-18T14:52:36.374032+0000","last_became_active":"2024-03-18T14:51:01.363916+0000","last_became_peered":"2024-03-18T14:51:01.363916+0000","last_unstale":"2024-03-18T14:52:36.374032+0000","last_undegraded":"2024-03-18T14:52:36.374032+0000","last_fullsized":"2024-03-18T14:52:36.374032+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:21:17.905819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'31","reported_seq":305,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:53:02.055197+0000","last_change":"2024-03-18T14:52:07.427262+0000","last_active":"2024-03-18T14:53:02.055197+0000","last_peered":"2024-03-18T14:53:02.055197+0000","last_clean":"2024-03-18T14:53:02.055197+0000","last_became_active":"2024-03-18T14:52:07.421388+0000","last_became_peered":"2024-03-18T14:52:07.421388+0000","last_unstale":"2024-03-18T14:53:02.055197+0000","last_undegraded":"2024-03-18T14:53:02.055197+0000","last_fullsized":"2024-03-18T14:53:02.055197+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:28.159787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59469,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":233,"num_read_kb":174,"num_write":32,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":58912,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.376345+0000","last_change":"2024-03-18T14:51:01.362426+0000","last_active":"2024-03-18T14:52:36.376345+0000","last_peered":"2024-03-18T14:52:36.376345+0000","last_clean":"2024-03-18T14:52:36.376345+0000","last_became_active":"2024-03-18T14:51:01.362130+0000","last_became_peered":"2024-03-18T14:51:01.362130+0000","last_unstale":"2024-03-18T14:52:36.376345+0000","last_undegraded":"2024-03-18T14:52:36.376345+0000","last_fullsized":"2024-03-18T14:52:36.376345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:14.545997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368774+0000","last_change":"2024-03-18T14:51:00.934028+0000","last_active":"2024-03-18T14:52:07.368774+0000","last_peered":"2024-03-18T14:52:07.368774+0000","last_clean":"2024-03-18T14:52:07.368774+0000","last_became_active":"2024-03-18T14:51:00.933594+0000","last_became_peered":"2024-03-18T14:51:00.933594+0000","last_unstale":"2024-03-18T14:52:07.368774+0000","last_undegraded":"2024-03-18T14:52:07.368774+0000","last_fullsized":"2024-03-18T14:52:07.368774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:55:08.675533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.374723+0000","last_change":"2024-03-18T14:51:01.353544+0000","last_active":"2024-03-18T14:52:36.374723+0000","last_peered":"2024-03-18T14:52:36.374723+0000","last_clean":"2024-03-18T14:52:36.374723+0000","last_became_active":"2024-03-18T14:51:01.353241+0000","last_became_peered":"2024-03-18T14:51:01.353241+0000","last_unstale":"2024-03-18T14:52:36.374723+0000","last_undegraded":"2024-03-18T14:52:36.374723+0000","last_fullsized":"2024-03-18T14:52:36.374723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:45:27.617681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423792+0000","last_change":"2024-03-18T14:51:00.944492+0000","last_active":"2024-03-18T14:52:07.423792+0000","last_peered":"2024-03-18T14:52:07.423792+0000","last_clean":"2024-03-18T14:52:07.423792+0000","last_became_active":"2024-03-18T14:51:00.944165+0000","last_became_peered":"2024-03-18T14:51:00.944165+0000","last_unstale":"2024-03-18T14:52:07.423792+0000","last_undegraded":"2024-03-18T14:52:07.423792+0000","last_fullsized":"2024-03-18T14:52:07.423792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:47:05.968966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.371403+0000","last_change":"2024-03-18T14:51:01.356782+0000","last_active":"2024-03-18T14:52:36.371403+0000","last_peered":"2024-03-18T14:52:36.371403+0000","last_clean":"2024-03-18T14:52:36.371403+0000","last_became_active":"2024-03-18T14:51:01.356242+0000","last_became_peered":"2024-03-18T14:51:01.356242+0000","last_unstale":"2024-03-18T14:52:36.371403+0000","last_undegraded":"2024-03-18T14:52:36.371403+0000","last_fullsized":"2024-03-18T14:52:36.371403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:56:07.662545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'6","reported_seq":149,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:53:04.168246+0000","last_change":"2024-03-18T14:51:00.937873+0000","last_active":"2024-03-18T14:53:04.168246+0000","last_peered":"2024-03-18T14:53:04.168246+0000","last_clean":"2024-03-18T14:53:04.168246+0000","last_became_active":"2024-03-18T14:51:00.937562+0000","last_became_peered":"2024-03-18T14:51:00.937562+0000","last_unstale":"2024-03-18T14:53:04.168246+0000","last_undegraded":"2024-03-18T14:53:04.168246+0000","last_fullsized":"2024-03-18T14:53:04.168246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:36:44.342414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1428,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375613+0000","last_change":"2024-03-18T14:51:01.358681+0000","last_active":"2024-03-18T14:52:36.375613+0000","last_peered":"2024-03-18T14:52:36.375613+0000","last_clean":"2024-03-18T14:52:36.375613+0000","last_became_active":"2024-03-18T14:51:01.358374+0000","last_became_peered":"2024-03-18T14:51:01.358374+0000","last_unstale":"2024-03-18T14:52:36.375613+0000","last_undegraded":"2024-03-18T14:52:36.375613+0000","last_fullsized":"2024-03-18T14:52:36.375613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:43:47.804271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370451+0000","last_change":"2024-03-18T14:51:00.935639+0000","last_active":"2024-03-18T14:52:07.370451+0000","last_peered":"2024-03-18T14:52:07.370451+0000","last_clean":"2024-03-18T14:52:07.370451+0000","last_became_active":"2024-03-18T14:51:00.935334+0000","last_became_peered":"2024-03-18T14:51:00.935334+0000","last_unstale":"2024-03-18T14:52:07.370451+0000","last_undegraded":"2024-03-18T14:52:07.370451+0000","last_fullsized":"2024-03-18T14:52:07.370451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:46.966812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.370423+0000","last_change":"2024-03-18T14:51:01.364755+0000","last_active":"2024-03-18T14:52:36.370423+0000","last_peered":"2024-03-18T14:52:36.370423+0000","last_clean":"2024-03-18T14:52:36.370423+0000","last_became_active":"2024-03-18T14:51:01.364448+0000","last_became_peered":"2024-03-18T14:51:01.364448+0000","last_unstale":"2024-03-18T14:52:36.370423+0000","last_undegraded":"2024-03-18T14:52:36.370423+0000","last_fullsized":"2024-03-18T14:52:36.370423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:37:49.480787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.371021+0000","last_change":"2024-03-18T14:51:00.933271+0000","last_active":"2024-03-18T14:52:07.371021+0000","last_peered":"2024-03-18T14:52:07.371021+0000","last_clean":"2024-03-18T14:52:07.371021+0000","last_became_active":"2024-03-18T14:51:00.932928+0000","last_became_peered":"2024-03-18T14:51:00.932928+0000","last_unstale":"2024-03-18T14:52:07.371021+0000","last_undegraded":"2024-03-18T14:52:07.371021+0000","last_fullsized":"2024-03-18T14:52:07.371021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:11:55.281763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375943+0000","last_change":"2024-03-18T14:51:01.362911+0000","last_active":"2024-03-18T14:52:36.375943+0000","last_peered":"2024-03-18T14:52:36.375943+0000","last_clean":"2024-03-18T14:52:36.375943+0000","last_became_active":"2024-03-18T14:51:01.362598+0000","last_became_peered":"2024-03-18T14:51:01.362598+0000","last_unstale":"2024-03-18T14:52:36.375943+0000","last_undegraded":"2024-03-18T14:52:36.375943+0000","last_fullsized":"2024-03-18T14:52:36.375943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:22:58.627753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.363811+0000","last_change":"2024-03-18T14:51:00.935554+0000","last_active":"2024-03-18T14:52:07.363811+0000","last_peered":"2024-03-18T14:52:07.363811+0000","last_clean":"2024-03-18T14:52:07.363811+0000","last_became_active":"2024-03-18T14:51:00.935184+0000","last_became_peered":"2024-03-18T14:51:00.935184+0000","last_unstale":"2024-03-18T14:52:07.363811+0000","last_undegraded":"2024-03-18T14:52:07.363811+0000","last_fullsized":"2024-03-18T14:52:07.363811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:21:13.973570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.361967+0000","last_change":"2024-03-18T14:51:01.355990+0000","last_active":"2024-03-18T14:52:36.361967+0000","last_peered":"2024-03-18T14:52:36.361967+0000","last_clean":"2024-03-18T14:52:36.361967+0000","last_became_active":"2024-03-18T14:51:01.355692+0000","last_became_peered":"2024-03-18T14:51:01.355692+0000","last_unstale":"2024-03-18T14:52:36.361967+0000","last_undegraded":"2024-03-18T14:52:36.361967+0000","last_fullsized":"2024-03-18T14:52:36.361967+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:27:19.742684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366919+0000","last_change":"2024-03-18T14:51:00.943947+0000","last_active":"2024-03-18T14:52:07.366919+0000","last_peered":"2024-03-18T14:52:07.366919+0000","last_clean":"2024-03-18T14:52:07.366919+0000","last_became_active":"2024-03-18T14:51:00.943565+0000","last_became_peered":"2024-03-18T14:51:00.943565+0000","last_unstale":"2024-03-18T14:52:07.366919+0000","last_undegraded":"2024-03-18T14:52:07.366919+0000","last_fullsized":"2024-03-18T14:52:07.366919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:33:06.103148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.367260+0000","last_change":"2024-03-18T14:51:01.363221+0000","last_active":"2024-03-18T14:52:36.367260+0000","last_peered":"2024-03-18T14:52:36.367260+0000","last_clean":"2024-03-18T14:52:36.367260+0000","last_became_active":"2024-03-18T14:51:01.362848+0000","last_became_peered":"2024-03-18T14:51:01.362848+0000","last_unstale":"2024-03-18T14:52:36.367260+0000","last_undegraded":"2024-03-18T14:52:36.367260+0000","last_fullsized":"2024-03-18T14:52:36.367260+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:04.139365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.365329+0000","last_change":"2024-03-18T14:51:01.356237+0000","last_active":"2024-03-18T14:52:36.365329+0000","last_peered":"2024-03-18T14:52:36.365329+0000","last_clean":"2024-03-18T14:52:36.365329+0000","last_became_active":"2024-03-18T14:51:01.355922+0000","last_became_peered":"2024-03-18T14:51:01.355922+0000","last_unstale":"2024-03-18T14:52:36.365329+0000","last_undegraded":"2024-03-18T14:52:36.365329+0000","last_fullsized":"2024-03-18T14:52:36.365329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:04:50.491752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'6","reported_seq":145,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:53:05.168942+0000","last_change":"2024-03-18T14:51:00.936058+0000","last_active":"2024-03-18T14:53:05.168942+0000","last_peered":"2024-03-18T14:53:05.168942+0000","last_clean":"2024-03-18T14:53:05.168942+0000","last_became_active":"2024-03-18T14:51:00.935731+0000","last_became_peered":"2024-03-18T14:51:00.935731+0000","last_unstale":"2024-03-18T14:53:05.168942+0000","last_undegraded":"2024-03-18T14:53:05.168942+0000","last_fullsized":"2024-03-18T14:53:05.168942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:47:49.023943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.374426+0000","last_change":"2024-03-18T14:51:01.359742+0000","last_active":"2024-03-18T14:52:36.374426+0000","last_peered":"2024-03-18T14:52:36.374426+0000","last_clean":"2024-03-18T14:52:36.374426+0000","last_became_active":"2024-03-18T14:51:01.358507+0000","last_became_peered":"2024-03-18T14:51:01.358507+0000","last_unstale":"2024-03-18T14:52:36.374426+0000","last_undegraded":"2024-03-18T14:52:36.374426+0000","last_fullsized":"2024-03-18T14:52:36.374426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:17.129695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.419738+0000","last_change":"2024-03-18T14:51:00.934593+0000","last_active":"2024-03-18T14:52:07.419738+0000","last_peered":"2024-03-18T14:52:07.419738+0000","last_clean":"2024-03-18T14:52:07.419738+0000","last_became_active":"2024-03-18T14:51:00.934253+0000","last_became_peered":"2024-03-18T14:51:00.934253+0000","last_unstale":"2024-03-18T14:52:07.419738+0000","last_undegraded":"2024-03-18T14:52:07.419738+0000","last_fullsized":"2024-03-18T14:52:07.419738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:43:26.323162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":352,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366664+0000","last_change":"2024-03-18T14:49:51.158965+0000","last_active":"2024-03-18T14:52:07.366664+0000","last_peered":"2024-03-18T14:52:07.366664+0000","last_clean":"2024-03-18T14:52:07.366664+0000","last_became_active":"2024-03-18T14:49:51.158580+0000","last_became_peered":"2024-03-18T14:49:51.158580+0000","last_unstale":"2024-03-18T14:52:07.366664+0000","last_undegraded":"2024-03-18T14:52:07.366664+0000","last_fullsized":"2024-03-18T14:52:07.366664+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:46:27.210244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:46:27.210244+0000","last_clean_scrub_stamp":"2024-03-18T14:46:27.210244+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:37:08.104617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.359433+0000","last_change":"2024-03-18T14:51:01.359972+0000","last_active":"2024-03-18T14:52:36.359433+0000","last_peered":"2024-03-18T14:52:36.359433+0000","last_clean":"2024-03-18T14:52:36.359433+0000","last_became_active":"2024-03-18T14:51:01.359430+0000","last_became_peered":"2024-03-18T14:51:01.359430+0000","last_unstale":"2024-03-18T14:52:36.359433+0000","last_undegraded":"2024-03-18T14:52:36.359433+0000","last_fullsized":"2024-03-18T14:52:36.359433+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:00:14.099499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366772+0000","last_change":"2024-03-18T14:51:00.934196+0000","last_active":"2024-03-18T14:52:07.366772+0000","last_peered":"2024-03-18T14:52:07.366772+0000","last_clean":"2024-03-18T14:52:07.366772+0000","last_became_active":"2024-03-18T14:51:00.933875+0000","last_became_peered":"2024-03-18T14:51:00.933875+0000","last_unstale":"2024-03-18T14:52:07.366772+0000","last_undegraded":"2024-03-18T14:52:07.366772+0000","last_fullsized":"2024-03-18T14:52:07.366772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:56.797227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370852+0000","last_change":"2024-03-18T14:51:00.941862+0000","last_active":"2024-03-18T14:52:07.370852+0000","last_peered":"2024-03-18T14:52:07.370852+0000","last_clean":"2024-03-18T14:52:07.370852+0000","last_became_active":"2024-03-18T14:51:00.941536+0000","last_became_peered":"2024-03-18T14:51:00.941536+0000","last_unstale":"2024-03-18T14:52:07.370852+0000","last_undegraded":"2024-03-18T14:52:07.370852+0000","last_fullsized":"2024-03-18T14:52:07.370852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:56.364552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.359217+0000","last_change":"2024-03-18T14:51:01.352404+0000","last_active":"2024-03-18T14:52:36.359217+0000","last_peered":"2024-03-18T14:52:36.359217+0000","last_clean":"2024-03-18T14:52:36.359217+0000","last_became_active":"2024-03-18T14:51:01.352077+0000","last_became_peered":"2024-03-18T14:51:01.352077+0000","last_unstale":"2024-03-18T14:52:36.359217+0000","last_undegraded":"2024-03-18T14:52:36.359217+0000","last_fullsized":"2024-03-18T14:52:36.359217+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:30:38.050712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.364118+0000","last_change":"2024-03-18T14:51:00.935564+0000","last_active":"2024-03-18T14:52:07.364118+0000","last_peered":"2024-03-18T14:52:07.364118+0000","last_clean":"2024-03-18T14:52:07.364118+0000","last_became_active":"2024-03-18T14:51:00.935268+0000","last_became_peered":"2024-03-18T14:51:00.935268+0000","last_unstale":"2024-03-18T14:52:07.364118+0000","last_undegraded":"2024-03-18T14:52:07.364118+0000","last_fullsized":"2024-03-18T14:52:07.364118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:22:33.504184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.372551+0000","last_change":"2024-03-18T14:51:01.356096+0000","last_active":"2024-03-18T14:52:36.372551+0000","last_peered":"2024-03-18T14:52:36.372551+0000","last_clean":"2024-03-18T14:52:36.372551+0000","last_became_active":"2024-03-18T14:51:01.355688+0000","last_became_peered":"2024-03-18T14:51:01.355688+0000","last_unstale":"2024-03-18T14:52:36.372551+0000","last_undegraded":"2024-03-18T14:52:36.372551+0000","last_fullsized":"2024-03-18T14:52:36.372551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:04:38.678482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.419726+0000","last_change":"2024-03-18T14:51:00.942610+0000","last_active":"2024-03-18T14:52:07.419726+0000","last_peered":"2024-03-18T14:52:07.419726+0000","last_clean":"2024-03-18T14:52:07.419726+0000","last_became_active":"2024-03-18T14:51:00.942203+0000","last_became_peered":"2024-03-18T14:51:00.942203+0000","last_unstale":"2024-03-18T14:52:07.419726+0000","last_undegraded":"2024-03-18T14:52:07.419726+0000","last_fullsized":"2024-03-18T14:52:07.419726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:17:26.324282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.367722+0000","last_change":"2024-03-18T14:51:01.362931+0000","last_active":"2024-03-18T14:52:36.367722+0000","last_peered":"2024-03-18T14:52:36.367722+0000","last_clean":"2024-03-18T14:52:36.367722+0000","last_became_active":"2024-03-18T14:51:01.362624+0000","last_became_peered":"2024-03-18T14:51:01.362624+0000","last_unstale":"2024-03-18T14:52:36.367722+0000","last_undegraded":"2024-03-18T14:52:36.367722+0000","last_fullsized":"2024-03-18T14:52:36.367722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:40:52.512871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370722+0000","last_change":"2024-03-18T14:51:00.938971+0000","last_active":"2024-03-18T14:52:07.370722+0000","last_peered":"2024-03-18T14:52:07.370722+0000","last_clean":"2024-03-18T14:52:07.370722+0000","last_became_active":"2024-03-18T14:51:00.938645+0000","last_became_peered":"2024-03-18T14:51:00.938645+0000","last_unstale":"2024-03-18T14:52:07.370722+0000","last_undegraded":"2024-03-18T14:52:07.370722+0000","last_fullsized":"2024-03-18T14:52:07.370722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:18:54.999927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.334973+0000","last_change":"2024-03-18T14:51:01.359510+0000","last_active":"2024-03-18T14:52:36.334973+0000","last_peered":"2024-03-18T14:52:36.334973+0000","last_clean":"2024-03-18T14:52:36.334973+0000","last_became_active":"2024-03-18T14:51:01.359192+0000","last_became_peered":"2024-03-18T14:51:01.359192+0000","last_unstale":"2024-03-18T14:52:36.334973+0000","last_undegraded":"2024-03-18T14:52:36.334973+0000","last_fullsized":"2024-03-18T14:52:36.334973+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:11:52.942825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368576+0000","last_change":"2024-03-18T14:51:00.940194+0000","last_active":"2024-03-18T14:52:07.368576+0000","last_peered":"2024-03-18T14:52:07.368576+0000","last_clean":"2024-03-18T14:52:07.368576+0000","last_became_active":"2024-03-18T14:51:00.939861+0000","last_became_peered":"2024-03-18T14:51:00.939861+0000","last_unstale":"2024-03-18T14:52:07.368576+0000","last_undegraded":"2024-03-18T14:52:07.368576+0000","last_fullsized":"2024-03-18T14:52:07.368576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:06:30.314102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.367604+0000","last_change":"2024-03-18T14:51:01.355098+0000","last_active":"2024-03-18T14:52:36.367604+0000","last_peered":"2024-03-18T14:52:36.367604+0000","last_clean":"2024-03-18T14:52:36.367604+0000","last_became_active":"2024-03-18T14:51:01.354452+0000","last_became_peered":"2024-03-18T14:51:01.354452+0000","last_unstale":"2024-03-18T14:52:36.367604+0000","last_undegraded":"2024-03-18T14:52:36.367604+0000","last_fullsized":"2024-03-18T14:52:36.367604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:40:54.650294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.381529+0000","last_change":"2024-03-18T14:52:07.381159+0000","last_active":"2024-03-18T14:52:07.381529+0000","last_peered":"2024-03-18T14:52:07.381529+0000","last_clean":"2024-03-18T14:52:07.381529+0000","last_became_active":"2024-03-18T14:52:07.371943+0000","last_became_peered":"2024-03-18T14:52:07.371943+0000","last_unstale":"2024-03-18T14:52:07.381529+0000","last_undegraded":"2024-03-18T14:52:07.381529+0000","last_fullsized":"2024-03-18T14:52:07.381529+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:11.329239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.359825+0000","last_change":"2024-03-18T14:51:01.357920+0000","last_active":"2024-03-18T14:52:36.359825+0000","last_peered":"2024-03-18T14:52:36.359825+0000","last_clean":"2024-03-18T14:52:36.359825+0000","last_became_active":"2024-03-18T14:51:01.357565+0000","last_became_peered":"2024-03-18T14:51:01.357565+0000","last_unstale":"2024-03-18T14:52:36.359825+0000","last_undegraded":"2024-03-18T14:52:36.359825+0000","last_fullsized":"2024-03-18T14:52:36.359825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:39:57.880178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370714+0000","last_change":"2024-03-18T14:51:00.937887+0000","last_active":"2024-03-18T14:52:07.370714+0000","last_peered":"2024-03-18T14:52:07.370714+0000","last_clean":"2024-03-18T14:52:07.370714+0000","last_became_active":"2024-03-18T14:51:00.937588+0000","last_became_peered":"2024-03-18T14:51:00.937588+0000","last_unstale":"2024-03-18T14:52:07.370714+0000","last_undegraded":"2024-03-18T14:52:07.370714+0000","last_fullsized":"2024-03-18T14:52:07.370714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:43:37.558412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375612+0000","last_change":"2024-03-18T14:51:01.354175+0000","last_active":"2024-03-18T14:52:36.375612+0000","last_peered":"2024-03-18T14:52:36.375612+0000","last_clean":"2024-03-18T14:52:36.375612+0000","last_became_active":"2024-03-18T14:51:01.353839+0000","last_became_peered":"2024-03-18T14:51:01.353839+0000","last_unstale":"2024-03-18T14:52:36.375612+0000","last_undegraded":"2024-03-18T14:52:36.375612+0000","last_fullsized":"2024-03-18T14:52:36.375612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:21:52.372938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369167+0000","last_change":"2024-03-18T14:51:00.941600+0000","last_active":"2024-03-18T14:52:07.369167+0000","last_peered":"2024-03-18T14:52:07.369167+0000","last_clean":"2024-03-18T14:52:07.369167+0000","last_became_active":"2024-03-18T14:51:00.941291+0000","last_became_peered":"2024-03-18T14:51:00.941291+0000","last_unstale":"2024-03-18T14:52:07.369167+0000","last_undegraded":"2024-03-18T14:52:07.369167+0000","last_fullsized":"2024-03-18T14:52:07.369167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:04:11.142029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.376178+0000","last_change":"2024-03-18T14:51:01.359058+0000","last_active":"2024-03-18T14:52:36.376178+0000","last_peered":"2024-03-18T14:52:36.376178+0000","last_clean":"2024-03-18T14:52:36.376178+0000","last_became_active":"2024-03-18T14:51:01.358437+0000","last_became_peered":"2024-03-18T14:51:01.358437+0000","last_unstale":"2024-03-18T14:52:36.376178+0000","last_undegraded":"2024-03-18T14:52:36.376178+0000","last_fullsized":"2024-03-18T14:52:36.376178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:39:58.945080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.424033+0000","last_change":"2024-03-18T14:51:00.933647+0000","last_active":"2024-03-18T14:52:07.424033+0000","last_peered":"2024-03-18T14:52:07.424033+0000","last_clean":"2024-03-18T14:52:07.424033+0000","last_became_active":"2024-03-18T14:51:00.933320+0000","last_became_peered":"2024-03-18T14:51:00.933320+0000","last_unstale":"2024-03-18T14:52:07.424033+0000","last_undegraded":"2024-03-18T14:52:07.424033+0000","last_fullsized":"2024-03-18T14:52:07.424033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:16:46.498845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.370719+0000","last_change":"2024-03-18T14:51:01.359874+0000","last_active":"2024-03-18T14:52:36.370719+0000","last_peered":"2024-03-18T14:52:36.370719+0000","last_clean":"2024-03-18T14:52:36.370719+0000","last_became_active":"2024-03-18T14:51:01.359335+0000","last_became_peered":"2024-03-18T14:51:01.359335+0000","last_unstale":"2024-03-18T14:52:36.370719+0000","last_undegraded":"2024-03-18T14:52:36.370719+0000","last_fullsized":"2024-03-18T14:52:36.370719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:03.583402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481395+0000","last_change":"2024-03-18T14:51:00.935755+0000","last_active":"2024-03-18T14:52:07.481395+0000","last_peered":"2024-03-18T14:52:07.481395+0000","last_clean":"2024-03-18T14:52:07.481395+0000","last_became_active":"2024-03-18T14:51:00.935421+0000","last_became_peered":"2024-03-18T14:51:00.935421+0000","last_unstale":"2024-03-18T14:52:07.481395+0000","last_undegraded":"2024-03-18T14:52:07.481395+0000","last_fullsized":"2024-03-18T14:52:07.481395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:46:11.680972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.373520+0000","last_change":"2024-03-18T14:51:01.355605+0000","last_active":"2024-03-18T14:52:36.373520+0000","last_peered":"2024-03-18T14:52:36.373520+0000","last_clean":"2024-03-18T14:52:36.373520+0000","last_became_active":"2024-03-18T14:51:01.355295+0000","last_became_peered":"2024-03-18T14:51:01.355295+0000","last_unstale":"2024-03-18T14:52:36.373520+0000","last_undegraded":"2024-03-18T14:52:36.373520+0000","last_fullsized":"2024-03-18T14:52:36.373520+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:15:20.466314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.379370+0000","last_change":"2024-03-18T14:51:01.359704+0000","last_active":"2024-03-18T14:52:36.379370+0000","last_peered":"2024-03-18T14:52:36.379370+0000","last_clean":"2024-03-18T14:52:36.379370+0000","last_became_active":"2024-03-18T14:51:01.359367+0000","last_became_peered":"2024-03-18T14:51:01.359367+0000","last_unstale":"2024-03-18T14:52:36.379370+0000","last_undegraded":"2024-03-18T14:52:36.379370+0000","last_fullsized":"2024-03-18T14:52:36.379370+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:26.933993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.365339+0000","last_change":"2024-03-18T14:52:07.420506+0000","last_active":"2024-03-18T14:52:36.365339+0000","last_peered":"2024-03-18T14:52:36.365339+0000","last_clean":"2024-03-18T14:52:36.365339+0000","last_became_active":"2024-03-18T14:52:07.420254+0000","last_became_peered":"2024-03-18T14:52:07.420254+0000","last_unstale":"2024-03-18T14:52:36.365339+0000","last_undegraded":"2024-03-18T14:52:36.365339+0000","last_fullsized":"2024-03-18T14:52:36.365339+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:32:08.702462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.369110+0000","last_change":"2024-03-18T14:51:01.355678+0000","last_active":"2024-03-18T14:52:36.369110+0000","last_peered":"2024-03-18T14:52:36.369110+0000","last_clean":"2024-03-18T14:52:36.369110+0000","last_became_active":"2024-03-18T14:51:01.355197+0000","last_became_peered":"2024-03-18T14:51:01.355197+0000","last_unstale":"2024-03-18T14:52:36.369110+0000","last_undegraded":"2024-03-18T14:52:36.369110+0000","last_fullsized":"2024-03-18T14:52:36.369110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:34.194834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.366017+0000","last_change":"2024-03-18T14:51:01.363964+0000","last_active":"2024-03-18T14:52:36.366017+0000","last_peered":"2024-03-18T14:52:36.366017+0000","last_clean":"2024-03-18T14:52:36.366017+0000","last_became_active":"2024-03-18T14:51:01.363590+0000","last_became_peered":"2024-03-18T14:51:01.363590+0000","last_unstale":"2024-03-18T14:52:36.366017+0000","last_undegraded":"2024-03-18T14:52:36.366017+0000","last_fullsized":"2024-03-18T14:52:36.366017+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:59.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.366199+0000","last_change":"2024-03-18T14:51:01.361599+0000","last_active":"2024-03-18T14:52:36.366199+0000","last_peered":"2024-03-18T14:52:36.366199+0000","last_clean":"2024-03-18T14:52:36.366199+0000","last_became_active":"2024-03-18T14:51:01.361280+0000","last_became_peered":"2024-03-18T14:51:01.361280+0000","last_unstale":"2024-03-18T14:52:36.366199+0000","last_undegraded":"2024-03-18T14:52:36.366199+0000","last_fullsized":"2024-03-18T14:52:36.366199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:52:50.572257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.376675+0000","last_change":"2024-03-18T14:51:01.360495+0000","last_active":"2024-03-18T14:52:36.376675+0000","last_peered":"2024-03-18T14:52:36.376675+0000","last_clean":"2024-03-18T14:52:36.376675+0000","last_became_active":"2024-03-18T14:51:01.360146+0000","last_became_peered":"2024-03-18T14:51:01.360146+0000","last_unstale":"2024-03-18T14:52:36.376675+0000","last_undegraded":"2024-03-18T14:52:36.376675+0000","last_fullsized":"2024-03-18T14:52:36.376675+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:52.048137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.363452+0000","last_change":"2024-03-18T14:51:01.357650+0000","last_active":"2024-03-18T14:52:36.363452+0000","last_peered":"2024-03-18T14:52:36.363452+0000","last_clean":"2024-03-18T14:52:36.363452+0000","last_became_active":"2024-03-18T14:51:01.356265+0000","last_became_peered":"2024-03-18T14:51:01.356265+0000","last_unstale":"2024-03-18T14:52:36.363452+0000","last_undegraded":"2024-03-18T14:52:36.363452+0000","last_fullsized":"2024-03-18T14:52:36.363452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:49.328404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.357983+0000","last_change":"2024-03-18T14:51:01.363753+0000","last_active":"2024-03-18T14:52:36.357983+0000","last_peered":"2024-03-18T14:52:36.357983+0000","last_clean":"2024-03-18T14:52:36.357983+0000","last_became_active":"2024-03-18T14:51:01.363411+0000","last_became_peered":"2024-03-18T14:51:01.363411+0000","last_unstale":"2024-03-18T14:52:36.357983+0000","last_undegraded":"2024-03-18T14:52:36.357983+0000","last_fullsized":"2024-03-18T14:52:36.357983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:55.452209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.365639+0000","last_change":"2024-03-18T14:51:01.360723+0000","last_active":"2024-03-18T14:52:36.365639+0000","last_peered":"2024-03-18T14:52:36.365639+0000","last_clean":"2024-03-18T14:52:36.365639+0000","last_became_active":"2024-03-18T14:51:01.360130+0000","last_became_peered":"2024-03-18T14:51:01.360130+0000","last_unstale":"2024-03-18T14:52:36.365639+0000","last_undegraded":"2024-03-18T14:52:36.365639+0000","last_fullsized":"2024-03-18T14:52:36.365639+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:51.634011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.372926+0000","last_change":"2024-03-18T14:51:01.356462+0000","last_active":"2024-03-18T14:52:36.372926+0000","last_peered":"2024-03-18T14:52:36.372926+0000","last_clean":"2024-03-18T14:52:36.372926+0000","last_became_active":"2024-03-18T14:51:01.355395+0000","last_became_peered":"2024-03-18T14:51:01.355395+0000","last_unstale":"2024-03-18T14:52:36.372926+0000","last_undegraded":"2024-03-18T14:52:36.372926+0000","last_fullsized":"2024-03-18T14:52:36.372926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:37:15.326354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.376909+0000","last_change":"2024-03-18T14:51:01.358626+0000","last_active":"2024-03-18T14:52:36.376909+0000","last_peered":"2024-03-18T14:52:36.376909+0000","last_clean":"2024-03-18T14:52:36.376909+0000","last_became_active":"2024-03-18T14:51:01.358293+0000","last_became_peered":"2024-03-18T14:51:01.358293+0000","last_unstale":"2024-03-18T14:52:36.376909+0000","last_undegraded":"2024-03-18T14:52:36.376909+0000","last_fullsized":"2024-03-18T14:52:36.376909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:52:41.139885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375009+0000","last_change":"2024-03-18T14:51:01.355276+0000","last_active":"2024-03-18T14:52:36.375009+0000","last_peered":"2024-03-18T14:52:36.375009+0000","last_clean":"2024-03-18T14:52:36.375009+0000","last_became_active":"2024-03-18T14:51:01.354661+0000","last_became_peered":"2024-03-18T14:51:01.354661+0000","last_unstale":"2024-03-18T14:52:36.375009+0000","last_undegraded":"2024-03-18T14:52:36.375009+0000","last_fullsized":"2024-03-18T14:52:36.375009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:28.569055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.377238+0000","last_change":"2024-03-18T14:52:07.372297+0000","last_active":"2024-03-18T14:52:36.377238+0000","last_peered":"2024-03-18T14:52:36.377238+0000","last_clean":"2024-03-18T14:52:36.377238+0000","last_became_active":"2024-03-18T14:52:07.371918+0000","last_became_peered":"2024-03-18T14:52:07.371918+0000","last_unstale":"2024-03-18T14:52:36.377238+0000","last_undegraded":"2024-03-18T14:52:36.377238+0000","last_fullsized":"2024-03-18T14:52:36.377238+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:28:23.431373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280632,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63919,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":727,"num_read_kb":430,"num_write":167,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":58912,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":200385,"data_compressed":13263,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351014,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":459161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612650,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":1110643,"data_compressed":37553,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776176,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037813,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":459317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":1051696,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462975,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":458786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":518048,"data_compressed":1859,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":460057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":459929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":459097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888138,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":523846,"data_compressed":9545,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724612,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":1050540,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63783,"data_compressed":1859,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":65267,"data_compressed":9545,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":61887,"data_compressed":1859,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:53:07.007 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-18T14:53:07.007 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:53:07.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.145+0000 7ff807868640 1 -- 172.21.15.76:0/4151082452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8001022b0 msgr2=0x7ff800108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:07.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.145+0000 7ff807868640 1 --2- 172.21.15.76:0/4151082452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8001022b0 0x7ff800108b40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8009900 tx=0x7ff7f802bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:07.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.145+0000 7ff807868640 1 -- 172.21.15.76:0/4151082452 shutdown_connections 2024-03-18T14:53:07.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.145+0000 7ff807868640 1 --2- 172.21.15.76:0/4151082452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8001022b0 0x7ff800108b40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.145+0000 7ff807868640 1 --2- 172.21.15.76:0/4151082452 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8001018f0 0x7ff800101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.145+0000 7ff807868640 1 --2- 172.21.15.76:0/4151082452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff800100700 0x7ff800100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.145+0000 7ff807868640 1 -- 172.21.15.76:0/4151082452 >> 172.21.15.76:0/4151082452 conn(0x7ff8000fbc70 msgr2=0x7ff8000fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:07.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.145+0000 7ff807868640 1 -- 172.21.15.76:0/4151082452 shutdown_connections 2024-03-18T14:53:07.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.145+0000 7ff807868640 1 -- 172.21.15.76:0/4151082452 wait complete. 2024-03-18T14:53:07.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.145+0000 7ff807868640 1 Processor -- start 2024-03-18T14:53:07.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.145+0000 7ff807868640 1 -- start start 2024-03-18T14:53:07.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.145+0000 7ff807868640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff800100700 0x7ff800197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:07.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.145+0000 7ff807868640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8001018f0 0x7ff800197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:07.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.145+0000 7ff807868640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8001022b0 0x7ff80019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:07.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.145+0000 7ff807868640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff80010d4a0 con 0x7ff8001022b0 2024-03-18T14:53:07.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.145+0000 7ff807868640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff80010d320 con 0x7ff800100700 2024-03-18T14:53:07.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.145+0000 7ff807868640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff80010d620 con 0x7ff8001018f0 2024-03-18T14:53:07.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.146+0000 7ff805e05640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8001022b0 0x7ff80019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:07.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.146+0000 7ff805e05640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8001022b0 0x7ff80019f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50426/0 (socket says 172.21.15.76:50426) 2024-03-18T14:53:07.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.146+0000 7ff805604640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff800100700 0x7ff800197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:07.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.146+0000 7ff805e05640 1 -- 172.21.15.76:0/3294980130 learned_addr learned my addr 172.21.15.76:0/3294980130 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:07.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.146+0000 7ff804e03640 1 --2- 172.21.15.76:0/3294980130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8001018f0 0x7ff800197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:07.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.146+0000 7ff805e05640 1 -- 172.21.15.76:0/3294980130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8001018f0 msgr2=0x7ff800197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:07.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.146+0000 7ff805e05640 1 --2- 172.21.15.76:0/3294980130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8001018f0 0x7ff800197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.146+0000 7ff805e05640 1 -- 172.21.15.76:0/3294980130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff800100700 msgr2=0x7ff800197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:07.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.146+0000 7ff805e05640 1 --2- 172.21.15.76:0/3294980130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff800100700 0x7ff800197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.146+0000 7ff805e05640 1 -- 172.21.15.76:0/3294980130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80019f6c0 con 0x7ff8001022b0 2024-03-18T14:53:07.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.146+0000 7ff805604640 1 --2- 172.21.15.76:0/3294980130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff800100700 0x7ff800197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:07.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.146+0000 7ff805e05640 1 --2- 172.21.15.76:0/3294980130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8001022b0 0x7ff80019f050 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8005ab0 tx=0x7ff7f802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:07.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.146+0000 7ff7f67fc640 1 -- 172.21.15.76:0/3294980130 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8041070 con 0x7ff8001022b0 2024-03-18T14:53:07.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.146+0000 7ff7f67fc640 1 -- 172.21.15.76:0/3294980130 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7f8004450 con 0x7ff8001022b0 2024-03-18T14:53:07.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.146+0000 7ff807868640 1 -- 172.21.15.76:0/3294980130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff80019f950 con 0x7ff8001022b0 2024-03-18T14:53:07.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.146+0000 7ff7f67fc640 1 -- 172.21.15.76:0/3294980130 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8003e60 con 0x7ff8001022b0 2024-03-18T14:53:07.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.146+0000 7ff807868640 1 -- 172.21.15.76:0/3294980130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff80019fe40 con 0x7ff8001022b0 2024-03-18T14:53:07.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.147+0000 7ff807868640 1 -- 172.21.15.76:0/3294980130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8001079c0 con 0x7ff8001022b0 2024-03-18T14:53:07.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.148+0000 7ff7f67fc640 1 -- 172.21.15.76:0/3294980130 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff7f8044070 con 0x7ff8001022b0 2024-03-18T14:53:07.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.149+0000 7ff7f67fc640 1 --2- 172.21.15.76:0/3294980130 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff7ec0a2800 0x7ff7ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:07.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.149+0000 7ff805604640 1 --2- 172.21.15.76:0/3294980130 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff7ec0a2800 0x7ff7ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:07.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.149+0000 7ff7f67fc640 1 -- 172.21.15.76:0/3294980130 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7ff7f80f0650 con 0x7ff8001022b0 2024-03-18T14:53:07.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.149+0000 7ff805604640 1 --2- 172.21.15.76:0/3294980130 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff7ec0a2800 0x7ff7ec0a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff800101750 tx=0x7ff7fc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:07.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.153+0000 7ff7f67fc640 1 -- 172.21.15.76:0/3294980130 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff7f80bd0d0 con 0x7ff8001022b0 2024-03-18T14:53:07.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.325+0000 7ff807868640 1 -- 172.21.15.76:0/3294980130 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff800061ce0 con 0x7ff7ec0a2800 2024-03-18T14:53:07.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.332+0000 7ff7f67fc640 1 -- 172.21.15.76:0/3294980130 <== mgr.14152 v2:172.21.15.76:6800/3010549595 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259063 (secure 0 0 0) 0x7ff800061ce0 con 0x7ff7ec0a2800 2024-03-18T14:53:07.335 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:53:07.336 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-03-18T14:53:07.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.335+0000 7ff807868640 1 -- 172.21.15.76:0/3294980130 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff7ec0a2800 msgr2=0x7ff7ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:07.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.335+0000 7ff807868640 1 --2- 172.21.15.76:0/3294980130 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff7ec0a2800 0x7ff7ec0a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff800101750 tx=0x7ff7fc009500 comp rx=0 tx=0).stop 2024-03-18T14:53:07.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.335+0000 7ff807868640 1 -- 172.21.15.76:0/3294980130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8001022b0 msgr2=0x7ff80019f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:07.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.335+0000 7ff807868640 1 --2- 172.21.15.76:0/3294980130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8001022b0 0x7ff80019f050 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8005ab0 tx=0x7ff7f802c6c0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.335+0000 7ff807868640 1 -- 172.21.15.76:0/3294980130 shutdown_connections 2024-03-18T14:53:07.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.335+0000 7ff807868640 1 --2- 172.21.15.76:0/3294980130 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7ff7ec0a2800 0x7ff7ec0a4cc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.335+0000 7ff807868640 1 --2- 172.21.15.76:0/3294980130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8001018f0 0x7ff800197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.335+0000 7ff807868640 1 --2- 172.21.15.76:0/3294980130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8001022b0 0x7ff80019f050 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.335+0000 7ff807868640 1 --2- 172.21.15.76:0/3294980130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff800100700 0x7ff800197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.335+0000 7ff807868640 1 -- 172.21.15.76:0/3294980130 >> 172.21.15.76:0/3294980130 conn(0x7ff8000fbc70 msgr2=0x7ff800104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:07.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.335+0000 7ff807868640 1 -- 172.21.15.76:0/3294980130 shutdown_connections 2024-03-18T14:53:07.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.335+0000 7ff807868640 1 -- 172.21.15.76:0/3294980130 wait complete. 2024-03-18T14:53:07.347 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-03-18T14:53:06.359535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654521,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":863,"num_read_kb":549,"num_write":350,"num_write_kb":3055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1239648,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":783,"ondisk_log_size":783,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369088,"kb_used_data":11616,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572736,"statfs":{"total":1151940427776,"available":1151562481664,"internally_reserved":0,"allocated":11894784,"data_stored":7470517,"data_compressed":120345,"data_compressed_allocated":155648,"data_compressed_original":1843200,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":36,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002927"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.371528+0000","last_change":"2024-03-18T14:51:01.365948+0000","last_active":"2024-03-18T14:52:36.371528+0000","last_peered":"2024-03-18T14:52:36.371528+0000","last_clean":"2024-03-18T14:52:36.371528+0000","last_became_active":"2024-03-18T14:51:01.364600+0000","last_became_peered":"2024-03-18T14:51:01.364600+0000","last_unstale":"2024-03-18T14:52:36.371528+0000","last_undegraded":"2024-03-18T14:52:36.371528+0000","last_fullsized":"2024-03-18T14:52:36.371528+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:14:45.198618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.359354+0000","last_change":"2024-03-18T14:51:01.360186+0000","last_active":"2024-03-18T14:52:36.359354+0000","last_peered":"2024-03-18T14:52:36.359354+0000","last_clean":"2024-03-18T14:52:36.359354+0000","last_became_active":"2024-03-18T14:51:01.359862+0000","last_became_peered":"2024-03-18T14:51:01.359862+0000","last_unstale":"2024-03-18T14:52:36.359354+0000","last_undegraded":"2024-03-18T14:52:36.359354+0000","last_fullsized":"2024-03-18T14:52:36.359354+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:46.206823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375396+0000","last_change":"2024-03-18T14:51:01.364635+0000","last_active":"2024-03-18T14:52:36.375396+0000","last_peered":"2024-03-18T14:52:36.375396+0000","last_clean":"2024-03-18T14:52:36.375396+0000","last_became_active":"2024-03-18T14:51:01.364320+0000","last_became_peered":"2024-03-18T14:51:01.364320+0000","last_unstale":"2024-03-18T14:52:36.375396+0000","last_undegraded":"2024-03-18T14:52:36.375396+0000","last_fullsized":"2024-03-18T14:52:36.375396+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:08:20.227639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.358624+0000","last_change":"2024-03-18T14:52:07.372090+0000","last_active":"2024-03-18T14:52:36.358624+0000","last_peered":"2024-03-18T14:52:36.358624+0000","last_clean":"2024-03-18T14:52:36.358624+0000","last_became_active":"2024-03-18T14:52:07.371775+0000","last_became_peered":"2024-03-18T14:52:07.371775+0000","last_unstale":"2024-03-18T14:52:36.358624+0000","last_undegraded":"2024-03-18T14:52:36.358624+0000","last_fullsized":"2024-03-18T14:52:36.358624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:48.128827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.356734+0000","last_change":"2024-03-18T14:51:01.356839+0000","last_active":"2024-03-18T14:52:36.356734+0000","last_peered":"2024-03-18T14:52:36.356734+0000","last_clean":"2024-03-18T14:52:36.356734+0000","last_became_active":"2024-03-18T14:51:01.356531+0000","last_became_peered":"2024-03-18T14:51:01.356531+0000","last_unstale":"2024-03-18T14:52:36.356734+0000","last_undegraded":"2024-03-18T14:52:36.356734+0000","last_fullsized":"2024-03-18T14:52:36.356734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:28.499573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.357939+0000","last_change":"2024-03-18T14:51:01.362912+0000","last_active":"2024-03-18T14:52:36.357939+0000","last_peered":"2024-03-18T14:52:36.357939+0000","last_clean":"2024-03-18T14:52:36.357939+0000","last_became_active":"2024-03-18T14:51:01.362590+0000","last_became_peered":"2024-03-18T14:51:01.362590+0000","last_unstale":"2024-03-18T14:52:36.357939+0000","last_undegraded":"2024-03-18T14:52:36.357939+0000","last_fullsized":"2024-03-18T14:52:36.357939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:49.199616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.370814+0000","last_change":"2024-03-18T14:51:01.358652+0000","last_active":"2024-03-18T14:52:36.370814+0000","last_peered":"2024-03-18T14:52:36.370814+0000","last_clean":"2024-03-18T14:52:36.370814+0000","last_became_active":"2024-03-18T14:51:01.358312+0000","last_became_peered":"2024-03-18T14:51:01.358312+0000","last_unstale":"2024-03-18T14:52:36.370814+0000","last_undegraded":"2024-03-18T14:52:36.370814+0000","last_fullsized":"2024-03-18T14:52:36.370814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:40.150240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.367332+0000","last_change":"2024-03-18T14:51:01.360082+0000","last_active":"2024-03-18T14:52:36.367332+0000","last_peered":"2024-03-18T14:52:36.367332+0000","last_clean":"2024-03-18T14:52:36.367332+0000","last_became_active":"2024-03-18T14:51:01.359769+0000","last_became_peered":"2024-03-18T14:51:01.359769+0000","last_unstale":"2024-03-18T14:52:36.367332+0000","last_undegraded":"2024-03-18T14:52:36.367332+0000","last_fullsized":"2024-03-18T14:52:36.367332+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:53:27.081546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.370348+0000","last_change":"2024-03-18T14:51:01.363947+0000","last_active":"2024-03-18T14:52:36.370348+0000","last_peered":"2024-03-18T14:52:36.370348+0000","last_clean":"2024-03-18T14:52:36.370348+0000","last_became_active":"2024-03-18T14:51:01.363106+0000","last_became_peered":"2024-03-18T14:51:01.363106+0000","last_unstale":"2024-03-18T14:52:36.370348+0000","last_undegraded":"2024-03-18T14:52:36.370348+0000","last_fullsized":"2024-03-18T14:52:36.370348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:24:50.478426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.363104+0000","last_change":"2024-03-18T14:51:01.357950+0000","last_active":"2024-03-18T14:52:36.363104+0000","last_peered":"2024-03-18T14:52:36.363104+0000","last_clean":"2024-03-18T14:52:36.363104+0000","last_became_active":"2024-03-18T14:51:01.357651+0000","last_became_peered":"2024-03-18T14:51:01.357651+0000","last_unstale":"2024-03-18T14:52:36.363104+0000","last_undegraded":"2024-03-18T14:52:36.363104+0000","last_fullsized":"2024-03-18T14:52:36.363104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:49:01.376008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375514+0000","last_change":"2024-03-18T14:51:01.360069+0000","last_active":"2024-03-18T14:52:36.375514+0000","last_peered":"2024-03-18T14:52:36.375514+0000","last_clean":"2024-03-18T14:52:36.375514+0000","last_became_active":"2024-03-18T14:51:01.358960+0000","last_became_peered":"2024-03-18T14:51:01.358960+0000","last_unstale":"2024-03-18T14:52:36.375514+0000","last_undegraded":"2024-03-18T14:52:36.375514+0000","last_fullsized":"2024-03-18T14:52:36.375514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:48:42.197329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.364663+0000","last_change":"2024-03-18T14:51:01.361200+0000","last_active":"2024-03-18T14:52:36.364663+0000","last_peered":"2024-03-18T14:52:36.364663+0000","last_clean":"2024-03-18T14:52:36.364663+0000","last_became_active":"2024-03-18T14:51:01.360879+0000","last_became_peered":"2024-03-18T14:51:01.360879+0000","last_unstale":"2024-03-18T14:52:36.364663+0000","last_undegraded":"2024-03-18T14:52:36.364663+0000","last_fullsized":"2024-03-18T14:52:36.364663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:05.335627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.364978+0000","last_change":"2024-03-18T14:51:01.352441+0000","last_active":"2024-03-18T14:52:36.364978+0000","last_peered":"2024-03-18T14:52:36.364978+0000","last_clean":"2024-03-18T14:52:36.364978+0000","last_became_active":"2024-03-18T14:51:01.352118+0000","last_became_peered":"2024-03-18T14:51:01.352118+0000","last_unstale":"2024-03-18T14:52:36.364978+0000","last_undegraded":"2024-03-18T14:52:36.364978+0000","last_fullsized":"2024-03-18T14:52:36.364978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:45:42.781439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.364844+0000","last_change":"2024-03-18T14:51:01.361257+0000","last_active":"2024-03-18T14:52:36.364844+0000","last_peered":"2024-03-18T14:52:36.364844+0000","last_clean":"2024-03-18T14:52:36.364844+0000","last_became_active":"2024-03-18T14:51:01.360898+0000","last_became_peered":"2024-03-18T14:51:01.360898+0000","last_unstale":"2024-03-18T14:52:36.364844+0000","last_undegraded":"2024-03-18T14:52:36.364844+0000","last_fullsized":"2024-03-18T14:52:36.364844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:38:26.200678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.367432+0000","last_change":"2024-03-18T14:52:07.374822+0000","last_active":"2024-03-18T14:52:36.367432+0000","last_peered":"2024-03-18T14:52:36.367432+0000","last_clean":"2024-03-18T14:52:36.367432+0000","last_became_active":"2024-03-18T14:52:07.374537+0000","last_became_peered":"2024-03-18T14:52:07.374537+0000","last_unstale":"2024-03-18T14:52:36.367432+0000","last_undegraded":"2024-03-18T14:52:36.367432+0000","last_fullsized":"2024-03-18T14:52:36.367432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:57:29.971366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.374959+0000","last_change":"2024-03-18T14:51:01.357223+0000","last_active":"2024-03-18T14:52:36.374959+0000","last_peered":"2024-03-18T14:52:36.374959+0000","last_clean":"2024-03-18T14:52:36.374959+0000","last_became_active":"2024-03-18T14:51:01.356934+0000","last_became_peered":"2024-03-18T14:51:01.356934+0000","last_unstale":"2024-03-18T14:52:36.374959+0000","last_undegraded":"2024-03-18T14:52:36.374959+0000","last_fullsized":"2024-03-18T14:52:36.374959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:28.683605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.372986+0000","last_change":"2024-03-18T14:51:01.366414+0000","last_active":"2024-03-18T14:52:36.372986+0000","last_peered":"2024-03-18T14:52:36.372986+0000","last_clean":"2024-03-18T14:52:36.372986+0000","last_became_active":"2024-03-18T14:51:01.365534+0000","last_became_peered":"2024-03-18T14:51:01.365534+0000","last_unstale":"2024-03-18T14:52:36.372986+0000","last_undegraded":"2024-03-18T14:52:36.372986+0000","last_fullsized":"2024-03-18T14:52:36.372986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:28:51.568662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.368646+0000","last_change":"2024-03-18T14:51:01.354084+0000","last_active":"2024-03-18T14:52:36.368646+0000","last_peered":"2024-03-18T14:52:36.368646+0000","last_clean":"2024-03-18T14:52:36.368646+0000","last_became_active":"2024-03-18T14:51:01.353599+0000","last_became_peered":"2024-03-18T14:51:01.353599+0000","last_unstale":"2024-03-18T14:52:36.368646+0000","last_undegraded":"2024-03-18T14:52:36.368646+0000","last_fullsized":"2024-03-18T14:52:36.368646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:40.502783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.366302+0000","last_change":"2024-03-18T14:51:01.353391+0000","last_active":"2024-03-18T14:52:36.366302+0000","last_peered":"2024-03-18T14:52:36.366302+0000","last_clean":"2024-03-18T14:52:36.366302+0000","last_became_active":"2024-03-18T14:51:01.353084+0000","last_became_peered":"2024-03-18T14:51:01.353084+0000","last_unstale":"2024-03-18T14:52:36.366302+0000","last_undegraded":"2024-03-18T14:52:36.366302+0000","last_fullsized":"2024-03-18T14:52:36.366302+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:18:02.774589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.363884+0000","last_change":"2024-03-18T14:51:00.942617+0000","last_active":"2024-03-18T14:52:07.363884+0000","last_peered":"2024-03-18T14:52:07.363884+0000","last_clean":"2024-03-18T14:52:07.363884+0000","last_became_active":"2024-03-18T14:51:00.942276+0000","last_became_peered":"2024-03-18T14:51:00.942276+0000","last_unstale":"2024-03-18T14:52:07.363884+0000","last_undegraded":"2024-03-18T14:52:07.363884+0000","last_fullsized":"2024-03-18T14:52:07.363884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:03:05.984066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.366394+0000","last_change":"2024-03-18T14:51:01.363887+0000","last_active":"2024-03-18T14:52:36.366394+0000","last_peered":"2024-03-18T14:52:36.366394+0000","last_clean":"2024-03-18T14:52:36.366394+0000","last_became_active":"2024-03-18T14:51:01.361359+0000","last_became_peered":"2024-03-18T14:51:01.361359+0000","last_unstale":"2024-03-18T14:52:36.366394+0000","last_undegraded":"2024-03-18T14:52:36.366394+0000","last_fullsized":"2024-03-18T14:52:36.366394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:22.996930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368943+0000","last_change":"2024-03-18T14:51:00.940392+0000","last_active":"2024-03-18T14:52:07.368943+0000","last_peered":"2024-03-18T14:52:07.368943+0000","last_clean":"2024-03-18T14:52:07.368943+0000","last_became_active":"2024-03-18T14:51:00.940055+0000","last_became_peered":"2024-03-18T14:51:00.940055+0000","last_unstale":"2024-03-18T14:52:07.368943+0000","last_undegraded":"2024-03-18T14:52:07.368943+0000","last_fullsized":"2024-03-18T14:52:07.368943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:57:28.523549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375190+0000","last_change":"2024-03-18T14:51:01.361458+0000","last_active":"2024-03-18T14:52:36.375190+0000","last_peered":"2024-03-18T14:52:36.375190+0000","last_clean":"2024-03-18T14:52:36.375190+0000","last_became_active":"2024-03-18T14:51:01.361105+0000","last_became_peered":"2024-03-18T14:51:01.361105+0000","last_unstale":"2024-03-18T14:52:36.375190+0000","last_undegraded":"2024-03-18T14:52:36.375190+0000","last_fullsized":"2024-03-18T14:52:36.375190+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:22:59.410080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.379071+0000","last_change":"2024-03-18T14:52:07.378642+0000","last_active":"2024-03-18T14:52:07.379071+0000","last_peered":"2024-03-18T14:52:07.379071+0000","last_clean":"2024-03-18T14:52:07.379071+0000","last_became_active":"2024-03-18T14:52:07.371947+0000","last_became_peered":"2024-03-18T14:52:07.371947+0000","last_unstale":"2024-03-18T14:52:07.379071+0000","last_undegraded":"2024-03-18T14:52:07.379071+0000","last_fullsized":"2024-03-18T14:52:07.379071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:46:00.898462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.371790+0000","last_change":"2024-03-18T14:51:01.366305+0000","last_active":"2024-03-18T14:52:36.371790+0000","last_peered":"2024-03-18T14:52:36.371790+0000","last_clean":"2024-03-18T14:52:36.371790+0000","last_became_active":"2024-03-18T14:51:01.364626+0000","last_became_peered":"2024-03-18T14:51:01.364626+0000","last_unstale":"2024-03-18T14:52:36.371790+0000","last_undegraded":"2024-03-18T14:52:36.371790+0000","last_fullsized":"2024-03-18T14:52:36.371790+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:06.212030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370835+0000","last_change":"2024-03-18T14:51:00.932514+0000","last_active":"2024-03-18T14:52:07.370835+0000","last_peered":"2024-03-18T14:52:07.370835+0000","last_clean":"2024-03-18T14:52:07.370835+0000","last_became_active":"2024-03-18T14:51:00.932183+0000","last_became_peered":"2024-03-18T14:51:00.932183+0000","last_unstale":"2024-03-18T14:52:07.370835+0000","last_undegraded":"2024-03-18T14:52:07.370835+0000","last_fullsized":"2024-03-18T14:52:07.370835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:36:37.230185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.372279+0000","last_change":"2024-03-18T14:51:01.358939+0000","last_active":"2024-03-18T14:52:36.372279+0000","last_peered":"2024-03-18T14:52:36.372279+0000","last_clean":"2024-03-18T14:52:36.372279+0000","last_became_active":"2024-03-18T14:51:01.358601+0000","last_became_peered":"2024-03-18T14:51:01.358601+0000","last_unstale":"2024-03-18T14:52:36.372279+0000","last_undegraded":"2024-03-18T14:52:36.372279+0000","last_fullsized":"2024-03-18T14:52:36.372279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:40:14.876143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":145,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:53:04.156527+0000","last_change":"2024-03-18T14:51:00.945116+0000","last_active":"2024-03-18T14:53:04.156527+0000","last_peered":"2024-03-18T14:53:04.156527+0000","last_clean":"2024-03-18T14:53:04.156527+0000","last_became_active":"2024-03-18T14:51:00.944791+0000","last_became_peered":"2024-03-18T14:51:00.944791+0000","last_unstale":"2024-03-18T14:53:04.156527+0000","last_undegraded":"2024-03-18T14:53:04.156527+0000","last_fullsized":"2024-03-18T14:53:04.156527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:06.637940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.357355+0000","last_change":"2024-03-18T14:51:01.358682+0000","last_active":"2024-03-18T14:52:36.357355+0000","last_peered":"2024-03-18T14:52:36.357355+0000","last_clean":"2024-03-18T14:52:36.357355+0000","last_became_active":"2024-03-18T14:51:01.358378+0000","last_became_peered":"2024-03-18T14:51:01.358378+0000","last_unstale":"2024-03-18T14:52:36.357355+0000","last_undegraded":"2024-03-18T14:52:36.357355+0000","last_fullsized":"2024-03-18T14:52:36.357355+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:42.697944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369158+0000","last_change":"2024-03-18T14:51:00.943976+0000","last_active":"2024-03-18T14:52:07.369158+0000","last_peered":"2024-03-18T14:52:07.369158+0000","last_clean":"2024-03-18T14:52:07.369158+0000","last_became_active":"2024-03-18T14:51:00.943614+0000","last_became_peered":"2024-03-18T14:51:00.943614+0000","last_unstale":"2024-03-18T14:52:07.369158+0000","last_undegraded":"2024-03-18T14:52:07.369158+0000","last_fullsized":"2024-03-18T14:52:07.369158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:07:00.744708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.365708+0000","last_change":"2024-03-18T14:51:01.365316+0000","last_active":"2024-03-18T14:52:36.365708+0000","last_peered":"2024-03-18T14:52:36.365708+0000","last_clean":"2024-03-18T14:52:36.365708+0000","last_became_active":"2024-03-18T14:51:01.365019+0000","last_became_peered":"2024-03-18T14:51:01.365019+0000","last_unstale":"2024-03-18T14:52:36.365708+0000","last_undegraded":"2024-03-18T14:52:36.365708+0000","last_fullsized":"2024-03-18T14:52:36.365708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:39:59.040907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481239+0000","last_change":"2024-03-18T14:51:00.936923+0000","last_active":"2024-03-18T14:52:07.481239+0000","last_peered":"2024-03-18T14:52:07.481239+0000","last_clean":"2024-03-18T14:52:07.481239+0000","last_became_active":"2024-03-18T14:51:00.936591+0000","last_became_peered":"2024-03-18T14:51:00.936591+0000","last_unstale":"2024-03-18T14:52:07.481239+0000","last_undegraded":"2024-03-18T14:52:07.481239+0000","last_fullsized":"2024-03-18T14:52:07.481239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:48:46.423033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.378104+0000","last_change":"2024-03-18T14:51:01.356827+0000","last_active":"2024-03-18T14:52:36.378104+0000","last_peered":"2024-03-18T14:52:36.378104+0000","last_clean":"2024-03-18T14:52:36.378104+0000","last_became_active":"2024-03-18T14:51:01.356148+0000","last_became_peered":"2024-03-18T14:51:01.356148+0000","last_unstale":"2024-03-18T14:52:36.378104+0000","last_undegraded":"2024-03-18T14:52:36.378104+0000","last_fullsized":"2024-03-18T14:52:36.378104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:18.477890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423674+0000","last_change":"2024-03-18T14:51:00.934024+0000","last_active":"2024-03-18T14:52:07.423674+0000","last_peered":"2024-03-18T14:52:07.423674+0000","last_clean":"2024-03-18T14:52:07.423674+0000","last_became_active":"2024-03-18T14:51:00.933739+0000","last_became_peered":"2024-03-18T14:51:00.933739+0000","last_unstale":"2024-03-18T14:52:07.423674+0000","last_undegraded":"2024-03-18T14:52:07.423674+0000","last_fullsized":"2024-03-18T14:52:07.423674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:03.967260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.362587+0000","last_change":"2024-03-18T14:51:01.356519+0000","last_active":"2024-03-18T14:52:36.362587+0000","last_peered":"2024-03-18T14:52:36.362587+0000","last_clean":"2024-03-18T14:52:36.362587+0000","last_became_active":"2024-03-18T14:51:01.356193+0000","last_became_peered":"2024-03-18T14:51:01.356193+0000","last_unstale":"2024-03-18T14:52:36.362587+0000","last_undegraded":"2024-03-18T14:52:36.362587+0000","last_fullsized":"2024-03-18T14:52:36.362587+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:34:59.633241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:53:01.153237+0000","last_change":"2024-03-18T14:51:00.940189+0000","last_active":"2024-03-18T14:53:01.153237+0000","last_peered":"2024-03-18T14:53:01.153237+0000","last_clean":"2024-03-18T14:53:01.153237+0000","last_became_active":"2024-03-18T14:51:00.939820+0000","last_became_peered":"2024-03-18T14:51:00.939820+0000","last_unstale":"2024-03-18T14:53:01.153237+0000","last_undegraded":"2024-03-18T14:53:01.153237+0000","last_fullsized":"2024-03-18T14:53:01.153237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:44:09.749747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.371264+0000","last_change":"2024-03-18T14:51:01.355756+0000","last_active":"2024-03-18T14:52:36.371264+0000","last_peered":"2024-03-18T14:52:36.371264+0000","last_clean":"2024-03-18T14:52:36.371264+0000","last_became_active":"2024-03-18T14:51:01.352871+0000","last_became_peered":"2024-03-18T14:51:01.352871+0000","last_unstale":"2024-03-18T14:52:36.371264+0000","last_undegraded":"2024-03-18T14:52:36.371264+0000","last_fullsized":"2024-03-18T14:52:36.371264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:20:26.533066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366850+0000","last_change":"2024-03-18T14:51:00.941908+0000","last_active":"2024-03-18T14:52:07.366850+0000","last_peered":"2024-03-18T14:52:07.366850+0000","last_clean":"2024-03-18T14:52:07.366850+0000","last_became_active":"2024-03-18T14:51:00.940978+0000","last_became_peered":"2024-03-18T14:51:00.940978+0000","last_unstale":"2024-03-18T14:52:07.366850+0000","last_undegraded":"2024-03-18T14:52:07.366850+0000","last_fullsized":"2024-03-18T14:52:07.366850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:52.705243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.366882+0000","last_change":"2024-03-18T14:51:01.363462+0000","last_active":"2024-03-18T14:52:36.366882+0000","last_peered":"2024-03-18T14:52:36.366882+0000","last_clean":"2024-03-18T14:52:36.366882+0000","last_became_active":"2024-03-18T14:51:01.362262+0000","last_became_peered":"2024-03-18T14:51:01.362262+0000","last_unstale":"2024-03-18T14:52:36.366882+0000","last_undegraded":"2024-03-18T14:52:36.366882+0000","last_fullsized":"2024-03-18T14:52:36.366882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:07:18.952876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.424170+0000","last_change":"2024-03-18T14:51:00.936050+0000","last_active":"2024-03-18T14:52:07.424170+0000","last_peered":"2024-03-18T14:52:07.424170+0000","last_clean":"2024-03-18T14:52:07.424170+0000","last_became_active":"2024-03-18T14:51:00.935676+0000","last_became_peered":"2024-03-18T14:51:00.935676+0000","last_unstale":"2024-03-18T14:52:07.424170+0000","last_undegraded":"2024-03-18T14:52:07.424170+0000","last_fullsized":"2024-03-18T14:52:07.424170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:29.423110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.374032+0000","last_change":"2024-03-18T14:51:01.364295+0000","last_active":"2024-03-18T14:52:36.374032+0000","last_peered":"2024-03-18T14:52:36.374032+0000","last_clean":"2024-03-18T14:52:36.374032+0000","last_became_active":"2024-03-18T14:51:01.363916+0000","last_became_peered":"2024-03-18T14:51:01.363916+0000","last_unstale":"2024-03-18T14:52:36.374032+0000","last_undegraded":"2024-03-18T14:52:36.374032+0000","last_fullsized":"2024-03-18T14:52:36.374032+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:21:17.905819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'31","reported_seq":305,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:53:02.055197+0000","last_change":"2024-03-18T14:52:07.427262+0000","last_active":"2024-03-18T14:53:02.055197+0000","last_peered":"2024-03-18T14:53:02.055197+0000","last_clean":"2024-03-18T14:53:02.055197+0000","last_became_active":"2024-03-18T14:52:07.421388+0000","last_became_peered":"2024-03-18T14:52:07.421388+0000","last_unstale":"2024-03-18T14:53:02.055197+0000","last_undegraded":"2024-03-18T14:53:02.055197+0000","last_fullsized":"2024-03-18T14:53:02.055197+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:28.159787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59469,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":233,"num_read_kb":174,"num_write":32,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":58912,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.376345+0000","last_change":"2024-03-18T14:51:01.362426+0000","last_active":"2024-03-18T14:52:36.376345+0000","last_peered":"2024-03-18T14:52:36.376345+0000","last_clean":"2024-03-18T14:52:36.376345+0000","last_became_active":"2024-03-18T14:51:01.362130+0000","last_became_peered":"2024-03-18T14:51:01.362130+0000","last_unstale":"2024-03-18T14:52:36.376345+0000","last_undegraded":"2024-03-18T14:52:36.376345+0000","last_fullsized":"2024-03-18T14:52:36.376345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:14.545997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368774+0000","last_change":"2024-03-18T14:51:00.934028+0000","last_active":"2024-03-18T14:52:07.368774+0000","last_peered":"2024-03-18T14:52:07.368774+0000","last_clean":"2024-03-18T14:52:07.368774+0000","last_became_active":"2024-03-18T14:51:00.933594+0000","last_became_peered":"2024-03-18T14:51:00.933594+0000","last_unstale":"2024-03-18T14:52:07.368774+0000","last_undegraded":"2024-03-18T14:52:07.368774+0000","last_fullsized":"2024-03-18T14:52:07.368774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:55:08.675533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.374723+0000","last_change":"2024-03-18T14:51:01.353544+0000","last_active":"2024-03-18T14:52:36.374723+0000","last_peered":"2024-03-18T14:52:36.374723+0000","last_clean":"2024-03-18T14:52:36.374723+0000","last_became_active":"2024-03-18T14:51:01.353241+0000","last_became_peered":"2024-03-18T14:51:01.353241+0000","last_unstale":"2024-03-18T14:52:36.374723+0000","last_undegraded":"2024-03-18T14:52:36.374723+0000","last_fullsized":"2024-03-18T14:52:36.374723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:45:27.617681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.423792+0000","last_change":"2024-03-18T14:51:00.944492+0000","last_active":"2024-03-18T14:52:07.423792+0000","last_peered":"2024-03-18T14:52:07.423792+0000","last_clean":"2024-03-18T14:52:07.423792+0000","last_became_active":"2024-03-18T14:51:00.944165+0000","last_became_peered":"2024-03-18T14:51:00.944165+0000","last_unstale":"2024-03-18T14:52:07.423792+0000","last_undegraded":"2024-03-18T14:52:07.423792+0000","last_fullsized":"2024-03-18T14:52:07.423792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:47:05.968966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.371403+0000","last_change":"2024-03-18T14:51:01.356782+0000","last_active":"2024-03-18T14:52:36.371403+0000","last_peered":"2024-03-18T14:52:36.371403+0000","last_clean":"2024-03-18T14:52:36.371403+0000","last_became_active":"2024-03-18T14:51:01.356242+0000","last_became_peered":"2024-03-18T14:51:01.356242+0000","last_unstale":"2024-03-18T14:52:36.371403+0000","last_undegraded":"2024-03-18T14:52:36.371403+0000","last_fullsized":"2024-03-18T14:52:36.371403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:56:07.662545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'6","reported_seq":149,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:53:04.168246+0000","last_change":"2024-03-18T14:51:00.937873+0000","last_active":"2024-03-18T14:53:04.168246+0000","last_peered":"2024-03-18T14:53:04.168246+0000","last_clean":"2024-03-18T14:53:04.168246+0000","last_became_active":"2024-03-18T14:51:00.937562+0000","last_became_peered":"2024-03-18T14:51:00.937562+0000","last_unstale":"2024-03-18T14:53:04.168246+0000","last_undegraded":"2024-03-18T14:53:04.168246+0000","last_fullsized":"2024-03-18T14:53:04.168246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:36:44.342414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1428,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375613+0000","last_change":"2024-03-18T14:51:01.358681+0000","last_active":"2024-03-18T14:52:36.375613+0000","last_peered":"2024-03-18T14:52:36.375613+0000","last_clean":"2024-03-18T14:52:36.375613+0000","last_became_active":"2024-03-18T14:51:01.358374+0000","last_became_peered":"2024-03-18T14:51:01.358374+0000","last_unstale":"2024-03-18T14:52:36.375613+0000","last_undegraded":"2024-03-18T14:52:36.375613+0000","last_fullsized":"2024-03-18T14:52:36.375613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:43:47.804271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370451+0000","last_change":"2024-03-18T14:51:00.935639+0000","last_active":"2024-03-18T14:52:07.370451+0000","last_peered":"2024-03-18T14:52:07.370451+0000","last_clean":"2024-03-18T14:52:07.370451+0000","last_became_active":"2024-03-18T14:51:00.935334+0000","last_became_peered":"2024-03-18T14:51:00.935334+0000","last_unstale":"2024-03-18T14:52:07.370451+0000","last_undegraded":"2024-03-18T14:52:07.370451+0000","last_fullsized":"2024-03-18T14:52:07.370451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:46.966812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.370423+0000","last_change":"2024-03-18T14:51:01.364755+0000","last_active":"2024-03-18T14:52:36.370423+0000","last_peered":"2024-03-18T14:52:36.370423+0000","last_clean":"2024-03-18T14:52:36.370423+0000","last_became_active":"2024-03-18T14:51:01.364448+0000","last_became_peered":"2024-03-18T14:51:01.364448+0000","last_unstale":"2024-03-18T14:52:36.370423+0000","last_undegraded":"2024-03-18T14:52:36.370423+0000","last_fullsized":"2024-03-18T14:52:36.370423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:37:49.480787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.371021+0000","last_change":"2024-03-18T14:51:00.933271+0000","last_active":"2024-03-18T14:52:07.371021+0000","last_peered":"2024-03-18T14:52:07.371021+0000","last_clean":"2024-03-18T14:52:07.371021+0000","last_became_active":"2024-03-18T14:51:00.932928+0000","last_became_peered":"2024-03-18T14:51:00.932928+0000","last_unstale":"2024-03-18T14:52:07.371021+0000","last_undegraded":"2024-03-18T14:52:07.371021+0000","last_fullsized":"2024-03-18T14:52:07.371021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:11:55.281763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375943+0000","last_change":"2024-03-18T14:51:01.362911+0000","last_active":"2024-03-18T14:52:36.375943+0000","last_peered":"2024-03-18T14:52:36.375943+0000","last_clean":"2024-03-18T14:52:36.375943+0000","last_became_active":"2024-03-18T14:51:01.362598+0000","last_became_peered":"2024-03-18T14:51:01.362598+0000","last_unstale":"2024-03-18T14:52:36.375943+0000","last_undegraded":"2024-03-18T14:52:36.375943+0000","last_fullsized":"2024-03-18T14:52:36.375943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:22:58.627753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.363811+0000","last_change":"2024-03-18T14:51:00.935554+0000","last_active":"2024-03-18T14:52:07.363811+0000","last_peered":"2024-03-18T14:52:07.363811+0000","last_clean":"2024-03-18T14:52:07.363811+0000","last_became_active":"2024-03-18T14:51:00.935184+0000","last_became_peered":"2024-03-18T14:51:00.935184+0000","last_unstale":"2024-03-18T14:52:07.363811+0000","last_undegraded":"2024-03-18T14:52:07.363811+0000","last_fullsized":"2024-03-18T14:52:07.363811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:21:13.973570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.361967+0000","last_change":"2024-03-18T14:51:01.355990+0000","last_active":"2024-03-18T14:52:36.361967+0000","last_peered":"2024-03-18T14:52:36.361967+0000","last_clean":"2024-03-18T14:52:36.361967+0000","last_became_active":"2024-03-18T14:51:01.355692+0000","last_became_peered":"2024-03-18T14:51:01.355692+0000","last_unstale":"2024-03-18T14:52:36.361967+0000","last_undegraded":"2024-03-18T14:52:36.361967+0000","last_fullsized":"2024-03-18T14:52:36.361967+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:27:19.742684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366919+0000","last_change":"2024-03-18T14:51:00.943947+0000","last_active":"2024-03-18T14:52:07.366919+0000","last_peered":"2024-03-18T14:52:07.366919+0000","last_clean":"2024-03-18T14:52:07.366919+0000","last_became_active":"2024-03-18T14:51:00.943565+0000","last_became_peered":"2024-03-18T14:51:00.943565+0000","last_unstale":"2024-03-18T14:52:07.366919+0000","last_undegraded":"2024-03-18T14:52:07.366919+0000","last_fullsized":"2024-03-18T14:52:07.366919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:33:06.103148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.367260+0000","last_change":"2024-03-18T14:51:01.363221+0000","last_active":"2024-03-18T14:52:36.367260+0000","last_peered":"2024-03-18T14:52:36.367260+0000","last_clean":"2024-03-18T14:52:36.367260+0000","last_became_active":"2024-03-18T14:51:01.362848+0000","last_became_peered":"2024-03-18T14:51:01.362848+0000","last_unstale":"2024-03-18T14:52:36.367260+0000","last_undegraded":"2024-03-18T14:52:36.367260+0000","last_fullsized":"2024-03-18T14:52:36.367260+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:04.139365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.365329+0000","last_change":"2024-03-18T14:51:01.356237+0000","last_active":"2024-03-18T14:52:36.365329+0000","last_peered":"2024-03-18T14:52:36.365329+0000","last_clean":"2024-03-18T14:52:36.365329+0000","last_became_active":"2024-03-18T14:51:01.355922+0000","last_became_peered":"2024-03-18T14:51:01.355922+0000","last_unstale":"2024-03-18T14:52:36.365329+0000","last_undegraded":"2024-03-18T14:52:36.365329+0000","last_fullsized":"2024-03-18T14:52:36.365329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:04:50.491752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'6","reported_seq":145,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:53:05.168942+0000","last_change":"2024-03-18T14:51:00.936058+0000","last_active":"2024-03-18T14:53:05.168942+0000","last_peered":"2024-03-18T14:53:05.168942+0000","last_clean":"2024-03-18T14:53:05.168942+0000","last_became_active":"2024-03-18T14:51:00.935731+0000","last_became_peered":"2024-03-18T14:51:00.935731+0000","last_unstale":"2024-03-18T14:53:05.168942+0000","last_undegraded":"2024-03-18T14:53:05.168942+0000","last_fullsized":"2024-03-18T14:53:05.168942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:47:49.023943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.374426+0000","last_change":"2024-03-18T14:51:01.359742+0000","last_active":"2024-03-18T14:52:36.374426+0000","last_peered":"2024-03-18T14:52:36.374426+0000","last_clean":"2024-03-18T14:52:36.374426+0000","last_became_active":"2024-03-18T14:51:01.358507+0000","last_became_peered":"2024-03-18T14:51:01.358507+0000","last_unstale":"2024-03-18T14:52:36.374426+0000","last_undegraded":"2024-03-18T14:52:36.374426+0000","last_fullsized":"2024-03-18T14:52:36.374426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:17.129695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.419738+0000","last_change":"2024-03-18T14:51:00.934593+0000","last_active":"2024-03-18T14:52:07.419738+0000","last_peered":"2024-03-18T14:52:07.419738+0000","last_clean":"2024-03-18T14:52:07.419738+0000","last_became_active":"2024-03-18T14:51:00.934253+0000","last_became_peered":"2024-03-18T14:51:00.934253+0000","last_unstale":"2024-03-18T14:52:07.419738+0000","last_undegraded":"2024-03-18T14:52:07.419738+0000","last_fullsized":"2024-03-18T14:52:07.419738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:43:26.323162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":352,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366664+0000","last_change":"2024-03-18T14:49:51.158965+0000","last_active":"2024-03-18T14:52:07.366664+0000","last_peered":"2024-03-18T14:52:07.366664+0000","last_clean":"2024-03-18T14:52:07.366664+0000","last_became_active":"2024-03-18T14:49:51.158580+0000","last_became_peered":"2024-03-18T14:49:51.158580+0000","last_unstale":"2024-03-18T14:52:07.366664+0000","last_undegraded":"2024-03-18T14:52:07.366664+0000","last_fullsized":"2024-03-18T14:52:07.366664+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:46:27.210244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:46:27.210244+0000","last_clean_scrub_stamp":"2024-03-18T14:46:27.210244+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:37:08.104617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.359433+0000","last_change":"2024-03-18T14:51:01.359972+0000","last_active":"2024-03-18T14:52:36.359433+0000","last_peered":"2024-03-18T14:52:36.359433+0000","last_clean":"2024-03-18T14:52:36.359433+0000","last_became_active":"2024-03-18T14:51:01.359430+0000","last_became_peered":"2024-03-18T14:51:01.359430+0000","last_unstale":"2024-03-18T14:52:36.359433+0000","last_undegraded":"2024-03-18T14:52:36.359433+0000","last_fullsized":"2024-03-18T14:52:36.359433+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:00:14.099499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.366772+0000","last_change":"2024-03-18T14:51:00.934196+0000","last_active":"2024-03-18T14:52:07.366772+0000","last_peered":"2024-03-18T14:52:07.366772+0000","last_clean":"2024-03-18T14:52:07.366772+0000","last_became_active":"2024-03-18T14:51:00.933875+0000","last_became_peered":"2024-03-18T14:51:00.933875+0000","last_unstale":"2024-03-18T14:52:07.366772+0000","last_undegraded":"2024-03-18T14:52:07.366772+0000","last_fullsized":"2024-03-18T14:52:07.366772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:56.797227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370852+0000","last_change":"2024-03-18T14:51:00.941862+0000","last_active":"2024-03-18T14:52:07.370852+0000","last_peered":"2024-03-18T14:52:07.370852+0000","last_clean":"2024-03-18T14:52:07.370852+0000","last_became_active":"2024-03-18T14:51:00.941536+0000","last_became_peered":"2024-03-18T14:51:00.941536+0000","last_unstale":"2024-03-18T14:52:07.370852+0000","last_undegraded":"2024-03-18T14:52:07.370852+0000","last_fullsized":"2024-03-18T14:52:07.370852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:56.364552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.359217+0000","last_change":"2024-03-18T14:51:01.352404+0000","last_active":"2024-03-18T14:52:36.359217+0000","last_peered":"2024-03-18T14:52:36.359217+0000","last_clean":"2024-03-18T14:52:36.359217+0000","last_became_active":"2024-03-18T14:51:01.352077+0000","last_became_peered":"2024-03-18T14:51:01.352077+0000","last_unstale":"2024-03-18T14:52:36.359217+0000","last_undegraded":"2024-03-18T14:52:36.359217+0000","last_fullsized":"2024-03-18T14:52:36.359217+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:30:38.050712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.364118+0000","last_change":"2024-03-18T14:51:00.935564+0000","last_active":"2024-03-18T14:52:07.364118+0000","last_peered":"2024-03-18T14:52:07.364118+0000","last_clean":"2024-03-18T14:52:07.364118+0000","last_became_active":"2024-03-18T14:51:00.935268+0000","last_became_peered":"2024-03-18T14:51:00.935268+0000","last_unstale":"2024-03-18T14:52:07.364118+0000","last_undegraded":"2024-03-18T14:52:07.364118+0000","last_fullsized":"2024-03-18T14:52:07.364118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:22:33.504184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.372551+0000","last_change":"2024-03-18T14:51:01.356096+0000","last_active":"2024-03-18T14:52:36.372551+0000","last_peered":"2024-03-18T14:52:36.372551+0000","last_clean":"2024-03-18T14:52:36.372551+0000","last_became_active":"2024-03-18T14:51:01.355688+0000","last_became_peered":"2024-03-18T14:51:01.355688+0000","last_unstale":"2024-03-18T14:52:36.372551+0000","last_undegraded":"2024-03-18T14:52:36.372551+0000","last_fullsized":"2024-03-18T14:52:36.372551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:04:38.678482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.419726+0000","last_change":"2024-03-18T14:51:00.942610+0000","last_active":"2024-03-18T14:52:07.419726+0000","last_peered":"2024-03-18T14:52:07.419726+0000","last_clean":"2024-03-18T14:52:07.419726+0000","last_became_active":"2024-03-18T14:51:00.942203+0000","last_became_peered":"2024-03-18T14:51:00.942203+0000","last_unstale":"2024-03-18T14:52:07.419726+0000","last_undegraded":"2024-03-18T14:52:07.419726+0000","last_fullsized":"2024-03-18T14:52:07.419726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:17:26.324282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.367722+0000","last_change":"2024-03-18T14:51:01.362931+0000","last_active":"2024-03-18T14:52:36.367722+0000","last_peered":"2024-03-18T14:52:36.367722+0000","last_clean":"2024-03-18T14:52:36.367722+0000","last_became_active":"2024-03-18T14:51:01.362624+0000","last_became_peered":"2024-03-18T14:51:01.362624+0000","last_unstale":"2024-03-18T14:52:36.367722+0000","last_undegraded":"2024-03-18T14:52:36.367722+0000","last_fullsized":"2024-03-18T14:52:36.367722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:40:52.512871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370722+0000","last_change":"2024-03-18T14:51:00.938971+0000","last_active":"2024-03-18T14:52:07.370722+0000","last_peered":"2024-03-18T14:52:07.370722+0000","last_clean":"2024-03-18T14:52:07.370722+0000","last_became_active":"2024-03-18T14:51:00.938645+0000","last_became_peered":"2024-03-18T14:51:00.938645+0000","last_unstale":"2024-03-18T14:52:07.370722+0000","last_undegraded":"2024-03-18T14:52:07.370722+0000","last_fullsized":"2024-03-18T14:52:07.370722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:18:54.999927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.334973+0000","last_change":"2024-03-18T14:51:01.359510+0000","last_active":"2024-03-18T14:52:36.334973+0000","last_peered":"2024-03-18T14:52:36.334973+0000","last_clean":"2024-03-18T14:52:36.334973+0000","last_became_active":"2024-03-18T14:51:01.359192+0000","last_became_peered":"2024-03-18T14:51:01.359192+0000","last_unstale":"2024-03-18T14:52:36.334973+0000","last_undegraded":"2024-03-18T14:52:36.334973+0000","last_fullsized":"2024-03-18T14:52:36.334973+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:11:52.942825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.368576+0000","last_change":"2024-03-18T14:51:00.940194+0000","last_active":"2024-03-18T14:52:07.368576+0000","last_peered":"2024-03-18T14:52:07.368576+0000","last_clean":"2024-03-18T14:52:07.368576+0000","last_became_active":"2024-03-18T14:51:00.939861+0000","last_became_peered":"2024-03-18T14:51:00.939861+0000","last_unstale":"2024-03-18T14:52:07.368576+0000","last_undegraded":"2024-03-18T14:52:07.368576+0000","last_fullsized":"2024-03-18T14:52:07.368576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:06:30.314102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.367604+0000","last_change":"2024-03-18T14:51:01.355098+0000","last_active":"2024-03-18T14:52:36.367604+0000","last_peered":"2024-03-18T14:52:36.367604+0000","last_clean":"2024-03-18T14:52:36.367604+0000","last_became_active":"2024-03-18T14:51:01.354452+0000","last_became_peered":"2024-03-18T14:51:01.354452+0000","last_unstale":"2024-03-18T14:52:36.367604+0000","last_undegraded":"2024-03-18T14:52:36.367604+0000","last_fullsized":"2024-03-18T14:52:36.367604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:40:54.650294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.381529+0000","last_change":"2024-03-18T14:52:07.381159+0000","last_active":"2024-03-18T14:52:07.381529+0000","last_peered":"2024-03-18T14:52:07.381529+0000","last_clean":"2024-03-18T14:52:07.381529+0000","last_became_active":"2024-03-18T14:52:07.371943+0000","last_became_peered":"2024-03-18T14:52:07.371943+0000","last_unstale":"2024-03-18T14:52:07.381529+0000","last_undegraded":"2024-03-18T14:52:07.381529+0000","last_fullsized":"2024-03-18T14:52:07.381529+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:11.329239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.359825+0000","last_change":"2024-03-18T14:51:01.357920+0000","last_active":"2024-03-18T14:52:36.359825+0000","last_peered":"2024-03-18T14:52:36.359825+0000","last_clean":"2024-03-18T14:52:36.359825+0000","last_became_active":"2024-03-18T14:51:01.357565+0000","last_became_peered":"2024-03-18T14:51:01.357565+0000","last_unstale":"2024-03-18T14:52:36.359825+0000","last_undegraded":"2024-03-18T14:52:36.359825+0000","last_fullsized":"2024-03-18T14:52:36.359825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:39:57.880178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.370714+0000","last_change":"2024-03-18T14:51:00.937887+0000","last_active":"2024-03-18T14:52:07.370714+0000","last_peered":"2024-03-18T14:52:07.370714+0000","last_clean":"2024-03-18T14:52:07.370714+0000","last_became_active":"2024-03-18T14:51:00.937588+0000","last_became_peered":"2024-03-18T14:51:00.937588+0000","last_unstale":"2024-03-18T14:52:07.370714+0000","last_undegraded":"2024-03-18T14:52:07.370714+0000","last_fullsized":"2024-03-18T14:52:07.370714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:43:37.558412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375612+0000","last_change":"2024-03-18T14:51:01.354175+0000","last_active":"2024-03-18T14:52:36.375612+0000","last_peered":"2024-03-18T14:52:36.375612+0000","last_clean":"2024-03-18T14:52:36.375612+0000","last_became_active":"2024-03-18T14:51:01.353839+0000","last_became_peered":"2024-03-18T14:51:01.353839+0000","last_unstale":"2024-03-18T14:52:36.375612+0000","last_undegraded":"2024-03-18T14:52:36.375612+0000","last_fullsized":"2024-03-18T14:52:36.375612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:21:52.372938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.369167+0000","last_change":"2024-03-18T14:51:00.941600+0000","last_active":"2024-03-18T14:52:07.369167+0000","last_peered":"2024-03-18T14:52:07.369167+0000","last_clean":"2024-03-18T14:52:07.369167+0000","last_became_active":"2024-03-18T14:51:00.941291+0000","last_became_peered":"2024-03-18T14:51:00.941291+0000","last_unstale":"2024-03-18T14:52:07.369167+0000","last_undegraded":"2024-03-18T14:52:07.369167+0000","last_fullsized":"2024-03-18T14:52:07.369167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:04:11.142029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.376178+0000","last_change":"2024-03-18T14:51:01.359058+0000","last_active":"2024-03-18T14:52:36.376178+0000","last_peered":"2024-03-18T14:52:36.376178+0000","last_clean":"2024-03-18T14:52:36.376178+0000","last_became_active":"2024-03-18T14:51:01.358437+0000","last_became_peered":"2024-03-18T14:51:01.358437+0000","last_unstale":"2024-03-18T14:52:36.376178+0000","last_undegraded":"2024-03-18T14:52:36.376178+0000","last_fullsized":"2024-03-18T14:52:36.376178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:39:58.945080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.424033+0000","last_change":"2024-03-18T14:51:00.933647+0000","last_active":"2024-03-18T14:52:07.424033+0000","last_peered":"2024-03-18T14:52:07.424033+0000","last_clean":"2024-03-18T14:52:07.424033+0000","last_became_active":"2024-03-18T14:51:00.933320+0000","last_became_peered":"2024-03-18T14:51:00.933320+0000","last_unstale":"2024-03-18T14:52:07.424033+0000","last_undegraded":"2024-03-18T14:52:07.424033+0000","last_fullsized":"2024-03-18T14:52:07.424033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:16:46.498845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.370719+0000","last_change":"2024-03-18T14:51:01.359874+0000","last_active":"2024-03-18T14:52:36.370719+0000","last_peered":"2024-03-18T14:52:36.370719+0000","last_clean":"2024-03-18T14:52:36.370719+0000","last_became_active":"2024-03-18T14:51:01.359335+0000","last_became_peered":"2024-03-18T14:51:01.359335+0000","last_unstale":"2024-03-18T14:52:36.370719+0000","last_undegraded":"2024-03-18T14:52:36.370719+0000","last_fullsized":"2024-03-18T14:52:36.370719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:03.583402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:07.481395+0000","last_change":"2024-03-18T14:51:00.935755+0000","last_active":"2024-03-18T14:52:07.481395+0000","last_peered":"2024-03-18T14:52:07.481395+0000","last_clean":"2024-03-18T14:52:07.481395+0000","last_became_active":"2024-03-18T14:51:00.935421+0000","last_became_peered":"2024-03-18T14:51:00.935421+0000","last_unstale":"2024-03-18T14:52:07.481395+0000","last_undegraded":"2024-03-18T14:52:07.481395+0000","last_fullsized":"2024-03-18T14:52:07.481395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:50:59.905745+0000","last_clean_scrub_stamp":"2024-03-18T14:50:59.905745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:46:11.680972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.373520+0000","last_change":"2024-03-18T14:51:01.355605+0000","last_active":"2024-03-18T14:52:36.373520+0000","last_peered":"2024-03-18T14:52:36.373520+0000","last_clean":"2024-03-18T14:52:36.373520+0000","last_became_active":"2024-03-18T14:51:01.355295+0000","last_became_peered":"2024-03-18T14:51:01.355295+0000","last_unstale":"2024-03-18T14:52:36.373520+0000","last_undegraded":"2024-03-18T14:52:36.373520+0000","last_fullsized":"2024-03-18T14:52:36.373520+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:15:20.466314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.379370+0000","last_change":"2024-03-18T14:51:01.359704+0000","last_active":"2024-03-18T14:52:36.379370+0000","last_peered":"2024-03-18T14:52:36.379370+0000","last_clean":"2024-03-18T14:52:36.379370+0000","last_became_active":"2024-03-18T14:51:01.359367+0000","last_became_peered":"2024-03-18T14:51:01.359367+0000","last_unstale":"2024-03-18T14:52:36.379370+0000","last_undegraded":"2024-03-18T14:52:36.379370+0000","last_fullsized":"2024-03-18T14:52:36.379370+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:26.933993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.365339+0000","last_change":"2024-03-18T14:52:07.420506+0000","last_active":"2024-03-18T14:52:36.365339+0000","last_peered":"2024-03-18T14:52:36.365339+0000","last_clean":"2024-03-18T14:52:36.365339+0000","last_became_active":"2024-03-18T14:52:07.420254+0000","last_became_peered":"2024-03-18T14:52:07.420254+0000","last_unstale":"2024-03-18T14:52:36.365339+0000","last_undegraded":"2024-03-18T14:52:36.365339+0000","last_fullsized":"2024-03-18T14:52:36.365339+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:32:08.702462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.369110+0000","last_change":"2024-03-18T14:51:01.355678+0000","last_active":"2024-03-18T14:52:36.369110+0000","last_peered":"2024-03-18T14:52:36.369110+0000","last_clean":"2024-03-18T14:52:36.369110+0000","last_became_active":"2024-03-18T14:51:01.355197+0000","last_became_peered":"2024-03-18T14:51:01.355197+0000","last_unstale":"2024-03-18T14:52:36.369110+0000","last_undegraded":"2024-03-18T14:52:36.369110+0000","last_fullsized":"2024-03-18T14:52:36.369110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:34.194834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.366017+0000","last_change":"2024-03-18T14:51:01.363964+0000","last_active":"2024-03-18T14:52:36.366017+0000","last_peered":"2024-03-18T14:52:36.366017+0000","last_clean":"2024-03-18T14:52:36.366017+0000","last_became_active":"2024-03-18T14:51:01.363590+0000","last_became_peered":"2024-03-18T14:51:01.363590+0000","last_unstale":"2024-03-18T14:52:36.366017+0000","last_undegraded":"2024-03-18T14:52:36.366017+0000","last_fullsized":"2024-03-18T14:52:36.366017+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:59.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.366199+0000","last_change":"2024-03-18T14:51:01.361599+0000","last_active":"2024-03-18T14:52:36.366199+0000","last_peered":"2024-03-18T14:52:36.366199+0000","last_clean":"2024-03-18T14:52:36.366199+0000","last_became_active":"2024-03-18T14:51:01.361280+0000","last_became_peered":"2024-03-18T14:51:01.361280+0000","last_unstale":"2024-03-18T14:52:36.366199+0000","last_undegraded":"2024-03-18T14:52:36.366199+0000","last_fullsized":"2024-03-18T14:52:36.366199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:52:50.572257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.376675+0000","last_change":"2024-03-18T14:51:01.360495+0000","last_active":"2024-03-18T14:52:36.376675+0000","last_peered":"2024-03-18T14:52:36.376675+0000","last_clean":"2024-03-18T14:52:36.376675+0000","last_became_active":"2024-03-18T14:51:01.360146+0000","last_became_peered":"2024-03-18T14:51:01.360146+0000","last_unstale":"2024-03-18T14:52:36.376675+0000","last_undegraded":"2024-03-18T14:52:36.376675+0000","last_fullsized":"2024-03-18T14:52:36.376675+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:52.048137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.363452+0000","last_change":"2024-03-18T14:51:01.357650+0000","last_active":"2024-03-18T14:52:36.363452+0000","last_peered":"2024-03-18T14:52:36.363452+0000","last_clean":"2024-03-18T14:52:36.363452+0000","last_became_active":"2024-03-18T14:51:01.356265+0000","last_became_peered":"2024-03-18T14:51:01.356265+0000","last_unstale":"2024-03-18T14:52:36.363452+0000","last_undegraded":"2024-03-18T14:52:36.363452+0000","last_fullsized":"2024-03-18T14:52:36.363452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:49.328404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.357983+0000","last_change":"2024-03-18T14:51:01.363753+0000","last_active":"2024-03-18T14:52:36.357983+0000","last_peered":"2024-03-18T14:52:36.357983+0000","last_clean":"2024-03-18T14:52:36.357983+0000","last_became_active":"2024-03-18T14:51:01.363411+0000","last_became_peered":"2024-03-18T14:51:01.363411+0000","last_unstale":"2024-03-18T14:52:36.357983+0000","last_undegraded":"2024-03-18T14:52:36.357983+0000","last_fullsized":"2024-03-18T14:52:36.357983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:55.452209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.365639+0000","last_change":"2024-03-18T14:51:01.360723+0000","last_active":"2024-03-18T14:52:36.365639+0000","last_peered":"2024-03-18T14:52:36.365639+0000","last_clean":"2024-03-18T14:52:36.365639+0000","last_became_active":"2024-03-18T14:51:01.360130+0000","last_became_peered":"2024-03-18T14:51:01.360130+0000","last_unstale":"2024-03-18T14:52:36.365639+0000","last_undegraded":"2024-03-18T14:52:36.365639+0000","last_fullsized":"2024-03-18T14:52:36.365639+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:51.634011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.372926+0000","last_change":"2024-03-18T14:51:01.356462+0000","last_active":"2024-03-18T14:52:36.372926+0000","last_peered":"2024-03-18T14:52:36.372926+0000","last_clean":"2024-03-18T14:52:36.372926+0000","last_became_active":"2024-03-18T14:51:01.355395+0000","last_became_peered":"2024-03-18T14:51:01.355395+0000","last_unstale":"2024-03-18T14:52:36.372926+0000","last_undegraded":"2024-03-18T14:52:36.372926+0000","last_fullsized":"2024-03-18T14:52:36.372926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:37:15.326354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.376909+0000","last_change":"2024-03-18T14:51:01.358626+0000","last_active":"2024-03-18T14:52:36.376909+0000","last_peered":"2024-03-18T14:52:36.376909+0000","last_clean":"2024-03-18T14:52:36.376909+0000","last_became_active":"2024-03-18T14:51:01.358293+0000","last_became_peered":"2024-03-18T14:51:01.358293+0000","last_unstale":"2024-03-18T14:52:36.376909+0000","last_undegraded":"2024-03-18T14:52:36.376909+0000","last_fullsized":"2024-03-18T14:52:36.376909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:52:41.139885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.375009+0000","last_change":"2024-03-18T14:51:01.355276+0000","last_active":"2024-03-18T14:52:36.375009+0000","last_peered":"2024-03-18T14:52:36.375009+0000","last_clean":"2024-03-18T14:52:36.375009+0000","last_became_active":"2024-03-18T14:51:01.354661+0000","last_became_peered":"2024-03-18T14:51:01.354661+0000","last_unstale":"2024-03-18T14:52:36.375009+0000","last_undegraded":"2024-03-18T14:52:36.375009+0000","last_fullsized":"2024-03-18T14:52:36.375009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:28.569055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T14:52:36.377238+0000","last_change":"2024-03-18T14:52:07.372297+0000","last_active":"2024-03-18T14:52:36.377238+0000","last_peered":"2024-03-18T14:52:36.377238+0000","last_clean":"2024-03-18T14:52:36.377238+0000","last_became_active":"2024-03-18T14:52:07.371918+0000","last_became_peered":"2024-03-18T14:52:07.371918+0000","last_unstale":"2024-03-18T14:52:36.377238+0000","last_undegraded":"2024-03-18T14:52:36.377238+0000","last_fullsized":"2024-03-18T14:52:36.377238+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:51:00.914572+0000","last_clean_scrub_stamp":"2024-03-18T14:51:00.914572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:28:23.431373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280632,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63919,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":727,"num_read_kb":430,"num_write":167,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":58912,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":200385,"data_compressed":13263,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351014,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":459161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612650,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":1110643,"data_compressed":37553,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776176,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037813,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":459317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":1051696,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462975,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":458786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":518048,"data_compressed":1859,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":460057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":459929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":459097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888138,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":523846,"data_compressed":9545,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724612,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":1050540,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35694,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63783,"data_compressed":1859,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":65267,"data_compressed":9545,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":61887,"data_compressed":1859,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:53:07.351 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:53:07.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.489+0000 7f6bd2634640 1 -- 172.21.15.76:0/2108016536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6bcc0741c0 msgr2=0x7f6bcc074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:07.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.489+0000 7f6bd2634640 1 --2- 172.21.15.76:0/2108016536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6bcc0741c0 0x7f6bcc074620 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0009900 tx=0x7f6bc002bcd0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.489+0000 7f6bd2634640 1 -- 172.21.15.76:0/2108016536 shutdown_connections 2024-03-18T14:53:07.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.489+0000 7f6bd2634640 1 --2- 172.21.15.76:0/2108016536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bcc1096b0 0x7f6bcc10bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.489+0000 7f6bd2634640 1 --2- 172.21.15.76:0/2108016536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6bcc0741c0 0x7f6bcc074620 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.489+0000 7f6bd2634640 1 --2- 172.21.15.76:0/2108016536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bcc0757a0 0x7f6bcc073bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.489+0000 7f6bd2634640 1 -- 172.21.15.76:0/2108016536 >> 172.21.15.76:0/2108016536 conn(0x7f6bcc0fcd90 msgr2=0x7f6bcc0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:07.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.489+0000 7f6bd2634640 1 -- 172.21.15.76:0/2108016536 shutdown_connections 2024-03-18T14:53:07.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.489+0000 7f6bd2634640 1 -- 172.21.15.76:0/2108016536 wait complete. 2024-03-18T14:53:07.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.490+0000 7f6bd2634640 1 Processor -- start 2024-03-18T14:53:07.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.490+0000 7f6bd2634640 1 -- start start 2024-03-18T14:53:07.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.490+0000 7f6bd2634640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bcc0741c0 0x7f6bcc199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:07.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.490+0000 7f6bd2634640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bcc0757a0 0x7f6bcc19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:07.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.490+0000 7f6bd2634640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6bcc1096b0 0x7f6bcc1a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:07.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.490+0000 7f6bd2634640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6bcc10f660 con 0x7f6bcc1096b0 2024-03-18T14:53:07.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.490+0000 7f6bd2634640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6bcc10f4e0 con 0x7f6bcc0741c0 2024-03-18T14:53:07.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.490+0000 7f6bd2634640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6bcc10f7e0 con 0x7f6bcc0757a0 2024-03-18T14:53:07.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.490+0000 7f6bd0bd1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6bcc1096b0 0x7f6bcc1a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:07.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.490+0000 7f6bd0bd1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6bcc1096b0 0x7f6bcc1a1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50448/0 (socket says 172.21.15.76:50448) 2024-03-18T14:53:07.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.490+0000 7f6bd0bd1640 1 -- 172.21.15.76:0/3409131602 learned_addr learned my addr 172.21.15.76:0/3409131602 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:07.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.490+0000 7f6bcb7fe640 1 --2- 172.21.15.76:0/3409131602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bcc0757a0 0x7f6bcc19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:07.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.490+0000 7f6bcbfff640 1 --2- 172.21.15.76:0/3409131602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bcc0741c0 0x7f6bcc199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:07.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.491+0000 7f6bd0bd1640 1 -- 172.21.15.76:0/3409131602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bcc0757a0 msgr2=0x7f6bcc19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:07.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.491+0000 7f6bd0bd1640 1 --2- 172.21.15.76:0/3409131602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bcc0757a0 0x7f6bcc19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.491+0000 7f6bd0bd1640 1 -- 172.21.15.76:0/3409131602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bcc0741c0 msgr2=0x7f6bcc199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:07.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.491+0000 7f6bd0bd1640 1 --2- 172.21.15.76:0/3409131602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bcc0741c0 0x7f6bcc199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.491+0000 7f6bd0bd1640 1 -- 172.21.15.76:0/3409131602 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bcc1a1920 con 0x7f6bcc1096b0 2024-03-18T14:53:07.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.491+0000 7f6bcb7fe640 1 --2- 172.21.15.76:0/3409131602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bcc0757a0 0x7f6bcc19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:07.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.491+0000 7f6bd0bd1640 1 --2- 172.21.15.76:0/3409131602 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6bcc1096b0 0x7f6bcc1a1280 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6bc400efd0 tx=0x7f6bc400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:07.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.491+0000 7f6bc97fa640 1 -- 172.21.15.76:0/3409131602 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc4003b70 con 0x7f6bcc1096b0 2024-03-18T14:53:07.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.491+0000 7f6bc97fa640 1 -- 172.21.15.76:0/3409131602 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bc4003d10 con 0x7f6bcc1096b0 2024-03-18T14:53:07.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.491+0000 7f6bd2634640 1 -- 172.21.15.76:0/3409131602 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bcc1a1c10 con 0x7f6bcc1096b0 2024-03-18T14:53:07.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.491+0000 7f6bc97fa640 1 -- 172.21.15.76:0/3409131602 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc40109f0 con 0x7f6bcc1096b0 2024-03-18T14:53:07.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.491+0000 7f6bd2634640 1 -- 172.21.15.76:0/3409131602 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bcc1a2110 con 0x7f6bcc1096b0 2024-03-18T14:53:07.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.492+0000 7f6bd2634640 1 -- 172.21.15.76:0/3409131602 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bcc108470 con 0x7f6bcc1096b0 2024-03-18T14:53:07.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.493+0000 7f6bc97fa640 1 -- 172.21.15.76:0/3409131602 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6bc4004020 con 0x7f6bcc1096b0 2024-03-18T14:53:07.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.494+0000 7f6bc97fa640 1 --2- 172.21.15.76:0/3409131602 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6bb40a2800 0x7f6bb40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:07.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.494+0000 7f6bcbfff640 1 --2- 172.21.15.76:0/3409131602 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6bb40a2800 0x7f6bb40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:07.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.494+0000 7f6bc97fa640 1 -- 172.21.15.76:0/3409131602 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6bc40ceec0 con 0x7f6bcc1096b0 2024-03-18T14:53:07.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.494+0000 7f6bcbfff640 1 --2- 172.21.15.76:0/3409131602 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6bb40a2800 0x7f6bb40a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc19b1e0 tx=0x7f6bbc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:07.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.498+0000 7f6bc97fa640 1 -- 172.21.15.76:0/3409131602 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6bc409bf90 con 0x7f6bcc1096b0 2024-03-18T14:53:07.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:07 smithi076 ceph-mon[32205]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:53:07.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:07 smithi076 ceph-mon[32205]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:53:07.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:07 smithi076 ceph-mon[32205]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:53:07.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.674+0000 7f6bd2634640 1 -- 172.21.15.76:0/3409131602 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6bcc062480 con 0x7f6bcc1096b0 2024-03-18T14:53:07.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.676+0000 7f6bc97fa640 1 -- 172.21.15.76:0/3409131602 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22580 (secure 0 0 0) 0x7f6bc409b890 con 0x7f6bcc1096b0 2024-03-18T14:53:07.677 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:53:07.677 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:52:07.361301+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:53:07.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.678+0000 7f6bd2634640 1 -- 172.21.15.76:0/3409131602 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6bb40a2800 msgr2=0x7f6bb40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:07.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.678+0000 7f6bd2634640 1 --2- 172.21.15.76:0/3409131602 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6bb40a2800 0x7f6bb40a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc19b1e0 tx=0x7f6bbc005f90 comp rx=0 tx=0).stop 2024-03-18T14:53:07.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.678+0000 7f6bd2634640 1 -- 172.21.15.76:0/3409131602 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6bcc1096b0 msgr2=0x7f6bcc1a1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:07.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.678+0000 7f6bd2634640 1 --2- 172.21.15.76:0/3409131602 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6bcc1096b0 0x7f6bcc1a1280 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6bc400efd0 tx=0x7f6bc400c6a0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.678+0000 7f6bd2634640 1 -- 172.21.15.76:0/3409131602 shutdown_connections 2024-03-18T14:53:07.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.678+0000 7f6bd2634640 1 --2- 172.21.15.76:0/3409131602 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6bb40a2800 0x7f6bb40a4cc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.678+0000 7f6bd2634640 1 --2- 172.21.15.76:0/3409131602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bcc0757a0 0x7f6bcc19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.678+0000 7f6bd2634640 1 --2- 172.21.15.76:0/3409131602 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6bcc1096b0 0x7f6bcc1a1280 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.678+0000 7f6bd2634640 1 --2- 172.21.15.76:0/3409131602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bcc0741c0 0x7f6bcc199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.678+0000 7f6bd2634640 1 -- 172.21.15.76:0/3409131602 >> 172.21.15.76:0/3409131602 conn(0x7f6bcc0fcd90 msgr2=0x7f6bcc105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:07.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.678+0000 7f6bd2634640 1 -- 172.21.15.76:0/3409131602 shutdown_connections 2024-03-18T14:53:07.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.678+0000 7f6bd2634640 1 -- 172.21.15.76:0/3409131602 wait complete. 2024-03-18T14:53:07.690 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:46:26.588384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:50:59.606852+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:51:00.465325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:53:07.690 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:53:07.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.794+0000 7efcab885640 1 -- 172.21.15.76:0/187428595 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efca41029e0 msgr2=0x7efca4102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:07.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.794+0000 7efcab885640 1 --2- 172.21.15.76:0/187428595 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efca41029e0 0x7efca4102e60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efc94009900 tx=0x7efc9402bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:07.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.794+0000 7efcab885640 1 -- 172.21.15.76:0/187428595 shutdown_connections 2024-03-18T14:53:07.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.794+0000 7efcab885640 1 --2- 172.21.15.76:0/187428595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efca41033a0 0x7efca4109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.794+0000 7efcab885640 1 --2- 172.21.15.76:0/187428595 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efca41029e0 0x7efca4102e60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.794+0000 7efcab885640 1 --2- 172.21.15.76:0/187428595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efca41017f0 0x7efca4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.794+0000 7efcab885640 1 -- 172.21.15.76:0/187428595 >> 172.21.15.76:0/187428595 conn(0x7efca40fcd80 msgr2=0x7efca40ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:07.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.794+0000 7efcab885640 1 -- 172.21.15.76:0/187428595 shutdown_connections 2024-03-18T14:53:07.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.794+0000 7efcab885640 1 -- 172.21.15.76:0/187428595 wait complete. 2024-03-18T14:53:07.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efcab885640 1 Processor -- start 2024-03-18T14:53:07.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efcab885640 1 -- start start 2024-03-18T14:53:07.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efcab885640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efca41017f0 0x7efca4197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:07.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efcab885640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efca41029e0 0x7efca4198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:07.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efcab885640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efca41033a0 0x7efca419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:07.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efcab885640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efca410d470 con 0x7efca41033a0 2024-03-18T14:53:07.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efcab885640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efca410d2f0 con 0x7efca41029e0 2024-03-18T14:53:07.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efcab885640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efca410d5f0 con 0x7efca41017f0 2024-03-18T14:53:07.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efca9e22640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efca41033a0 0x7efca419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:07.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efca9621640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efca41017f0 0x7efca4197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:07.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efca9e22640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efca41033a0 0x7efca419f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50476/0 (socket says 172.21.15.76:50476) 2024-03-18T14:53:07.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efca9e22640 1 -- 172.21.15.76:0/4101166994 learned_addr learned my addr 172.21.15.76:0/4101166994 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:07.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efca8e20640 1 --2- 172.21.15.76:0/4101166994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efca41029e0 0x7efca4198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:07.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efca9e22640 1 -- 172.21.15.76:0/4101166994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efca41017f0 msgr2=0x7efca4197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:07.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efca9e22640 1 --2- 172.21.15.76:0/4101166994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efca41017f0 0x7efca4197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efca9e22640 1 -- 172.21.15.76:0/4101166994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efca41029e0 msgr2=0x7efca4198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:07.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efca9e22640 1 --2- 172.21.15.76:0/4101166994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efca41029e0 0x7efca4198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efca9e22640 1 -- 172.21.15.76:0/4101166994 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca419f730 con 0x7efca41033a0 2024-03-18T14:53:07.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efca9621640 1 --2- 172.21.15.76:0/4101166994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efca41017f0 0x7efca4197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:07.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efca9e22640 1 --2- 172.21.15.76:0/4101166994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efca41033a0 0x7efca419f0c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00eb10 tx=0x7efc9c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:07.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.795+0000 7efc9a7fc640 1 -- 172.21.15.76:0/4101166994 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c004b20 con 0x7efca41033a0 2024-03-18T14:53:07.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.796+0000 7efc9a7fc640 1 -- 172.21.15.76:0/4101166994 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc9c004cc0 con 0x7efca41033a0 2024-03-18T14:53:07.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.796+0000 7efcab885640 1 -- 172.21.15.76:0/4101166994 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca419fa20 con 0x7efca41033a0 2024-03-18T14:53:07.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.796+0000 7efc9a7fc640 1 -- 172.21.15.76:0/4101166994 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c010760 con 0x7efca41033a0 2024-03-18T14:53:07.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.796+0000 7efcab885640 1 -- 172.21.15.76:0/4101166994 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca419fef0 con 0x7efca41033a0 2024-03-18T14:53:07.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.796+0000 7efcab885640 1 -- 172.21.15.76:0/4101166994 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca4108b90 con 0x7efca41033a0 2024-03-18T14:53:07.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.798+0000 7efc9a7fc640 1 -- 172.21.15.76:0/4101166994 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7efc9c005020 con 0x7efca41033a0 2024-03-18T14:53:07.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.798+0000 7efc9a7fc640 1 --2- 172.21.15.76:0/4101166994 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efc900a2800 0x7efc900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:07.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.798+0000 7efca9621640 1 --2- 172.21.15.76:0/4101166994 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efc900a2800 0x7efc900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:07.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.798+0000 7efc9a7fc640 1 -- 172.21.15.76:0/4101166994 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7efc9c0cefa0 con 0x7efca41033a0 2024-03-18T14:53:07.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.799+0000 7efca9621640 1 --2- 172.21.15.76:0/4101166994 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efc900a2800 0x7efc900a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efca4102840 tx=0x7efca0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:07.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.802+0000 7efc9a7fc640 1 -- 172.21.15.76:0/4101166994 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efc9c09c070 con 0x7efca41033a0 2024-03-18T14:53:07.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:07 smithi174 ceph-mon[36787]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:53:07.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:07 smithi174 ceph-mon[36787]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:53:07.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:07 smithi174 ceph-mon[36787]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:53:07.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:07 smithi077 ceph-mon[35865]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:53:07.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:07 smithi077 ceph-mon[35865]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:53:07.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:07 smithi077 ceph-mon[35865]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T14:53:07.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.977+0000 7efcab885640 1 -- 172.21.15.76:0/4101166994 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efca4062480 con 0x7efca41033a0 2024-03-18T14:53:07.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.978+0000 7efc9a7fc640 1 -- 172.21.15.76:0/4101166994 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7efc9c09b970 con 0x7efca41033a0 2024-03-18T14:53:07.979 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-18T14:53:07.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.980+0000 7efcab885640 1 -- 172.21.15.76:0/4101166994 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efc900a2800 msgr2=0x7efc900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:07.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.980+0000 7efcab885640 1 --2- 172.21.15.76:0/4101166994 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efc900a2800 0x7efc900a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efca4102840 tx=0x7efca0005f90 comp rx=0 tx=0).stop 2024-03-18T14:53:07.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.980+0000 7efcab885640 1 -- 172.21.15.76:0/4101166994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efca41033a0 msgr2=0x7efca419f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:07.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.980+0000 7efcab885640 1 --2- 172.21.15.76:0/4101166994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efca41033a0 0x7efca419f0c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00eb10 tx=0x7efc9c00c2d0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.980+0000 7efcab885640 1 -- 172.21.15.76:0/4101166994 shutdown_connections 2024-03-18T14:53:07.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.980+0000 7efcab885640 1 --2- 172.21.15.76:0/4101166994 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7efc900a2800 0x7efc900a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.980+0000 7efcab885640 1 --2- 172.21.15.76:0/4101166994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efca41033a0 0x7efca419f0c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.980+0000 7efcab885640 1 --2- 172.21.15.76:0/4101166994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efca41029e0 0x7efca4198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.980+0000 7efcab885640 1 --2- 172.21.15.76:0/4101166994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efca41017f0 0x7efca4197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:07.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.980+0000 7efcab885640 1 -- 172.21.15.76:0/4101166994 >> 172.21.15.76:0/4101166994 conn(0x7efca40fcd80 msgr2=0x7efca4105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:07.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.980+0000 7efcab885640 1 -- 172.21.15.76:0/4101166994 shutdown_connections 2024-03-18T14:53:07.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:07.980+0000 7efcab885640 1 -- 172.21.15.76:0/4101166994 wait complete. 2024-03-18T14:53:07.992 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:53:08.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.096+0000 7f42a831b640 1 -- 172.21.15.76:0/1074269841 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a01022b0 msgr2=0x7f42a0108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.096+0000 7f42a831b640 1 --2- 172.21.15.76:0/1074269841 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a01022b0 0x7f42a0108b40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f429c009ca0 tx=0x7f429c02c0f0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.096+0000 7f42a831b640 1 -- 172.21.15.76:0/1074269841 shutdown_connections 2024-03-18T14:53:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.096+0000 7f42a831b640 1 --2- 172.21.15.76:0/1074269841 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a01022b0 0x7f42a0108b40 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.096+0000 7f42a831b640 1 --2- 172.21.15.76:0/1074269841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a01018f0 0x7f42a0101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.096+0000 7f42a831b640 1 --2- 172.21.15.76:0/1074269841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42a0100700 0x7f42a0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.096+0000 7f42a831b640 1 -- 172.21.15.76:0/1074269841 >> 172.21.15.76:0/1074269841 conn(0x7f42a00fbc90 msgr2=0x7f42a00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.096+0000 7f42a831b640 1 -- 172.21.15.76:0/1074269841 shutdown_connections 2024-03-18T14:53:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.096+0000 7f42a831b640 1 -- 172.21.15.76:0/1074269841 wait complete. 2024-03-18T14:53:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.096+0000 7f42a831b640 1 Processor -- start 2024-03-18T14:53:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.096+0000 7f42a831b640 1 -- start start 2024-03-18T14:53:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.096+0000 7f42a831b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42a0100700 0x7f42a0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.096+0000 7f42a831b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a01018f0 0x7f42a0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.096+0000 7f42a831b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a01022b0 0x7f42a019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.096+0000 7f42a831b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f42a010c380 con 0x7f42a01022b0 2024-03-18T14:53:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.096+0000 7f42a831b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f42a010c200 con 0x7f42a01018f0 2024-03-18T14:53:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.096+0000 7f42a831b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f42a010c500 con 0x7f42a0100700 2024-03-18T14:53:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.097+0000 7f42a68b8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a01022b0 0x7f42a019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.097+0000 7f42a60b7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42a0100700 0x7f42a0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.097+0000 7f42a68b8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a01022b0 0x7f42a019f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50504/0 (socket says 172.21.15.76:50504) 2024-03-18T14:53:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.097+0000 7f42a60b7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42a0100700 0x7f42a0197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:42616/0 (socket says 172.21.15.76:42616) 2024-03-18T14:53:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.097+0000 7f42a68b8640 1 -- 172.21.15.76:0/3339735455 learned_addr learned my addr 172.21.15.76:0/3339735455 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.097+0000 7f42a58b6640 1 --2- 172.21.15.76:0/3339735455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a01018f0 0x7f42a0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.097+0000 7f42a68b8640 1 -- 172.21.15.76:0/3339735455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42a0100700 msgr2=0x7f42a0197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.097+0000 7f42a68b8640 1 --2- 172.21.15.76:0/3339735455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42a0100700 0x7f42a0197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.097+0000 7f42a68b8640 1 -- 172.21.15.76:0/3339735455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a01018f0 msgr2=0x7f42a0198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.097+0000 7f42a68b8640 1 --2- 172.21.15.76:0/3339735455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a01018f0 0x7f42a0198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.097+0000 7f42a68b8640 1 -- 172.21.15.76:0/3339735455 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a019f6e0 con 0x7f42a01022b0 2024-03-18T14:53:08.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.097+0000 7f42a68b8640 1 --2- 172.21.15.76:0/3339735455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a01022b0 0x7f42a019f070 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f429c037040 tx=0x7f429c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:08.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.097+0000 7f42977fe640 1 -- 172.21.15.76:0/3339735455 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429c041020 con 0x7f42a01022b0 2024-03-18T14:53:08.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.097+0000 7f42977fe640 1 -- 172.21.15.76:0/3339735455 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f429c034cd0 con 0x7f42a01022b0 2024-03-18T14:53:08.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.097+0000 7f42a831b640 1 -- 172.21.15.76:0/3339735455 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42a019f970 con 0x7f42a01022b0 2024-03-18T14:53:08.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.097+0000 7f42977fe640 1 -- 172.21.15.76:0/3339735455 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429c039040 con 0x7f42a01022b0 2024-03-18T14:53:08.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.097+0000 7f42a831b640 1 -- 172.21.15.76:0/3339735455 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42a019fe60 con 0x7f42a01022b0 2024-03-18T14:53:08.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.098+0000 7f42a831b640 1 -- 172.21.15.76:0/3339735455 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42a01a0090 con 0x7f42a01022b0 2024-03-18T14:53:08.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.099+0000 7f42977fe640 1 -- 172.21.15.76:0/3339735455 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f429c044020 con 0x7f42a01022b0 2024-03-18T14:53:08.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.100+0000 7f42977fe640 1 --2- 172.21.15.76:0/3339735455 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f428c0a2800 0x7f428c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:08.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.100+0000 7f42a60b7640 1 --2- 172.21.15.76:0/3339735455 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f428c0a2800 0x7f428c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:08.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.100+0000 7f42977fe640 1 -- 172.21.15.76:0/3339735455 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f429c0f0d10 con 0x7f42a01022b0 2024-03-18T14:53:08.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.100+0000 7f42a60b7640 1 --2- 172.21.15.76:0/3339735455 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f428c0a2800 0x7f428c0a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f42a0101750 tx=0x7f4298005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:08.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.103+0000 7f42977fe640 1 -- 172.21.15.76:0/3339735455 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f429c04b020 con 0x7f42a01022b0 2024-03-18T14:53:08.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.278+0000 7f42a831b640 1 -- 172.21.15.76:0/3339735455 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f42a01a0420 con 0x7f42a01022b0 2024-03-18T14:53:08.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.278+0000 7f42977fe640 1 -- 172.21.15.76:0/3339735455 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f429c0bde70 con 0x7f42a01022b0 2024-03-18T14:53:08.279 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-18T14:53:08.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.280+0000 7f42a831b640 1 -- 172.21.15.76:0/3339735455 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f428c0a2800 msgr2=0x7f428c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:08.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.280+0000 7f42a831b640 1 --2- 172.21.15.76:0/3339735455 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f428c0a2800 0x7f428c0a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f42a0101750 tx=0x7f4298005f90 comp rx=0 tx=0).stop 2024-03-18T14:53:08.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.280+0000 7f42a831b640 1 -- 172.21.15.76:0/3339735455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a01022b0 msgr2=0x7f42a019f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:08.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.280+0000 7f42a831b640 1 --2- 172.21.15.76:0/3339735455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a01022b0 0x7f42a019f070 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f429c037040 tx=0x7f429c032f40 comp rx=0 tx=0).stop 2024-03-18T14:53:08.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.280+0000 7f42a831b640 1 -- 172.21.15.76:0/3339735455 shutdown_connections 2024-03-18T14:53:08.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.280+0000 7f42a831b640 1 --2- 172.21.15.76:0/3339735455 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f428c0a2800 0x7f428c0a4cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.280+0000 7f42a831b640 1 --2- 172.21.15.76:0/3339735455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a01022b0 0x7f42a019f070 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.280+0000 7f42a831b640 1 --2- 172.21.15.76:0/3339735455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a01018f0 0x7f42a0198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.280+0000 7f42a831b640 1 --2- 172.21.15.76:0/3339735455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42a0100700 0x7f42a0197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.280+0000 7f42a831b640 1 -- 172.21.15.76:0/3339735455 >> 172.21.15.76:0/3339735455 conn(0x7f42a00fbc90 msgr2=0x7f42a0069db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:08.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.280+0000 7f42a831b640 1 -- 172.21.15.76:0/3339735455 shutdown_connections 2024-03-18T14:53:08.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.280+0000 7f42a831b640 1 -- 172.21.15.76:0/3339735455 wait complete. 2024-03-18T14:53:08.292 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:53:08.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.396+0000 7f2fa546e640 1 -- 172.21.15.76:0/895084671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2fa01018d0 msgr2=0x7f2fa0101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:08.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.396+0000 7f2fa546e640 1 --2- 172.21.15.76:0/895084671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2fa01018d0 0x7f2fa0101d50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009900 tx=0x7f2f9402bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:08.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.396+0000 7f2fa546e640 1 -- 172.21.15.76:0/895084671 shutdown_connections 2024-03-18T14:53:08.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.396+0000 7f2fa546e640 1 --2- 172.21.15.76:0/895084671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fa0102290 0x7f2fa0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.396+0000 7f2fa546e640 1 --2- 172.21.15.76:0/895084671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2fa01018d0 0x7f2fa0101d50 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.396+0000 7f2fa546e640 1 --2- 172.21.15.76:0/895084671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fa01006e0 0x7f2fa0100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.396+0000 7f2fa546e640 1 -- 172.21.15.76:0/895084671 >> 172.21.15.76:0/895084671 conn(0x7f2fa00fbc70 msgr2=0x7f2fa00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:08.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.396+0000 7f2fa546e640 1 -- 172.21.15.76:0/895084671 shutdown_connections 2024-03-18T14:53:08.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.396+0000 7f2fa546e640 1 -- 172.21.15.76:0/895084671 wait complete. 2024-03-18T14:53:08.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.397+0000 7f2fa546e640 1 Processor -- start 2024-03-18T14:53:08.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.397+0000 7f2fa546e640 1 -- start start 2024-03-18T14:53:08.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.397+0000 7f2fa546e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fa01006e0 0x7f2fa0197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:08.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.397+0000 7f2fa546e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fa01018d0 0x7f2fa0198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:08.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.397+0000 7f2fa546e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2fa0102290 0x7f2fa019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:08.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.397+0000 7f2fa546e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa010c360 con 0x7f2fa0102290 2024-03-18T14:53:08.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.397+0000 7f2fa546e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa010c1e0 con 0x7f2fa01018d0 2024-03-18T14:53:08.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.397+0000 7f2fa546e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa010c4e0 con 0x7f2fa01006e0 2024-03-18T14:53:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.397+0000 7f2f9f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2fa0102290 0x7f2fa019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.397+0000 7f2f9effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fa01006e0 0x7f2fa0197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.397+0000 7f2f9f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2fa0102290 0x7f2fa019f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50532/0 (socket says 172.21.15.76:50532) 2024-03-18T14:53:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.397+0000 7f2f9f7fe640 1 -- 172.21.15.76:0/858824079 learned_addr learned my addr 172.21.15.76:0/858824079 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.397+0000 7f2f9e7fc640 1 --2- 172.21.15.76:0/858824079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fa01018d0 0x7f2fa0198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.398+0000 7f2f9f7fe640 1 -- 172.21.15.76:0/858824079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fa01006e0 msgr2=0x7f2fa0197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.398+0000 7f2f9f7fe640 1 --2- 172.21.15.76:0/858824079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fa01006e0 0x7f2fa0197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.398+0000 7f2f9f7fe640 1 -- 172.21.15.76:0/858824079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fa01018d0 msgr2=0x7f2fa0198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.398+0000 7f2f9f7fe640 1 --2- 172.21.15.76:0/858824079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fa01018d0 0x7f2fa0198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.398+0000 7f2f9f7fe640 1 -- 172.21.15.76:0/858824079 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa019f730 con 0x7f2fa0102290 2024-03-18T14:53:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.398+0000 7f2f9effd640 1 --2- 172.21.15.76:0/858824079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fa01006e0 0x7f2fa0197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.398+0000 7f2f9f7fe640 1 --2- 172.21.15.76:0/858824079 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2fa0102290 0x7f2fa019f0c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f2f9800e970 tx=0x7f2f9800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.398+0000 7f2f87fff640 1 -- 172.21.15.76:0/858824079 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9800ccc0 con 0x7f2fa0102290 2024-03-18T14:53:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.398+0000 7f2fa546e640 1 -- 172.21.15.76:0/858824079 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa019fa20 con 0x7f2fa0102290 2024-03-18T14:53:08.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.398+0000 7f2f87fff640 1 -- 172.21.15.76:0/858824079 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f9800ce60 con 0x7f2fa0102290 2024-03-18T14:53:08.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.398+0000 7f2f87fff640 1 -- 172.21.15.76:0/858824079 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f98010490 con 0x7f2fa0102290 2024-03-18T14:53:08.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.398+0000 7f2fa546e640 1 -- 172.21.15.76:0/858824079 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa019ff40 con 0x7f2fa0102290 2024-03-18T14:53:08.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.399+0000 7f2fa546e640 1 -- 172.21.15.76:0/858824079 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fa01a01d0 con 0x7f2fa0102290 2024-03-18T14:53:08.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.400+0000 7f2f87fff640 1 -- 172.21.15.76:0/858824079 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2f98005020 con 0x7f2fa0102290 2024-03-18T14:53:08.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.400+0000 7f2f87fff640 1 --2- 172.21.15.76:0/858824079 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2f880a2800 0x7f2f880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:08.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.401+0000 7f2f9effd640 1 --2- 172.21.15.76:0/858824079 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2f880a2800 0x7f2f880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:08.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.401+0000 7f2f87fff640 1 -- 172.21.15.76:0/858824079 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f2f980ceeb0 con 0x7f2fa0102290 2024-03-18T14:53:08.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.401+0000 7f2f9effd640 1 --2- 172.21.15.76:0/858824079 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2f880a2800 0x7f2f880a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0101730 tx=0x7f2f90009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:08.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.405+0000 7f2f87fff640 1 -- 172.21.15.76:0/858824079 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2f9809bf80 con 0x7f2fa0102290 2024-03-18T14:53:08.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.584+0000 7f2fa546e640 1 -- 172.21.15.76:0/858824079 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2fa01a03a0 con 0x7f2fa0102290 2024-03-18T14:53:08.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.584+0000 7f2f87fff640 1 -- 172.21.15.76:0/858824079 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f2f9809b880 con 0x7f2fa0102290 2024-03-18T14:53:08.585 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-18T14:53:08.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.586+0000 7f2fa546e640 1 -- 172.21.15.76:0/858824079 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2f880a2800 msgr2=0x7f2f880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:08.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.586+0000 7f2fa546e640 1 --2- 172.21.15.76:0/858824079 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2f880a2800 0x7f2f880a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0101730 tx=0x7f2f90009500 comp rx=0 tx=0).stop 2024-03-18T14:53:08.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.586+0000 7f2fa546e640 1 -- 172.21.15.76:0/858824079 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2fa0102290 msgr2=0x7f2fa019f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:08.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.586+0000 7f2fa546e640 1 --2- 172.21.15.76:0/858824079 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2fa0102290 0x7f2fa019f0c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f2f9800e970 tx=0x7f2f9800ed70 comp rx=0 tx=0).stop 2024-03-18T14:53:08.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.586+0000 7f2fa546e640 1 -- 172.21.15.76:0/858824079 shutdown_connections 2024-03-18T14:53:08.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.586+0000 7f2fa546e640 1 --2- 172.21.15.76:0/858824079 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2f880a2800 0x7f2f880a4cc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.586+0000 7f2fa546e640 1 --2- 172.21.15.76:0/858824079 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2fa0102290 0x7f2fa019f0c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.586+0000 7f2fa546e640 1 --2- 172.21.15.76:0/858824079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fa01018d0 0x7f2fa0198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.586+0000 7f2fa546e640 1 --2- 172.21.15.76:0/858824079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fa01006e0 0x7f2fa0197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.586+0000 7f2fa546e640 1 -- 172.21.15.76:0/858824079 >> 172.21.15.76:0/858824079 conn(0x7f2fa00fbc70 msgr2=0x7f2fa0068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:08.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.586+0000 7f2fa546e640 1 -- 172.21.15.76:0/858824079 shutdown_connections 2024-03-18T14:53:08.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.586+0000 7f2fa546e640 1 -- 172.21.15.76:0/858824079 wait complete. 2024-03-18T14:53:08.598 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:53:08.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.699+0000 7fbed41d3640 1 -- 172.21.15.76:0/2140357823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbecc1022e0 msgr2=0x7fbecc1046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.699+0000 7fbed41d3640 1 --2- 172.21.15.76:0/2140357823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbecc1022e0 0x7fbecc1046d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009900 tx=0x7fbec802bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.699+0000 7fbed41d3640 1 -- 172.21.15.76:0/2140357823 shutdown_connections 2024-03-18T14:53:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.699+0000 7fbed41d3640 1 --2- 172.21.15.76:0/2140357823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbecc107540 0x7fbecc109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.699+0000 7fbed41d3640 1 --2- 172.21.15.76:0/2140357823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbecc104c10 0x7fbecc107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.699+0000 7fbed41d3640 1 --2- 172.21.15.76:0/2140357823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbecc1022e0 0x7fbecc1046d0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.699+0000 7fbed41d3640 1 -- 172.21.15.76:0/2140357823 >> 172.21.15.76:0/2140357823 conn(0x7fbecc0fbc70 msgr2=0x7fbecc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.699+0000 7fbed41d3640 1 -- 172.21.15.76:0/2140357823 shutdown_connections 2024-03-18T14:53:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.699+0000 7fbed41d3640 1 -- 172.21.15.76:0/2140357823 wait complete. 2024-03-18T14:53:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed41d3640 1 Processor -- start 2024-03-18T14:53:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed41d3640 1 -- start start 2024-03-18T14:53:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed41d3640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbecc1022e0 0x7fbecc197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed41d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbecc104c10 0x7fbecc198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed41d3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbecc107540 0x7fbecc19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed41d3640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc10d4a0 con 0x7fbecc1022e0 2024-03-18T14:53:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed41d3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc10d320 con 0x7fbecc104c10 2024-03-18T14:53:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed41d3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc10d620 con 0x7fbecc107540 2024-03-18T14:53:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed1f6f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbecc1022e0 0x7fbecc197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed1f6f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbecc1022e0 0x7fbecc197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44692/0 (socket says 172.21.15.76:44692) 2024-03-18T14:53:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed1f6f640 1 -- 172.21.15.76:0/416094150 learned_addr learned my addr 172.21.15.76:0/416094150 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed176e640 1 --2- 172.21.15.76:0/416094150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbecc104c10 0x7fbecc198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed2770640 1 --2- 172.21.15.76:0/416094150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbecc107540 0x7fbecc19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed1f6f640 1 -- 172.21.15.76:0/416094150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbecc107540 msgr2=0x7fbecc19f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed1f6f640 1 --2- 172.21.15.76:0/416094150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbecc107540 0x7fbecc19f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed1f6f640 1 -- 172.21.15.76:0/416094150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbecc104c10 msgr2=0x7fbecc198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed1f6f640 1 --2- 172.21.15.76:0/416094150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbecc104c10 0x7fbecc198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed1f6f640 1 -- 172.21.15.76:0/416094150 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbecc19f740 con 0x7fbecc1022e0 2024-03-18T14:53:08.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.700+0000 7fbed2770640 1 --2- 172.21.15.76:0/416094150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbecc107540 0x7fbecc19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:53:08.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.701+0000 7fbed1f6f640 1 --2- 172.21.15.76:0/416094150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbecc1022e0 0x7fbecc197b20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009230 tx=0x7fbec802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:08.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.701+0000 7fbec2ffd640 1 -- 172.21.15.76:0/416094150 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8041070 con 0x7fbecc1022e0 2024-03-18T14:53:08.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.701+0000 7fbec2ffd640 1 -- 172.21.15.76:0/416094150 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbec8004450 con 0x7fbecc1022e0 2024-03-18T14:53:08.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.701+0000 7fbed41d3640 1 -- 172.21.15.76:0/416094150 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbecc19f9d0 con 0x7fbecc1022e0 2024-03-18T14:53:08.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.701+0000 7fbec2ffd640 1 -- 172.21.15.76:0/416094150 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8003e20 con 0x7fbecc1022e0 2024-03-18T14:53:08.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.701+0000 7fbed41d3640 1 -- 172.21.15.76:0/416094150 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbecc19fec0 con 0x7fbecc1022e0 2024-03-18T14:53:08.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.702+0000 7fbed41d3640 1 -- 172.21.15.76:0/416094150 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbecc0fd760 con 0x7fbecc1022e0 2024-03-18T14:53:08.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.703+0000 7fbec2ffd640 1 -- 172.21.15.76:0/416094150 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbec8044070 con 0x7fbecc1022e0 2024-03-18T14:53:08.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.704+0000 7fbec2ffd640 1 --2- 172.21.15.76:0/416094150 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fbeb80a2800 0x7fbeb80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:08.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.704+0000 7fbed176e640 1 --2- 172.21.15.76:0/416094150 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fbeb80a2800 0x7fbeb80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:08.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.704+0000 7fbec2ffd640 1 -- 172.21.15.76:0/416094150 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fbec80f09e0 con 0x7fbecc1022e0 2024-03-18T14:53:08.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.704+0000 7fbed176e640 1 --2- 172.21.15.76:0/416094150 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fbeb80a2800 0x7fbeb80a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbecc199030 tx=0x7fbebc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:08.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.707+0000 7fbec2ffd640 1 -- 172.21.15.76:0/416094150 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbec80bdab0 con 0x7fbecc1022e0 2024-03-18T14:53:08.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:08 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3409131602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:53:08.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:08 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/4101166994' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:53:08.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:08 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3339735455' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:53:08.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:08 smithi076 ceph-mon[32205]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-18T14:53:08.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.920+0000 7fbed41d3640 1 -- 172.21.15.76:0/416094150 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbecc02d3b0 con 0x7fbecc1022e0 2024-03-18T14:53:08.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.921+0000 7fbec2ffd640 1 -- 172.21.15.76:0/416094150 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11863 (secure 0 0 0) 0x7fbec80bd3b0 con 0x7fbecc1022e0 2024-03-18T14:53:08.922 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:53:08.922 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/2801069369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":2801069369},{"type":"v1","addr":"172.21.15.76:6839","nonce":2801069369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2645229232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2645229232},{"type":"v1","addr":"172.21.15.76:6841","nonce":2645229232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T14:51:01.293009+0000","modified":"2024-03-18T14:53:02.728776+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24490,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6839/399810652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":399810652},{"type":"v1","addr":"172.21.15.174:6839","nonce":399810652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1685041985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1685041985},{"type":"v1","addr":"172.21.15.77:6839","nonce":1685041985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524,24463,24490]},"id":1}]} 2024-03-18T14:53:08.922 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 24 2024-03-18T14:53:08.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.923+0000 7fbed41d3640 1 -- 172.21.15.76:0/416094150 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fbeb80a2800 msgr2=0x7fbeb80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:08.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.923+0000 7fbed41d3640 1 --2- 172.21.15.76:0/416094150 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fbeb80a2800 0x7fbeb80a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbecc199030 tx=0x7fbebc005f90 comp rx=0 tx=0).stop 2024-03-18T14:53:08.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.923+0000 7fbed41d3640 1 -- 172.21.15.76:0/416094150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbecc1022e0 msgr2=0x7fbecc197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:08.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.923+0000 7fbed41d3640 1 --2- 172.21.15.76:0/416094150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbecc1022e0 0x7fbecc197b20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009230 tx=0x7fbec802c650 comp rx=0 tx=0).stop 2024-03-18T14:53:08.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.923+0000 7fbed41d3640 1 -- 172.21.15.76:0/416094150 shutdown_connections 2024-03-18T14:53:08.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.923+0000 7fbed41d3640 1 --2- 172.21.15.76:0/416094150 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fbeb80a2800 0x7fbeb80a4cc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.923+0000 7fbed41d3640 1 --2- 172.21.15.76:0/416094150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbecc107540 0x7fbecc19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.923+0000 7fbed41d3640 1 --2- 172.21.15.76:0/416094150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbecc104c10 0x7fbecc198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.923+0000 7fbed41d3640 1 --2- 172.21.15.76:0/416094150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbecc1022e0 0x7fbecc197b20 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:08.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.923+0000 7fbed41d3640 1 -- 172.21.15.76:0/416094150 >> 172.21.15.76:0/416094150 conn(0x7fbecc0fbc70 msgr2=0x7fbecc102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:08.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.923+0000 7fbed41d3640 1 -- 172.21.15.76:0/416094150 shutdown_connections 2024-03-18T14:53:08.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:08.923+0000 7fbed41d3640 1 -- 172.21.15.76:0/416094150 wait complete. 2024-03-18T14:53:08.935 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:53:08.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:08 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3409131602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:53:08.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:08 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/4101166994' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:53:08.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:08 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3339735455' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:53:08.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:08 smithi174 ceph-mon[36787]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-18T14:53:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:08 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3409131602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:53:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:08 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/4101166994' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T14:53:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:08 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3339735455' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T14:53:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:08 smithi077 ceph-mon[35865]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-18T14:53:09.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.037+0000 7fa637753640 1 -- 172.21.15.76:0/680061053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa630101800 msgr2=0x7fa630101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.037+0000 7fa637753640 1 --2- 172.21.15.76:0/680061053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa630101800 0x7fa630101c20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa62c009900 tx=0x7fa62c02bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:09.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.037+0000 7fa637753640 1 -- 172.21.15.76:0/680061053 shutdown_connections 2024-03-18T14:53:09.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.037+0000 7fa637753640 1 --2- 172.21.15.76:0/680061053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6301029f0 0x7fa630102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.037+0000 7fa637753640 1 --2- 172.21.15.76:0/680061053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6301033b0 0x7fa630109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.037+0000 7fa637753640 1 --2- 172.21.15.76:0/680061053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa630101800 0x7fa630101c20 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.037+0000 7fa637753640 1 -- 172.21.15.76:0/680061053 >> 172.21.15.76:0/680061053 conn(0x7fa6300fcd90 msgr2=0x7fa6300ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:09.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.037+0000 7fa637753640 1 -- 172.21.15.76:0/680061053 shutdown_connections 2024-03-18T14:53:09.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.037+0000 7fa637753640 1 -- 172.21.15.76:0/680061053 wait complete. 2024-03-18T14:53:09.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.037+0000 7fa637753640 1 Processor -- start 2024-03-18T14:53:09.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa637753640 1 -- start start 2024-03-18T14:53:09.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa637753640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa630101800 0x7fa630197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa637753640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6301029f0 0x7fa630198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa637753640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6301033b0 0x7fa63019f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa637753640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa63010d470 con 0x7fa6301033b0 2024-03-18T14:53:09.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa637753640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa63010d2f0 con 0x7fa630101800 2024-03-18T14:53:09.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa637753640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa63010d5f0 con 0x7fa6301029f0 2024-03-18T14:53:09.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa635cf0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6301033b0 0x7fa63019f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa6354ef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa630101800 0x7fa630197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa635cf0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6301033b0 0x7fa63019f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44706/0 (socket says 172.21.15.76:44706) 2024-03-18T14:53:09.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa635cf0640 1 -- 172.21.15.76:0/663042561 learned_addr learned my addr 172.21.15.76:0/663042561 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:09.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa634cee640 1 --2- 172.21.15.76:0/663042561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6301029f0 0x7fa630198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa635cf0640 1 -- 172.21.15.76:0/663042561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6301029f0 msgr2=0x7fa630198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa635cf0640 1 --2- 172.21.15.76:0/663042561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6301029f0 0x7fa630198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa635cf0640 1 -- 172.21.15.76:0/663042561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa630101800 msgr2=0x7fa630197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa635cf0640 1 --2- 172.21.15.76:0/663042561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa630101800 0x7fa630197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa635cf0640 1 -- 172.21.15.76:0/663042561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63019f740 con 0x7fa6301033b0 2024-03-18T14:53:09.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa6354ef640 1 --2- 172.21.15.76:0/663042561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa630101800 0x7fa630197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:09.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa635cf0640 1 --2- 172.21.15.76:0/663042561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6301033b0 0x7fa63019f0d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa62800eb10 tx=0x7fa62800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:09.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa6267fc640 1 -- 172.21.15.76:0/663042561 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628004b20 con 0x7fa6301033b0 2024-03-18T14:53:09.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa637753640 1 -- 172.21.15.76:0/663042561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa63019fa30 con 0x7fa6301033b0 2024-03-18T14:53:09.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa6267fc640 1 -- 172.21.15.76:0/663042561 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa628004cc0 con 0x7fa6301033b0 2024-03-18T14:53:09.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.038+0000 7fa6267fc640 1 -- 172.21.15.76:0/663042561 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628010760 con 0x7fa6301033b0 2024-03-18T14:53:09.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.039+0000 7fa637753640 1 -- 172.21.15.76:0/663042561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa63019ff30 con 0x7fa6301033b0 2024-03-18T14:53:09.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.039+0000 7fa637753640 1 -- 172.21.15.76:0/663042561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa630108b40 con 0x7fa6301033b0 2024-03-18T14:53:09.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.041+0000 7fa6267fc640 1 -- 172.21.15.76:0/663042561 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa628005000 con 0x7fa6301033b0 2024-03-18T14:53:09.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.041+0000 7fa6267fc640 1 --2- 172.21.15.76:0/663042561 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa61c0a2800 0x7fa61c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.041+0000 7fa6354ef640 1 --2- 172.21.15.76:0/663042561 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa61c0a2800 0x7fa61c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.041+0000 7fa6267fc640 1 -- 172.21.15.76:0/663042561 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fa6280cf1d0 con 0x7fa6301033b0 2024-03-18T14:53:09.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.042+0000 7fa6354ef640 1 --2- 172.21.15.76:0/663042561 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa61c0a2800 0x7fa61c0a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa62c000c00 tx=0x7fa62c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:09.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.045+0000 7fa6267fc640 1 -- 172.21.15.76:0/663042561 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa628053740 con 0x7fa6301033b0 2024-03-18T14:53:09.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.220+0000 7fa637753640 1 -- 172.21.15.76:0/663042561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa63002d400 con 0x7fa6301033b0 2024-03-18T14:53:09.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.221+0000 7fa6267fc640 1 -- 172.21.15.76:0/663042561 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22580 (secure 0 0 0) 0x7fa62809c2a0 con 0x7fa6301033b0 2024-03-18T14:53:09.222 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:53:09.222 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T14:52:07.361301+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:53:09.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.223+0000 7fa637753640 1 -- 172.21.15.76:0/663042561 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa61c0a2800 msgr2=0x7fa61c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.223+0000 7fa637753640 1 --2- 172.21.15.76:0/663042561 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa61c0a2800 0x7fa61c0a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa62c000c00 tx=0x7fa62c005fd0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.223+0000 7fa637753640 1 -- 172.21.15.76:0/663042561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6301033b0 msgr2=0x7fa63019f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.223+0000 7fa637753640 1 --2- 172.21.15.76:0/663042561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6301033b0 0x7fa63019f0d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa62800eb10 tx=0x7fa62800c2d0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.223+0000 7fa637753640 1 -- 172.21.15.76:0/663042561 shutdown_connections 2024-03-18T14:53:09.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.223+0000 7fa637753640 1 --2- 172.21.15.76:0/663042561 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa61c0a2800 0x7fa61c0a4cc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.223+0000 7fa637753640 1 --2- 172.21.15.76:0/663042561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6301029f0 0x7fa630198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.223+0000 7fa637753640 1 --2- 172.21.15.76:0/663042561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6301033b0 0x7fa63019f0d0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.223+0000 7fa637753640 1 --2- 172.21.15.76:0/663042561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa630101800 0x7fa630197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.223+0000 7fa637753640 1 -- 172.21.15.76:0/663042561 >> 172.21.15.76:0/663042561 conn(0x7fa6300fcd90 msgr2=0x7fa630105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:09.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.223+0000 7fa637753640 1 -- 172.21.15.76:0/663042561 shutdown_connections 2024-03-18T14:53:09.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.224+0000 7fa637753640 1 -- 172.21.15.76:0/663042561 wait complete. 2024-03-18T14:53:09.236 DEBUG:tasks.fwd_scrub:Yielding 2024-03-18T14:53:09.236 INFO:teuthology.run_tasks:Running task exec... 2024-03-18T14:53:09.246 INFO:teuthology.task.exec:Executing custom commands... 2024-03-18T14:53:09.246 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi076.front.sepia.ceph.com 2024-03-18T14:53:09.246 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-18T14:53:09.247 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-18T14:53:09.247 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T14:53:09.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:53:09.247 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T14:53:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.350+0000 7f2079c19640 1 -- 172.21.15.76:0/1450639477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2074104c50 msgr2=0x7f2074107040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.350+0000 7f2079c19640 1 --2- 172.21.15.76:0/1450639477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2074104c50 0x7f2074107040 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2068009860 tx=0x7f206802bb70 comp rx=0 tx=0).stop 2024-03-18T14:53:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.350+0000 7f2079c19640 1 -- 172.21.15.76:0/1450639477 shutdown_connections 2024-03-18T14:53:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.350+0000 7f2079c19640 1 --2- 172.21.15.76:0/1450639477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2074107580 0x7f2074109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.350+0000 7f2079c19640 1 --2- 172.21.15.76:0/1450639477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2074104c50 0x7f2074107040 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.350+0000 7f2079c19640 1 --2- 172.21.15.76:0/1450639477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2074102320 0x7f2074104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.350+0000 7f2079c19640 1 -- 172.21.15.76:0/1450639477 >> 172.21.15.76:0/1450639477 conn(0x7f20740fbc70 msgr2=0x7f20740fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.350+0000 7f2079c19640 1 -- 172.21.15.76:0/1450639477 shutdown_connections 2024-03-18T14:53:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.350+0000 7f2079c19640 1 -- 172.21.15.76:0/1450639477 wait complete. 2024-03-18T14:53:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.350+0000 7f2079c19640 1 Processor -- start 2024-03-18T14:53:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.351+0000 7f2079c19640 1 -- start start 2024-03-18T14:53:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.351+0000 7f2079c19640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2074102320 0x7f2074197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.351+0000 7f2079c19640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2074104c50 0x7f2074197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.351+0000 7f2079c19640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2074107580 0x7f207419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.351+0000 7f2079c19640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f207410d470 con 0x7f2074104c50 2024-03-18T14:53:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.351+0000 7f2079c19640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f207410d2f0 con 0x7f2074102320 2024-03-18T14:53:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.351+0000 7f2079c19640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f207410d5f0 con 0x7f2074107580 2024-03-18T14:53:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.351+0000 7f2072ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2074104c50 0x7f2074197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.351+0000 7f2072ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2074104c50 0x7f2074197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44732/0 (socket says 172.21.15.76:44732) 2024-03-18T14:53:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.351+0000 7f2072ffd640 1 -- 172.21.15.76:0/3975778803 learned_addr learned my addr 172.21.15.76:0/3975778803 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.351+0000 7f2073fff640 1 --2- 172.21.15.76:0/3975778803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2074107580 0x7f207419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.351+0000 7f20737fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2074102320 0x7f2074197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.351+0000 7f2073fff640 1 -- 172.21.15.76:0/3975778803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2074102320 msgr2=0x7f2074197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.351+0000 7f2073fff640 1 --2- 172.21.15.76:0/3975778803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2074102320 0x7f2074197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.351+0000 7f2073fff640 1 -- 172.21.15.76:0/3975778803 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2074104c50 msgr2=0x7f2074197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.351+0000 7f2073fff640 1 --2- 172.21.15.76:0/3975778803 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2074104c50 0x7f2074197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.351+0000 7f2073fff640 1 -- 172.21.15.76:0/3975778803 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f207419f6d0 con 0x7f2074107580 2024-03-18T14:53:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.352+0000 7f2072ffd640 1 --2- 172.21.15.76:0/3975778803 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2074104c50 0x7f2074197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.352+0000 7f2073fff640 1 --2- 172.21.15.76:0/3975778803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2074107580 0x7f207419f060 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f206c009fd0 tx=0x7f206c00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.352+0000 7f2070ff9640 1 -- 172.21.15.76:0/3975778803 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206c00cd60 con 0x7f2074107580 2024-03-18T14:53:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.352+0000 7f2079c19640 1 -- 172.21.15.76:0/3975778803 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f207419f9c0 con 0x7f2074107580 2024-03-18T14:53:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.352+0000 7f2070ff9640 1 -- 172.21.15.76:0/3975778803 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f206c005020 con 0x7f2074107580 2024-03-18T14:53:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.352+0000 7f2070ff9640 1 -- 172.21.15.76:0/3975778803 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206c010690 con 0x7f2074107580 2024-03-18T14:53:09.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.352+0000 7f2079c19640 1 -- 172.21.15.76:0/3975778803 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f207419ff10 con 0x7f2074107580 2024-03-18T14:53:09.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.353+0000 7f2079c19640 1 -- 172.21.15.76:0/3975778803 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20740fd760 con 0x7f2074107580 2024-03-18T14:53:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.353+0000 7f2c2d4e8640 1 -- 172.21.15.76:0/1255192386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c280ff850 msgr2=0x7f2c280ffcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.354+0000 7f2c2d4e8640 1 --2- 172.21.15.76:0/1255192386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c280ff850 0x7f2c280ffcd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009860 tx=0x7f2c1c02bb70 comp rx=0 tx=0).stop 2024-03-18T14:53:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.354+0000 7f2c2d4e8640 1 -- 172.21.15.76:0/1255192386 shutdown_connections 2024-03-18T14:53:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.354+0000 7f2c2d4e8640 1 --2- 172.21.15.76:0/1255192386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c280ff850 0x7f2c280ffcd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.354+0000 7f2c2d4e8640 1 --2- 172.21.15.76:0/1255192386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c28100210 0x7f2c281006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.354+0000 7f2c2d4e8640 1 --2- 172.21.15.76:0/1255192386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c28101d60 0x7f2c28102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.354+0000 7f2c2d4e8640 1 -- 172.21.15.76:0/1255192386 >> 172.21.15.76:0/1255192386 conn(0x7f2c280fb3d0 msgr2=0x7f2c280fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.354+0000 7f2c2d4e8640 1 -- 172.21.15.76:0/1255192386 shutdown_connections 2024-03-18T14:53:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.354+0000 7f2c2d4e8640 1 -- 172.21.15.76:0/1255192386 wait complete. 2024-03-18T14:53:09.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.354+0000 7f2c2d4e8640 1 Processor -- start 2024-03-18T14:53:09.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.354+0000 7f2070ff9640 1 -- 172.21.15.76:0/3975778803 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f206c005540 con 0x7f2074107580 2024-03-18T14:53:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.354+0000 7f2c2d4e8640 1 -- start start 2024-03-18T14:53:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2c2d4e8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c280ff850 0x7f2c281064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2c2d4e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c28100210 0x7f2c281069e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2c2d4e8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c28101d60 0x7f2c28103010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2c2d4e8640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2810f650 con 0x7f2c28101d60 2024-03-18T14:53:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2c2d4e8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2810f4d0 con 0x7f2c28100210 2024-03-18T14:53:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2c2d4e8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2810f7d0 con 0x7f2c280ff850 2024-03-18T14:53:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2c26ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c280ff850 0x7f2c281064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2c26ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c280ff850 0x7f2c281064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:36082/0 (socket says 172.21.15.76:36082) 2024-03-18T14:53:09.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2c267fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c28100210 0x7f2c281069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2c26ffd640 1 -- 172.21.15.76:0/1250558716 learned_addr learned my addr 172.21.15.76:0/1250558716 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:09.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2c277fe640 1 --2- 172.21.15.76:0/1250558716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c28101d60 0x7f2c28103010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2c26ffd640 1 -- 172.21.15.76:0/1250558716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c28100210 msgr2=0x7f2c281069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2c26ffd640 1 --2- 172.21.15.76:0/1250558716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c28100210 0x7f2c281069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2c26ffd640 1 -- 172.21.15.76:0/1250558716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c28101d60 msgr2=0x7f2c28103010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2c26ffd640 1 --2- 172.21.15.76:0/1250558716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c28101d60 0x7f2c28103010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2070ff9640 1 --2- 172.21.15.76:0/3975778803 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f205c0a28a0 0x7f205c0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2c26ffd640 1 -- 172.21.15.76:0/1250558716 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c281038d0 con 0x7f2c280ff850 2024-03-18T14:53:09.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2c277fe640 1 --2- 172.21.15.76:0/1250558716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c28101d60 0x7f2c28103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:09.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f20737fe640 1 --2- 172.21.15.76:0/3975778803 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f205c0a28a0 0x7f205c0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.355+0000 7f2070ff9640 1 -- 172.21.15.76:0/3975778803 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f206c0cefc0 con 0x7f2074107580 2024-03-18T14:53:09.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.356+0000 7f2c26ffd640 1 --2- 172.21.15.76:0/1250558716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c280ff850 0x7f2c281064a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800e9c0 tx=0x7f2c1800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.356+0000 7f20737fe640 1 --2- 172.21.15.76:0/3975778803 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f205c0a28a0 0x7f205c0a4d60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2064005fd0 tx=0x7f2064009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.361+0000 7f2c0ffff640 1 -- 172.21.15.76:0/1250558716 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1800cde0 con 0x7f2c280ff850 2024-03-18T14:53:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.361+0000 7f2c2d4e8640 1 -- 172.21.15.76:0/1250558716 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c281a6150 con 0x7f2c280ff850 2024-03-18T14:53:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.361+0000 7f2c0ffff640 1 -- 172.21.15.76:0/1250558716 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c180035f0 con 0x7f2c280ff850 2024-03-18T14:53:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.361+0000 7f2c0ffff640 1 -- 172.21.15.76:0/1250558716 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c18010650 con 0x7f2c280ff850 2024-03-18T14:53:09.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.361+0000 7f2c2d4e8640 1 -- 172.21.15.76:0/1250558716 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c281a65b0 con 0x7f2c280ff850 2024-03-18T14:53:09.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.361+0000 7f2070ff9640 1 -- 172.21.15.76:0/3975778803 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f206c09c130 con 0x7f2074107580 2024-03-18T14:53:09.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.364+0000 7f2c0ffff640 1 -- 172.21.15.76:0/1250558716 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2c18010810 con 0x7f2c280ff850 2024-03-18T14:53:09.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.364+0000 7f2c301ab740 1 Processor -- start 2024-03-18T14:53:09.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.364+0000 7f2c301ab740 1 -- start start 2024-03-18T14:53:09.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.364+0000 7f2c301ab740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b09db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.364+0000 7f2c301ab740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0a380 0x564e48b0ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.364+0000 7f2c301ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0f4f0 0x564e48b11980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.364+0000 7f2c301ab740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564e48a6e0d0 con 0x564e48b09990 2024-03-18T14:53:09.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.364+0000 7f2c301ab740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564e48a16c70 con 0x564e48b0a380 2024-03-18T14:53:09.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.364+0000 7f2c301ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564e48a6a170 con 0x564e48b0f4f0 2024-03-18T14:53:09.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.364+0000 7f2c277fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b09db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.364+0000 7f2c277fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b09db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44766/0 (socket says 172.21.15.76:44766) 2024-03-18T14:53:09.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.364+0000 7f2c26ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0a380 0x564e48b0ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.364+0000 7f2c277fe640 1 -- 172.21.15.76:0/2130421065 learned_addr learned my addr 172.21.15.76:0/2130421065 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:09.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.364+0000 7f2c267fc640 1 --2- 172.21.15.76:0/2130421065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0f4f0 0x564e48b11980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.364+0000 7f2c0ffff640 1 --2- 172.21.15.76:0/1250558716 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2c100a2800 0x7f2c100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.365+0000 7f2c277fe640 1 -- 172.21.15.76:0/2130421065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0f4f0 msgr2=0x564e48b11980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:09.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.365+0000 7f2c277fe640 1 --2- 172.21.15.76:0/2130421065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0f4f0 0x564e48b11980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.365+0000 7f2c277fe640 1 -- 172.21.15.76:0/2130421065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0a380 msgr2=0x564e48b0ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:09.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.365+0000 7f2c277fe640 1 --2- 172.21.15.76:0/2130421065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0a380 0x564e48b0ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.365+0000 7f2c277fe640 1 -- 172.21.15.76:0/2130421065 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e4884a7e0 con 0x564e48b09990 2024-03-18T14:53:09.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.365+0000 7f2c267fc640 1 --2- 172.21.15.76:0/2130421065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0f4f0 0x564e48b11980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:09.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.365+0000 7f2c0ffff640 1 -- 172.21.15.76:0/1250558716 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f2c180cedf0 con 0x7f2c280ff850 2024-03-18T14:53:09.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.365+0000 7f2c277fe640 1 --2- 172.21.15.76:0/1250558716 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2c100a2800 0x7f2c100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.365+0000 7f2c277fe640 1 --2- 172.21.15.76:0/2130421065 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b09db0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2c20011940 tx=0x7f2c20011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e7910276269474 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:09.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.365+0000 7f2c277fe640 1 --2- 172.21.15.76:0/1250558716 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2c100a2800 0x7f2c100a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2c20023000 tx=0x7f2c20024000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:09.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.365+0000 7f2c0d7fa640 1 -- 172.21.15.76:0/2130421065 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c20017070 con 0x564e48b09990 2024-03-18T14:53:09.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.366+0000 7f2c0d7fa640 1 -- 172.21.15.76:0/2130421065 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c20011250 con 0x564e48b09990 2024-03-18T14:53:09.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.366+0000 7f2c0d7fa640 1 -- 172.21.15.76:0/2130421065 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c20010ba0 con 0x564e48b09990 2024-03-18T14:53:09.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.366+0000 7f2c301ab740 1 -- 172.21.15.76:0/2130421065 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 msgr2=0x564e48b09db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.366+0000 7f2c301ab740 1 --2- 172.21.15.76:0/2130421065 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b09db0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2c20011940 tx=0x7f2c20011e00 comp rx=0 tx=0).stop 2024-03-18T14:53:09.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.366+0000 7f2c301ab740 1 -- 172.21.15.76:0/2130421065 shutdown_connections 2024-03-18T14:53:09.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.366+0000 7f2c301ab740 1 --2- 172.21.15.76:0/2130421065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0f4f0 0x564e48b11980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.366+0000 7f2c301ab740 1 --2- 172.21.15.76:0/2130421065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0a380 0x564e48b0ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.366+0000 7f2c301ab740 1 --2- 172.21.15.76:0/2130421065 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b09db0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.366+0000 7f2c301ab740 1 -- 172.21.15.76:0/2130421065 >> 172.21.15.76:0/2130421065 conn(0x564e48ad1c50 msgr2=0x564e48b05300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:09.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.366+0000 7f2c301ab740 1 -- 172.21.15.76:0/2130421065 shutdown_connections 2024-03-18T14:53:09.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.366+0000 7f2c301ab740 1 -- 172.21.15.76:0/2130421065 wait complete. 2024-03-18T14:53:09.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c301ab740 1 -- start start 2024-03-18T14:53:09.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c301ab740 1 Processor -- start 2024-03-18T14:53:09.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c301ab740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b13f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c301ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0a380 0x564e48b14480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c301ab740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0f4f0 0x564e48b1b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c301ab740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564e48b124a0 con 0x564e48b09990 2024-03-18T14:53:09.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c301ab740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564e48a2a0d0 con 0x564e48b0f4f0 2024-03-18T14:53:09.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c301ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564e48b12620 con 0x564e48b0a380 2024-03-18T14:53:09.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c301ab740 10 client.0 _ll_register_callbacks cb 0x564e489f3760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:09.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c26ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b13f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c277fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0f4f0 0x564e48b1b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c277fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0f4f0 0x564e48b1b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:47432/0 (socket says 172.21.15.76:47432) 2024-03-18T14:53:09.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c277fe640 1 -- 172.21.15.76:0/1448171928 learned_addr learned my addr 172.21.15.76:0/1448171928 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:09.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c267fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0a380 0x564e48b14480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c26ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b13f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44770/0 (socket says 172.21.15.76:44770) 2024-03-18T14:53:09.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c277fe640 1 -- 172.21.15.76:0/1448171928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0a380 msgr2=0x564e48b14480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:09.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c277fe640 1 --2- 172.21.15.76:0/1448171928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0a380 0x564e48b14480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c277fe640 1 -- 172.21.15.76:0/1448171928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 msgr2=0x564e48b13f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:09.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c277fe640 1 --2- 172.21.15.76:0/1448171928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b13f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.367+0000 7f2c277fe640 1 -- 172.21.15.76:0/1448171928 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e488aaf70 con 0x564e48b0f4f0 2024-03-18T14:53:09.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.368+0000 7f2c277fe640 1 --2- 172.21.15.76:0/1448171928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0f4f0 0x564e48b1b4c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000a010 tx=0x7f2c200094f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24b6ee5b38da2043 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:09.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.368+0000 7f2c0aff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:53:09.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.368+0000 7f2c0aff5640 1 -- 172.21.15.76:0/1448171928 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c20017070 con 0x564e48b0f4f0 2024-03-18T14:53:09.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.368+0000 7f2c301ab740 1 -- 172.21.15.76:0/1448171928 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c20011250 con 0x564e48b0f4f0 2024-03-18T14:53:09.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.368+0000 7f2c0aff5640 1 -- 172.21.15.76:0/1448171928 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f2c200044a0 con 0x564e48b0f4f0 2024-03-18T14:53:09.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.368+0000 7f2c0aff5640 1 -- 172.21.15.76:0/1448171928 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2c200046a0 con 0x564e48b0f4f0 2024-03-18T14:53:09.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.368+0000 7f2c301ab740 10 client.25259 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:09.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.368+0000 7f2c301ab740 1 -- 172.21.15.76:0/1448171928 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564e48742170 con 0x564e48b0f4f0 2024-03-18T14:53:09.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.368+0000 7f2c0aff5640 1 -- 172.21.15.76:0/1448171928 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c20011250 con 0x564e48b0f4f0 2024-03-18T14:53:09.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.368+0000 7f2c0aff5640 1 -- 172.21.15.76:0/1448171928 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f2c2001b070 con 0x564e48b0f4f0 2024-03-18T14:53:09.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.368+0000 7f2c301ab740 10 client.25259 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:09.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.368+0000 7f2c301ab740 10 client.25259 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:09.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.368+0000 7f2c301ab740 20 client.25259 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:09.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.368+0000 7f2c301ab740 1 --2- 172.21.15.76:0/1448171928 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x564e48b1f830 0x564e48b21cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.368+0000 7f2c301ab740 4 client.25259 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:53:09.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.368+0000 7f2c301ab740 1 -- 172.21.15.76:0/1448171928 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e48a7b610 con 0x564e48b1f830 2024-03-18T14:53:09.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.369+0000 7f2c26ffd640 1 --2- 172.21.15.76:0/1448171928 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x564e48b1f830 0x564e48b21cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.369+0000 7f2c0aff5640 10 client.25259 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:09.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.369+0000 7f2c26ffd640 1 --2- 172.21.15.76:0/1448171928 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x564e48b1f830 0x564e48b21cb0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1acf2e05c470bce3 server_cookie=5fbda11cb643d230 in_seq=0 out_seq=0 2024-03-18T14:53:09.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.370+0000 7f2c0aff5640 1 -- 172.21.15.76:0/1448171928 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564e48a7b610 con 0x564e48b1f830 2024-03-18T14:53:09.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.370+0000 7f2c0aff5640 10 client.25259 handle_command_reply: tid=0 2024-03-18T14:53:09.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.370+0000 7f2c301ab740 1 client.25259 shutdown 2024-03-18T14:53:09.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.370+0000 7f2c301ab740 1 -- 172.21.15.76:0/1448171928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0f4f0 msgr2=0x564e48b1b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.370+0000 7f2c301ab740 1 --2- 172.21.15.76:0/1448171928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0f4f0 0x564e48b1b4c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000a010 tx=0x7f2c200094f0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.370+0000 7f2c301ab740 1 -- 172.21.15.76:0/1448171928 shutdown_connections 2024-03-18T14:53:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.370+0000 7f2c301ab740 1 --2- 172.21.15.76:0/1448171928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0a380 0x564e48b14480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.370+0000 7f2c301ab740 1 --2- 172.21.15.76:0/1448171928 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x564e48b1f830 0x564e48b21cb0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.370+0000 7f2c301ab740 1 --2- 172.21.15.76:0/1448171928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0f4f0 0x564e48b1b4c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.370+0000 7f2c0aff5640 0 client.25259 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.370+0000 7f2c301ab740 1 --2- 172.21.15.76:0/1448171928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b13f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.370+0000 7f2c301ab740 1 -- 172.21.15.76:0/1448171928 >> 172.21.15.76:0/1448171928 conn(0x564e48ad1c50 msgr2=0x564e48b044e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.370+0000 7f2c301ab740 1 -- 172.21.15.76:0/1448171928 shutdown_connections 2024-03-18T14:53:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.370+0000 7f2c301ab740 1 -- 172.21.15.76:0/1448171928 wait complete. 2024-03-18T14:53:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.370+0000 7f2c301ab740 20 client.25259 trim_cache size 0 max 16384 2024-03-18T14:53:09.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.387+0000 7f2c301ab740 1 Processor -- start 2024-03-18T14:53:09.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.387+0000 7f2c301ab740 1 -- start start 2024-03-18T14:53:09.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.387+0000 7f2c301ab740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b08540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.387+0000 7f2c301ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0a380 0x564e48b1b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.387+0000 7f2c301ab740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0f4f0 0x564e48b08a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.387+0000 7f2c301ab740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2c281b1b50 con 0x564e48b09990 2024-03-18T14:53:09.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.387+0000 7f2c301ab740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2c281af420 con 0x564e48b0f4f0 2024-03-18T14:53:09.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.387+0000 7f2c301ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2c281ad270 con 0x564e48b0a380 2024-03-18T14:53:09.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.387+0000 7f2c26ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b08540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.387+0000 7f2c26ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b08540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44778/0 (socket says 172.21.15.76:44778) 2024-03-18T14:53:09.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.387+0000 7f2c26ffd640 1 -- 172.21.15.76:0/3734542398 learned_addr learned my addr 172.21.15.76:0/3734542398 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:09.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.387+0000 7f2c277fe640 1 --2- 172.21.15.76:0/3734542398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0f4f0 0x564e48b08a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.387+0000 7f2c267fc640 1 --2- 172.21.15.76:0/3734542398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0a380 0x564e48b1b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.388+0000 7f2c26ffd640 1 -- 172.21.15.76:0/3734542398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0a380 msgr2=0x564e48b1b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:09.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.388+0000 7f2c26ffd640 1 --2- 172.21.15.76:0/3734542398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0a380 0x564e48b1b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.388+0000 7f2c26ffd640 1 -- 172.21.15.76:0/3734542398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0f4f0 msgr2=0x564e48b08a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:09.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.388+0000 7f2c26ffd640 1 --2- 172.21.15.76:0/3734542398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0f4f0 0x564e48b08a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.388+0000 7f2c26ffd640 1 -- 172.21.15.76:0/3734542398 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e48b095d0 con 0x564e48b09990 2024-03-18T14:53:09.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.388+0000 7f2c277fe640 1 --2- 172.21.15.76:0/3734542398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0f4f0 0x564e48b08a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:09.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.388+0000 7f2c26ffd640 1 --2- 172.21.15.76:0/3734542398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b08540 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800c6a0 tx=0x7f2c18086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f133faaf526a29c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:09.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.388+0000 7f2c0cff9640 1 -- 172.21.15.76:0/3734542398 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c180b3070 con 0x564e48b09990 2024-03-18T14:53:09.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.388+0000 7f2c0cff9640 1 -- 172.21.15.76:0/3734542398 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c18099b40 con 0x564e48b09990 2024-03-18T14:53:09.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.388+0000 7f2c0cff9640 1 -- 172.21.15.76:0/3734542398 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c18098040 con 0x564e48b09990 2024-03-18T14:53:09.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 1 -- 172.21.15.76:0/3734542398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 msgr2=0x564e48b08540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 1 --2- 172.21.15.76:0/3734542398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b08540 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800c6a0 tx=0x7f2c18086040 comp rx=0 tx=0).stop 2024-03-18T14:53:09.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 1 -- 172.21.15.76:0/3734542398 shutdown_connections 2024-03-18T14:53:09.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 1 --2- 172.21.15.76:0/3734542398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0a380 0x564e48b1b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 1 --2- 172.21.15.76:0/3734542398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0f4f0 0x564e48b08a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 1 --2- 172.21.15.76:0/3734542398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b08540 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 1 -- 172.21.15.76:0/3734542398 >> 172.21.15.76:0/3734542398 conn(0x564e48ad1c50 msgr2=0x564e48b0b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:09.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 1 -- 172.21.15.76:0/3734542398 shutdown_connections 2024-03-18T14:53:09.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 1 -- 172.21.15.76:0/3734542398 wait complete. 2024-03-18T14:53:09.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 1 -- start start 2024-03-18T14:53:09.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 1 Processor -- start 2024-03-18T14:53:09.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b193d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0a380 0x564e48b19910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0f4f0 0x564e48b1a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2c281ad8e0 con 0x564e48b09990 2024-03-18T14:53:09.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2c281aed50 con 0x564e48b0a380 2024-03-18T14:53:09.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2c281adfb0 con 0x564e48b0f4f0 2024-03-18T14:53:09.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c301ab740 10 client.0 _ll_register_callbacks cb 0x7f2c1809b690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:09.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.389+0000 7f2c26ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b193d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c26ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b193d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44790/0 (socket says 172.21.15.76:44790) 2024-03-18T14:53:09.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c26ffd640 1 -- 172.21.15.76:0/715425423 learned_addr learned my addr 172.21.15.76:0/715425423 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:09.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c277fe640 1 --2- 172.21.15.76:0/715425423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0f4f0 0x564e48b1a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c267fc640 1 --2- 172.21.15.76:0/715425423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0a380 0x564e48b19910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c26ffd640 1 -- 172.21.15.76:0/715425423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0f4f0 msgr2=0x564e48b1a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:09.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c26ffd640 1 --2- 172.21.15.76:0/715425423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0f4f0 0x564e48b1a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c26ffd640 1 -- 172.21.15.76:0/715425423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0a380 msgr2=0x564e48b19910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:09.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c26ffd640 1 --2- 172.21.15.76:0/715425423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0a380 0x564e48b19910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c26ffd640 1 -- 172.21.15.76:0/715425423 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e48b1af10 con 0x564e48b09990 2024-03-18T14:53:09.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c26ffd640 1 --2- 172.21.15.76:0/715425423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b193d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2c1809b7d0 tx=0x7f2c180a3fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba6a73405bfc54c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:09.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c0aff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:53:09.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c0aff5640 1 -- 172.21.15.76:0/715425423 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1807e020 con 0x564e48b09990 2024-03-18T14:53:09.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c0aff5640 1 -- 172.21.15.76:0/715425423 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f2c18003b10 con 0x564e48b09990 2024-03-18T14:53:09.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c301ab740 1 -- 172.21.15.76:0/715425423 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c18099b40 con 0x564e48b09990 2024-03-18T14:53:09.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c0aff5640 1 -- 172.21.15.76:0/715425423 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2c180b3040 con 0x564e48b09990 2024-03-18T14:53:09.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c301ab740 10 client.15393 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:09.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c301ab740 1 -- 172.21.15.76:0/715425423 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564e48b86460 con 0x564e48b09990 2024-03-18T14:53:09.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.390+0000 7f2c0aff5640 1 -- 172.21.15.76:0/715425423 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c18099b40 con 0x564e48b09990 2024-03-18T14:53:09.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.391+0000 7f2c0aff5640 1 -- 172.21.15.76:0/715425423 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f2c1809ac90 con 0x564e48b09990 2024-03-18T14:53:09.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.391+0000 7f2c301ab740 10 client.15393 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:09.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.391+0000 7f2c301ab740 10 client.15393 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:09.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.391+0000 7f2c301ab740 20 client.15393 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:09.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.391+0000 7f2c301ab740 1 --2- 172.21.15.76:0/715425423 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x564e48b86d30 0x564e48b89150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.391+0000 7f2c301ab740 4 client.15393 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T14:53:09.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.391+0000 7f2c301ab740 1 -- 172.21.15.76:0/715425423 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564e48b00b40 con 0x564e48b86d30 2024-03-18T14:53:09.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.391+0000 7f2c267fc640 1 --2- 172.21.15.76:0/715425423 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x564e48b86d30 0x564e48b89150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.391+0000 7f2c267fc640 1 --2- 172.21.15.76:0/715425423 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x564e48b86d30 0x564e48b89150 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a01a2397a7cff82 server_cookie=257f028355006ca0 in_seq=0 out_seq=0 2024-03-18T14:53:09.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.391+0000 7f2c0aff5640 10 client.15393 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:09.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.392+0000 7f2c0aff5640 1 -- 172.21.15.76:0/715425423 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564e48b00b40 con 0x564e48b86d30 2024-03-18T14:53:09.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.392+0000 7f2c0aff5640 10 client.15393 handle_command_reply: tid=0 2024-03-18T14:53:09.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.392+0000 7f2c301ab740 1 client.15393 shutdown 2024-03-18T14:53:09.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.392+0000 7f2c301ab740 1 -- 172.21.15.76:0/715425423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 msgr2=0x564e48b193d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.392+0000 7f2c301ab740 1 --2- 172.21.15.76:0/715425423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b193d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2c1809b7d0 tx=0x7f2c180a3fc0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.392+0000 7f2c301ab740 1 -- 172.21.15.76:0/715425423 shutdown_connections 2024-03-18T14:53:09.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.392+0000 7f2c301ab740 1 --2- 172.21.15.76:0/715425423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e48b0f4f0 0x564e48b1a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.392+0000 7f2c301ab740 1 --2- 172.21.15.76:0/715425423 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x564e48b86d30 0x564e48b89150 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.392+0000 7f2c301ab740 1 --2- 172.21.15.76:0/715425423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e48b0a380 0x564e48b19910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.392+0000 7f2c0aff5640 0 client.15393 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:09.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.392+0000 7f2c301ab740 1 --2- 172.21.15.76:0/715425423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e48b09990 0x564e48b193d0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.392+0000 7f2c301ab740 1 -- 172.21.15.76:0/715425423 >> 172.21.15.76:0/715425423 conn(0x564e48ad1c50 msgr2=0x564e48b08540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:09.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.392+0000 7f2c301ab740 1 -- 172.21.15.76:0/715425423 shutdown_connections 2024-03-18T14:53:09.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.392+0000 7f2c301ab740 1 -- 172.21.15.76:0/715425423 wait complete. 2024-03-18T14:53:09.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.392+0000 7f2c301ab740 20 client.15393 trim_cache size 0 max 16384 2024-03-18T14:53:09.399 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T14:53:09.399 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-18T14:53:09.399 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "9422f42f-d368-45a8-9759-4b7b52dfab42", 2024-03-18T14:53:09.399 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-18T14:53:09.399 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T14:53:09.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.393+0000 7f2c2d4e8640 1 -- 172.21.15.76:0/1250558716 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2c100a2800 msgr2=0x7f2c100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.393+0000 7f2c2d4e8640 1 --2- 172.21.15.76:0/1250558716 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2c100a2800 0x7f2c100a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2c20023000 tx=0x7f2c20024000 comp rx=0 tx=0).stop 2024-03-18T14:53:09.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.393+0000 7f2c2d4e8640 1 -- 172.21.15.76:0/1250558716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c280ff850 msgr2=0x7f2c281064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.393+0000 7f2c2d4e8640 1 --2- 172.21.15.76:0/1250558716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c280ff850 0x7f2c281064a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800e9c0 tx=0x7f2c1800ee80 comp rx=0 tx=0).stop 2024-03-18T14:53:09.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.393+0000 7f2c2d4e8640 1 -- 172.21.15.76:0/1250558716 shutdown_connections 2024-03-18T14:53:09.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.393+0000 7f2c2d4e8640 1 --2- 172.21.15.76:0/1250558716 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f2c100a2800 0x7f2c100a4cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.393+0000 7f2c2d4e8640 1 --2- 172.21.15.76:0/1250558716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c28101d60 0x7f2c28103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.393+0000 7f2c2d4e8640 1 --2- 172.21.15.76:0/1250558716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c28100210 0x7f2c281069e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.393+0000 7f2c2d4e8640 1 --2- 172.21.15.76:0/1250558716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c280ff850 0x7f2c281064a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.393+0000 7f2c2d4e8640 1 -- 172.21.15.76:0/1250558716 >> 172.21.15.76:0/1250558716 conn(0x7f2c280fb3d0 msgr2=0x7f2c280fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:09.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.393+0000 7f2c2d4e8640 1 -- 172.21.15.76:0/1250558716 shutdown_connections 2024-03-18T14:53:09.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.393+0000 7f2c2d4e8640 1 -- 172.21.15.76:0/1250558716 wait complete. 2024-03-18T14:53:09.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T14:53:09.405 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T14:53:09.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.509+0000 7f6cb5537640 1 -- 172.21.15.76:0/3364670345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cb0102b50 msgr2=0x7f6cb0102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.509+0000 7f6cb5537640 1 --2- 172.21.15.76:0/3364670345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cb0102b50 0x7f6cb0102fd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009930 tx=0x7f6ca002bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:09.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.510+0000 7f6cb5537640 1 -- 172.21.15.76:0/3364670345 shutdown_connections 2024-03-18T14:53:09.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.510+0000 7f6cb5537640 1 --2- 172.21.15.76:0/3364670345 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6cb0103510 0x7f6cb0109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.510+0000 7f6cb5537640 1 --2- 172.21.15.76:0/3364670345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cb0102b50 0x7f6cb0102fd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.510+0000 7f6cb5537640 1 --2- 172.21.15.76:0/3364670345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb00ffae0 0x7f6cb00fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.510+0000 7f6cb5537640 1 -- 172.21.15.76:0/3364670345 >> 172.21.15.76:0/3364670345 conn(0x7f6cb00fb3f0 msgr2=0x7f6cb00fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.510+0000 7f6cb5537640 1 -- 172.21.15.76:0/3364670345 shutdown_connections 2024-03-18T14:53:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.510+0000 7f6cb5537640 1 -- 172.21.15.76:0/3364670345 wait complete. 2024-03-18T14:53:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.510+0000 7f6cb5537640 1 Processor -- start 2024-03-18T14:53:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.510+0000 7f6cb5537640 1 -- start start 2024-03-18T14:53:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.510+0000 7f6cb5537640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb00ffae0 0x7f6cb0197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.510+0000 7f6cb5537640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6cb0102b50 0x7f6cb0198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.510+0000 7f6cb5537640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cb0103510 0x7f6cb019f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.510+0000 7f6cb5537640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb010d470 con 0x7f6cb0102b50 2024-03-18T14:53:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.510+0000 7f6cb5537640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb010d2f0 con 0x7f6cb0103510 2024-03-18T14:53:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.510+0000 7f6cb5537640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb010d5f0 con 0x7f6cb00ffae0 2024-03-18T14:53:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.511+0000 7f6cae7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6cb0102b50 0x7f6cb0198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.511+0000 7f6cae7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6cb0102b50 0x7f6cb0198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44804/0 (socket says 172.21.15.76:44804) 2024-03-18T14:53:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.511+0000 7f6cae7fc640 1 -- 172.21.15.76:0/1486983610 learned_addr learned my addr 172.21.15.76:0/1486983610 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.511+0000 7f6caeffd640 1 --2- 172.21.15.76:0/1486983610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb00ffae0 0x7f6cb0197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.511+0000 7f6caf7fe640 1 --2- 172.21.15.76:0/1486983610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cb0103510 0x7f6cb019f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.511+0000 7f6cae7fc640 1 -- 172.21.15.76:0/1486983610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb00ffae0 msgr2=0x7f6cb0197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.511+0000 7f6cae7fc640 1 --2- 172.21.15.76:0/1486983610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb00ffae0 0x7f6cb0197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.511+0000 7f6cae7fc640 1 -- 172.21.15.76:0/1486983610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cb0103510 msgr2=0x7f6cb019f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.511+0000 7f6cae7fc640 1 --2- 172.21.15.76:0/1486983610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cb0103510 0x7f6cb019f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.511+0000 7f6cae7fc640 1 -- 172.21.15.76:0/1486983610 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb019f740 con 0x7f6cb0102b50 2024-03-18T14:53:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.511+0000 7f6caf7fe640 1 --2- 172.21.15.76:0/1486983610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cb0103510 0x7f6cb019f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.511+0000 7f6cae7fc640 1 --2- 172.21.15.76:0/1486983610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6cb0102b50 0x7f6cb0198060 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0000c00 tx=0x7f6ca002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.511+0000 7f6ca7fff640 1 -- 172.21.15.76:0/1486983610 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0041070 con 0x7f6cb0102b50 2024-03-18T14:53:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.511+0000 7f6cb5537640 1 -- 172.21.15.76:0/1486983610 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cb019f9d0 con 0x7f6cb0102b50 2024-03-18T14:53:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.511+0000 7f6ca7fff640 1 -- 172.21.15.76:0/1486983610 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ca0004450 con 0x7f6cb0102b50 2024-03-18T14:53:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.511+0000 7f6ca7fff640 1 -- 172.21.15.76:0/1486983610 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0034440 con 0x7f6cb0102b50 2024-03-18T14:53:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.511+0000 7f6cb5537640 1 -- 172.21.15.76:0/1486983610 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cb019ff10 con 0x7f6cb0102b50 2024-03-18T14:53:09.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.513+0000 7f6ca7fff640 1 -- 172.21.15.76:0/1486983610 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6ca0044070 con 0x7f6cb0102b50 2024-03-18T14:53:09.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.514+0000 7f6cb81fb740 1 Processor -- start 2024-03-18T14:53:09.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.514+0000 7f6cb81fb740 1 -- start start 2024-03-18T14:53:09.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.514+0000 7f6cb81fb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea053170 0x5556ea053590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.514+0000 7f6cb81fb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea053b60 0x5556ea058660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.514+0000 7f6cb81fb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea058cd0 0x5556ea05b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.514+0000 7f6cb81fb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5556e9fb8500 con 0x5556ea053b60 2024-03-18T14:53:09.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.514+0000 7f6cb81fb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5556e9f60be0 con 0x5556ea058cd0 2024-03-18T14:53:09.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.514+0000 7f6cb81fb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5556e9fb45a0 con 0x5556ea053170 2024-03-18T14:53:09.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.514+0000 7f6ca7fff640 1 --2- 172.21.15.76:0/1486983610 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6c9c0a2800 0x7f6c9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.514+0000 7f6caeffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea053b60 0x5556ea058660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.514+0000 7f6caeffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea053b60 0x5556ea058660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44814/0 (socket says 172.21.15.76:44814) 2024-03-18T14:53:09.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.514+0000 7f6caeffd640 1 -- 172.21.15.76:0/709266731 learned_addr learned my addr 172.21.15.76:0/709266731 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:09.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.514+0000 7f6cae7fc640 1 --2- 172.21.15.76:0/709266731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea058cd0 0x5556ea05b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.514+0000 7f6caf7fe640 1 --2- 172.21.15.76:0/1486983610 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6c9c0a2800 0x7f6c9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.515+0000 7f6caf7fe640 1 --2- 172.21.15.76:0/709266731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea053170 0x5556ea053590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.515+0000 7f6caeffd640 1 -- 172.21.15.76:0/709266731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea053170 msgr2=0x5556ea053590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.515+0000 7f6caeffd640 1 --2- 172.21.15.76:0/709266731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea053170 0x5556ea053590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.515+0000 7f6ca7fff640 1 -- 172.21.15.76:0/1486983610 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6ca00f04d0 con 0x7f6cb0102b50 2024-03-18T14:53:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.515+0000 7f6caeffd640 1 -- 172.21.15.76:0/709266731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea058cd0 msgr2=0x5556ea05b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.515+0000 7f6caeffd640 1 --2- 172.21.15.76:0/709266731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea058cd0 0x5556ea05b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.515+0000 7f6caeffd640 1 -- 172.21.15.76:0/709266731 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556e9f744b0 con 0x5556ea053b60 2024-03-18T14:53:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.515+0000 7f6caeffd640 1 --2- 172.21.15.76:0/709266731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea053b60 0x5556ea058660 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009ca0 tx=0x7f6c98011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f04f16edf26c8a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.515+0000 7f6caf7fe640 1 --2- 172.21.15.76:0/1486983610 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6c9c0a2800 0x7f6c9c0a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x5556ea0543c0 tx=0x7f6ca8011440 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.515+0000 7f6ca57fa640 1 -- 172.21.15.76:0/709266731 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98004220 con 0x5556ea053b60 2024-03-18T14:53:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.515+0000 7f6ca57fa640 1 -- 172.21.15.76:0/709266731 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c980043c0 con 0x5556ea053b60 2024-03-18T14:53:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.515+0000 7f6ca57fa640 1 -- 172.21.15.76:0/709266731 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98019890 con 0x5556ea053b60 2024-03-18T14:53:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 1 -- 172.21.15.76:0/709266731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea053b60 msgr2=0x5556ea058660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/709266731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea053b60 0x5556ea058660 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009ca0 tx=0x7f6c98011810 comp rx=0 tx=0).stop 2024-03-18T14:53:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 1 -- 172.21.15.76:0/709266731 shutdown_connections 2024-03-18T14:53:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/709266731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea058cd0 0x5556ea05b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/709266731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea053b60 0x5556ea058660 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/709266731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea053170 0x5556ea053590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 1 -- 172.21.15.76:0/709266731 >> 172.21.15.76:0/709266731 conn(0x5556ea04e6e0 msgr2=0x5556ea04eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 1 -- 172.21.15.76:0/709266731 shutdown_connections 2024-03-18T14:53:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 1 -- 172.21.15.76:0/709266731 wait complete. 2024-03-18T14:53:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 1 -- start start 2024-03-18T14:53:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 1 Processor -- start 2024-03-18T14:53:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea053170 0x5556ea05d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea053b60 0x5556ea05dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea058cd0 0x5556ea064c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5556ea05bc80 con 0x5556ea058cd0 2024-03-18T14:53:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5556e9f74160 con 0x5556ea053b60 2024-03-18T14:53:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5556ea05be00 con 0x5556ea053170 2024-03-18T14:53:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.516+0000 7f6cb81fb740 10 client.0 _ll_register_callbacks cb 0x5556e9f58150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.517+0000 7f6caf7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea058cd0 0x5556ea064c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.517+0000 7f6caf7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea058cd0 0x5556ea064c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44818/0 (socket says 172.21.15.76:44818) 2024-03-18T14:53:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.517+0000 7f6caf7fe640 1 -- 172.21.15.76:0/709663742 learned_addr learned my addr 172.21.15.76:0/709663742 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.517+0000 7f6caeffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea053170 0x5556ea05d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.517+0000 7f6cae7fc640 1 --2- 172.21.15.76:0/709663742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea053b60 0x5556ea05dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.517+0000 7f6caf7fe640 1 -- 172.21.15.76:0/709663742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea053170 msgr2=0x5556ea05d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.517+0000 7f6caf7fe640 1 --2- 172.21.15.76:0/709663742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea053170 0x5556ea05d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.517+0000 7f6caf7fe640 1 -- 172.21.15.76:0/709663742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea053b60 msgr2=0x5556ea05dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.517+0000 7f6caf7fe640 1 --2- 172.21.15.76:0/709663742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea053b60 0x5556ea05dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.517+0000 7f6caf7fe640 1 -- 172.21.15.76:0/709663742 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556e9d967d0 con 0x5556ea058cd0 2024-03-18T14:53:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.517+0000 7f6cae7fc640 1 --2- 172.21.15.76:0/709663742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea053b60 0x5556ea05dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.517+0000 7f6caf7fe640 1 --2- 172.21.15.76:0/709663742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea058cd0 0x5556ea064c30 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009fd0 tx=0x7f6ca8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51219c7926ca7fcb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.517+0000 7f6c967fc640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:53:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.517+0000 7f6c967fc640 1 -- 172.21.15.76:0/709663742 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca8003e40 con 0x5556ea058cd0 2024-03-18T14:53:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.517+0000 7f6c967fc640 1 -- 172.21.15.76:0/709663742 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6ca80045c0 con 0x5556ea058cd0 2024-03-18T14:53:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.517+0000 7f6cb81fb740 1 -- 172.21.15.76:0/709663742 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c980043c0 con 0x5556ea058cd0 2024-03-18T14:53:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.517+0000 7f6c967fc640 1 -- 172.21.15.76:0/709663742 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6ca80047c0 con 0x5556ea058cd0 2024-03-18T14:53:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.518+0000 7f6cb81fb740 10 client.15405 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.518+0000 7f6cb81fb740 1 -- 172.21.15.76:0/709663742 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5556e9ccc3f0 con 0x5556ea058cd0 2024-03-18T14:53:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.518+0000 7f6c967fc640 1 -- 172.21.15.76:0/709663742 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c980043c0 con 0x5556ea058cd0 2024-03-18T14:53:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.518+0000 7f6c967fc640 1 -- 172.21.15.76:0/709663742 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f6ca80335e0 con 0x5556ea058cd0 2024-03-18T14:53:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.518+0000 7f6cb81fb740 10 client.15405 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.518+0000 7f6cb81fb740 10 client.15405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.518+0000 7f6cb81fb740 20 client.15405 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.518+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/709663742 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5556ea0692a0 0x5556ea06b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.518+0000 7f6cb81fb740 4 client.15405 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:53:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.518+0000 7f6cb81fb740 1 -- 172.21.15.76:0/709663742 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556e9fc5a40 con 0x5556ea0692a0 2024-03-18T14:53:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.518+0000 7f6caeffd640 1 --2- 172.21.15.76:0/709663742 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5556ea0692a0 0x5556ea06b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.519+0000 7f6c967fc640 10 client.15405 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.519+0000 7f6caeffd640 1 --2- 172.21.15.76:0/709663742 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5556ea0692a0 0x5556ea06b700 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8ac9ac15518dd1e server_cookie=c70ae1f89acad7fd in_seq=0 out_seq=0 2024-03-18T14:53:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.519+0000 7f6c967fc640 1 -- 172.21.15.76:0/709663742 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5556e9fc5a40 con 0x5556ea0692a0 2024-03-18T14:53:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.519+0000 7f6c967fc640 10 client.15405 handle_command_reply: tid=0 2024-03-18T14:53:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.519+0000 7f6cb81fb740 1 client.15405 shutdown 2024-03-18T14:53:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.519+0000 7f6cb81fb740 1 -- 172.21.15.76:0/709663742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea058cd0 msgr2=0x5556ea064c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.519+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/709663742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea058cd0 0x5556ea064c30 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009fd0 tx=0x7f6ca8004a40 comp rx=0 tx=0).stop 2024-03-18T14:53:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.519+0000 7f6cb81fb740 1 -- 172.21.15.76:0/709663742 shutdown_connections 2024-03-18T14:53:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.519+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/709663742 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5556ea0692a0 0x5556ea06b700 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.519+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/709663742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea058cd0 0x5556ea064c30 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.519+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/709663742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea053b60 0x5556ea05dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.519+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/709663742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea053170 0x5556ea05d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.519+0000 7f6c967fc640 0 client.15405 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.519+0000 7f6cb81fb740 1 -- 172.21.15.76:0/709663742 >> 172.21.15.76:0/709663742 conn(0x5556ea04e6e0 msgr2=0x5556ea04d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:09.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.519+0000 7f6cb81fb740 1 -- 172.21.15.76:0/709663742 shutdown_connections 2024-03-18T14:53:09.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.519+0000 7f6cb81fb740 1 -- 172.21.15.76:0/709663742 wait complete. 2024-03-18T14:53:09.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.519+0000 7f6cb81fb740 20 client.15405 trim_cache size 0 max 16384 2024-03-18T14:53:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.536+0000 7f6cb81fb740 1 Processor -- start 2024-03-18T14:53:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.536+0000 7f6cb81fb740 1 -- start start 2024-03-18T14:53:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6cb81fb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea053170 0x5556ea052050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6cb81fb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea053b60 0x5556ea064bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6cb81fb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea058cd0 0x5556ea06ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6cb81fb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb01ab570 con 0x5556ea058cd0 2024-03-18T14:53:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6cb81fb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb01a8e40 con 0x5556ea053170 2024-03-18T14:53:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6cb81fb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb01a6c90 con 0x5556ea053b60 2024-03-18T14:53:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6caf7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea058cd0 0x5556ea06ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6cae7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea053b60 0x5556ea064bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6caeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea053170 0x5556ea052050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6cae7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea053b60 0x5556ea064bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:36144/0 (socket says 172.21.15.76:36144) 2024-03-18T14:53:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6caf7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea058cd0 0x5556ea06ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44832/0 (socket says 172.21.15.76:44832) 2024-03-18T14:53:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6cae7fc640 1 -- 172.21.15.76:0/1815844372 learned_addr learned my addr 172.21.15.76:0/1815844372 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6caeffd640 1 -- 172.21.15.76:0/1815844372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea053b60 msgr2=0x5556ea064bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6caeffd640 1 --2- 172.21.15.76:0/1815844372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea053b60 0x5556ea064bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6caeffd640 1 -- 172.21.15.76:0/1815844372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea058cd0 msgr2=0x5556ea06ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6caeffd640 1 --2- 172.21.15.76:0/1815844372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea058cd0 0x5556ea06ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6caeffd640 1 -- 172.21.15.76:0/1815844372 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556e9c8c0f0 con 0x5556ea053170 2024-03-18T14:53:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6caf7fe640 1 --2- 172.21.15.76:0/1815844372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea058cd0 0x5556ea06ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6caeffd640 1 --2- 172.21.15.76:0/1815844372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea053170 0x5556ea052050 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5556ea04f650 tx=0x7f6c98005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f73a079aa8ae5389 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6ca4ff9640 1 -- 172.21.15.76:0/1815844372 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98035070 con 0x5556ea053170 2024-03-18T14:53:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.537+0000 7f6ca4ff9640 1 -- 172.21.15.76:0/1815844372 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c98005520 con 0x5556ea053170 2024-03-18T14:53:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.538+0000 7f6ca4ff9640 1 -- 172.21.15.76:0/1815844372 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98019430 con 0x5556ea053170 2024-03-18T14:53:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.538+0000 7f6cb81fb740 1 -- 172.21.15.76:0/1815844372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea053170 msgr2=0x5556ea052050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.538+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/1815844372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea053170 0x5556ea052050 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5556ea04f650 tx=0x7f6c98005fd0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.538+0000 7f6cb81fb740 1 -- 172.21.15.76:0/1815844372 shutdown_connections 2024-03-18T14:53:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.538+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/1815844372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea053b60 0x5556ea064bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.538+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/1815844372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea058cd0 0x5556ea06ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.538+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/1815844372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea053170 0x5556ea052050 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.538+0000 7f6cb81fb740 1 -- 172.21.15.76:0/1815844372 >> 172.21.15.76:0/1815844372 conn(0x5556ea04e6e0 msgr2=0x5556ea051940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.538+0000 7f6cb81fb740 1 -- 172.21.15.76:0/1815844372 shutdown_connections 2024-03-18T14:53:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.538+0000 7f6cb81fb740 1 -- 172.21.15.76:0/1815844372 wait complete. 2024-03-18T14:53:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cb81fb740 1 -- start start 2024-03-18T14:53:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cb81fb740 1 Processor -- start 2024-03-18T14:53:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cb81fb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea053170 0x5556ea062850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cb81fb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea053b60 0x5556ea062d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cb81fb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea058cd0 0x5556ea063da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cb81fb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb01a7300 con 0x5556ea053b60 2024-03-18T14:53:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cb81fb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb01a8770 con 0x5556ea053170 2024-03-18T14:53:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cb81fb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb01a79d0 con 0x5556ea058cd0 2024-03-18T14:53:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cb81fb740 10 client.0 _ll_register_callbacks cb 0x7f6ca80118b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cae7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea053b60 0x5556ea062d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cae7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea053b60 0x5556ea062d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44844/0 (socket says 172.21.15.76:44844) 2024-03-18T14:53:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cae7fc640 1 -- 172.21.15.76:0/965163395 learned_addr learned my addr 172.21.15.76:0/965163395 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6caf7fe640 1 --2- 172.21.15.76:0/965163395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea058cd0 0x5556ea063da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6caeffd640 1 --2- 172.21.15.76:0/965163395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea053170 0x5556ea062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cae7fc640 1 -- 172.21.15.76:0/965163395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea058cd0 msgr2=0x5556ea063da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cae7fc640 1 --2- 172.21.15.76:0/965163395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea058cd0 0x5556ea063da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cae7fc640 1 -- 172.21.15.76:0/965163395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea053170 msgr2=0x5556ea062850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cae7fc640 1 --2- 172.21.15.76:0/965163395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea053170 0x5556ea062850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cae7fc640 1 -- 172.21.15.76:0/965163395 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556e9f7f9d0 con 0x5556ea053b60 2024-03-18T14:53:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6cae7fc640 1 --2- 172.21.15.76:0/965163395 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea053b60 0x5556ea062d90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6ca00ee050 tx=0x7f6ca0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a66210e2471bf7d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.539+0000 7f6c967fc640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:53:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.540+0000 7f6c967fc640 1 -- 172.21.15.76:0/965163395 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca004b040 con 0x5556ea053b60 2024-03-18T14:53:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.540+0000 7f6c967fc640 1 -- 172.21.15.76:0/965163395 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6ca00bd240 con 0x5556ea053b60 2024-03-18T14:53:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.540+0000 7f6cb81fb740 1 -- 172.21.15.76:0/965163395 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c98005520 con 0x5556ea053b60 2024-03-18T14:53:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.540+0000 7f6c967fc640 1 -- 172.21.15.76:0/965163395 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6ca00bc940 con 0x5556ea053b60 2024-03-18T14:53:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.540+0000 7f6cb81fb740 10 client.15411 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.540+0000 7f6cb81fb740 1 -- 172.21.15.76:0/965163395 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5556ea0cf9c0 con 0x5556ea053b60 2024-03-18T14:53:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.540+0000 7f6c967fc640 1 -- 172.21.15.76:0/965163395 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c98005520 con 0x5556ea053b60 2024-03-18T14:53:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.540+0000 7f6c967fc640 1 -- 172.21.15.76:0/965163395 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f6ca00c51b0 con 0x5556ea053b60 2024-03-18T14:53:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.540+0000 7f6cb81fb740 10 client.15411 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.540+0000 7f6cb81fb740 10 client.15411 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.540+0000 7f6cb81fb740 20 client.15411 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.540+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/965163395 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5556ea0d03c0 0x5556ea0d27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.540+0000 7f6cb81fb740 4 client.15411 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T14:53:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.540+0000 7f6cb81fb740 1 -- 172.21.15.76:0/965163395 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5556e9fbfd90 con 0x5556ea0d03c0 2024-03-18T14:53:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.540+0000 7f6caeffd640 1 --2- 172.21.15.76:0/965163395 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5556ea0d03c0 0x5556ea0d27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6caeffd640 1 --2- 172.21.15.76:0/965163395 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5556ea0d03c0 0x5556ea0d27e0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81b99154f5049088 server_cookie=a243002ab0767f6a in_seq=0 out_seq=0 2024-03-18T14:53:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6c967fc640 10 client.15411 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6c967fc640 1 -- 172.21.15.76:0/965163395 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5556e9fbfd90 con 0x5556ea0d03c0 2024-03-18T14:53:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6c967fc640 10 client.15411 handle_command_reply: tid=0 2024-03-18T14:53:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6cb81fb740 1 client.15411 shutdown 2024-03-18T14:53:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6cb81fb740 1 -- 172.21.15.76:0/965163395 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea053b60 msgr2=0x5556ea062d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/965163395 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea053b60 0x5556ea062d90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6ca00ee050 tx=0x7f6ca0009260 comp rx=0 tx=0).stop 2024-03-18T14:53:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6cb81fb740 1 -- 172.21.15.76:0/965163395 shutdown_connections 2024-03-18T14:53:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/965163395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556ea058cd0 0x5556ea063da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/965163395 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5556ea0d03c0 0x5556ea0d27e0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/965163395 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556ea053b60 0x5556ea062d90 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6c967fc640 0 client.15411 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6cb81fb740 1 --2- 172.21.15.76:0/965163395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556ea053170 0x5556ea062850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6cb81fb740 1 -- 172.21.15.76:0/965163395 >> 172.21.15.76:0/965163395 conn(0x5556ea04e6e0 msgr2=0x5556ea0521d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6cb81fb740 1 -- 172.21.15.76:0/965163395 shutdown_connections 2024-03-18T14:53:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6cb81fb740 1 -- 172.21.15.76:0/965163395 wait complete. 2024-03-18T14:53:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6cb81fb740 20 client.15411 trim_cache size 0 max 16384 2024-03-18T14:53:09.549 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T14:53:09.549 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T14:53:09.549 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-18T14:53:09.549 INFO:teuthology.orchestra.run.smithi076.stdout: "9422f42f-d368-45a8-9759-4b7b52dfab42": { 2024-03-18T14:53:09.549 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-18T14:53:09.549 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "9422f42f-d368-45a8-9759-4b7b52dfab42", 2024-03-18T14:53:09.549 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-18T14:53:09.549 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T14:53:09.549 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T14:53:09.549 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T14:53:09.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.541+0000 7f6cb5537640 1 -- 172.21.15.76:0/1486983610 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6c9c0a2800 msgr2=0x7f6c9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.542+0000 7f6cb5537640 1 --2- 172.21.15.76:0/1486983610 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6c9c0a2800 0x7f6c9c0a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x5556ea0543c0 tx=0x7f6ca8011440 comp rx=0 tx=0).stop 2024-03-18T14:53:09.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.542+0000 7f6cb5537640 1 -- 172.21.15.76:0/1486983610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6cb0102b50 msgr2=0x7f6cb0198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:09.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.542+0000 7f6cb5537640 1 --2- 172.21.15.76:0/1486983610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6cb0102b50 0x7f6cb0198060 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0000c00 tx=0x7f6ca002c640 comp rx=0 tx=0).stop 2024-03-18T14:53:09.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.542+0000 7f6cb5537640 1 -- 172.21.15.76:0/1486983610 shutdown_connections 2024-03-18T14:53:09.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.542+0000 7f6cb5537640 1 --2- 172.21.15.76:0/1486983610 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f6c9c0a2800 0x7f6c9c0a4cc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.542+0000 7f6cb5537640 1 --2- 172.21.15.76:0/1486983610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cb0103510 0x7f6cb019f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.542+0000 7f6cb5537640 1 --2- 172.21.15.76:0/1486983610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6cb0102b50 0x7f6cb0198060 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.542+0000 7f6cb5537640 1 --2- 172.21.15.76:0/1486983610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb00ffae0 0x7f6cb0197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:09.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.542+0000 7f6cb5537640 1 -- 172.21.15.76:0/1486983610 >> 172.21.15.76:0/1486983610 conn(0x7f6cb00fb3f0 msgr2=0x7f6cb0105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:09.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.542+0000 7f6cb5537640 1 -- 172.21.15.76:0/1486983610 shutdown_connections 2024-03-18T14:53:09.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.542+0000 7f6cb5537640 1 -- 172.21.15.76:0/1486983610 wait complete. 2024-03-18T14:53:09.554 INFO:tasks.cephfs.filesystem:scrub status for tag:9422f42f-d368-45a8-9759-4b7b52dfab42 - {'path': '/', 'tag': '9422f42f-d368-45a8-9759-4b7b52dfab42', 'options': 'recursive,force'} 2024-03-18T14:53:09.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:09.601+0000 7f2079c19640 1 -- 172.21.15.76:0/3975778803 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f20740623e0 con 0x7f2074107580 2024-03-18T14:53:09.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:09 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/858824079' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:53:09.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:09 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/416094150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:53:09.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:09 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/663042561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:53:09.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:09 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/858824079' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:53:09.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:09 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/416094150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:53:09.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:09 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/663042561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:53:09.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:09 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/858824079' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T14:53:09.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:09 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/416094150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:53:09.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:09 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/663042561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T14:53:10.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.561+0000 7f2070ff9640 1 -- 172.21.15.76:0/3975778803 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mgrmap(e 19) ==== 137307+0+0 (secure 0 0 0) 0x7f206c0980a0 con 0x7f2074107580 2024-03-18T14:53:10.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.562+0000 7f2070ff9640 1 -- 172.21.15.76:0/3975778803 <== mon.1 v2:172.21.15.174:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f206c0d1080 con 0x7f2074107580 2024-03-18T14:53:10.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.565+0000 7f2079c19640 1 -- 172.21.15.76:0/3975778803 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f205c0a28a0 msgr2=0x7f205c0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:10.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.565+0000 7f2079c19640 1 --2- 172.21.15.76:0/3975778803 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f205c0a28a0 0x7f205c0a4d60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2064005fd0 tx=0x7f2064009260 comp rx=0 tx=0).stop 2024-03-18T14:53:10.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.565+0000 7f2079c19640 1 -- 172.21.15.76:0/3975778803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2074107580 msgr2=0x7f207419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:10.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.565+0000 7f2079c19640 1 --2- 172.21.15.76:0/3975778803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2074107580 0x7f207419f060 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f206c009fd0 tx=0x7f206c00ee50 comp rx=0 tx=0).stop 2024-03-18T14:53:10.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.565+0000 7f2079c19640 1 -- 172.21.15.76:0/3975778803 shutdown_connections 2024-03-18T14:53:10.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.565+0000 7f2079c19640 1 --2- 172.21.15.76:0/3975778803 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f205c0a28a0 0x7f205c0a4d60 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:10.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.565+0000 7f2079c19640 1 --2- 172.21.15.76:0/3975778803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2074107580 0x7f207419f060 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:10.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.565+0000 7f2079c19640 1 --2- 172.21.15.76:0/3975778803 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2074104c50 0x7f2074197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:10.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.565+0000 7f2079c19640 1 --2- 172.21.15.76:0/3975778803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2074102320 0x7f2074197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:10.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.565+0000 7f2079c19640 1 -- 172.21.15.76:0/3975778803 >> 172.21.15.76:0/3975778803 conn(0x7f20740fbc70 msgr2=0x7f2074102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:10.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.566+0000 7f2079c19640 1 -- 172.21.15.76:0/3975778803 shutdown_connections 2024-03-18T14:53:10.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.566+0000 7f2079c19640 1 -- 172.21.15.76:0/3975778803 wait complete. 2024-03-18T14:53:10.583 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-18T14:53:10.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.692+0000 7f30ed322640 1 -- 172.21.15.76:0/1262907465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e8104c10 msgr2=0x7f30e8107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:10.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.692+0000 7f30ed322640 1 --2- 172.21.15.76:0/1262907465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e8104c10 0x7f30e8107000 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f30d0009900 tx=0x7f30d002bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:10.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.692+0000 7f30ed322640 1 -- 172.21.15.76:0/1262907465 shutdown_connections 2024-03-18T14:53:10.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.692+0000 7f30ed322640 1 --2- 172.21.15.76:0/1262907465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30e8107540 0x7f30e8109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:10.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.692+0000 7f30ed322640 1 --2- 172.21.15.76:0/1262907465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e8104c10 0x7f30e8107000 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:10.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.692+0000 7f30ed322640 1 --2- 172.21.15.76:0/1262907465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e81022e0 0x7f30e81046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:10.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.692+0000 7f30ed322640 1 -- 172.21.15.76:0/1262907465 >> 172.21.15.76:0/1262907465 conn(0x7f30e80fbc70 msgr2=0x7f30e80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:10.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.692+0000 7f30ed322640 1 -- 172.21.15.76:0/1262907465 shutdown_connections 2024-03-18T14:53:10.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.692+0000 7f30ed322640 1 -- 172.21.15.76:0/1262907465 wait complete. 2024-03-18T14:53:10.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.693+0000 7f30ed322640 1 Processor -- start 2024-03-18T14:53:10.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.693+0000 7f30ed322640 1 -- start start 2024-03-18T14:53:10.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.693+0000 7f30ed322640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e81022e0 0x7f30e8101c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:10.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.693+0000 7f30ed322640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e8104c10 0x7f30e8100290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:10.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.693+0000 7f30ed322640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30e8107540 0x7f30e81007d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:10.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.693+0000 7f30ed322640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f30e810d480 con 0x7f30e8104c10 2024-03-18T14:53:10.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.693+0000 7f30ed322640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f30e810d300 con 0x7f30e81022e0 2024-03-18T14:53:10.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.693+0000 7f30ed322640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f30e810d600 con 0x7f30e8107540 2024-03-18T14:53:10.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.693+0000 7f30de7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e8104c10 0x7f30e8100290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:10.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.693+0000 7f30e6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e81022e0 0x7f30e8101c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:10.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.693+0000 7f30de7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e8104c10 0x7f30e8100290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44902/0 (socket says 172.21.15.76:44902) 2024-03-18T14:53:10.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.693+0000 7f30de7fc640 1 -- 172.21.15.76:0/2659083687 learned_addr learned my addr 172.21.15.76:0/2659083687 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:10.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.693+0000 7f30e77fe640 1 --2- 172.21.15.76:0/2659083687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30e8107540 0x7f30e81007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:10.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.694+0000 7f30e6ffd640 1 -- 172.21.15.76:0/2659083687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30e8107540 msgr2=0x7f30e81007d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:10.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.694+0000 7f30e6ffd640 1 --2- 172.21.15.76:0/2659083687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30e8107540 0x7f30e81007d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:10.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.694+0000 7f30e6ffd640 1 -- 172.21.15.76:0/2659083687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e8104c10 msgr2=0x7f30e8100290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:10.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.694+0000 7f30e6ffd640 1 --2- 172.21.15.76:0/2659083687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e8104c10 0x7f30e8100290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:10.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.694+0000 7f30e6ffd640 1 -- 172.21.15.76:0/2659083687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e8101060 con 0x7f30e81022e0 2024-03-18T14:53:10.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.694+0000 7f30de7fc640 1 --2- 172.21.15.76:0/2659083687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e8104c10 0x7f30e8100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:10.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.694+0000 7f30e6ffd640 1 --2- 172.21.15.76:0/2659083687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e81022e0 0x7f30e8101c50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f30d800e9f0 tx=0x7f30d800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:10.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.694+0000 7f30e4ff9640 1 -- 172.21.15.76:0/2659083687 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d800ccc0 con 0x7f30e81022e0 2024-03-18T14:53:10.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.694+0000 7f30ed322640 1 -- 172.21.15.76:0/2659083687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e81a1ba0 con 0x7f30e81022e0 2024-03-18T14:53:10.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.694+0000 7f30e4ff9640 1 -- 172.21.15.76:0/2659083687 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30d8010910 con 0x7f30e81022e0 2024-03-18T14:53:10.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.694+0000 7f30e4ff9640 1 -- 172.21.15.76:0/2659083687 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d8018960 con 0x7f30e81022e0 2024-03-18T14:53:10.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.694+0000 7f30ed322640 1 -- 172.21.15.76:0/2659083687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e81a20c0 con 0x7f30e81022e0 2024-03-18T14:53:10.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.695+0000 7f30ed322640 1 -- 172.21.15.76:0/2659083687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30e80fd760 con 0x7f30e81022e0 2024-03-18T14:53:10.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.696+0000 7f30e4ff9640 1 -- 172.21.15.76:0/2659083687 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137307+0+0 (secure 0 0 0) 0x7f30d8010ac0 con 0x7f30e81022e0 2024-03-18T14:53:10.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.697+0000 7f30e4ff9640 1 --2- 172.21.15.76:0/2659083687 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f30d40a28f0 0x7f30d40a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:10.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.697+0000 7f30de7fc640 1 -- 172.21.15.76:0/2659083687 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f30d40a28f0 msgr2=0x7f30d40a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3010549595 2024-03-18T14:53:10.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.697+0000 7f30e4ff9640 1 -- 172.21.15.76:0/2659083687 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f30d80cee40 con 0x7f30e81022e0 2024-03-18T14:53:10.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.697+0000 7f30de7fc640 1 --2- 172.21.15.76:0/2659083687 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f30d40a28f0 0x7f30d40a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T14:53:10.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.706+0000 7f30e4ff9640 1 -- 172.21.15.76:0/2659083687 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f30d809b8c0 con 0x7f30e81022e0 2024-03-18T14:53:10.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.889+0000 7f30ed322640 1 -- 172.21.15.76:0/2659083687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f30e802d1b0 con 0x7f30e81022e0 2024-03-18T14:53:10.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.898+0000 7f30de7fc640 1 -- 172.21.15.76:0/2659083687 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f30d40a28f0 msgr2=0x7f30d40a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3010549595 2024-03-18T14:53:10.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.898+0000 7f30de7fc640 1 --2- 172.21.15.76:0/2659083687 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f30d40a28f0 0x7f30d40a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T14:53:10.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.899+0000 7f30e4ff9640 1 -- 172.21.15.76:0/2659083687 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) ==== 138+0+0 (secure 0 0 0) 0x7f30d809f960 con 0x7f30e81022e0 2024-03-18T14:53:10.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.901+0000 7f30ed322640 1 -- 172.21.15.76:0/2659083687 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f30d40a28f0 msgr2=0x7f30d40a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T14:53:10.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.901+0000 7f30ed322640 1 --2- 172.21.15.76:0/2659083687 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f30d40a28f0 0x7f30d40a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:10.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.901+0000 7f30ed322640 1 -- 172.21.15.76:0/2659083687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e81022e0 msgr2=0x7f30e8101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:10.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.901+0000 7f30ed322640 1 --2- 172.21.15.76:0/2659083687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e81022e0 0x7f30e8101c50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f30d800e9f0 tx=0x7f30d800eeb0 comp rx=0 tx=0).stop 2024-03-18T14:53:10.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.901+0000 7f30ed322640 1 -- 172.21.15.76:0/2659083687 shutdown_connections 2024-03-18T14:53:10.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.901+0000 7f30ed322640 1 --2- 172.21.15.76:0/2659083687 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f30d40a28f0 0x7f30d40a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:10.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.901+0000 7f30ed322640 1 --2- 172.21.15.76:0/2659083687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30e8107540 0x7f30e81007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:10.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.901+0000 7f30ed322640 1 --2- 172.21.15.76:0/2659083687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e8104c10 0x7f30e8100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:10.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.901+0000 7f30ed322640 1 --2- 172.21.15.76:0/2659083687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e81022e0 0x7f30e8101c50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:10.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.901+0000 7f30ed322640 1 -- 172.21.15.76:0/2659083687 >> 172.21.15.76:0/2659083687 conn(0x7f30e80fbc70 msgr2=0x7f30e8102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:10.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.901+0000 7f30ed322640 1 -- 172.21.15.76:0/2659083687 shutdown_connections 2024-03-18T14:53:10.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:10.901+0000 7f30ed322640 1 -- 172.21.15.76:0/2659083687 wait complete. 2024-03-18T14:53:10.911 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:10 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: ignoring --setuser ceph since I am not root 2024-03-18T14:53:10.911 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:10 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: ignoring --setgroup ceph since I am not root 2024-03-18T14:53:10.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:10 smithi076 ceph-mon[32205]: scrub queued for path: / 2024-03-18T14:53:10.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:10 smithi076 ceph-mon[32205]: scrub summary: idle+waiting paths [/] 2024-03-18T14:53:10.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:10 smithi076 ceph-mon[32205]: scrub summary: active paths [/] 2024-03-18T14:53:10.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:10 smithi076 ceph-mon[32205]: scrub summary: idle+waiting paths [/] 2024-03-18T14:53:10.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:10 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3975778803' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T14:53:10.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:10 smithi076 ceph-mon[32205]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T14:53:10.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:10 smithi076 ceph-mon[32205]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-18T14:53:10.913 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-18T14:53:10.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:10 smithi174 ceph-mon[36787]: scrub queued for path: / 2024-03-18T14:53:10.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:10 smithi174 ceph-mon[36787]: scrub summary: idle+waiting paths [/] 2024-03-18T14:53:10.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:10 smithi174 ceph-mon[36787]: scrub summary: active paths [/] 2024-03-18T14:53:10.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:10 smithi174 ceph-mon[36787]: scrub summary: idle+waiting paths [/] 2024-03-18T14:53:10.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:10 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3975778803' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T14:53:10.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:10 smithi174 ceph-mon[36787]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T14:53:10.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:10 smithi174 ceph-mon[36787]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-18T14:53:10.968 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:10 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: ignoring --setuser ceph since I am not root 2024-03-18T14:53:10.968 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:10 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: ignoring --setgroup ceph since I am not root 2024-03-18T14:53:10.968 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:10 smithi174 ceph-mgr[38646]: -- 172.21.15.174:0/4005357795 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d0b4f2e5a0 con 0x55d0b4f02800 2024-03-18T14:53:10.977 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:10 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: ignoring --setuser ceph since I am not root 2024-03-18T14:53:10.977 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:10 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: ignoring --setgroup ceph since I am not root 2024-03-18T14:53:10.977 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:10 smithi077 ceph-mgr[37256]: -- 172.21.15.77:0/3918657868 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56364e0625a0 con 0x56364e036800 2024-03-18T14:53:10.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:10 smithi077 ceph-mon[35865]: scrub queued for path: / 2024-03-18T14:53:10.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:10 smithi077 ceph-mon[35865]: scrub summary: idle+waiting paths [/] 2024-03-18T14:53:10.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:10 smithi077 ceph-mon[35865]: scrub summary: active paths [/] 2024-03-18T14:53:10.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:10 smithi077 ceph-mon[35865]: scrub summary: idle+waiting paths [/] 2024-03-18T14:53:10.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:10 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3975778803' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T14:53:10.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:10 smithi077 ceph-mon[35865]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T14:53:10.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:10 smithi077 ceph-mon[35865]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-18T14:53:11.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.016+0000 7fa8aa4f8640 1 -- 172.21.15.76:0/2065477810 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8a41018f0 msgr2=0x7fa8a4101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:11.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.016+0000 7fa8aa4f8640 1 --2- 172.21.15.76:0/2065477810 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8a41018f0 0x7fa8a4101d70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa898009900 tx=0x7fa89802bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:11.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.016+0000 7fa8aa4f8640 1 -- 172.21.15.76:0/2065477810 shutdown_connections 2024-03-18T14:53:11.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.016+0000 7fa8aa4f8640 1 --2- 172.21.15.76:0/2065477810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8a41022b0 0x7fa8a4108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:11.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.016+0000 7fa8aa4f8640 1 --2- 172.21.15.76:0/2065477810 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8a41018f0 0x7fa8a4101d70 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:11.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.016+0000 7fa8aa4f8640 1 --2- 172.21.15.76:0/2065477810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8a4100700 0x7fa8a4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:11.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.016+0000 7fa8aa4f8640 1 -- 172.21.15.76:0/2065477810 >> 172.21.15.76:0/2065477810 conn(0x7fa8a40fbc70 msgr2=0x7fa8a40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:11.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.016+0000 7fa8aa4f8640 1 -- 172.21.15.76:0/2065477810 shutdown_connections 2024-03-18T14:53:11.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.016+0000 7fa8aa4f8640 1 -- 172.21.15.76:0/2065477810 wait complete. 2024-03-18T14:53:11.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8aa4f8640 1 Processor -- start 2024-03-18T14:53:11.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8aa4f8640 1 -- start start 2024-03-18T14:53:11.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8aa4f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8a4100700 0x7fa8a4069a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:11.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8aa4f8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8a41018f0 0x7fa8a419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:11.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8aa4f8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8a41022b0 0x7fa8a419ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:11.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8aa4f8640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a410c3d0 con 0x7fa8a41018f0 2024-03-18T14:53:11.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8aa4f8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a410c250 con 0x7fa8a4100700 2024-03-18T14:53:11.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8aa4f8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a410c550 con 0x7fa8a41022b0 2024-03-18T14:53:11.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8a37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8a41018f0 0x7fa8a419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:11.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8a37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8a41018f0 0x7fa8a419c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44930/0 (socket says 172.21.15.76:44930) 2024-03-18T14:53:11.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8a37fe640 1 -- 172.21.15.76:0/1193331768 learned_addr learned my addr 172.21.15.76:0/1193331768 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:11.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8a3fff640 1 --2- 172.21.15.76:0/1193331768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8a4100700 0x7fa8a4069a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:11.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8a8a95640 1 --2- 172.21.15.76:0/1193331768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8a41022b0 0x7fa8a419ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:11.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8a37fe640 1 -- 172.21.15.76:0/1193331768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8a41022b0 msgr2=0x7fa8a419ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:11.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8a37fe640 1 --2- 172.21.15.76:0/1193331768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8a41022b0 0x7fa8a419ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:11.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8a37fe640 1 -- 172.21.15.76:0/1193331768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8a4100700 msgr2=0x7fa8a4069a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:11.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8a37fe640 1 --2- 172.21.15.76:0/1193331768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8a4100700 0x7fa8a4069a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:11.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8a37fe640 1 -- 172.21.15.76:0/1193331768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8a419f6a0 con 0x7fa8a41018f0 2024-03-18T14:53:11.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.017+0000 7fa8a3fff640 1 --2- 172.21.15.76:0/1193331768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8a4100700 0x7fa8a4069a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:11.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.018+0000 7fa8a37fe640 1 --2- 172.21.15.76:0/1193331768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8a41018f0 0x7fa8a419c770 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa898000c00 tx=0x7fa89802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:11.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.018+0000 7fa8a17fa640 1 -- 172.21.15.76:0/1193331768 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa898043020 con 0x7fa8a41018f0 2024-03-18T14:53:11.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.018+0000 7fa8aa4f8640 1 -- 172.21.15.76:0/1193331768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8a419f930 con 0x7fa8a41018f0 2024-03-18T14:53:11.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.018+0000 7fa8a17fa640 1 -- 172.21.15.76:0/1193331768 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8980054e0 con 0x7fa8a41018f0 2024-03-18T14:53:11.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.018+0000 7fa8a17fa640 1 -- 172.21.15.76:0/1193331768 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa898003c40 con 0x7fa8a41018f0 2024-03-18T14:53:11.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.018+0000 7fa8aa4f8640 1 -- 172.21.15.76:0/1193331768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8a419fe80 con 0x7fa8a41018f0 2024-03-18T14:53:11.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.019+0000 7fa8aa4f8640 1 -- 172.21.15.76:0/1193331768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8a41079c0 con 0x7fa8a41018f0 2024-03-18T14:53:11.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.021+0000 7fa8a17fa640 1 -- 172.21.15.76:0/1193331768 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 19) ==== 137307+0+0 (secure 0 0 0) 0x7fa898046020 con 0x7fa8a41018f0 2024-03-18T14:53:11.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.021+0000 7fa8a17fa640 1 --2- 172.21.15.76:0/1193331768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa88c0a28f0 0x7fa88c0a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:11.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.021+0000 7fa8a3fff640 1 -- 172.21.15.76:0/1193331768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa88c0a28f0 msgr2=0x7fa88c0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3010549595 2024-03-18T14:53:11.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.021+0000 7fa8a17fa640 1 -- 172.21.15.76:0/1193331768 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fa8980f0ad0 con 0x7fa8a41018f0 2024-03-18T14:53:11.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.021+0000 7fa8a3fff640 1 --2- 172.21.15.76:0/1193331768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa88c0a28f0 0x7fa88c0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T14:53:11.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.026+0000 7fa8a17fa640 1 -- 172.21.15.76:0/1193331768 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa89804d020 con 0x7fa8a41018f0 2024-03-18T14:53:11.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.205+0000 7fa8aa4f8640 1 -- 172.21.15.76:0/1193331768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fa8a40623e0 con 0x7fa8a41018f0 2024-03-18T14:53:11.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.211+0000 7fa8a17fa640 1 -- 172.21.15.76:0/1193331768 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v32)=0 v32) ==== 133+0+0 (secure 0 0 0) 0x7fa8980bdba0 con 0x7fa8a41018f0 2024-03-18T14:53:11.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.213+0000 7fa8aa4f8640 1 -- 172.21.15.76:0/1193331768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa88c0a28f0 msgr2=0x7fa88c0a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T14:53:11.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.213+0000 7fa8aa4f8640 1 --2- 172.21.15.76:0/1193331768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa88c0a28f0 0x7fa88c0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:11.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.213+0000 7fa8aa4f8640 1 -- 172.21.15.76:0/1193331768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8a41018f0 msgr2=0x7fa8a419c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:11.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.213+0000 7fa8aa4f8640 1 --2- 172.21.15.76:0/1193331768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8a41018f0 0x7fa8a419c770 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa898000c00 tx=0x7fa89802c650 comp rx=0 tx=0).stop 2024-03-18T14:53:11.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.213+0000 7fa8aa4f8640 1 -- 172.21.15.76:0/1193331768 shutdown_connections 2024-03-18T14:53:11.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.213+0000 7fa8aa4f8640 1 --2- 172.21.15.76:0/1193331768 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7fa88c0a28f0 0x7fa88c0a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:11.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.213+0000 7fa8aa4f8640 1 --2- 172.21.15.76:0/1193331768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8a41022b0 0x7fa8a419ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:11.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.213+0000 7fa8aa4f8640 1 --2- 172.21.15.76:0/1193331768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8a41018f0 0x7fa8a419c770 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:11.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.213+0000 7fa8aa4f8640 1 --2- 172.21.15.76:0/1193331768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8a4100700 0x7fa8a4069a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:11.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.213+0000 7fa8aa4f8640 1 -- 172.21.15.76:0/1193331768 >> 172.21.15.76:0/1193331768 conn(0x7fa8a40fbc70 msgr2=0x7fa8a4104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:11.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.213+0000 7fa8aa4f8640 1 -- 172.21.15.76:0/1193331768 shutdown_connections 2024-03-18T14:53:11.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.213+0000 7fa8aa4f8640 1 -- 172.21.15.76:0/1193331768 wait complete. 2024-03-18T14:53:11.224 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:11 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:11.152+0000 7f9008daf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T14:53:11.225 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-18T14:53:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.330+0000 7f662823f640 1 -- 172.21.15.76:0/347647539 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66201022b0 msgr2=0x7f6620108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.330+0000 7f662823f640 1 --2- 172.21.15.76:0/347647539 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66201022b0 0x7f6620108b40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6618009ca0 tx=0x7f661802c100 comp rx=0 tx=0).stop 2024-03-18T14:53:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.330+0000 7f662823f640 1 -- 172.21.15.76:0/347647539 shutdown_connections 2024-03-18T14:53:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.330+0000 7f662823f640 1 --2- 172.21.15.76:0/347647539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66201018f0 0x7f6620101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.330+0000 7f662823f640 1 --2- 172.21.15.76:0/347647539 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66201022b0 0x7f6620108b40 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.330+0000 7f662823f640 1 --2- 172.21.15.76:0/347647539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6620100700 0x7f6620100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.330+0000 7f662823f640 1 -- 172.21.15.76:0/347647539 >> 172.21.15.76:0/347647539 conn(0x7f66200fbc70 msgr2=0x7f66200fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.331+0000 7f662823f640 1 -- 172.21.15.76:0/347647539 shutdown_connections 2024-03-18T14:53:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.331+0000 7f662823f640 1 -- 172.21.15.76:0/347647539 wait complete. 2024-03-18T14:53:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.331+0000 7f662823f640 1 Processor -- start 2024-03-18T14:53:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.331+0000 7f662823f640 1 -- start start 2024-03-18T14:53:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.331+0000 7f662823f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6620100700 0x7f66200699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.331+0000 7f662823f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66201018f0 0x7f662019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.331+0000 7f662823f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66201022b0 0x7f66201a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.331+0000 7f662823f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f662010c360 con 0x7f6620100700 2024-03-18T14:53:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.331+0000 7f662823f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f662010c1e0 con 0x7f66201022b0 2024-03-18T14:53:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.331+0000 7f662823f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f662010c4e0 con 0x7f66201018f0 2024-03-18T14:53:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.331+0000 7f6625fdb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6620100700 0x7f66200699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.331+0000 7f6625fdb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6620100700 0x7f66200699d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44944/0 (socket says 172.21.15.76:44944) 2024-03-18T14:53:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.331+0000 7f6625fdb640 1 -- 172.21.15.76:0/965485915 learned_addr learned my addr 172.21.15.76:0/965485915 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.331+0000 7f66267dc640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66201022b0 0x7f66201a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.331+0000 7f66257da640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66201018f0 0x7f662019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.332+0000 7f6625fdb640 1 -- 172.21.15.76:0/965485915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66201018f0 msgr2=0x7f662019eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.332+0000 7f6625fdb640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66201018f0 0x7f662019eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.332+0000 7f6625fdb640 1 -- 172.21.15.76:0/965485915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66201022b0 msgr2=0x7f66201a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.332+0000 7f6625fdb640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66201022b0 0x7f66201a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.332+0000 7f6625fdb640 1 -- 172.21.15.76:0/965485915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66201a18a0 con 0x7f6620100700 2024-03-18T14:53:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.332+0000 7f6625fdb640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6620100700 0x7f66200699d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f661c00d770 tx=0x7f661c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.332+0000 7f6616ffd640 1 -- 172.21.15.76:0/965485915 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661c017070 con 0x7f6620100700 2024-03-18T14:53:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.332+0000 7f6616ffd640 1 -- 172.21.15.76:0/965485915 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f661c009c30 con 0x7f6620100700 2024-03-18T14:53:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.332+0000 7f662823f640 1 -- 172.21.15.76:0/965485915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66201a1b30 con 0x7f6620100700 2024-03-18T14:53:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.332+0000 7f6616ffd640 1 -- 172.21.15.76:0/965485915 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661c003dd0 con 0x7f6620100700 2024-03-18T14:53:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.332+0000 7f662823f640 1 -- 172.21.15.76:0/965485915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66201a2080 con 0x7f6620100700 2024-03-18T14:53:11.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.333+0000 7f662823f640 1 -- 172.21.15.76:0/965485915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66201079c0 con 0x7f6620100700 2024-03-18T14:53:11.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.335+0000 7f6616ffd640 1 -- 172.21.15.76:0/965485915 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 19) ==== 137307+0+0 (secure 0 0 0) 0x7f661c003f70 con 0x7f6620100700 2024-03-18T14:53:11.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.335+0000 7f6616ffd640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f660c0a2920 0x7f660c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:11.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.335+0000 7f66257da640 1 -- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f660c0a2920 msgr2=0x7f660c0a4de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3010549595 2024-03-18T14:53:11.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.335+0000 7f6616ffd640 1 -- 172.21.15.76:0/965485915 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f661c0ce360 con 0x7f6620100700 2024-03-18T14:53:11.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.335+0000 7f66257da640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f660c0a2920 0x7f660c0a4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T14:53:11.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.340+0000 7f6616ffd640 1 -- 172.21.15.76:0/965485915 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f661c009de0 con 0x7f6620100700 2024-03-18T14:53:11.466 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:11 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:11.139+0000 7fa3e1316200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T14:53:11.467 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:11 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:11.237+0000 7fa3e1316200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T14:53:11.477 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:11 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:11.129+0000 7fab866fd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T14:53:11.477 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:11 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:11.227+0000 7fab866fd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T14:53:11.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.536+0000 7f66257da640 1 -- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f660c0a2920 msgr2=0x7f660c0a4de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3010549595 2024-03-18T14:53:11.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.536+0000 7f66257da640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f660c0a2920 0x7f660c0a4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T14:53:11.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.544+0000 7f662823f640 1 -- 172.21.15.76:0/965485915 --> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f6620061c20 con 0x7f660c0a2920 2024-03-18T14:53:11.571 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:11 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:11.251+0000 7f9008daf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T14:53:11.903 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:11 smithi076 ceph-mon[32205]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-18T14:53:11.903 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:11 smithi076 ceph-mon[32205]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-18T14:53:11.904 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:11 smithi076 ceph-mon[32205]: from='client.? ' entity='client.admin' 2024-03-18T14:53:11.904 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:11 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1193331768' entity='client.admin' 2024-03-18T14:53:11.934 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:11 smithi174 ceph-mon[36787]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-18T14:53:11.935 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:11 smithi174 ceph-mon[36787]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-18T14:53:11.935 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:11 smithi174 ceph-mon[36787]: from='client.? ' entity='client.admin' 2024-03-18T14:53:11.935 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:11 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1193331768' entity='client.admin' 2024-03-18T14:53:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.937+0000 7f66257da640 1 -- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f660c0a2920 msgr2=0x7f660c0a4de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3010549595 2024-03-18T14:53:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:11.937+0000 7f66257da640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f660c0a2920 0x7f660c0a4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T14:53:11.942 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:11 smithi077 ceph-mon[35865]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-18T14:53:11.942 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:11 smithi077 ceph-mon[35865]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-18T14:53:11.942 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:11 smithi077 ceph-mon[35865]: from='client.? ' entity='client.admin' 2024-03-18T14:53:11.942 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:11 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1193331768' entity='client.admin' 2024-03-18T14:53:12.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:11 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:11.902+0000 7f9008daf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T14:53:12.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:11 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:11.996+0000 7f9008daf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T14:53:12.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:12 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:12.137+0000 7f9008daf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T14:53:12.216 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:11 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:11.934+0000 7fa3e1316200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T14:53:12.216 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:12.029+0000 7fa3e1316200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T14:53:12.217 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:12.174+0000 7fa3e1316200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T14:53:12.227 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:11 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:11.941+0000 7fab866fd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T14:53:12.227 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:12 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:12.036+0000 7fab866fd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T14:53:12.227 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:12 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:12.178+0000 7fab866fd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T14:53:12.581 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:12 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:12.234+0000 7f9008daf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T14:53:12.581 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:12 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:12.317+0000 7f9008daf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T14:53:12.624 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:12.271+0000 7fa3e1316200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T14:53:12.624 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:12.355+0000 7fa3e1316200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T14:53:12.653 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:12 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:12.276+0000 7fab866fd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T14:53:12.653 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:12 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:12.360+0000 7fab866fd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T14:53:12.653 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:12 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:12.652+0000 7fab866fd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T14:53:12.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:12.738+0000 7f66257da640 1 -- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f660c0a2920 msgr2=0x7f660c0a4de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3010549595 2024-03-18T14:53:12.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:12.738+0000 7f66257da640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f660c0a2920 0x7f660c0a4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T14:53:12.873 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:12 smithi076 ceph-mon[32205]: scrub summary: idle 2024-03-18T14:53:12.873 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:12 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:12.580+0000 7f9008daf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T14:53:12.873 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:12 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:12.690+0000 7f9008daf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T14:53:12.873 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:12 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:12.777+0000 7f9008daf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T14:53:12.873 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:12 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:12.872+0000 7f9008daf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T14:53:12.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:12 smithi174 ceph-mon[36787]: scrub summary: idle 2024-03-18T14:53:12.967 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:12.623+0000 7fa3e1316200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T14:53:12.967 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:12.734+0000 7fa3e1316200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T14:53:12.967 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:12.856+0000 7fa3e1316200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T14:53:12.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:12 smithi077 ceph-mon[35865]: scrub summary: idle 2024-03-18T14:53:12.977 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:12 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:12.783+0000 7fab866fd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T14:53:12.977 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:12 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:12.875+0000 7fab866fd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T14:53:13.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:13 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:13.047+0000 7f9008daf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T14:53:13.467 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:13 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:13.009+0000 7fa3e1316200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T14:53:13.467 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:13 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:13.212+0000 7fa3e1316200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T14:53:13.477 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:12 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:12.995+0000 7fab866fd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T14:53:13.477 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:13 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:13.173+0000 7fab866fd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T14:53:14.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:14.339+0000 7f66257da640 1 -- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f660c0a2920 msgr2=0x7f660c0a4de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3010549595 2024-03-18T14:53:14.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:14.339+0000 7f66257da640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f660c0a2920 0x7f660c0a4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T14:53:14.410 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:14 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:14.013+0000 7f9008daf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T14:53:14.411 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:14 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:14.145+0000 7f9008daf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T14:53:14.466 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:14.215+0000 7fa3e1316200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T14:53:14.466 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:14.348+0000 7fa3e1316200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T14:53:14.477 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:14 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:14.146+0000 7fab866fd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T14:53:14.477 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:14 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:14.278+0000 7fab866fd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T14:53:14.772 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:14 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:14.480+0000 7f9008daf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T14:53:14.772 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:14 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:14.572+0000 7f9008daf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T14:53:14.773 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:14 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:14.771+0000 7f9008daf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:53:14.900 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:14 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:14.611+0000 7fab866fd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T14:53:14.900 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:14 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:14.701+0000 7fab866fd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T14:53:14.967 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:14.693+0000 7fa3e1316200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T14:53:14.967 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:14.785+0000 7fa3e1316200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T14:53:15.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:14 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:14.959+0000 7f9008daf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T14:53:15.227 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:14 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:14.899+0000 7fab866fd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:53:15.227 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:15 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:15.085+0000 7fab866fd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T14:53:15.438 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:15 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:15.257+0000 7f9008daf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:53:15.466 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:15 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:15.008+0000 7fa3e1316200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:53:15.467 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:15 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:15.210+0000 7fa3e1316200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T14:53:15.637 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:15 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:15.373+0000 7fab866fd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:53:15.764 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:15 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:15.499+0000 7fa3e1316200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:53:15.785 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:15 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:15.530+0000 7f9008daf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T14:53:15.785 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:15 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:15.784+0000 7f9008daf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T14:53:15.897 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:15 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:15.636+0000 7fab866fd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T14:53:16.076 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:15 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:15.763+0000 7fa3e1316200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T14:53:16.227 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:15 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:15.896+0000 7fab866fd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T14:53:16.466 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:16 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:16.075+0000 7fa3e1316200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T14:53:16.661 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:16 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:16.623+0000 7f9008daf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T14:53:16.977 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:16 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:16.709+0000 7fab866fd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T14:53:17.367 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:17.043+0000 7fa3e1316200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T14:53:17.522 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:17.200+0000 7f9008daf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T14:53:17.523 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:17.300+0000 7f9008daf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T14:53:17.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:17.542+0000 7f66257da640 1 -- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f660c0a2920 msgr2=0x7f660c0a4de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3010549595 2024-03-18T14:53:17.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:17.542+0000 7f66257da640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f660c0a2920 0x7f660c0a4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-18T14:53:17.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:17.546+0000 7f6616ffd640 1 -- 172.21.15.76:0/965485915 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 20) ==== 136975+0+0 (secure 0 0 0) 0x7f661c004260 con 0x7f6620100700 2024-03-18T14:53:17.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:17.547+0000 7f6616ffd640 1 -- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f660c0a2920 msgr2=0x7f660c0a4de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T14:53:17.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:17.547+0000 7f6616ffd640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3010549595,v1:172.21.15.76:6801/3010549595] conn(0x7f660c0a2920 0x7f660c0a4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:17.590 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:17.292+0000 7fab866fd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T14:53:17.590 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:17.393+0000 7fab866fd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T14:53:17.691 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: Active manager daemon x restarted 2024-03-18T14:53:17.691 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: Activating manager daemon x 2024-03-18T14:53:17.691 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T14:53:17.692 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: mgrmap e20: x(active, starting, since 0.0119995s), standbys: z, y 2024-03-18T14:53:17.692 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:53:17.692 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:53:17.692 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:53:17.692 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T14:53:17.692 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T14:53:17.692 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T14:53:17.692 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T14:53:17.692 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T14:53:17.692 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T14:53:17.692 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T14:53:17.692 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T14:53:17.692 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:53:17.692 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T14:53:17.692 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T14:53:17.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T14:53:17.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T14:53:17.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T14:53:17.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T14:53:17.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:53:17.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:53:17.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:53:17.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:53:17.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:53:17.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:53:17.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:53:17.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:53:17.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:53:17.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:53:17.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:53:17.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:53:17.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T14:53:17.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T14:53:17.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T14:53:17.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-mon[36787]: Manager daemon x is now available 2024-03-18T14:53:17.786 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:17.522+0000 7f9008daf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T14:53:17.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: Active manager daemon x restarted 2024-03-18T14:53:17.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: Activating manager daemon x 2024-03-18T14:53:17.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T14:53:17.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: mgrmap e20: x(active, starting, since 0.0119995s), standbys: z, y 2024-03-18T14:53:17.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:53:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:53:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:53:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T14:53:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T14:53:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T14:53:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T14:53:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T14:53:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T14:53:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T14:53:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T14:53:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:53:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T14:53:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T14:53:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T14:53:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T14:53:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T14:53:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T14:53:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:53:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:53:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:53:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:53:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:53:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:53:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:53:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:53:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:53:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:53:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:53:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:53:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T14:53:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T14:53:17.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T14:53:17.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-mon[32205]: Manager daemon x is now available 2024-03-18T14:53:17.967 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:17.689+0000 7fa3e1316200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T14:53:17.967 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:17 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:17.822+0000 7fa3e1316200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T14:53:17.977 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y[37252]: 2024-03-18T14:53:17.631+0000 7fab866fd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T14:53:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: Active manager daemon x restarted 2024-03-18T14:53:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: Activating manager daemon x 2024-03-18T14:53:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T14:53:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: mgrmap e20: x(active, starting, since 0.0119995s), standbys: z, y 2024-03-18T14:53:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T14:53:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T14:53:17.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T14:53:17.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T14:53:17.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T14:53:17.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T14:53:17.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T14:53:17.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T14:53:17.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T14:53:17.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T14:53:17.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T14:53:17.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:53:17.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T14:53:17.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T14:53:17.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T14:53:17.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T14:53:17.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T14:53:17.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T14:53:17.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T14:53:17.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T14:53:17.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T14:53:17.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T14:53:17.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T14:53:17.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T14:53:17.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T14:53:17.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T14:53:17.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T14:53:17.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T14:53:17.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T14:53:17.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T14:53:17.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T14:53:17.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T14:53:17.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T14:53:17.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:17 smithi077 ceph-mon[35865]: Manager daemon x is now available 2024-03-18T14:53:18.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:17.786+0000 7f8f6b003700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:53:18.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:17.786+0000 7f8f6b003700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:53:18.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:17.786+0000 7f8f6b003700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:53:18.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:17.786+0000 7f8f6b003700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:53:18.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:17.786+0000 7f8f6b003700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:53:18.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:17.787+0000 7f8f687fe700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:53:18.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:17.787+0000 7f8f687fe700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:53:18.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:17.787+0000 7f8f687fe700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:53:18.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:17.787+0000 7f8f687fe700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:53:18.161 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:53:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:53:17.787+0000 7f8f687fe700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:53:18.467 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z[38642]: 2024-03-18T14:53:18.067+0000 7fa3e1316200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T14:53:18.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.556+0000 7f6616ffd640 1 -- 172.21.15.76:0/965485915 <== mon.0 v2:172.21.15.76:3300/0 8 ==== mgrmap(e 21) ==== 137102+0+0 (secure 0 0 0) 0x7f661c097450 con 0x7f6620100700 2024-03-18T14:53:18.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.557+0000 7f6616ffd640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f660c051330 0x7f660c051710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:18.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.557+0000 7f6616ffd640 1 -- 172.21.15.76:0/965485915 --> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f660c03ca70 con 0x7f660c051330 2024-03-18T14:53:18.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.561+0000 7f66257da640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f660c051330 0x7f660c051710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:18.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.563+0000 7f66257da640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f660c051330 0x7f660c051710 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6620069740 tx=0x7f661000b390 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:18.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.630+0000 7f6616ffd640 1 -- 172.21.15.76:0/965485915 <== mgr.25298 v2:172.21.15.76:6800/3052007196 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f660c03ca70 con 0x7f660c051330 2024-03-18T14:53:18.631 INFO:teuthology.orchestra.run.smithi076.stdout:Schedule set for path / 2024-03-18T14:53:18.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.633+0000 7f662823f640 1 -- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f660c051330 msgr2=0x7f660c051710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:18.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.633+0000 7f662823f640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f660c051330 0x7f660c051710 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6620069740 tx=0x7f661000b390 comp rx=0 tx=0).stop 2024-03-18T14:53:18.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.633+0000 7f662823f640 1 -- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6620100700 msgr2=0x7f66200699d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:18.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.633+0000 7f662823f640 1 --2- 172.21.15.76:0/965485915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6620100700 0x7f66200699d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f661c00d770 tx=0x7f661c00dc30 comp rx=0 tx=0).stop 2024-03-18T14:53:18.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.633+0000 7f66267dc640 1 -- 172.21.15.76:0/965485915 reap_dead start 2024-03-18T14:53:18.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.633+0000 7f662823f640 1 -- 172.21.15.76:0/965485915 shutdown_connections 2024-03-18T14:53:18.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.633+0000 7f662823f640 1 -- 172.21.15.76:0/965485915 >> 172.21.15.76:0/965485915 conn(0x7f66200fbc70 msgr2=0x7f6620104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:18.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.633+0000 7f662823f640 1 -- 172.21.15.76:0/965485915 shutdown_connections 2024-03-18T14:53:18.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.633+0000 7f662823f640 1 -- 172.21.15.76:0/965485915 wait complete. 2024-03-18T14:53:18.645 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-18T14:53:18.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.753+0000 7fe22e7e6640 1 -- 172.21.15.76:0/1665125249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2281018f0 msgr2=0x7fe228101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:18.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.753+0000 7fe22e7e6640 1 --2- 172.21.15.76:0/1665125249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2281018f0 0x7fe228101d70 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe218009860 tx=0x7fe21802bb70 comp rx=0 tx=0).stop 2024-03-18T14:53:18.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.753+0000 7fe22e7e6640 1 -- 172.21.15.76:0/1665125249 shutdown_connections 2024-03-18T14:53:18.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.753+0000 7fe22e7e6640 1 --2- 172.21.15.76:0/1665125249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2281022b0 0x7fe228108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:18.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.753+0000 7fe22e7e6640 1 --2- 172.21.15.76:0/1665125249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2281018f0 0x7fe228101d70 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:18.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.753+0000 7fe22e7e6640 1 --2- 172.21.15.76:0/1665125249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe228100700 0x7fe228100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:18.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.753+0000 7fe22e7e6640 1 -- 172.21.15.76:0/1665125249 >> 172.21.15.76:0/1665125249 conn(0x7fe2280fbc70 msgr2=0x7fe2280fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:18.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.753+0000 7fe22e7e6640 1 -- 172.21.15.76:0/1665125249 shutdown_connections 2024-03-18T14:53:18.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.753+0000 7fe22e7e6640 1 -- 172.21.15.76:0/1665125249 wait complete. 2024-03-18T14:53:18.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.753+0000 7fe22e7e6640 1 Processor -- start 2024-03-18T14:53:18.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.753+0000 7fe22e7e6640 1 -- start start 2024-03-18T14:53:18.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe22e7e6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe228100700 0x7fe228197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:18.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe22e7e6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2281018f0 0x7fe228198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:18.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe22e7e6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2281022b0 0x7fe22819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:18.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe22e7e6640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe22810d4a0 con 0x7fe2281018f0 2024-03-18T14:53:18.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe22e7e6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe22810d320 con 0x7fe2281022b0 2024-03-18T14:53:18.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe22e7e6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe22810d620 con 0x7fe228100700 2024-03-18T14:53:18.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe2277fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2281018f0 0x7fe228198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:18.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe2277fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2281018f0 0x7fe228198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41966/0 (socket says 172.21.15.76:41966) 2024-03-18T14:53:18.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe2277fe640 1 -- 172.21.15.76:0/2998400574 learned_addr learned my addr 172.21.15.76:0/2998400574 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:18.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe227fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe228100700 0x7fe228197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:18.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe22cd83640 1 --2- 172.21.15.76:0/2998400574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2281022b0 0x7fe22819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:18.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe2277fe640 1 -- 172.21.15.76:0/2998400574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe228100700 msgr2=0x7fe228197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:18.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe2277fe640 1 --2- 172.21.15.76:0/2998400574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe228100700 0x7fe228197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:18.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe2277fe640 1 -- 172.21.15.76:0/2998400574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2281022b0 msgr2=0x7fe22819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:18.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe2277fe640 1 --2- 172.21.15.76:0/2998400574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2281022b0 0x7fe22819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:18.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe2277fe640 1 -- 172.21.15.76:0/2998400574 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe22819f6e0 con 0x7fe2281018f0 2024-03-18T14:53:18.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe22cd83640 1 --2- 172.21.15.76:0/2998400574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2281022b0 0x7fe22819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:18.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.754+0000 7fe2277fe640 1 --2- 172.21.15.76:0/2998400574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2281018f0 0x7fe228198000 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe218005940 tx=0x7fe218032470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:18.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.755+0000 7fe2257fa640 1 -- 172.21.15.76:0/2998400574 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe218041020 con 0x7fe2281018f0 2024-03-18T14:53:18.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.755+0000 7fe22e7e6640 1 -- 172.21.15.76:0/2998400574 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe22819f970 con 0x7fe2281018f0 2024-03-18T14:53:18.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.755+0000 7fe2257fa640 1 -- 172.21.15.76:0/2998400574 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe21802c9f0 con 0x7fe2281018f0 2024-03-18T14:53:18.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.755+0000 7fe2257fa640 1 -- 172.21.15.76:0/2998400574 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe218039040 con 0x7fe2281018f0 2024-03-18T14:53:18.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.755+0000 7fe22e7e6640 1 -- 172.21.15.76:0/2998400574 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe22819fe60 con 0x7fe2281018f0 2024-03-18T14:53:18.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.756+0000 7fe22e7e6640 1 -- 172.21.15.76:0/2998400574 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe228107aa0 con 0x7fe2281018f0 2024-03-18T14:53:18.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.757+0000 7fe2257fa640 1 -- 172.21.15.76:0/2998400574 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 21) ==== 137102+0+0 (secure 0 0 0) 0x7fe218044020 con 0x7fe2281018f0 2024-03-18T14:53:18.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.758+0000 7fe2257fa640 1 --2- 172.21.15.76:0/2998400574 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe2140a26b0 0x7fe2140a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:18.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.758+0000 7fe2257fa640 1 -- 172.21.15.76:0/2998400574 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fe2180f04f0 con 0x7fe2281018f0 2024-03-18T14:53:18.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.758+0000 7fe227fff640 1 --2- 172.21.15.76:0/2998400574 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe2140a26b0 0x7fe2140a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:18.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.758+0000 7fe227fff640 1 --2- 172.21.15.76:0/2998400574 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe2140a26b0 0x7fe2140a4b70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe228101750 tx=0x7fe210005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:18.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.762+0000 7fe2257fa640 1 -- 172.21.15.76:0/2998400574 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe21804b020 con 0x7fe2281018f0 2024-03-18T14:53:18.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:53:18.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:53:18.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T14:53:18.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-mon[36787]: Standby manager daemon y restarted 2024-03-18T14:53:18.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-mon[36787]: Standby manager daemon y started 2024-03-18T14:53:18.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-mon[36787]: from='mgr.? 172.21.15.77:0/2801637298' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T14:53:18.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-mon[36787]: from='mgr.? 172.21.15.77:0/2801637298' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T14:53:18.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T14:53:18.890 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-mon[36787]: from='mgr.? 172.21.15.77:0/2801637298' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T14:53:18.890 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-mon[36787]: from='mgr.? 172.21.15.77:0/2801637298' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T14:53:18.890 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-mon[36787]: Standby manager daemon z restarted 2024-03-18T14:53:18.890 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-mon[36787]: Standby manager daemon z started 2024-03-18T14:53:18.890 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-mon[36787]: from='mgr.? 172.21.15.174:0/921287682' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T14:53:18.890 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-mon[36787]: from='mgr.? 172.21.15.174:0/921287682' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T14:53:18.890 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-mon[36787]: from='mgr.? 172.21.15.174:0/921287682' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T14:53:18.890 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-mon[36787]: from='mgr.? 172.21.15.174:0/921287682' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T14:53:18.890 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:18 smithi174 ceph-mon[36787]: mgrmap e21: x(active, since 1.02693s), standbys: z, y 2024-03-18T14:53:18.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:18 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:53:18.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:18 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:53:18.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:18 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T14:53:18.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:18 smithi076 ceph-mon[32205]: Standby manager daemon y restarted 2024-03-18T14:53:18.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:18 smithi076 ceph-mon[32205]: Standby manager daemon y started 2024-03-18T14:53:18.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:18 smithi076 ceph-mon[32205]: from='mgr.? 172.21.15.77:0/2801637298' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T14:53:18.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:18 smithi076 ceph-mon[32205]: from='mgr.? 172.21.15.77:0/2801637298' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T14:53:18.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:18 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T14:53:18.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:18 smithi076 ceph-mon[32205]: from='mgr.? 172.21.15.77:0/2801637298' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T14:53:18.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:18 smithi076 ceph-mon[32205]: from='mgr.? 172.21.15.77:0/2801637298' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T14:53:18.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:18 smithi076 ceph-mon[32205]: Standby manager daemon z restarted 2024-03-18T14:53:18.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:18 smithi076 ceph-mon[32205]: Standby manager daemon z started 2024-03-18T14:53:18.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:18 smithi076 ceph-mon[32205]: from='mgr.? 172.21.15.174:0/921287682' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T14:53:18.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:18 smithi076 ceph-mon[32205]: from='mgr.? 172.21.15.174:0/921287682' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T14:53:18.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:18 smithi076 ceph-mon[32205]: from='mgr.? 172.21.15.174:0/921287682' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T14:53:18.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:18 smithi076 ceph-mon[32205]: from='mgr.? 172.21.15.174:0/921287682' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T14:53:18.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:18 smithi076 ceph-mon[32205]: mgrmap e21: x(active, since 1.02693s), standbys: z, y 2024-03-18T14:53:18.943 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:18 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:53:18.943 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:18 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:53:18.943 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:18 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T14:53:18.943 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:18 smithi077 ceph-mon[35865]: Standby manager daemon y restarted 2024-03-18T14:53:18.943 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:18 smithi077 ceph-mon[35865]: Standby manager daemon y started 2024-03-18T14:53:18.943 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:18 smithi077 ceph-mon[35865]: from='mgr.? 172.21.15.77:0/2801637298' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T14:53:18.943 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:18 smithi077 ceph-mon[35865]: from='mgr.? 172.21.15.77:0/2801637298' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T14:53:18.943 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:18 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T14:53:18.943 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:18 smithi077 ceph-mon[35865]: from='mgr.? 172.21.15.77:0/2801637298' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T14:53:18.943 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:18 smithi077 ceph-mon[35865]: from='mgr.? 172.21.15.77:0/2801637298' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T14:53:18.944 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:18 smithi077 ceph-mon[35865]: Standby manager daemon z restarted 2024-03-18T14:53:18.944 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:18 smithi077 ceph-mon[35865]: Standby manager daemon z started 2024-03-18T14:53:18.944 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:18 smithi077 ceph-mon[35865]: from='mgr.? 172.21.15.174:0/921287682' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T14:53:18.944 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:18 smithi077 ceph-mon[35865]: from='mgr.? 172.21.15.174:0/921287682' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T14:53:18.944 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:18 smithi077 ceph-mon[35865]: from='mgr.? 172.21.15.174:0/921287682' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T14:53:18.944 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:18 smithi077 ceph-mon[35865]: from='mgr.? 172.21.15.174:0/921287682' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T14:53:18.944 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:18 smithi077 ceph-mon[35865]: mgrmap e21: x(active, since 1.02693s), standbys: z, y 2024-03-18T14:53:18.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:18.970+0000 7fe22e7e6640 1 -- 172.21.15.76:0/2998400574 --> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fe228061cc0 con 0x7fe2140a26b0 2024-03-18T14:53:19.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.024+0000 7fe2257fa640 1 -- 172.21.15.76:0/2998400574 <== mgr.25298 v2:172.21.15.76:6800/3052007196 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fe228061cc0 con 0x7fe2140a26b0 2024-03-18T14:53:19.025 INFO:teuthology.orchestra.run.smithi076.stdout:Retention added to path / 2024-03-18T14:53:19.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.026+0000 7fe22e7e6640 1 -- 172.21.15.76:0/2998400574 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe2140a26b0 msgr2=0x7fe2140a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:19.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.026+0000 7fe22e7e6640 1 --2- 172.21.15.76:0/2998400574 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe2140a26b0 0x7fe2140a4b70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe228101750 tx=0x7fe210005f90 comp rx=0 tx=0).stop 2024-03-18T14:53:19.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.026+0000 7fe22e7e6640 1 -- 172.21.15.76:0/2998400574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2281018f0 msgr2=0x7fe228198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:19.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.026+0000 7fe22e7e6640 1 --2- 172.21.15.76:0/2998400574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2281018f0 0x7fe228198000 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe218005940 tx=0x7fe218032470 comp rx=0 tx=0).stop 2024-03-18T14:53:19.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.026+0000 7fe22e7e6640 1 -- 172.21.15.76:0/2998400574 shutdown_connections 2024-03-18T14:53:19.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.026+0000 7fe22e7e6640 1 --2- 172.21.15.76:0/2998400574 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe2140a26b0 0x7fe2140a4b70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.026+0000 7fe22e7e6640 1 --2- 172.21.15.76:0/2998400574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2281022b0 0x7fe22819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.026+0000 7fe22e7e6640 1 --2- 172.21.15.76:0/2998400574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2281018f0 0x7fe228198000 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.026+0000 7fe22e7e6640 1 --2- 172.21.15.76:0/2998400574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe228100700 0x7fe228197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.026+0000 7fe22e7e6640 1 -- 172.21.15.76:0/2998400574 >> 172.21.15.76:0/2998400574 conn(0x7fe2280fbc70 msgr2=0x7fe228104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:19.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.026+0000 7fe22e7e6640 1 -- 172.21.15.76:0/2998400574 shutdown_connections 2024-03-18T14:53:19.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.026+0000 7fe22e7e6640 1 -- 172.21.15.76:0/2998400574 wait complete. 2024-03-18T14:53:19.039 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-18T14:53:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.142+0000 7f91df7df640 1 -- 172.21.15.76:0/459366114 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f91d81018f0 msgr2=0x7f91d8101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.142+0000 7f91df7df640 1 --2- 172.21.15.76:0/459366114 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f91d81018f0 0x7f91d8101d70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f91c8009900 tx=0x7f91c802bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.142+0000 7f91df7df640 1 -- 172.21.15.76:0/459366114 shutdown_connections 2024-03-18T14:53:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.142+0000 7f91df7df640 1 --2- 172.21.15.76:0/459366114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91d81022b0 0x7f91d8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.142+0000 7f91df7df640 1 --2- 172.21.15.76:0/459366114 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f91d81018f0 0x7f91d8101d70 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.142+0000 7f91df7df640 1 --2- 172.21.15.76:0/459366114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91d8100700 0x7f91d8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.142+0000 7f91df7df640 1 -- 172.21.15.76:0/459366114 >> 172.21.15.76:0/459366114 conn(0x7f91d80fbc70 msgr2=0x7f91d80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.142+0000 7f91df7df640 1 -- 172.21.15.76:0/459366114 shutdown_connections 2024-03-18T14:53:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.142+0000 7f91df7df640 1 -- 172.21.15.76:0/459366114 wait complete. 2024-03-18T14:53:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91df7df640 1 Processor -- start 2024-03-18T14:53:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91df7df640 1 -- start start 2024-03-18T14:53:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91df7df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91d8100700 0x7f91d8197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91df7df640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91d81018f0 0x7f91d8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91df7df640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f91d81022b0 0x7f91d819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91df7df640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f91d810c380 con 0x7f91d81022b0 2024-03-18T14:53:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91df7df640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f91d810c200 con 0x7f91d8100700 2024-03-18T14:53:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91df7df640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f91d810c500 con 0x7f91d81018f0 2024-03-18T14:53:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91ddd7c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f91d81022b0 0x7f91d819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91ddd7c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f91d81022b0 0x7f91d819f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41984/0 (socket says 172.21.15.76:41984) 2024-03-18T14:53:19.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91ddd7c640 1 -- 172.21.15.76:0/2235140562 learned_addr learned my addr 172.21.15.76:0/2235140562 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:19.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91dd57b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91d8100700 0x7f91d8197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:19.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91dcd7a640 1 --2- 172.21.15.76:0/2235140562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91d81018f0 0x7f91d8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:19.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91ddd7c640 1 -- 172.21.15.76:0/2235140562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91d81018f0 msgr2=0x7f91d8197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:19.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91ddd7c640 1 --2- 172.21.15.76:0/2235140562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91d81018f0 0x7f91d8197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91ddd7c640 1 -- 172.21.15.76:0/2235140562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91d8100700 msgr2=0x7f91d8197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:19.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91ddd7c640 1 --2- 172.21.15.76:0/2235140562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91d8100700 0x7f91d8197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.143+0000 7f91ddd7c640 1 -- 172.21.15.76:0/2235140562 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d819f6b0 con 0x7f91d81022b0 2024-03-18T14:53:19.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.144+0000 7f91ddd7c640 1 --2- 172.21.15.76:0/2235140562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f91d81022b0 0x7f91d819f040 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f91d000eb10 tx=0x7f91d000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:19.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.144+0000 7f91ce7fc640 1 -- 172.21.15.76:0/2235140562 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d0004b20 con 0x7f91d81022b0 2024-03-18T14:53:19.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.144+0000 7f91ce7fc640 1 -- 172.21.15.76:0/2235140562 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91d0004cc0 con 0x7f91d81022b0 2024-03-18T14:53:19.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.144+0000 7f91df7df640 1 -- 172.21.15.76:0/2235140562 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91d819f9a0 con 0x7f91d81022b0 2024-03-18T14:53:19.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.144+0000 7f91ce7fc640 1 -- 172.21.15.76:0/2235140562 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d0010760 con 0x7f91d81022b0 2024-03-18T14:53:19.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.144+0000 7f91df7df640 1 -- 172.21.15.76:0/2235140562 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91d819ff10 con 0x7f91d81022b0 2024-03-18T14:53:19.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.145+0000 7f91df7df640 1 -- 172.21.15.76:0/2235140562 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91d81a01a0 con 0x7f91d81022b0 2024-03-18T14:53:19.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.146+0000 7f91ce7fc640 1 -- 172.21.15.76:0/2235140562 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 21) ==== 137102+0+0 (secure 0 0 0) 0x7f91d0005020 con 0x7f91d81022b0 2024-03-18T14:53:19.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.146+0000 7f91ce7fc640 1 --2- 172.21.15.76:0/2235140562 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f91c40a2730 0x7f91c40a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:19.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.146+0000 7f91dd57b640 1 --2- 172.21.15.76:0/2235140562 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f91c40a2730 0x7f91c40a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:19.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.146+0000 7f91ce7fc640 1 -- 172.21.15.76:0/2235140562 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f91d00ced80 con 0x7f91d81022b0 2024-03-18T14:53:19.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.147+0000 7f91dd57b640 1 --2- 172.21.15.76:0/2235140562 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f91c40a2730 0x7f91c40a4bf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f91d8101750 tx=0x7f91d4005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:19.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.151+0000 7f91ce7fc640 1 -- 172.21.15.76:0/2235140562 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f91d009c040 con 0x7f91d81022b0 2024-03-18T14:53:19.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.360+0000 7f91df7df640 1 -- 172.21.15.76:0/2235140562 --> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f91d81a0370 con 0x7f91c40a2730 2024-03-18T14:53:19.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.367+0000 7f91ce7fc640 1 -- 172.21.15.76:0/2235140562 <== mgr.25298 v2:172.21.15.76:6800/3052007196 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f91d81a0370 con 0x7f91c40a2730 2024-03-18T14:53:19.368 INFO:teuthology.orchestra.run.smithi076.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T14:53:18", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-18T14:53:19.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.369+0000 7f91df7df640 1 -- 172.21.15.76:0/2235140562 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f91c40a2730 msgr2=0x7f91c40a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:19.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.369+0000 7f91df7df640 1 --2- 172.21.15.76:0/2235140562 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f91c40a2730 0x7f91c40a4bf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f91d8101750 tx=0x7f91d4005f90 comp rx=0 tx=0).stop 2024-03-18T14:53:19.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.369+0000 7f91df7df640 1 -- 172.21.15.76:0/2235140562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f91d81022b0 msgr2=0x7f91d819f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:19.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.369+0000 7f91df7df640 1 --2- 172.21.15.76:0/2235140562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f91d81022b0 0x7f91d819f040 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f91d000eb10 tx=0x7f91d000c2d0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.370+0000 7f91df7df640 1 -- 172.21.15.76:0/2235140562 shutdown_connections 2024-03-18T14:53:19.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.370+0000 7f91df7df640 1 --2- 172.21.15.76:0/2235140562 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f91c40a2730 0x7f91c40a4bf0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.370+0000 7f91df7df640 1 --2- 172.21.15.76:0/2235140562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91d81018f0 0x7f91d8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.370+0000 7f91df7df640 1 --2- 172.21.15.76:0/2235140562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f91d81022b0 0x7f91d819f040 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.370+0000 7f91df7df640 1 --2- 172.21.15.76:0/2235140562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91d8100700 0x7f91d8197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.370+0000 7f91df7df640 1 -- 172.21.15.76:0/2235140562 >> 172.21.15.76:0/2235140562 conn(0x7f91d80fbc70 msgr2=0x7f91d8068e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:19.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.370+0000 7f91df7df640 1 -- 172.21.15.76:0/2235140562 shutdown_connections 2024-03-18T14:53:19.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.370+0000 7f91df7df640 1 -- 172.21.15.76:0/2235140562 wait complete. 2024-03-18T14:53:19.382 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-18T14:53:19.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.492+0000 7fe86b463640 1 -- 172.21.15.76:0/2497051435 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8641022b0 msgr2=0x7fe864108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:19.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.492+0000 7fe86b463640 1 --2- 172.21.15.76:0/2497051435 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8641022b0 0x7fe864108b40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009ca0 tx=0x7fe85c02c0f0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.492+0000 7fe86b463640 1 -- 172.21.15.76:0/2497051435 shutdown_connections 2024-03-18T14:53:19.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.492+0000 7fe86b463640 1 --2- 172.21.15.76:0/2497051435 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8641022b0 0x7fe864108b40 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.492+0000 7fe86b463640 1 --2- 172.21.15.76:0/2497051435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8641018f0 0x7fe864101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.492+0000 7fe86b463640 1 --2- 172.21.15.76:0/2497051435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe864100700 0x7fe864100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.492+0000 7fe86b463640 1 -- 172.21.15.76:0/2497051435 >> 172.21.15.76:0/2497051435 conn(0x7fe8640fbc70 msgr2=0x7fe8640fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:19.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.493+0000 7fe86b463640 1 -- 172.21.15.76:0/2497051435 shutdown_connections 2024-03-18T14:53:19.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.493+0000 7fe86b463640 1 -- 172.21.15.76:0/2497051435 wait complete. 2024-03-18T14:53:19.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.493+0000 7fe86b463640 1 Processor -- start 2024-03-18T14:53:19.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.493+0000 7fe86b463640 1 -- start start 2024-03-18T14:53:19.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.493+0000 7fe86b463640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe864100700 0x7fe8640699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:19.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.493+0000 7fe86b463640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8641018f0 0x7fe86419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:19.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.493+0000 7fe86b463640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8641022b0 0x7fe8641a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:19.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.493+0000 7fe86b463640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe86410c360 con 0x7fe864100700 2024-03-18T14:53:19.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.493+0000 7fe86b463640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe86410c1e0 con 0x7fe8641022b0 2024-03-18T14:53:19.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.493+0000 7fe86b463640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe86410c4e0 con 0x7fe8641018f0 2024-03-18T14:53:19.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.494+0000 7fe869a00640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8641022b0 0x7fe8641a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:19.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.494+0000 7fe8691ff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe864100700 0x7fe8640699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:19.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.494+0000 7fe869a00640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8641022b0 0x7fe8641a10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:42990/0 (socket says 172.21.15.76:42990) 2024-03-18T14:53:19.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.494+0000 7fe869a00640 1 -- 172.21.15.76:0/1078617825 learned_addr learned my addr 172.21.15.76:0/1078617825 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:19.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.494+0000 7fe8689fe640 1 --2- 172.21.15.76:0/1078617825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8641018f0 0x7fe86419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:19.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.494+0000 7fe8691ff640 1 -- 172.21.15.76:0/1078617825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8641018f0 msgr2=0x7fe86419eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:19.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.494+0000 7fe8691ff640 1 --2- 172.21.15.76:0/1078617825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8641018f0 0x7fe86419eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.494+0000 7fe8691ff640 1 -- 172.21.15.76:0/1078617825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8641022b0 msgr2=0x7fe8641a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:19.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.494+0000 7fe8691ff640 1 --2- 172.21.15.76:0/1078617825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8641022b0 0x7fe8641a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.494+0000 7fe8691ff640 1 -- 172.21.15.76:0/1078617825 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8641a18b0 con 0x7fe864100700 2024-03-18T14:53:19.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.494+0000 7fe8691ff640 1 --2- 172.21.15.76:0/1078617825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe864100700 0x7fe8640699d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe864101750 tx=0x7fe86000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:19.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.494+0000 7fe85a7fc640 1 -- 172.21.15.76:0/1078617825 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe860017070 con 0x7fe864100700 2024-03-18T14:53:19.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.494+0000 7fe86b463640 1 -- 172.21.15.76:0/1078617825 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8641a1b40 con 0x7fe864100700 2024-03-18T14:53:19.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.494+0000 7fe85a7fc640 1 -- 172.21.15.76:0/1078617825 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe860009c30 con 0x7fe864100700 2024-03-18T14:53:19.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.494+0000 7fe85a7fc640 1 -- 172.21.15.76:0/1078617825 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86000f620 con 0x7fe864100700 2024-03-18T14:53:19.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.494+0000 7fe86b463640 1 -- 172.21.15.76:0/1078617825 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8641a2090 con 0x7fe864100700 2024-03-18T14:53:19.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.496+0000 7fe86b463640 1 -- 172.21.15.76:0/1078617825 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe86404f540 con 0x7fe864100700 2024-03-18T14:53:19.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.496+0000 7fe85a7fc640 1 -- 172.21.15.76:0/1078617825 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 21) ==== 137102+0+0 (secure 0 0 0) 0x7fe86001a070 con 0x7fe864100700 2024-03-18T14:53:19.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.497+0000 7fe85a7fc640 1 --2- 172.21.15.76:0/1078617825 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe8500a26e0 0x7fe8500a4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:19.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.497+0000 7fe8689fe640 1 --2- 172.21.15.76:0/1078617825 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe8500a26e0 0x7fe8500a4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:19.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.497+0000 7fe85a7fc640 1 -- 172.21.15.76:0/1078617825 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fe8600ce060 con 0x7fe864100700 2024-03-18T14:53:19.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.498+0000 7fe8689fe640 1 --2- 172.21.15.76:0/1078617825 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe8500a26e0 0x7fe8500a4ba0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe864069740 tx=0x7fe854005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:19.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.502+0000 7fe85a7fc640 1 -- 172.21.15.76:0/1078617825 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe860021030 con 0x7fe864100700 2024-03-18T14:53:19.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:19 smithi076 ceph-mon[32205]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:53:19.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:19 smithi076 ceph-mon[32205]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:53:19.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.748+0000 7fe86b463640 1 -- 172.21.15.76:0/1078617825 --> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fe864061c20 con 0x7fe8500a26e0 2024-03-18T14:53:19.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.755+0000 7fe85a7fc640 1 -- 172.21.15.76:0/1078617825 <== mgr.25298 v2:172.21.15.76:6800/3052007196 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fe864061c20 con 0x7fe8500a26e0 2024-03-18T14:53:19.756 INFO:teuthology.orchestra.run.smithi076.stdout:/ 1m 6m3h 2024-03-18T14:53:19.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.757+0000 7fe86b463640 1 -- 172.21.15.76:0/1078617825 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe8500a26e0 msgr2=0x7fe8500a4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:19.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.757+0000 7fe86b463640 1 --2- 172.21.15.76:0/1078617825 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe8500a26e0 0x7fe8500a4ba0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe864069740 tx=0x7fe854005f90 comp rx=0 tx=0).stop 2024-03-18T14:53:19.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.757+0000 7fe86b463640 1 -- 172.21.15.76:0/1078617825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe864100700 msgr2=0x7fe8640699d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:19.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.757+0000 7fe86b463640 1 --2- 172.21.15.76:0/1078617825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe864100700 0x7fe8640699d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe864101750 tx=0x7fe86000dde0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.758+0000 7fe86b463640 1 -- 172.21.15.76:0/1078617825 shutdown_connections 2024-03-18T14:53:19.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.758+0000 7fe86b463640 1 --2- 172.21.15.76:0/1078617825 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe8500a26e0 0x7fe8500a4ba0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.758+0000 7fe86b463640 1 --2- 172.21.15.76:0/1078617825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8641018f0 0x7fe86419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.758+0000 7fe86b463640 1 --2- 172.21.15.76:0/1078617825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8641022b0 0x7fe8641a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.758+0000 7fe86b463640 1 --2- 172.21.15.76:0/1078617825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe864100700 0x7fe8640699d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:19.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.758+0000 7fe86b463640 1 -- 172.21.15.76:0/1078617825 >> 172.21.15.76:0/1078617825 conn(0x7fe8640fbc70 msgr2=0x7fe864104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:19.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.758+0000 7fe86b463640 1 -- 172.21.15.76:0/1078617825 shutdown_connections 2024-03-18T14:53:19.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:19.758+0000 7fe86b463640 1 -- 172.21.15.76:0/1078617825 wait complete. 2024-03-18T14:53:19.770 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-18T14:53:19.799 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-18T14:53:19.809 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-18T14:53:19.831 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi076.front.sepia.ceph.com 2024-03-18T14:53:19.831 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-18T14:53:19.841 INFO:teuthology.run_tasks:Running task workunit... 2024-03-18T14:53:19.856 INFO:tasks.workunit:Pulling workunits from ref 4e19f704afd9957ea657ab858f9ecf76401fef6f 2024-03-18T14:53:19.856 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-18T14:53:19.856 INFO:tasks.workunit:timeout=3h 2024-03-18T14:53:19.857 INFO:tasks.workunit:cleanup=True 2024-03-18T14:53:19.857 DEBUG:teuthology.orchestra.run.smithi076:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-18T14:53:19.872 INFO:teuthology.orchestra.run.smithi076.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-18T14:53:19.873 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-18T14:53:19.873 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-18T14:53:19.873 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:53:19.873 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-18T14:53:19.873 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-03-18 14:51:08.407002909 +0000 2024-03-18T14:53:19.873 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-03-18 14:51:08.407002909 +0000 2024-03-18T14:53:19.873 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-03-18 14:53:02.050970559 +0000 2024-03-18T14:53:19.873 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-03-18 14:51:08.407002909 +0000 2024-03-18T14:53:19.873 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-18T14:53:19.873 DEBUG:teuthology.orchestra.run.smithi076:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-18T14:53:19.949 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4e19f704afd9957ea657ab858f9ecf76401fef6f 2024-03-18T14:53:19.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:19 smithi174 ceph-mon[36787]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:53:19.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:19 smithi174 ceph-mon[36787]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:53:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:19 smithi077 ceph-mon[35865]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:53:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:19 smithi077 ceph-mon[35865]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:53:20.004 INFO:tasks.workunit.client.0.smithi076.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-18T14:53:20.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:20 smithi076 ceph-mon[32205]: [18/Mar/2024:14:53:18] ENGINE Bus STARTING 2024-03-18T14:53:20.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:20 smithi076 ceph-mon[32205]: [18/Mar/2024:14:53:18] ENGINE Serving on https://172.21.15.76:7150 2024-03-18T14:53:20.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:20 smithi076 ceph-mon[32205]: [18/Mar/2024:14:53:18] ENGINE Serving on http://172.21.15.76:8765 2024-03-18T14:53:20.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:20 smithi076 ceph-mon[32205]: [18/Mar/2024:14:53:18] ENGINE Bus STARTED 2024-03-18T14:53:20.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:20 smithi076 ceph-mon[32205]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:53:20.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:20 smithi076 ceph-mon[32205]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:53:20.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:20 smithi076 ceph-mon[32205]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:53:20.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:20 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:20 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:20 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:20 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:20 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:20 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:20 smithi076 ceph-mon[32205]: mgrmap e22: x(active, since 3s), standbys: z, y 2024-03-18T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:20 smithi174 ceph-mon[36787]: [18/Mar/2024:14:53:18] ENGINE Bus STARTING 2024-03-18T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:20 smithi174 ceph-mon[36787]: [18/Mar/2024:14:53:18] ENGINE Serving on https://172.21.15.76:7150 2024-03-18T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:20 smithi174 ceph-mon[36787]: [18/Mar/2024:14:53:18] ENGINE Serving on http://172.21.15.76:8765 2024-03-18T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:20 smithi174 ceph-mon[36787]: [18/Mar/2024:14:53:18] ENGINE Bus STARTED 2024-03-18T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:20 smithi174 ceph-mon[36787]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:20 smithi174 ceph-mon[36787]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:20 smithi174 ceph-mon[36787]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:20 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:20 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:20 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:20 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:20 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:20 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:20 smithi174 ceph-mon[36787]: mgrmap e22: x(active, since 3s), standbys: z, y 2024-03-18T14:53:20.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:20 smithi077 ceph-mon[35865]: [18/Mar/2024:14:53:18] ENGINE Bus STARTING 2024-03-18T14:53:20.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:20 smithi077 ceph-mon[35865]: [18/Mar/2024:14:53:18] ENGINE Serving on https://172.21.15.76:7150 2024-03-18T14:53:20.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:20 smithi077 ceph-mon[35865]: [18/Mar/2024:14:53:18] ENGINE Serving on http://172.21.15.76:8765 2024-03-18T14:53:20.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:20 smithi077 ceph-mon[35865]: [18/Mar/2024:14:53:18] ENGINE Bus STARTED 2024-03-18T14:53:20.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:20 smithi077 ceph-mon[35865]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:53:20.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:20 smithi077 ceph-mon[35865]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:53:20.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:20 smithi077 ceph-mon[35865]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:53:20.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:20 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:20 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:20 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:20 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:20 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:20 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:20.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:20 smithi077 ceph-mon[35865]: mgrmap e22: x(active, since 3s), standbys: z, y 2024-03-18T14:53:21.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:53:21.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:53:21.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:21 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:53:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:53:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:53:21.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:21 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:53:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T14:53:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:21.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T14:53:21.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:53:21.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:21 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:53:22.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-03-18T14:53:22.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: Unable to set osd_memory_target on smithi076 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-03-18T14:53:22.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi077 to 174.1M 2024-03-18T14:53:22.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: Unable to set osd_memory_target on smithi077 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-18T14:53:22.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi174 to 174.1M 2024-03-18T14:53:22.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: Unable to set osd_memory_target on smithi174 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-18T14:53:22.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: Updating smithi076:/etc/ceph/ceph.conf 2024-03-18T14:53:22.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: Updating smithi077:/etc/ceph/ceph.conf 2024-03-18T14:53:22.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-18T14:53:22.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: Updating smithi174:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:53:22.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: Updating smithi076:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:53:22.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: Updating smithi077:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:53:22.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:53:22.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:53:22.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:53:22.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:53:22.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-03-18T14:53:22.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:53:22.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:53:22.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:53:22.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:22 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-03-18T14:53:22.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: Unable to set osd_memory_target on smithi076 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-03-18T14:53:22.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi077 to 174.1M 2024-03-18T14:53:22.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: Unable to set osd_memory_target on smithi077 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-18T14:53:22.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi174 to 174.1M 2024-03-18T14:53:22.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: Unable to set osd_memory_target on smithi174 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-18T14:53:22.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: Updating smithi076:/etc/ceph/ceph.conf 2024-03-18T14:53:22.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: Updating smithi077:/etc/ceph/ceph.conf 2024-03-18T14:53:22.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-18T14:53:22.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: Updating smithi174:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:53:22.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: Updating smithi076:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:53:22.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: Updating smithi077:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:53:22.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:53:22.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:53:22.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:53:22.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:53:22.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-03-18T14:53:22.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:53:22.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:53:22.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:53:22.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:22 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-03-18T14:53:22.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: Unable to set osd_memory_target on smithi076 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-03-18T14:53:22.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi077 to 174.1M 2024-03-18T14:53:22.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: Unable to set osd_memory_target on smithi077 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-18T14:53:22.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi174 to 174.1M 2024-03-18T14:53:22.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: Unable to set osd_memory_target on smithi174 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-18T14:53:22.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: Updating smithi076:/etc/ceph/ceph.conf 2024-03-18T14:53:22.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: Updating smithi077:/etc/ceph/ceph.conf 2024-03-18T14:53:22.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-18T14:53:22.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: Updating smithi174:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:53:22.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: Updating smithi076:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:53:22.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: Updating smithi077:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T14:53:22.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:53:22.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:53:22.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-03-18T14:53:22.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T14:53:22.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-03-18T14:53:22.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:22.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:53:22.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:53:22.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:53:22.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:22 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:53:23.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:23 smithi076 ceph-mon[32205]: Updating smithi174:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T14:53:23.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:23 smithi076 ceph-mon[32205]: Updating smithi076:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T14:53:23.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:23 smithi076 ceph-mon[32205]: Updating smithi077:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T14:53:23.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:23 smithi174 ceph-mon[36787]: Updating smithi174:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T14:53:23.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:23 smithi174 ceph-mon[36787]: Updating smithi076:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T14:53:23.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:23 smithi174 ceph-mon[36787]: Updating smithi077:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T14:53:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:23 smithi077 ceph-mon[35865]: Updating smithi174:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T14:53:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:23 smithi077 ceph-mon[35865]: Updating smithi076:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T14:53:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:23 smithi077 ceph-mon[35865]: Updating smithi077:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T14:53:24.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:24 smithi076 ceph-mon[32205]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-18T14:53:24.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:24 smithi174 ceph-mon[36787]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-18T14:53:24.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:24 smithi077 ceph-mon[35865]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-18T14:53:26.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:26 smithi076 ceph-mon[32205]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 76 op/s 2024-03-18T14:53:26.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:26 smithi174 ceph-mon[36787]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 76 op/s 2024-03-18T14:53:26.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:26 smithi077 ceph-mon[35865]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 76 op/s 2024-03-18T14:53:28.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:28 smithi076 ceph-mon[32205]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-18T14:53:28.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:28 smithi174 ceph-mon[36787]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-18T14:53:28.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:28 smithi077 ceph-mon[35865]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-18T14:53:30.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:30 smithi076 ceph-mon[32205]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-18T14:53:30.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:30 smithi174 ceph-mon[36787]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-18T14:53:30.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:30 smithi077 ceph-mon[35865]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-18T14:53:32.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:32 smithi076 ceph-mon[32205]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-18T14:53:32.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:32 smithi174 ceph-mon[36787]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-18T14:53:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.975+0000 7f863c419640 1 -- 172.21.15.76:0/3038741298 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8634102140 msgr2=0x7f863410e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.975+0000 7f863c419640 1 --2- 172.21.15.76:0/3038741298 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8634102140 0x7f863410e650 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f862c009cd0 tx=0x7f862c02c0f0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.975+0000 7f863c419640 1 -- 172.21.15.76:0/3038741298 shutdown_connections 2024-03-18T14:53:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.975+0000 7f863c419640 1 --2- 172.21.15.76:0/3038741298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8634101780 0x7f8634101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.975+0000 7f863c419640 1 --2- 172.21.15.76:0/3038741298 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8634102140 0x7f863410e650 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.975+0000 7f863c419640 1 --2- 172.21.15.76:0/3038741298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8634107790 0x7f8634107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.975+0000 7f863c419640 1 -- 172.21.15.76:0/3038741298 >> 172.21.15.76:0/3038741298 conn(0x7f86340fd270 msgr2=0x7f86340ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.975+0000 7f863c419640 1 -- 172.21.15.76:0/3038741298 shutdown_connections 2024-03-18T14:53:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.975+0000 7f863c419640 1 -- 172.21.15.76:0/3038741298 wait complete. 2024-03-18T14:53:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f863c419640 1 Processor -- start 2024-03-18T14:53:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f863c419640 1 -- start start 2024-03-18T14:53:32.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:32 smithi077 ceph-mon[35865]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-18T14:53:32.980 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f863c419640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8634101780 0x7f8634197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:32.980 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f863c419640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8634102140 0x7f86341984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:32.980 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f863c419640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8634107790 0x7f863419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:32.980 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f863c419640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8634111b60 con 0x7f8634101780 2024-03-18T14:53:32.980 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f863c419640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f86341119e0 con 0x7f8634107790 2024-03-18T14:53:32.980 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f863c419640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8634111ce0 con 0x7f8634102140 2024-03-18T14:53:32.980 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f863a1b5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8634101780 0x7f8634197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:32.980 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f863a1b5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8634101780 0x7f8634197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45400/0 (socket says 172.21.15.76:45400) 2024-03-18T14:53:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f863a1b5640 1 -- 172.21.15.76:0/972720605 learned_addr learned my addr 172.21.15.76:0/972720605 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f86399b4640 1 --2- 172.21.15.76:0/972720605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8634102140 0x7f86341984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f863a9b6640 1 --2- 172.21.15.76:0/972720605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8634107790 0x7f863419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f863a1b5640 1 -- 172.21.15.76:0/972720605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8634102140 msgr2=0x7f86341984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f863a1b5640 1 --2- 172.21.15.76:0/972720605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8634102140 0x7f86341984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f863a1b5640 1 -- 172.21.15.76:0/972720605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8634107790 msgr2=0x7f863419c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f863a1b5640 1 --2- 172.21.15.76:0/972720605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8634107790 0x7f863419c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.976+0000 7f863a1b5640 1 -- 172.21.15.76:0/972720605 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f863419cf70 con 0x7f8634101780 2024-03-18T14:53:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.977+0000 7f863a1b5640 1 --2- 172.21.15.76:0/972720605 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8634101780 0x7f8634197f80 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f863000e9f0 tx=0x7f863000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.977+0000 7f862b7fe640 1 -- 172.21.15.76:0/972720605 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863000cd80 con 0x7f8634101780 2024-03-18T14:53:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.977+0000 7f862b7fe640 1 -- 172.21.15.76:0/972720605 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8630010910 con 0x7f8634101780 2024-03-18T14:53:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.977+0000 7f862b7fe640 1 -- 172.21.15.76:0/972720605 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8630018990 con 0x7f8634101780 2024-03-18T14:53:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.977+0000 7f863c419640 1 -- 172.21.15.76:0/972720605 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f863419d260 con 0x7f8634101780 2024-03-18T14:53:32.982 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.977+0000 7f863c419640 1 -- 172.21.15.76:0/972720605 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86341a4af0 con 0x7f8634101780 2024-03-18T14:53:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.979+0000 7f862b7fe640 1 -- 172.21.15.76:0/972720605 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 23) ==== 137466+0+0 (secure 0 0 0) 0x7f8630010ac0 con 0x7f8634101780 2024-03-18T14:53:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.979+0000 7f863f007740 1 Processor -- start 2024-03-18T14:53:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.980+0000 7f863f007740 1 -- start start 2024-03-18T14:53:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.980+0000 7f863f007740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbae130 0x55a7adbae510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.980+0000 7f863f007740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbaea50 0x55a7adbb72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.980+0000 7f863f007740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbb9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.980+0000 7f863f007740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a7adb12a40 con 0x55a7adbb7820 2024-03-18T14:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.980+0000 7f863f007740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a7adabb920 con 0x55a7adbae130 2024-03-18T14:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.980+0000 7f863f007740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a7adb0eae0 con 0x55a7adbaea50 2024-03-18T14:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.980+0000 7f86399b4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbb9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.980+0000 7f863a1b5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbaea50 0x55a7adbb72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.980+0000 7f86399b4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbb9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45402/0 (socket says 172.21.15.76:45402) 2024-03-18T14:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.980+0000 7f863a9b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbae130 0x55a7adbae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.980+0000 7f86399b4640 1 -- 172.21.15.76:0/2657452362 learned_addr learned my addr 172.21.15.76:0/2657452362 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.980+0000 7f862b7fe640 1 --2- 172.21.15.76:0/972720605 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f86200a2b30 0x7f86200a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.980+0000 7f863a9b6640 1 --2- 172.21.15.76:0/972720605 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f86200a2b30 0x7f86200a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.980+0000 7f86399b4640 1 -- 172.21.15.76:0/2657452362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbaea50 msgr2=0x55a7adbb72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.981+0000 7f86399b4640 1 --2- 172.21.15.76:0/2657452362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbaea50 0x55a7adbb72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.981+0000 7f86399b4640 1 -- 172.21.15.76:0/2657452362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbae130 msgr2=0x55a7adbae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.981+0000 7f86399b4640 1 --2- 172.21.15.76:0/2657452362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbae130 0x55a7adbae510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.981+0000 7f86399b4640 1 -- 172.21.15.76:0/2657452362 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7adad3460 con 0x55a7adbb7820 2024-03-18T14:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.981+0000 7f863a1b5640 1 --2- 172.21.15.76:0/2657452362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbaea50 0x55a7adbb72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.981+0000 7f862b7fe640 1 -- 172.21.15.76:0/972720605 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f86300cf1c0 con 0x7f8634101780 2024-03-18T14:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.981+0000 7f86399b4640 1 --2- 172.21.15.76:0/2657452362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbb9c10 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8624009c90 tx=0x7f8624011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b17e772105aa795 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.981+0000 7f863a9b6640 1 --2- 172.21.15.76:0/972720605 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f86200a2b30 0x7f86200a4ff0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f862c005cc0 tx=0x7f862c02c0f0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.981+0000 7f8628bfe640 1 -- 172.21.15.76:0/2657452362 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862401e070 con 0x55a7adbb7820 2024-03-18T14:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.981+0000 7f8628bfe640 1 -- 172.21.15.76:0/2657452362 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8624019b10 con 0x55a7adbb7820 2024-03-18T14:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.981+0000 7f8628bfe640 1 -- 172.21.15.76:0/2657452362 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862400ea70 con 0x55a7adbb7820 2024-03-18T14:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.982+0000 7f863f007740 1 -- 172.21.15.76:0/2657452362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 msgr2=0x55a7adbb9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.982+0000 7f863f007740 1 --2- 172.21.15.76:0/2657452362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbb9c10 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8624009c90 tx=0x7f8624011810 comp rx=0 tx=0).stop 2024-03-18T14:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.982+0000 7f863f007740 1 -- 172.21.15.76:0/2657452362 shutdown_connections 2024-03-18T14:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.982+0000 7f863f007740 1 --2- 172.21.15.76:0/2657452362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbaea50 0x55a7adbb72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.982+0000 7f863f007740 1 --2- 172.21.15.76:0/2657452362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbb9c10 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.982+0000 7f863f007740 1 --2- 172.21.15.76:0/2657452362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbae130 0x55a7adbae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.982+0000 7f863f007740 1 -- 172.21.15.76:0/2657452362 >> 172.21.15.76:0/2657452362 conn(0x55a7adba8a50 msgr2=0x55a7adba8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.982+0000 7f863f007740 1 -- 172.21.15.76:0/2657452362 shutdown_connections 2024-03-18T14:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.982+0000 7f863f007740 1 -- 172.21.15.76:0/2657452362 wait complete. 2024-03-18T14:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.982+0000 7f863f007740 1 -- start start 2024-03-18T14:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f863f007740 1 Processor -- start 2024-03-18T14:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f863f007740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbae130 0x55a7adbb4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f863f007740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbaea50 0x55a7adbb4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f863f007740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbb5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f863f007740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a7adbba450 con 0x55a7adbb7820 2024-03-18T14:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f863f007740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a7adaced70 con 0x55a7adbae130 2024-03-18T14:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f863f007740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a7adbba5d0 con 0x55a7adbaea50 2024-03-18T14:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f863f007740 10 client.0 _ll_register_callbacks cb 0x55a7ad826a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f863a9b6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbb5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f86399b4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbaea50 0x55a7adbb4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f863a9b6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbb5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45408/0 (socket says 172.21.15.76:45408) 2024-03-18T14:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f863a9b6640 1 -- 172.21.15.76:0/4118282700 learned_addr learned my addr 172.21.15.76:0/4118282700 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f863a1b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbae130 0x55a7adbb4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f863a9b6640 1 -- 172.21.15.76:0/4118282700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbaea50 msgr2=0x55a7adbb4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f863a9b6640 1 --2- 172.21.15.76:0/4118282700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbaea50 0x55a7adbb4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f863a9b6640 1 -- 172.21.15.76:0/4118282700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbae130 msgr2=0x55a7adbb4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f863a9b6640 1 --2- 172.21.15.76:0/4118282700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbae130 0x55a7adbb4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.983+0000 7f863a9b6640 1 -- 172.21.15.76:0/4118282700 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7ad827210 con 0x55a7adbb7820 2024-03-18T14:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.984+0000 7f863a9b6640 1 --2- 172.21.15.76:0/4118282700 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbb5310 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f862c036000 tx=0x7f862c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee5bd99266e6a27b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.984+0000 7f861dffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.984+0000 7f861dffb640 1 -- 172.21.15.76:0/4118282700 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c053060 con 0x55a7adbb7820 2024-03-18T14:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.984+0000 7f861dffb640 1 -- 172.21.15.76:0/4118282700 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f862c004040 con 0x55a7adbb7820 2024-03-18T14:53:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.984+0000 7f863f007740 1 -- 172.21.15.76:0/4118282700 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8624019b10 con 0x55a7adbb7820 2024-03-18T14:53:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.984+0000 7f861dffb640 1 -- 172.21.15.76:0/4118282700 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f862c047070 con 0x55a7adbb7820 2024-03-18T14:53:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.984+0000 7f863f007740 10 client.15519 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.984+0000 7f863f007740 1 -- 172.21.15.76:0/4118282700 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a7ad8f3620 con 0x55a7adbb7820 2024-03-18T14:53:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.984+0000 7f861dffb640 1 -- 172.21.15.76:0/4118282700 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8624019b10 con 0x55a7adbb7820 2024-03-18T14:53:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.984+0000 7f861dffb640 1 -- 172.21.15.76:0/4118282700 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f862c03f070 con 0x55a7adbb7820 2024-03-18T14:53:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.984+0000 7f863f007740 10 client.15519 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.984+0000 7f863f007740 10 client.15519 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.984+0000 7f863f007740 20 client.15519 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.984+0000 7f863f007740 1 --2- 172.21.15.76:0/4118282700 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a7adbc7a40 0x55a7adbc9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.984+0000 7f863f007740 4 client.15519 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:53:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.984+0000 7f863f007740 1 -- 172.21.15.76:0/4118282700 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7adb1ff80 con 0x55a7adbc7a40 2024-03-18T14:53:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.985+0000 7f863a1b5640 1 --2- 172.21.15.76:0/4118282700 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a7adbc7a40 0x55a7adbc9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.985+0000 7f861dffb640 10 client.15519 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.985+0000 7f863a1b5640 1 --2- 172.21.15.76:0/4118282700 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a7adbc7a40 0x55a7adbc9e50 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fda7dbd83830270 server_cookie=8a6d369de68a52b in_seq=0 out_seq=0 2024-03-18T14:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.986+0000 7f861dffb640 1 -- 172.21.15.76:0/4118282700 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a7adb1ff80 con 0x55a7adbc7a40 2024-03-18T14:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.986+0000 7f861dffb640 10 client.15519 handle_command_reply: tid=0 2024-03-18T14:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.986+0000 7f863f007740 1 client.15519 shutdown 2024-03-18T14:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.986+0000 7f863f007740 1 -- 172.21.15.76:0/4118282700 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 msgr2=0x55a7adbb5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.986+0000 7f863f007740 1 --2- 172.21.15.76:0/4118282700 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbb5310 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f862c036000 tx=0x7f862c0095c0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.986+0000 7f863f007740 1 -- 172.21.15.76:0/4118282700 shutdown_connections 2024-03-18T14:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.986+0000 7f863f007740 1 --2- 172.21.15.76:0/4118282700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbaea50 0x55a7adbb4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.986+0000 7f863f007740 1 --2- 172.21.15.76:0/4118282700 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a7adbc7a40 0x55a7adbc9e50 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.986+0000 7f863f007740 1 --2- 172.21.15.76:0/4118282700 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbb5310 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.986+0000 7f863f007740 1 --2- 172.21.15.76:0/4118282700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbae130 0x55a7adbb4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.986+0000 7f861dffb640 0 client.15519 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.986+0000 7f863f007740 1 -- 172.21.15.76:0/4118282700 >> 172.21.15.76:0/4118282700 conn(0x55a7adba8a50 msgr2=0x55a7adbaa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.986+0000 7f863f007740 1 -- 172.21.15.76:0/4118282700 shutdown_connections 2024-03-18T14:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.986+0000 7f863f007740 1 -- 172.21.15.76:0/4118282700 wait complete. 2024-03-18T14:53:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:32.986+0000 7f863f007740 20 client.15519 trim_cache size 0 max 16384 2024-03-18T14:53:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863f007740 1 Processor -- start 2024-03-18T14:53:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863f007740 1 -- start start 2024-03-18T14:53:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863f007740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbae130 0x55a7adbb6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863f007740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbaea50 0x55a7adbc31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863f007740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbcf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863f007740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f86341afc10 con 0x55a7adbb7820 2024-03-18T14:53:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863f007740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f86341ad4e0 con 0x55a7adbaea50 2024-03-18T14:53:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863f007740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f86341ab330 con 0x55a7adbae130 2024-03-18T14:53:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863a9b6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbcf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863a1b5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbae130 0x55a7adbb6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863a9b6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbcf760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45420/0 (socket says 172.21.15.76:45420) 2024-03-18T14:53:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863a9b6640 1 -- 172.21.15.76:0/2912991784 learned_addr learned my addr 172.21.15.76:0/2912991784 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f86399b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbaea50 0x55a7adbc31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863a9b6640 1 -- 172.21.15.76:0/2912991784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbae130 msgr2=0x55a7adbb6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863a9b6640 1 --2- 172.21.15.76:0/2912991784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbae130 0x55a7adbb6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863a9b6640 1 -- 172.21.15.76:0/2912991784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbaea50 msgr2=0x55a7adbc31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863a9b6640 1 --2- 172.21.15.76:0/2912991784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbaea50 0x55a7adbc31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863a9b6640 1 -- 172.21.15.76:0/2912991784 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7adb9a140 con 0x55a7adbb7820 2024-03-18T14:53:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f86399b4640 1 --2- 172.21.15.76:0/2912991784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbaea50 0x55a7adbc31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863a1b5640 1 --2- 172.21.15.76:0/2912991784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbae130 0x55a7adbb6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.016+0000 7f863a9b6640 1 --2- 172.21.15.76:0/2912991784 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbcf760 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f862c009cd0 tx=0x7f862c0324e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d09075ebb61cfeff server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.017+0000 7f861ffff640 1 -- 172.21.15.76:0/2912991784 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c053040 con 0x55a7adbb7820 2024-03-18T14:53:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.017+0000 7f861ffff640 1 -- 172.21.15.76:0/2912991784 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f862c0098e0 con 0x55a7adbb7820 2024-03-18T14:53:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.017+0000 7f861ffff640 1 -- 172.21.15.76:0/2912991784 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c039040 con 0x55a7adbb7820 2024-03-18T14:53:33.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.017+0000 7f863f007740 1 -- 172.21.15.76:0/2912991784 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 msgr2=0x55a7adbcf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:33.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.017+0000 7f863f007740 1 --2- 172.21.15.76:0/2912991784 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbcf760 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f862c009cd0 tx=0x7f862c0324e0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863f007740 1 -- 172.21.15.76:0/2912991784 shutdown_connections 2024-03-18T14:53:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863f007740 1 --2- 172.21.15.76:0/2912991784 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbb7820 0x55a7adbcf760 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863f007740 1 --2- 172.21.15.76:0/2912991784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbaea50 0x55a7adbc31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863f007740 1 --2- 172.21.15.76:0/2912991784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbae130 0x55a7adbb6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863f007740 1 -- 172.21.15.76:0/2912991784 >> 172.21.15.76:0/2912991784 conn(0x55a7adba8a50 msgr2=0x55a7adba7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863f007740 1 -- 172.21.15.76:0/2912991784 shutdown_connections 2024-03-18T14:53:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863f007740 1 -- 172.21.15.76:0/2912991784 wait complete. 2024-03-18T14:53:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863f007740 1 -- start start 2024-03-18T14:53:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863f007740 1 Processor -- start 2024-03-18T14:53:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863f007740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbae130 0x55a7adbbf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863f007740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbaea50 0x55a7adbbf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863f007740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbb7820 0x55a7adbbb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863f007740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f86341ab9a0 con 0x55a7adbae130 2024-03-18T14:53:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863f007740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f86341ace10 con 0x55a7adbaea50 2024-03-18T14:53:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863f007740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f86341ac070 con 0x55a7adbb7820 2024-03-18T14:53:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863f007740 10 client.0 _ll_register_callbacks cb 0x7f8624005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863a1b5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbae130 0x55a7adbbf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863a1b5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbae130 0x55a7adbbf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45432/0 (socket says 172.21.15.76:45432) 2024-03-18T14:53:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f86399b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbaea50 0x55a7adbbf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863a1b5640 1 -- 172.21.15.76:0/3078505372 learned_addr learned my addr 172.21.15.76:0/3078505372 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.018+0000 7f863a9b6640 1 --2- 172.21.15.76:0/3078505372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbb7820 0x55a7adbbb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.019+0000 7f863a1b5640 1 -- 172.21.15.76:0/3078505372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbb7820 msgr2=0x55a7adbbb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.019+0000 7f863a1b5640 1 --2- 172.21.15.76:0/3078505372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbb7820 0x55a7adbbb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.019+0000 7f863a1b5640 1 -- 172.21.15.76:0/3078505372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbaea50 msgr2=0x55a7adbbf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.019+0000 7f863a1b5640 1 --2- 172.21.15.76:0/3078505372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbaea50 0x55a7adbbf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.019+0000 7f863a1b5640 1 -- 172.21.15.76:0/3078505372 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7adbc3730 con 0x55a7adbae130 2024-03-18T14:53:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.019+0000 7f863a1b5640 1 --2- 172.21.15.76:0/3078505372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbae130 0x55a7adbbf3a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f863009b4b0 tx=0x7f8630086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d46b23ea3c299e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.019+0000 7f861dffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:53:33.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.019+0000 7f861dffb640 1 -- 172.21.15.76:0/3078505372 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863009b7b0 con 0x55a7adbae130 2024-03-18T14:53:33.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.019+0000 7f861dffb640 1 -- 172.21.15.76:0/3078505372 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f8630097e20 con 0x55a7adbae130 2024-03-18T14:53:33.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.019+0000 7f863f007740 1 -- 172.21.15.76:0/3078505372 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f862c0098e0 con 0x55a7adbae130 2024-03-18T14:53:33.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.019+0000 7f861dffb640 1 -- 172.21.15.76:0/3078505372 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f86300ccb50 con 0x55a7adbae130 2024-03-18T14:53:33.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.019+0000 7f863f007740 10 client.15525 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:33.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.019+0000 7f863f007740 1 -- 172.21.15.76:0/3078505372 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a7adc2e6b0 con 0x55a7adbae130 2024-03-18T14:53:33.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.019+0000 7f861dffb640 1 -- 172.21.15.76:0/3078505372 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f862c0098e0 con 0x55a7adbae130 2024-03-18T14:53:33.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.019+0000 7f861dffb640 1 -- 172.21.15.76:0/3078505372 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f863009ac00 con 0x55a7adbae130 2024-03-18T14:53:33.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.020+0000 7f863f007740 10 client.15525 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:33.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.020+0000 7f863f007740 10 client.15525 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:33.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.020+0000 7f863f007740 20 client.15525 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:33.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.020+0000 7f863f007740 1 --2- 172.21.15.76:0/3078505372 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a7adc2ed80 0x55a7adc311a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:33.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.020+0000 7f863f007740 4 client.15525 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:53:33.027 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.020+0000 7f863f007740 1 -- 172.21.15.76:0/3078505372 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a7adb1a2d0 con 0x55a7adc2ed80 2024-03-18T14:53:33.027 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.020+0000 7f86399b4640 1 --2- 172.21.15.76:0/3078505372 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a7adc2ed80 0x55a7adc311a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:33.027 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.020+0000 7f86399b4640 1 --2- 172.21.15.76:0/3078505372 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a7adc2ed80 0x55a7adc311a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91a83f9ab330d31c server_cookie=431fcc052c98d3e5 in_seq=0 out_seq=0 2024-03-18T14:53:33.027 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.020+0000 7f861dffb640 10 client.15525 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.030+0000 7f861dffb640 1 -- 172.21.15.76:0/3078505372 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a7adb1a2d0 con 0x55a7adc2ed80 2024-03-18T14:53:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.030+0000 7f861dffb640 10 client.15525 handle_command_reply: tid=0 2024-03-18T14:53:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.030+0000 7f863f007740 1 client.15525 shutdown 2024-03-18T14:53:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.030+0000 7f863f007740 1 -- 172.21.15.76:0/3078505372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbae130 msgr2=0x55a7adbbf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.030+0000 7f863f007740 1 --2- 172.21.15.76:0/3078505372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbae130 0x55a7adbbf3a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f863009b4b0 tx=0x7f8630086040 comp rx=0 tx=0).stop 2024-03-18T14:53:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.030+0000 7f863f007740 1 -- 172.21.15.76:0/3078505372 shutdown_connections 2024-03-18T14:53:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.030+0000 7f863f007740 1 --2- 172.21.15.76:0/3078505372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a7adbb7820 0x55a7adbbb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.030+0000 7f863f007740 1 --2- 172.21.15.76:0/3078505372 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a7adc2ed80 0x55a7adc311a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.030+0000 7f863f007740 1 --2- 172.21.15.76:0/3078505372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7adbaea50 0x55a7adbbf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.030+0000 7f863f007740 1 --2- 172.21.15.76:0/3078505372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a7adbae130 0x55a7adbbf3a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.030+0000 7f861dffb640 0 client.15525 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.030+0000 7f863f007740 1 -- 172.21.15.76:0/3078505372 >> 172.21.15.76:0/3078505372 conn(0x55a7adba8a50 msgr2=0x55a7adba9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.030+0000 7f863f007740 1 -- 172.21.15.76:0/3078505372 shutdown_connections 2024-03-18T14:53:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.030+0000 7f863f007740 1 -- 172.21.15.76:0/3078505372 wait complete. 2024-03-18T14:53:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.030+0000 7f863f007740 20 client.15525 trim_cache size 0 max 16384 2024-03-18T14:53:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:53:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:53:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:53:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:53:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.031+0000 7f863c419640 1 -- 172.21.15.76:0/972720605 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f86200a2b30 msgr2=0x7f86200a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.031+0000 7f863c419640 1 --2- 172.21.15.76:0/972720605 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f86200a2b30 0x7f86200a4ff0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f862c005cc0 tx=0x7f862c02c0f0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.031+0000 7f863c419640 1 -- 172.21.15.76:0/972720605 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8634101780 msgr2=0x7f8634197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.031+0000 7f863c419640 1 --2- 172.21.15.76:0/972720605 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8634101780 0x7f8634197f80 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f863000e9f0 tx=0x7f863000eeb0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.031+0000 7f863c419640 1 -- 172.21.15.76:0/972720605 shutdown_connections 2024-03-18T14:53:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.031+0000 7f863c419640 1 --2- 172.21.15.76:0/972720605 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f86200a2b30 0x7f86200a4ff0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.031+0000 7f863c419640 1 --2- 172.21.15.76:0/972720605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8634102140 0x7f86341984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.031+0000 7f863c419640 1 --2- 172.21.15.76:0/972720605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8634107790 0x7f863419c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.031+0000 7f863c419640 1 --2- 172.21.15.76:0/972720605 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8634101780 0x7f8634197f80 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.031+0000 7f863c419640 1 -- 172.21.15.76:0/972720605 >> 172.21.15.76:0/972720605 conn(0x7f86340fd270 msgr2=0x7f86340fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.031+0000 7f863c419640 1 -- 172.21.15.76:0/972720605 shutdown_connections 2024-03-18T14:53:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:33.031+0000 7f863c419640 1 -- 172.21.15.76:0/972720605 wait complete. 2024-03-18T14:53:34.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:34 smithi076 ceph-mon[32205]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-03-18T14:53:34.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:34 smithi174 ceph-mon[36787]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-03-18T14:53:34.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:34 smithi077 ceph-mon[35865]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-03-18T14:53:36.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:36 smithi076 ceph-mon[32205]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 78 KiB/s wr, 44 op/s 2024-03-18T14:53:36.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:36 smithi174 ceph-mon[36787]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 78 KiB/s wr, 44 op/s 2024-03-18T14:53:36.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:36 smithi077 ceph-mon[35865]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 78 KiB/s wr, 44 op/s 2024-03-18T14:53:38.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:38 smithi076 ceph-mon[32205]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-18T14:53:38.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:38 smithi174 ceph-mon[36787]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-18T14:53:38.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:38 smithi077 ceph-mon[35865]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-18T14:53:39.555 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T14:53:39.555 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T14:53:39.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.664+0000 7fd071073640 1 -- 172.21.15.76:0/1356726687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c0689e0 msgr2=0x7fd06c068e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:39.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.664+0000 7fd071073640 1 --2- 172.21.15.76:0/1356726687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c0689e0 0x7fd06c068e60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd060009930 tx=0x7fd06002bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:39.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.664+0000 7fd071073640 1 -- 172.21.15.76:0/1356726687 shutdown_connections 2024-03-18T14:53:39.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.664+0000 7fd071073640 1 --2- 172.21.15.76:0/1356726687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd06c0693a0 0x7fd06c10e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.664+0000 7fd071073640 1 --2- 172.21.15.76:0/1356726687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c0689e0 0x7fd06c068e60 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.664+0000 7fd071073640 1 --2- 172.21.15.76:0/1356726687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd06c107830 0x7fd06c107c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.664+0000 7fd071073640 1 -- 172.21.15.76:0/1356726687 >> 172.21.15.76:0/1356726687 conn(0x7fd06c0777e0 msgr2=0x7fd06c077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:39.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.664+0000 7fd071073640 1 -- 172.21.15.76:0/1356726687 shutdown_connections 2024-03-18T14:53:39.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.664+0000 7fd071073640 1 -- 172.21.15.76:0/1356726687 wait complete. 2024-03-18T14:53:39.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.665+0000 7fd071073640 1 Processor -- start 2024-03-18T14:53:39.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.665+0000 7fd071073640 1 -- start start 2024-03-18T14:53:39.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd071073640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c0689e0 0x7fd06c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd071073640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd06c0693a0 0x7fd06c1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd071073640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd06c107830 0x7fd06c19c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd071073640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c111b40 con 0x7fd06c0689e0 2024-03-18T14:53:39.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd071073640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c1119c0 con 0x7fd06c0693a0 2024-03-18T14:53:39.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd071073640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c111cc0 con 0x7fd06c107830 2024-03-18T14:53:39.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd06ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c0689e0 0x7fd06c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd06a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd06c0693a0 0x7fd06c1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd06ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c0689e0 0x7fd06c198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39198/0 (socket says 172.21.15.76:39198) 2024-03-18T14:53:39.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd06a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd06c0693a0 0x7fd06c1985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41004/0 (socket says 172.21.15.76:41004) 2024-03-18T14:53:39.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd06ad9d640 1 -- 172.21.15.76:0/3662811853 learned_addr learned my addr 172.21.15.76:0/3662811853 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:39.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd06b59e640 1 --2- 172.21.15.76:0/3662811853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd06c107830 0x7fd06c19c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd06a59c640 1 -- 172.21.15.76:0/3662811853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd06c107830 msgr2=0x7fd06c19c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:39.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd06a59c640 1 --2- 172.21.15.76:0/3662811853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd06c107830 0x7fd06c19c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd06a59c640 1 -- 172.21.15.76:0/3662811853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c0689e0 msgr2=0x7fd06c198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:39.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd06a59c640 1 --2- 172.21.15.76:0/3662811853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c0689e0 0x7fd06c198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd06a59c640 1 -- 172.21.15.76:0/3662811853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd06c19d070 con 0x7fd06c0693a0 2024-03-18T14:53:39.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd06a59c640 1 --2- 172.21.15.76:0/3662811853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd06c0693a0 0x7fd06c1985c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd060009230 tx=0x7fd060004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:39.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.666+0000 7fd053fff640 1 -- 172.21.15.76:0/3662811853 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd060041070 con 0x7fd06c0693a0 2024-03-18T14:53:39.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.667+0000 7fd071073640 1 -- 172.21.15.76:0/3662811853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd06c19d300 con 0x7fd06c0693a0 2024-03-18T14:53:39.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.667+0000 7fd053fff640 1 -- 172.21.15.76:0/3662811853 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0600054e0 con 0x7fd06c0693a0 2024-03-18T14:53:39.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.667+0000 7fd053fff640 1 -- 172.21.15.76:0/3662811853 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd060003bf0 con 0x7fd06c0693a0 2024-03-18T14:53:39.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.667+0000 7fd071073640 1 -- 172.21.15.76:0/3662811853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd06c1a4bf0 con 0x7fd06c0693a0 2024-03-18T14:53:39.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.669+0000 7fd053fff640 1 -- 172.21.15.76:0/3662811853 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 23) ==== 137466+0+0 (secure 0 0 0) 0x7fd060044070 con 0x7fd06c0693a0 2024-03-18T14:53:39.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.670+0000 7fd053fff640 1 --2- 172.21.15.76:0/3662811853 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fd0540a2a80 0x7fd0540a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.670+0000 7fd06ad9d640 1 --2- 172.21.15.76:0/3662811853 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fd0540a2a80 0x7fd0540a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.670+0000 7fd053fff640 1 -- 172.21.15.76:0/3662811853 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fd0600f08c0 con 0x7fd06c0693a0 2024-03-18T14:53:39.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.670+0000 7fd073c5d740 1 Processor -- start 2024-03-18T14:53:39.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.670+0000 7fd073c5d740 1 -- start start 2024-03-18T14:53:39.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.670+0000 7fd073c5d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb46150 0x5612eeb46530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.670+0000 7fd073c5d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46a70 0x5612eeb4f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.670+0000 7fd073c5d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb4f650 0x5612eeb51a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.670+0000 7fd073c5d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5612eeaaa500 con 0x5612eeb4f650 2024-03-18T14:53:39.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.670+0000 7fd073c5d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5612eea52be0 con 0x5612eeb46150 2024-03-18T14:53:39.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.670+0000 7fd073c5d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5612eeaa65a0 con 0x5612eeb46a70 2024-03-18T14:53:39.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.671+0000 7fd06ad9d640 1 --2- 172.21.15.76:0/3662811853 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fd0540a2a80 0x7fd0540a4f40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd05c009c90 tx=0x7fd05c005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:39.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.671+0000 7fd06b59e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb4f650 0x5612eeb51a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.671+0000 7fd06a59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46a70 0x5612eeb4f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.671+0000 7fd06a59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46a70 0x5612eeb4f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:48962/0 (socket says 172.21.15.76:48962) 2024-03-18T14:53:39.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.671+0000 7fd06a59c640 1 -- 172.21.15.76:0/396337905 learned_addr learned my addr 172.21.15.76:0/396337905 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:39.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.671+0000 7fd06ad9d640 1 --2- 172.21.15.76:0/396337905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb46150 0x5612eeb46530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.671+0000 7fd06b59e640 1 -- 172.21.15.76:0/396337905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46a70 msgr2=0x5612eeb4f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:39.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.671+0000 7fd06b59e640 1 --2- 172.21.15.76:0/396337905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46a70 0x5612eeb4f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.671+0000 7fd06b59e640 1 -- 172.21.15.76:0/396337905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb46150 msgr2=0x5612eeb46530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:39.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.671+0000 7fd06b59e640 1 --2- 172.21.15.76:0/396337905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb46150 0x5612eeb46530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.671+0000 7fd06b59e640 1 -- 172.21.15.76:0/396337905 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612eea664b0 con 0x5612eeb4f650 2024-03-18T14:53:39.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.671+0000 7fd06a59c640 1 --2- 172.21.15.76:0/396337905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46a70 0x5612eeb4f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:39.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.671+0000 7fd06b59e640 1 --2- 172.21.15.76:0/396337905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb4f650 0x5612eeb51a40 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd064009f00 tx=0x7fd064011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91828c415d84703c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:39.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.671+0000 7fd0517fa640 1 -- 172.21.15.76:0/396337905 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd064004d20 con 0x5612eeb4f650 2024-03-18T14:53:39.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.671+0000 7fd0517fa640 1 -- 172.21.15.76:0/396337905 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd064004030 con 0x5612eeb4f650 2024-03-18T14:53:39.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.671+0000 7fd0517fa640 1 -- 172.21.15.76:0/396337905 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd064005280 con 0x5612eeb4f650 2024-03-18T14:53:39.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.672+0000 7fd073c5d740 1 -- 172.21.15.76:0/396337905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb4f650 msgr2=0x5612eeb51a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:39.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.672+0000 7fd073c5d740 1 --2- 172.21.15.76:0/396337905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb4f650 0x5612eeb51a40 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd064009f00 tx=0x7fd064011810 comp rx=0 tx=0).stop 2024-03-18T14:53:39.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.672+0000 7fd073c5d740 1 -- 172.21.15.76:0/396337905 shutdown_connections 2024-03-18T14:53:39.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.672+0000 7fd073c5d740 1 --2- 172.21.15.76:0/396337905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46a70 0x5612eeb4f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.672+0000 7fd073c5d740 1 --2- 172.21.15.76:0/396337905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb4f650 0x5612eeb51a40 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.672+0000 7fd073c5d740 1 --2- 172.21.15.76:0/396337905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb46150 0x5612eeb46530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.672+0000 7fd073c5d740 1 -- 172.21.15.76:0/396337905 >> 172.21.15.76:0/396337905 conn(0x5612eeb40880 msgr2=0x5612eeb40c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:39.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.673+0000 7fd073c5d740 1 -- 172.21.15.76:0/396337905 shutdown_connections 2024-03-18T14:53:39.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.673+0000 7fd073c5d740 1 -- 172.21.15.76:0/396337905 wait complete. 2024-03-18T14:53:39.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.673+0000 7fd073c5d740 1 -- start start 2024-03-18T14:53:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.673+0000 7fd073c5d740 1 Processor -- start 2024-03-18T14:53:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.673+0000 7fd073c5d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46150 0x5612eeb4c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.673+0000 7fd073c5d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb46a70 0x5612eeb4cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.673+0000 7fd073c5d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb4f650 0x5612eeb4d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.673+0000 7fd073c5d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5612eeb52440 con 0x5612eeb46a70 2024-03-18T14:53:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd073c5d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5612eea66160 con 0x5612eeb4f650 2024-03-18T14:53:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd073c5d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5612eeb525c0 con 0x5612eeb46150 2024-03-18T14:53:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd06a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb46a70 0x5612eeb4cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd06ad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46150 0x5612eeb4c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd06ad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46150 0x5612eeb4c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:48970/0 (socket says 172.21.15.76:48970) 2024-03-18T14:53:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd06ad9d640 1 -- 172.21.15.76:0/3971934751 learned_addr learned my addr 172.21.15.76:0/3971934751 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd06b59e640 1 --2- 172.21.15.76:0/3971934751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb4f650 0x5612eeb4d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd073c5d740 10 client.0 _ll_register_callbacks cb 0x5612eea4a150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd06b59e640 1 -- 172.21.15.76:0/3971934751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46150 msgr2=0x5612eeb4c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd06b59e640 1 --2- 172.21.15.76:0/3971934751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46150 0x5612eeb4c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd06b59e640 1 -- 172.21.15.76:0/3971934751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb46a70 msgr2=0x5612eeb4cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd06b59e640 1 --2- 172.21.15.76:0/3971934751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb46a70 0x5612eeb4cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd06b59e640 1 -- 172.21.15.76:0/3971934751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612ee8887d0 con 0x5612eeb4f650 2024-03-18T14:53:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd06ad9d640 1 --2- 172.21.15.76:0/3971934751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46150 0x5612eeb4c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd06a59c640 1 --2- 172.21.15.76:0/3971934751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb46a70 0x5612eeb4cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd06b59e640 1 --2- 172.21.15.76:0/3971934751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb4f650 0x5612eeb4d0d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd064009270 tx=0x7fd064004ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40f6ae9d82aa5b43 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd04eff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:53:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd04eff5640 1 -- 172.21.15.76:0/3971934751 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd064027020 con 0x5612eeb4f650 2024-03-18T14:53:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd04eff5640 1 -- 172.21.15.76:0/3971934751 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fd064003e00 con 0x5612eeb4f650 2024-03-18T14:53:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.674+0000 7fd073c5d740 1 -- 172.21.15.76:0/3971934751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd064004030 con 0x5612eeb4f650 2024-03-18T14:53:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.675+0000 7fd04eff5640 1 -- 172.21.15.76:0/3971934751 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd064032050 con 0x5612eeb4f650 2024-03-18T14:53:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.675+0000 7fd073c5d740 10 client.25397 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.675+0000 7fd073c5d740 1 -- 172.21.15.76:0/3971934751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5612ee7be3f0 con 0x5612eeb4f650 2024-03-18T14:53:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.675+0000 7fd04eff5640 1 -- 172.21.15.76:0/3971934751 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0640046c0 con 0x5612eeb4f650 2024-03-18T14:53:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.675+0000 7fd04eff5640 1 -- 172.21.15.76:0/3971934751 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fd064018070 con 0x5612eeb4f650 2024-03-18T14:53:39.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.675+0000 7fd073c5d740 10 client.25397 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:39.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.675+0000 7fd073c5d740 10 client.25397 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:39.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.675+0000 7fd073c5d740 20 client.25397 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:39.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.675+0000 7fd073c5d740 1 --2- 172.21.15.76:0/3971934751 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5612eeb5f560 0x5612eeb61950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.675+0000 7fd073c5d740 4 client.25397 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:53:39.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.675+0000 7fd073c5d740 1 -- 172.21.15.76:0/3971934751 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd064004030 con 0x5612eeb5f560 2024-03-18T14:53:39.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.675+0000 7fd06ad9d640 1 --2- 172.21.15.76:0/3971934751 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5612eeb5f560 0x5612eeb61950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.676+0000 7fd04eff5640 10 client.25397 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:39.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.676+0000 7fd06ad9d640 1 --2- 172.21.15.76:0/3971934751 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5612eeb5f560 0x5612eeb61950 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2728c942b0c7e030 server_cookie=eec1d107b20109b5 in_seq=0 out_seq=0 2024-03-18T14:53:39.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.676+0000 7fd04eff5640 1 -- 172.21.15.76:0/3971934751 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fd064004030 con 0x5612eeb5f560 2024-03-18T14:53:39.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.676+0000 7fd04eff5640 10 client.25397 handle_command_reply: tid=0 2024-03-18T14:53:39.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.676+0000 7fd073c5d740 1 client.25397 shutdown 2024-03-18T14:53:39.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.677+0000 7fd073c5d740 1 -- 172.21.15.76:0/3971934751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb4f650 msgr2=0x5612eeb4d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:39.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.677+0000 7fd073c5d740 1 --2- 172.21.15.76:0/3971934751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb4f650 0x5612eeb4d0d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd064009270 tx=0x7fd064004ef0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.677+0000 7fd073c5d740 1 -- 172.21.15.76:0/3971934751 shutdown_connections 2024-03-18T14:53:39.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.677+0000 7fd073c5d740 1 --2- 172.21.15.76:0/3971934751 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5612eeb5f560 0x5612eeb61950 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.677+0000 7fd073c5d740 1 --2- 172.21.15.76:0/3971934751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb4f650 0x5612eeb4d0d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.677+0000 7fd073c5d740 1 --2- 172.21.15.76:0/3971934751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb46a70 0x5612eeb4cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.677+0000 7fd04eff5640 0 client.25397 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:39.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.677+0000 7fd073c5d740 1 --2- 172.21.15.76:0/3971934751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46150 0x5612eeb4c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.677+0000 7fd073c5d740 1 -- 172.21.15.76:0/3971934751 >> 172.21.15.76:0/3971934751 conn(0x5612eeb40880 msgr2=0x5612eeb425d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:39.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.677+0000 7fd073c5d740 1 -- 172.21.15.76:0/3971934751 shutdown_connections 2024-03-18T14:53:39.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.677+0000 7fd073c5d740 1 -- 172.21.15.76:0/3971934751 wait complete. 2024-03-18T14:53:39.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.677+0000 7fd073c5d740 20 client.25397 trim_cache size 0 max 16384 2024-03-18T14:53:39.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.694+0000 7fd073c5d740 1 Processor -- start 2024-03-18T14:53:39.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.694+0000 7fd073c5d740 1 -- start start 2024-03-18T14:53:39.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.694+0000 7fd073c5d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb46150 0x5612eeb4ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.694+0000 7fd073c5d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb46a70 0x5612eeb5b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.694+0000 7fd073c5d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb4f650 0x5612eeb4b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.695+0000 7fd073c5d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c1b1e40 con 0x5612eeb46a70 2024-03-18T14:53:39.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.695+0000 7fd073c5d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c1af710 con 0x5612eeb46150 2024-03-18T14:53:39.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.695+0000 7fd073c5d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c1ad580 con 0x5612eeb4f650 2024-03-18T14:53:39.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.695+0000 7fd06a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb46a70 0x5612eeb5b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.695+0000 7fd06b59e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb4f650 0x5612eeb4b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.695+0000 7fd06a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb46a70 0x5612eeb5b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39230/0 (socket says 172.21.15.76:39230) 2024-03-18T14:53:39.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.695+0000 7fd06a59c640 1 -- 172.21.15.76:0/2147887616 learned_addr learned my addr 172.21.15.76:0/2147887616 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:39.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.695+0000 7fd06ad9d640 1 --2- 172.21.15.76:0/2147887616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb46150 0x5612eeb4ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.695+0000 7fd06ad9d640 1 -- 172.21.15.76:0/2147887616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb4f650 msgr2=0x5612eeb4b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:39.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.695+0000 7fd06ad9d640 1 --2- 172.21.15.76:0/2147887616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb4f650 0x5612eeb4b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.695+0000 7fd06ad9d640 1 -- 172.21.15.76:0/2147887616 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb46a70 msgr2=0x5612eeb5b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:39.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.695+0000 7fd06ad9d640 1 --2- 172.21.15.76:0/2147887616 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb46a70 0x5612eeb5b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.695+0000 7fd06ad9d640 1 -- 172.21.15.76:0/2147887616 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612ee77e0f0 con 0x5612eeb46150 2024-03-18T14:53:39.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.695+0000 7fd06b59e640 1 --2- 172.21.15.76:0/2147887616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb4f650 0x5612eeb4b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:39.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.695+0000 7fd06a59c640 1 --2- 172.21.15.76:0/2147887616 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb46a70 0x5612eeb5b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:39.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.695+0000 7fd06ad9d640 1 --2- 172.21.15.76:0/2147887616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb46150 0x5612eeb4ed80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x5612eeb526a0 tx=0x7fd05c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7568bc9788c9caf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:39.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.695+0000 7fd050ff9640 1 -- 172.21.15.76:0/2147887616 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c00eb40 con 0x5612eeb46150 2024-03-18T14:53:39.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.696+0000 7fd050ff9640 1 -- 172.21.15.76:0/2147887616 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd05c00ece0 con 0x5612eeb46150 2024-03-18T14:53:39.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.696+0000 7fd050ff9640 1 -- 172.21.15.76:0/2147887616 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c029810 con 0x5612eeb46150 2024-03-18T14:53:39.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.696+0000 7fd073c5d740 1 -- 172.21.15.76:0/2147887616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb46150 msgr2=0x5612eeb4ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:39.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.696+0000 7fd073c5d740 1 --2- 172.21.15.76:0/2147887616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb46150 0x5612eeb4ed80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x5612eeb526a0 tx=0x7fd05c014000 comp rx=0 tx=0).stop 2024-03-18T14:53:39.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.696+0000 7fd073c5d740 1 -- 172.21.15.76:0/2147887616 shutdown_connections 2024-03-18T14:53:39.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.696+0000 7fd073c5d740 1 --2- 172.21.15.76:0/2147887616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb4f650 0x5612eeb4b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.696+0000 7fd073c5d740 1 --2- 172.21.15.76:0/2147887616 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb46a70 0x5612eeb5b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.696+0000 7fd073c5d740 1 --2- 172.21.15.76:0/2147887616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb46150 0x5612eeb4ed80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.696+0000 7fd073c5d740 1 -- 172.21.15.76:0/2147887616 >> 172.21.15.76:0/2147887616 conn(0x5612eeb40880 msgr2=0x5612eeb3f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:39.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.696+0000 7fd073c5d740 1 -- 172.21.15.76:0/2147887616 shutdown_connections 2024-03-18T14:53:39.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.696+0000 7fd073c5d740 1 -- 172.21.15.76:0/2147887616 wait complete. 2024-03-18T14:53:39.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd073c5d740 1 -- start start 2024-03-18T14:53:39.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd073c5d740 1 Processor -- start 2024-03-18T14:53:39.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd073c5d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46150 0x5612eeb57a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd073c5d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb46a70 0x5612eeb52e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd073c5d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb4f650 0x5612eeb53390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd073c5d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c1adbd0 con 0x5612eeb4f650 2024-03-18T14:53:39.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd073c5d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c1af040 con 0x5612eeb46a70 2024-03-18T14:53:39.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd073c5d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c1ae2a0 con 0x5612eeb46150 2024-03-18T14:53:39.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd073c5d740 10 client.0 _ll_register_callbacks cb 0x7fd05c0099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:39.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd06b59e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb4f650 0x5612eeb53390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd06ad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46150 0x5612eeb57a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd06b59e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb4f650 0x5612eeb53390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39238/0 (socket says 172.21.15.76:39238) 2024-03-18T14:53:39.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd06ad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46150 0x5612eeb57a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:48984/0 (socket says 172.21.15.76:48984) 2024-03-18T14:53:39.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd06b59e640 1 -- 172.21.15.76:0/2406164771 learned_addr learned my addr 172.21.15.76:0/2406164771 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:39.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd06a59c640 1 --2- 172.21.15.76:0/2406164771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb46a70 0x5612eeb52e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd06b59e640 1 -- 172.21.15.76:0/2406164771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46150 msgr2=0x5612eeb57a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:39.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd06b59e640 1 --2- 172.21.15.76:0/2406164771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46150 0x5612eeb57a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd06b59e640 1 -- 172.21.15.76:0/2406164771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb46a70 msgr2=0x5612eeb52e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:39.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd06b59e640 1 --2- 172.21.15.76:0/2406164771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb46a70 0x5612eeb52e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.697+0000 7fd06b59e640 1 -- 172.21.15.76:0/2406164771 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612eea719d0 con 0x5612eeb4f650 2024-03-18T14:53:39.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.698+0000 7fd06ad9d640 1 --2- 172.21.15.76:0/2406164771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46150 0x5612eeb57a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:39.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.698+0000 7fd06b59e640 1 --2- 172.21.15.76:0/2406164771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb4f650 0x5612eeb53390 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd06402e540 tx=0x7fd06402e7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d26a5db940c99af server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:39.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.698+0000 7fd04eff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:53:39.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.698+0000 7fd04eff5640 1 -- 172.21.15.76:0/2406164771 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd064009480 con 0x5612eeb4f650 2024-03-18T14:53:39.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.698+0000 7fd073c5d740 1 -- 172.21.15.76:0/2406164771 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd05c00ece0 con 0x5612eeb4f650 2024-03-18T14:53:39.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.698+0000 7fd04eff5640 1 -- 172.21.15.76:0/2406164771 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fd06402e0d0 con 0x5612eeb4f650 2024-03-18T14:53:39.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.698+0000 7fd04eff5640 1 -- 172.21.15.76:0/2406164771 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd06403d050 con 0x5612eeb4f650 2024-03-18T14:53:39.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.698+0000 7fd073c5d740 10 client.15543 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:39.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.698+0000 7fd073c5d740 1 -- 172.21.15.76:0/2406164771 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5612eebc54c0 con 0x5612eeb4f650 2024-03-18T14:53:39.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.698+0000 7fd04eff5640 1 -- 172.21.15.76:0/2406164771 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd05c00ece0 con 0x5612eeb4f650 2024-03-18T14:53:39.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.698+0000 7fd04eff5640 1 -- 172.21.15.76:0/2406164771 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fd064018040 con 0x5612eeb4f650 2024-03-18T14:53:39.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.698+0000 7fd073c5d740 10 client.15543 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:39.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.698+0000 7fd073c5d740 10 client.15543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:39.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.698+0000 7fd073c5d740 20 client.15543 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:39.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.698+0000 7fd073c5d740 1 --2- 172.21.15.76:0/2406164771 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5612eebc5bb0 0x5612eebc7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.699+0000 7fd073c5d740 4 client.15543 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T14:53:39.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.699+0000 7fd073c5d740 1 -- 172.21.15.76:0/2406164771 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5612eeab7a40 con 0x5612eebc5bb0 2024-03-18T14:53:39.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.699+0000 7fd06ad9d640 1 --2- 172.21.15.76:0/2406164771 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5612eebc5bb0 0x5612eebc7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.699+0000 7fd04eff5640 10 client.15543 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:39.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.699+0000 7fd06ad9d640 1 --2- 172.21.15.76:0/2406164771 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5612eebc5bb0 0x5612eebc7fd0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d8f3ccdd78ac03 server_cookie=9f71e84d60cf3dff in_seq=0 out_seq=0 2024-03-18T14:53:39.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.699+0000 7fd04eff5640 1 -- 172.21.15.76:0/2406164771 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5612eeab7a40 con 0x5612eebc5bb0 2024-03-18T14:53:39.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.699+0000 7fd04eff5640 10 client.15543 handle_command_reply: tid=0 2024-03-18T14:53:39.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.699+0000 7fd073c5d740 1 client.15543 shutdown 2024-03-18T14:53:39.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.699+0000 7fd073c5d740 1 -- 172.21.15.76:0/2406164771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb4f650 msgr2=0x5612eeb53390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:39.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.699+0000 7fd073c5d740 1 --2- 172.21.15.76:0/2406164771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb4f650 0x5612eeb53390 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd06402e540 tx=0x7fd06402e7d0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd073c5d740 1 -- 172.21.15.76:0/2406164771 shutdown_connections 2024-03-18T14:53:39.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd073c5d740 1 --2- 172.21.15.76:0/2406164771 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5612eebc5bb0 0x5612eebc7fd0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd04eff5640 0 client.15543 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:39.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd073c5d740 1 --2- 172.21.15.76:0/2406164771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612eeb4f650 0x5612eeb53390 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd073c5d740 1 --2- 172.21.15.76:0/2406164771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612eeb46a70 0x5612eeb52e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd073c5d740 1 --2- 172.21.15.76:0/2406164771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612eeb46150 0x5612eeb57a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd073c5d740 1 -- 172.21.15.76:0/2406164771 >> 172.21.15.76:0/2406164771 conn(0x5612eeb40880 msgr2=0x5612eeb41c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:39.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd073c5d740 1 -- 172.21.15.76:0/2406164771 shutdown_connections 2024-03-18T14:53:39.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd073c5d740 1 -- 172.21.15.76:0/2406164771 wait complete. 2024-03-18T14:53:39.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd073c5d740 20 client.15543 trim_cache size 0 max 16384 2024-03-18T14:53:39.706 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T14:53:39.706 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-18T14:53:39.706 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-18T14:53:39.706 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T14:53:39.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd071073640 1 -- 172.21.15.76:0/3662811853 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fd0540a2a80 msgr2=0x7fd0540a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:39.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd071073640 1 --2- 172.21.15.76:0/3662811853 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fd0540a2a80 0x7fd0540a4f40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd05c009c90 tx=0x7fd05c005f90 comp rx=0 tx=0).stop 2024-03-18T14:53:39.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd071073640 1 -- 172.21.15.76:0/3662811853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd06c0693a0 msgr2=0x7fd06c1985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:39.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd071073640 1 --2- 172.21.15.76:0/3662811853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd06c0693a0 0x7fd06c1985c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd060009230 tx=0x7fd060004270 comp rx=0 tx=0).stop 2024-03-18T14:53:39.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd071073640 1 -- 172.21.15.76:0/3662811853 shutdown_connections 2024-03-18T14:53:39.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd071073640 1 --2- 172.21.15.76:0/3662811853 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fd0540a2a80 0x7fd0540a4f40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd071073640 1 --2- 172.21.15.76:0/3662811853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd06c107830 0x7fd06c19c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd071073640 1 --2- 172.21.15.76:0/3662811853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd06c0693a0 0x7fd06c1985c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd071073640 1 --2- 172.21.15.76:0/3662811853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c0689e0 0x7fd06c198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.700+0000 7fd071073640 1 -- 172.21.15.76:0/3662811853 >> 172.21.15.76:0/3662811853 conn(0x7fd06c0777e0 msgr2=0x7fd06c0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:39.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.701+0000 7fd071073640 1 -- 172.21.15.76:0/3662811853 shutdown_connections 2024-03-18T14:53:39.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.701+0000 7fd071073640 1 -- 172.21.15.76:0/3662811853 wait complete. 2024-03-18T14:53:39.712 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T14:53:39.712 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:53:39.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.817+0000 7efd0444b640 1 -- 172.21.15.76:0/3217570370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcfc102a50 msgr2=0x7efcfc102e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:39.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.817+0000 7efd0444b640 1 --2- 172.21.15.76:0/3217570370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcfc102a50 0x7efcfc102e30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efcec00fb60 tx=0x7efcec02fcf0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.817+0000 7efd0444b640 1 -- 172.21.15.76:0/3217570370 shutdown_connections 2024-03-18T14:53:39.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.817+0000 7efd0444b640 1 --2- 172.21.15.76:0/3217570370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcfc103370 0x7efcfc10b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.817+0000 7efd0444b640 1 --2- 172.21.15.76:0/3217570370 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcfc10be20 0x7efcfc10e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.817+0000 7efd0444b640 1 --2- 172.21.15.76:0/3217570370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcfc102a50 0x7efcfc102e30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.817+0000 7efd0444b640 1 -- 172.21.15.76:0/3217570370 >> 172.21.15.76:0/3217570370 conn(0x7efcfc0777e0 msgr2=0x7efcfc077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:39.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.817+0000 7efd0444b640 1 -- 172.21.15.76:0/3217570370 shutdown_connections 2024-03-18T14:53:39.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.817+0000 7efd0444b640 1 -- 172.21.15.76:0/3217570370 wait complete. 2024-03-18T14:53:39.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.817+0000 7efd0444b640 1 Processor -- start 2024-03-18T14:53:39.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.817+0000 7efd0444b640 1 -- start start 2024-03-18T14:53:39.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd0444b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcfc102a50 0x7efcfc195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd0444b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcfc103370 0x7efcfc196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd0444b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcfc10be20 0x7efcfc19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd0444b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efcfc10fa10 con 0x7efcfc10be20 2024-03-18T14:53:39.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd0444b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efcfc10f890 con 0x7efcfc102a50 2024-03-18T14:53:39.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd0444b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efcfc10fb90 con 0x7efcfc103370 2024-03-18T14:53:39.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd029e8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcfc10be20 0x7efcfc19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd019e6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcfc103370 0x7efcfc196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd029e8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcfc10be20 0x7efcfc19a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39256/0 (socket says 172.21.15.76:39256) 2024-03-18T14:53:39.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd029e8640 1 -- 172.21.15.76:0/1301051792 learned_addr learned my addr 172.21.15.76:0/1301051792 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:39.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd021e7640 1 --2- 172.21.15.76:0/1301051792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcfc102a50 0x7efcfc195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd029e8640 1 -- 172.21.15.76:0/1301051792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcfc103370 msgr2=0x7efcfc196450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:39.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd029e8640 1 --2- 172.21.15.76:0/1301051792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcfc103370 0x7efcfc196450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd029e8640 1 -- 172.21.15.76:0/1301051792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcfc102a50 msgr2=0x7efcfc195f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:39.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd029e8640 1 --2- 172.21.15.76:0/1301051792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcfc102a50 0x7efcfc195f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:39.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd029e8640 1 -- 172.21.15.76:0/1301051792 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcfc19aeb0 con 0x7efcfc10be20 2024-03-18T14:53:39.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd021e7640 1 --2- 172.21.15.76:0/1301051792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcfc102a50 0x7efcfc195f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:39.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.818+0000 7efd029e8640 1 --2- 172.21.15.76:0/1301051792 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcfc10be20 0x7efcfc19a780 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7efcf400eb10 tx=0x7efcf400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:39.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.819+0000 7efcf37fe640 1 -- 172.21.15.76:0/1301051792 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4004b20 con 0x7efcfc10be20 2024-03-18T14:53:39.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.819+0000 7efcf37fe640 1 -- 172.21.15.76:0/1301051792 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcf4004cc0 con 0x7efcfc10be20 2024-03-18T14:53:39.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.819+0000 7efcf37fe640 1 -- 172.21.15.76:0/1301051792 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4010760 con 0x7efcfc10be20 2024-03-18T14:53:39.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.819+0000 7efd0444b640 1 -- 172.21.15.76:0/1301051792 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcfc19b1a0 con 0x7efcfc10be20 2024-03-18T14:53:39.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.819+0000 7efd0444b640 1 -- 172.21.15.76:0/1301051792 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcfc1a2a30 con 0x7efcfc10be20 2024-03-18T14:53:39.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.821+0000 7efd0444b640 1 -- 172.21.15.76:0/1301051792 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcfc0685d0 con 0x7efcfc10be20 2024-03-18T14:53:39.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.821+0000 7efcf37fe640 1 -- 172.21.15.76:0/1301051792 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 23) ==== 137466+0+0 (secure 0 0 0) 0x7efcf4005000 con 0x7efcfc10be20 2024-03-18T14:53:39.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.822+0000 7efcf37fe640 1 --2- 172.21.15.76:0/1301051792 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7efce80a2a80 0x7efce80a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:39.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.822+0000 7efd021e7640 1 --2- 172.21.15.76:0/1301051792 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7efce80a2a80 0x7efce80a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:39.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.822+0000 7efcf37fe640 1 -- 172.21.15.76:0/1301051792 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7efcf40cf520 con 0x7efcfc10be20 2024-03-18T14:53:39.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.823+0000 7efd021e7640 1 --2- 172.21.15.76:0/1301051792 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7efce80a2a80 0x7efce80a4f40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efcec030300 tx=0x7efcec032040 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:39.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:39.827+0000 7efcf37fe640 1 -- 172.21.15.76:0/1301051792 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efcf409c160 con 0x7efcfc10be20 2024-03-18T14:53:40.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.049+0000 7efd0444b640 1 -- 172.21.15.76:0/1301051792 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efcfc106f10 con 0x7efcfc10be20 2024-03-18T14:53:40.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.050+0000 7efcf37fe640 1 -- 172.21.15.76:0/1301051792 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11863 (secure 0 0 0) 0x7efcf409ba60 con 0x7efcfc10be20 2024-03-18T14:53:40.050 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:53:40.051 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/2801069369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":2801069369},{"type":"v1","addr":"172.21.15.76:6839","nonce":2801069369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2645229232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2645229232},{"type":"v1","addr":"172.21.15.76:6841","nonce":2645229232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T14:51:01.293009+0000","modified":"2024-03-18T14:53:02.728776+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24490,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6839/399810652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":399810652},{"type":"v1","addr":"172.21.15.174:6839","nonce":399810652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1685041985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1685041985},{"type":"v1","addr":"172.21.15.77:6839","nonce":1685041985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524,24463,24490]},"id":1}]} 2024-03-18T14:53:40.051 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 24 2024-03-18T14:53:40.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.052+0000 7efd0444b640 1 -- 172.21.15.76:0/1301051792 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7efce80a2a80 msgr2=0x7efce80a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.052+0000 7efd0444b640 1 --2- 172.21.15.76:0/1301051792 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7efce80a2a80 0x7efce80a4f40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efcec030300 tx=0x7efcec032040 comp rx=0 tx=0).stop 2024-03-18T14:53:40.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.052+0000 7efd0444b640 1 -- 172.21.15.76:0/1301051792 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcfc10be20 msgr2=0x7efcfc19a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.052+0000 7efd0444b640 1 --2- 172.21.15.76:0/1301051792 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcfc10be20 0x7efcfc19a780 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7efcf400eb10 tx=0x7efcf400c2d0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.053+0000 7efd0444b640 1 -- 172.21.15.76:0/1301051792 shutdown_connections 2024-03-18T14:53:40.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.053+0000 7efd0444b640 1 --2- 172.21.15.76:0/1301051792 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7efce80a2a80 0x7efce80a4f40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.053+0000 7efd0444b640 1 --2- 172.21.15.76:0/1301051792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcfc103370 0x7efcfc196450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.053+0000 7efd0444b640 1 --2- 172.21.15.76:0/1301051792 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcfc10be20 0x7efcfc19a780 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.053+0000 7efd0444b640 1 --2- 172.21.15.76:0/1301051792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcfc102a50 0x7efcfc195f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.053+0000 7efd0444b640 1 -- 172.21.15.76:0/1301051792 >> 172.21.15.76:0/1301051792 conn(0x7efcfc0777e0 msgr2=0x7efcfc0fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.053+0000 7efd0444b640 1 -- 172.21.15.76:0/1301051792 shutdown_connections 2024-03-18T14:53:40.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.053+0000 7efd0444b640 1 -- 172.21.15.76:0/1301051792 wait complete. 2024-03-18T14:53:40.068 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T14:53:40.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.176+0000 7f3c801f3640 1 -- 172.21.15.76:0/1140641049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c780693a0 msgr2=0x7f3c7810e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.176+0000 7f3c801f3640 1 --2- 172.21.15.76:0/1140641049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c780693a0 0x7f3c7810e640 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3c70009cd0 tx=0x7f3c7002c0f0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.176+0000 7f3c801f3640 1 -- 172.21.15.76:0/1140641049 shutdown_connections 2024-03-18T14:53:40.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.176+0000 7f3c801f3640 1 --2- 172.21.15.76:0/1140641049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c780693a0 0x7f3c7810e640 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.176+0000 7f3c801f3640 1 --2- 172.21.15.76:0/1140641049 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c780689e0 0x7f3c78068e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.176+0000 7f3c801f3640 1 --2- 172.21.15.76:0/1140641049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c78107830 0x7f3c78107c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.176+0000 7f3c801f3640 1 -- 172.21.15.76:0/1140641049 >> 172.21.15.76:0/1140641049 conn(0x7f3c780777e0 msgr2=0x7f3c78077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.176+0000 7f3c801f3640 1 -- 172.21.15.76:0/1140641049 shutdown_connections 2024-03-18T14:53:40.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.176+0000 7f3c801f3640 1 -- 172.21.15.76:0/1140641049 wait complete. 2024-03-18T14:53:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.177+0000 7f3c801f3640 1 Processor -- start 2024-03-18T14:53:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.177+0000 7f3c801f3640 1 -- start start 2024-03-18T14:53:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.177+0000 7f3c801f3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c780689e0 0x7f3c78198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.177+0000 7f3c801f3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c780693a0 0x7f3c78198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.177+0000 7f3c801f3640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c78107830 0x7f3c7819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.177+0000 7f3c7df8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c780689e0 0x7f3c78198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.177+0000 7f3c7df8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c780689e0 0x7f3c78198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41100/0 (socket says 172.21.15.76:41100) 2024-03-18T14:53:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.177+0000 7f3c7df8f640 1 -- 172.21.15.76:0/1155513243 learned_addr learned my addr 172.21.15.76:0/1155513243 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.178+0000 7f3c7e790640 1 --2- 172.21.15.76:0/1155513243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c78107830 0x7f3c7819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.178+0000 7f3c7d78e640 1 --2- 172.21.15.76:0/1155513243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c780693a0 0x7f3c78198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.178+0000 7f3c801f3640 1 -- 172.21.15.76:0/1155513243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3c78111b80 con 0x7f3c78107830 2024-03-18T14:53:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.178+0000 7f3c801f3640 1 -- 172.21.15.76:0/1155513243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3c78111a00 con 0x7f3c780689e0 2024-03-18T14:53:40.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.178+0000 7f3c801f3640 1 -- 172.21.15.76:0/1155513243 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c78111d00 con 0x7f3c780693a0 2024-03-18T14:53:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.178+0000 7f3c7df8f640 1 -- 172.21.15.76:0/1155513243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c780693a0 msgr2=0x7f3c78198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.178+0000 7f3c7df8f640 1 --2- 172.21.15.76:0/1155513243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c780693a0 0x7f3c78198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.178+0000 7f3c7df8f640 1 -- 172.21.15.76:0/1155513243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c78107830 msgr2=0x7f3c7819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.178+0000 7f3c7df8f640 1 --2- 172.21.15.76:0/1155513243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c78107830 0x7f3c7819c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.178+0000 7f3c7df8f640 1 -- 172.21.15.76:0/1155513243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c7819cff0 con 0x7f3c780689e0 2024-03-18T14:53:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.178+0000 7f3c7d78e640 1 --2- 172.21.15.76:0/1155513243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c780693a0 0x7f3c78198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.178+0000 7f3c7df8f640 1 --2- 172.21.15.76:0/1155513243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c780689e0 0x7f3c78198000 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400e9f0 tx=0x7f3c7400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.178+0000 7f3c6effd640 1 -- 172.21.15.76:0/1155513243 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7400cd80 con 0x7f3c780689e0 2024-03-18T14:53:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.178+0000 7f3c6effd640 1 -- 172.21.15.76:0/1155513243 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c74005500 con 0x7f3c780689e0 2024-03-18T14:53:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.178+0000 7f3c801f3640 1 -- 172.21.15.76:0/1155513243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c7819d2e0 con 0x7f3c780689e0 2024-03-18T14:53:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.178+0000 7f3c6effd640 1 -- 172.21.15.76:0/1155513243 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c74010690 con 0x7f3c780689e0 2024-03-18T14:53:40.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.178+0000 7f3c801f3640 1 -- 172.21.15.76:0/1155513243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c781a4b70 con 0x7f3c780689e0 2024-03-18T14:53:40.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.181+0000 7f3c6effd640 1 -- 172.21.15.76:0/1155513243 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 23) ==== 137466+0+0 (secure 0 0 0) 0x7f3c740046c0 con 0x7f3c780689e0 2024-03-18T14:53:40.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.182+0000 7f3c82ddb740 1 Processor -- start 2024-03-18T14:53:40.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.182+0000 7f3c6effd640 1 --2- 172.21.15.76:0/1155513243 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3c640a2a80 0x7f3c640a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.182+0000 7f3c6effd640 1 -- 172.21.15.76:0/1155513243 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f3c740cf320 con 0x7f3c780689e0 2024-03-18T14:53:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.182+0000 7f3c82ddb740 1 -- start start 2024-03-18T14:53:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.182+0000 7f3c7e790640 1 --2- 172.21.15.76:0/1155513243 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3c640a2a80 0x7f3c640a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.182+0000 7f3c82ddb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7670830 0x55b2e7670c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.182+0000 7f3c82ddb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7671150 0x55b2e76797f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.182+0000 7f3c82ddb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7679d30 0x55b2e767c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.182+0000 7f3c82ddb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b2e75d4490 con 0x55b2e7679d30 2024-03-18T14:53:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.182+0000 7f3c82ddb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b2e757cc90 con 0x55b2e7671150 2024-03-18T14:53:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.182+0000 7f3c82ddb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b2e75d0530 con 0x55b2e7670830 2024-03-18T14:53:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.183+0000 7f3c7d78e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7671150 0x55b2e76797f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.183+0000 7f3c7d78e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7671150 0x55b2e76797f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41112/0 (socket says 172.21.15.76:41112) 2024-03-18T14:53:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.183+0000 7f3c7d78e640 1 -- 172.21.15.76:0/3053981371 learned_addr learned my addr 172.21.15.76:0/3053981371 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.183+0000 7f3c7e790640 1 --2- 172.21.15.76:0/3053981371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7679d30 0x55b2e767c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.183+0000 7f3c7df8f640 1 --2- 172.21.15.76:0/3053981371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7670830 0x55b2e7670c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.183+0000 7f3c7d78e640 1 -- 172.21.15.76:0/3053981371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7670830 msgr2=0x55b2e7670c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.183+0000 7f3c7d78e640 1 --2- 172.21.15.76:0/3053981371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7670830 0x55b2e7670c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.183+0000 7f3c7d78e640 1 -- 172.21.15.76:0/3053981371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7679d30 msgr2=0x55b2e767c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.183+0000 7f3c7d78e640 1 --2- 172.21.15.76:0/3053981371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7679d30 0x55b2e767c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.183+0000 7f3c7d78e640 1 -- 172.21.15.76:0/3053981371 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2e7637500 con 0x55b2e7671150 2024-03-18T14:53:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.183+0000 7f3c7df8f640 1 --2- 172.21.15.76:0/3053981371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7670830 0x55b2e7670c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:53:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.183+0000 7f3c7d78e640 1 --2- 172.21.15.76:0/3053981371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7671150 0x55b2e76797f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3c680114f0 tx=0x7f3c680119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=814288d8e944a836 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.183+0000 7f3c7e790640 1 --2- 172.21.15.76:0/1155513243 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3c640a2a80 0x7f3c640a4f40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3c70005cc0 tx=0x7f3c70032e50 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.183+0000 7f3c63fff640 1 -- 172.21.15.76:0/3053981371 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c68011030 con 0x55b2e7671150 2024-03-18T14:53:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.183+0000 7f3c63fff640 1 -- 172.21.15.76:0/3053981371 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c680111d0 con 0x55b2e7671150 2024-03-18T14:53:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.183+0000 7f3c63fff640 1 -- 172.21.15.76:0/3053981371 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c6800f5c0 con 0x55b2e7671150 2024-03-18T14:53:40.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.184+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3053981371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7671150 msgr2=0x55b2e76797f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.184+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3053981371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7671150 0x55b2e76797f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3c680114f0 tx=0x7f3c680119b0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.184+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3053981371 shutdown_connections 2024-03-18T14:53:40.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.184+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3053981371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7679d30 0x55b2e767c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.184+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3053981371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7671150 0x55b2e76797f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.184+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3053981371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7670830 0x55b2e7670c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.184+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3053981371 >> 172.21.15.76:0/3053981371 conn(0x55b2e766af60 msgr2=0x55b2e766b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.185+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3053981371 shutdown_connections 2024-03-18T14:53:40.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.185+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3053981371 wait complete. 2024-03-18T14:53:40.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.185+0000 7f3c82ddb740 1 -- start start 2024-03-18T14:53:40.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.185+0000 7f3c82ddb740 1 Processor -- start 2024-03-18T14:53:40.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c82ddb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7670830 0x55b2e7676c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c82ddb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7671150 0x55b2e76771d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c82ddb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7679d30 0x55b2e7677710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c82ddb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b2e7680f30 con 0x55b2e7679d30 2024-03-18T14:53:40.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c82ddb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b2e7590230 con 0x55b2e7670830 2024-03-18T14:53:40.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c82ddb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b2e76810b0 con 0x55b2e7671150 2024-03-18T14:53:40.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c82ddb740 10 client.0 _ll_register_callbacks cb 0x55b2e75cc010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:40.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c7df8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7670830 0x55b2e7676c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c7e790640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7679d30 0x55b2e7677710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c7df8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7670830 0x55b2e7676c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41128/0 (socket says 172.21.15.76:41128) 2024-03-18T14:53:40.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c7df8f640 1 -- 172.21.15.76:0/3173990444 learned_addr learned my addr 172.21.15.76:0/3173990444 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:40.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c7d78e640 1 --2- 172.21.15.76:0/3173990444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7671150 0x55b2e76771d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c7df8f640 1 -- 172.21.15.76:0/3173990444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7671150 msgr2=0x55b2e76771d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c7df8f640 1 --2- 172.21.15.76:0/3173990444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7671150 0x55b2e76771d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c7df8f640 1 -- 172.21.15.76:0/3173990444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7679d30 msgr2=0x55b2e7677710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c7df8f640 1 --2- 172.21.15.76:0/3173990444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7679d30 0x55b2e7677710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c7df8f640 1 -- 172.21.15.76:0/3173990444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2e759f100 con 0x55b2e7670830 2024-03-18T14:53:40.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c7d78e640 1 --2- 172.21.15.76:0/3173990444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7671150 0x55b2e76771d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:53:40.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c7df8f640 1 --2- 172.21.15.76:0/3173990444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7670830 0x55b2e7676c90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3c7409b560 tx=0x7f3c7409b280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5ce3d67970f0ea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c617fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:53:40.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c617fa640 1 -- 172.21.15.76:0/3173990444 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c740cceb0 con 0x55b2e7670830 2024-03-18T14:53:40.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.186+0000 7f3c617fa640 1 -- 172.21.15.76:0/3173990444 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f3c740a3b70 con 0x55b2e7670830 2024-03-18T14:53:40.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.187+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3173990444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c680111d0 con 0x55b2e7670830 2024-03-18T14:53:40.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.187+0000 7f3c617fa640 1 -- 172.21.15.76:0/3173990444 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3c7409f9b0 con 0x55b2e7670830 2024-03-18T14:53:40.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.187+0000 7f3c82ddb740 10 client.25421 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:40.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.187+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3173990444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b2e72e83f0 con 0x55b2e7670830 2024-03-18T14:53:40.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.187+0000 7f3c617fa640 1 -- 172.21.15.76:0/3173990444 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c680111d0 con 0x55b2e7670830 2024-03-18T14:53:40.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.187+0000 7f3c617fa640 1 -- 172.21.15.76:0/3173990444 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3c7409acb0 con 0x55b2e7670830 2024-03-18T14:53:40.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.188+0000 7f3c82ddb740 10 client.25421 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:40.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.188+0000 7f3c82ddb740 10 client.25421 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T14:53:40.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.188+0000 7f3c82ddb740 20 client.25421 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:40.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.188+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3173990444 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55b2e768e3f0 0x55b2e7690840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.188+0000 7f3c82ddb740 4 client.25421 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:53:40.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.188+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3173990444 --> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2e75e19d0 con 0x55b2e768e3f0 2024-03-18T14:53:40.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.188+0000 7f3c7d78e640 1 --2- 172.21.15.76:0/3173990444 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55b2e768e3f0 0x55b2e7690840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.188+0000 7f3c7d78e640 1 --2- 172.21.15.76:0/3173990444 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55b2e768e3f0 0x55b2e7690840 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=da98512f219636f2 server_cookie=642f448125bbce3 in_seq=0 out_seq=0 2024-03-18T14:53:40.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.188+0000 7f3c617fa640 10 client.25421 ms_handle_connect on v2:172.21.15.174:6836/239386414 2024-03-18T14:53:40.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.189+0000 7f3c617fa640 1 -- 172.21.15.76:0/3173990444 <== mds.2 v2:172.21.15.174:6836/239386414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b2e75e19d0 con 0x55b2e768e3f0 2024-03-18T14:53:40.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.189+0000 7f3c617fa640 10 client.25421 handle_command_reply: tid=0 2024-03-18T14:53:40.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.189+0000 7f3c82ddb740 1 client.25421 shutdown 2024-03-18T14:53:40.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.189+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3173990444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7670830 msgr2=0x55b2e7676c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.189+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3173990444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7670830 0x55b2e7676c90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3c7409b560 tx=0x7f3c7409b280 comp rx=0 tx=0).stop 2024-03-18T14:53:40.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.189+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3173990444 shutdown_connections 2024-03-18T14:53:40.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.189+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3173990444 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55b2e768e3f0 0x55b2e7690840 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.189+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3173990444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7671150 0x55b2e76771d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.189+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3173990444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7679d30 0x55b2e7677710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.189+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3173990444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7670830 0x55b2e7676c90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.189+0000 7f3c617fa640 0 client.25421 ms_handle_reset on v2:172.21.15.174:6836/239386414 2024-03-18T14:53:40.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.189+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3173990444 >> 172.21.15.76:0/3173990444 conn(0x55b2e766af60 msgr2=0x55b2e766ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.189+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3173990444 shutdown_connections 2024-03-18T14:53:40.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.189+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3173990444 wait complete. 2024-03-18T14:53:40.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.189+0000 7f3c82ddb740 20 client.25421 trim_cache size 0 max 16384 2024-03-18T14:53:40.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.215+0000 7f3c82ddb740 1 Processor -- start 2024-03-18T14:53:40.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.215+0000 7f3c82ddb740 1 -- start start 2024-03-18T14:53:40.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.215+0000 7f3c82ddb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7670830 0x55b2e76793c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.215+0000 7f3c82ddb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7671150 0x55b2e7676570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.215+0000 7f3c82ddb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7679d30 0x55b2e7689d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.215+0000 7f3c82ddb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3c781adab0 con 0x55b2e7670830 2024-03-18T14:53:40.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.215+0000 7f3c82ddb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3c781ab380 con 0x55b2e7671150 2024-03-18T14:53:40.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.215+0000 7f3c82ddb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c781a91d0 con 0x55b2e7679d30 2024-03-18T14:53:40.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.216+0000 7f3c7d78e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7671150 0x55b2e7676570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.216+0000 7f3c7e790640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7679d30 0x55b2e7689d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.216+0000 7f3c7d78e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7671150 0x55b2e7676570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41136/0 (socket says 172.21.15.76:41136) 2024-03-18T14:53:40.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.216+0000 7f3c7d78e640 1 -- 172.21.15.76:0/3803017510 learned_addr learned my addr 172.21.15.76:0/3803017510 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:40.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.216+0000 7f3c7df8f640 1 --2- 172.21.15.76:0/3803017510 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7670830 0x55b2e76793c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.216+0000 7f3c7d78e640 1 -- 172.21.15.76:0/3803017510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7679d30 msgr2=0x55b2e7689d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.216+0000 7f3c7d78e640 1 --2- 172.21.15.76:0/3803017510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7679d30 0x55b2e7689d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.216+0000 7f3c7d78e640 1 -- 172.21.15.76:0/3803017510 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7670830 msgr2=0x55b2e76793c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.216+0000 7f3c7d78e640 1 --2- 172.21.15.76:0/3803017510 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7670830 0x55b2e76793c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.216+0000 7f3c7d78e640 1 -- 172.21.15.76:0/3803017510 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2e7670f80 con 0x55b2e7671150 2024-03-18T14:53:40.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.216+0000 7f3c7d78e640 1 --2- 172.21.15.76:0/3803017510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7671150 0x55b2e7676570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x55b2e7670300 tx=0x7f3c68003a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=513e2c89ccc4d7d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.216+0000 7f3c637fe640 1 -- 172.21.15.76:0/3803017510 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c68009b50 con 0x55b2e7671150 2024-03-18T14:53:40.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.216+0000 7f3c637fe640 1 -- 172.21.15.76:0/3803017510 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c68003b80 con 0x55b2e7671150 2024-03-18T14:53:40.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.216+0000 7f3c637fe640 1 -- 172.21.15.76:0/3803017510 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c6802b640 con 0x55b2e7671150 2024-03-18T14:53:40.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.217+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3803017510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7671150 msgr2=0x55b2e7676570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.217+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3803017510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7671150 0x55b2e7676570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x55b2e7670300 tx=0x7f3c68003a70 comp rx=0 tx=0).stop 2024-03-18T14:53:40.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.217+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3803017510 shutdown_connections 2024-03-18T14:53:40.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.217+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3803017510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7679d30 0x55b2e7689d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.217+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3803017510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7671150 0x55b2e7676570 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.217+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3803017510 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7670830 0x55b2e76793c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.217+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3803017510 >> 172.21.15.76:0/3803017510 conn(0x55b2e766af60 msgr2=0x55b2e7669fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.218+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3803017510 shutdown_connections 2024-03-18T14:53:40.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.218+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3803017510 wait complete. 2024-03-18T14:53:40.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.218+0000 7f3c82ddb740 1 -- start start 2024-03-18T14:53:40.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.218+0000 7f3c82ddb740 1 Processor -- start 2024-03-18T14:53:40.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.218+0000 7f3c82ddb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7670830 0x55b2e7686290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c82ddb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7671150 0x55b2e7681720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c82ddb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7679d30 0x55b2e7681c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c7df8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7670830 0x55b2e7686290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c7df8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7670830 0x55b2e7686290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41148/0 (socket says 172.21.15.76:41148) 2024-03-18T14:53:40.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c7d78e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7671150 0x55b2e7681720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c7e790640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7679d30 0x55b2e7681c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c7d78e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7671150 0x55b2e7681720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:49054/0 (socket says 172.21.15.76:49054) 2024-03-18T14:53:40.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c7e790640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7679d30 0x55b2e7681c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39324/0 (socket says 172.21.15.76:39324) 2024-03-18T14:53:40.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c7df8f640 1 -- 172.21.15.76:0/3484986220 learned_addr learned my addr 172.21.15.76:0/3484986220 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:40.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3484986220 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3c781a9840 con 0x55b2e7679d30 2024-03-18T14:53:40.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3484986220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3c781aacb0 con 0x55b2e7670830 2024-03-18T14:53:40.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3484986220 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c781a9f10 con 0x55b2e7671150 2024-03-18T14:53:40.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c7df8f640 1 -- 172.21.15.76:0/3484986220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7671150 msgr2=0x55b2e7681720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c7df8f640 1 --2- 172.21.15.76:0/3484986220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7671150 0x55b2e7681720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c7df8f640 1 -- 172.21.15.76:0/3484986220 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7679d30 msgr2=0x55b2e7681c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c7df8f640 1 --2- 172.21.15.76:0/3484986220 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7679d30 0x55b2e7681c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c7df8f640 1 -- 172.21.15.76:0/3484986220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2e73b27d0 con 0x55b2e7670830 2024-03-18T14:53:40.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c7e790640 1 --2- 172.21.15.76:0/3484986220 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7679d30 0x55b2e7681c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:40.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.219+0000 7f3c82ddb740 10 client.0 _ll_register_callbacks cb 0x7f3c740cc890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:40.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.220+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3484986220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c68003b80 con 0x55b2e7670830 2024-03-18T14:53:40.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.220+0000 7f3c7df8f640 1 --2- 172.21.15.76:0/3484986220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7670830 0x55b2e7686290 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3c7409b7f0 tx=0x7f3c74004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8797ef00713aae46 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.220+0000 7f3c617fa640 10 client.25427 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:53:40.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.220+0000 7f3c617fa640 1 -- 172.21.15.76:0/3484986220 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c740ccec0 con 0x55b2e7670830 2024-03-18T14:53:40.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.220+0000 7f3c617fa640 1 -- 172.21.15.76:0/3484986220 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f3c740a3b70 con 0x55b2e7670830 2024-03-18T14:53:40.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.220+0000 7f3c617fa640 1 -- 172.21.15.76:0/3484986220 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3c740a3dd0 con 0x55b2e7670830 2024-03-18T14:53:40.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.220+0000 7f3c82ddb740 10 client.25427 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:40.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.220+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3484986220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({fsmap=24}) -- 0x55b2e76f4090 con 0x55b2e7670830 2024-03-18T14:53:40.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.220+0000 7f3c617fa640 1 -- 172.21.15.76:0/3484986220 <== mon.2 v2:172.21.15.77:3300/0 4 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3c7409cb90 con 0x55b2e7670830 2024-03-18T14:53:40.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.220+0000 7f3c82ddb740 10 client.25427 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:40.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.220+0000 7f3c82ddb740 10 client.25427 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T14:53:40.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.220+0000 7f3c82ddb740 20 client.25427 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:40.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.220+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3484986220 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55b2e76f46d0 0x55b2e76f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.220+0000 7f3c82ddb740 4 client.25427 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T14:53:40.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.220+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3484986220 --> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b2e75dbd20 con 0x55b2e76f46d0 2024-03-18T14:53:40.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.221+0000 7f3c7d78e640 1 --2- 172.21.15.76:0/3484986220 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55b2e76f46d0 0x55b2e76f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.221+0000 7f3c617fa640 10 client.25427 ms_handle_connect on v2:172.21.15.174:6836/239386414 2024-03-18T14:53:40.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.221+0000 7f3c7d78e640 1 --2- 172.21.15.76:0/3484986220 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55b2e76f46d0 0x55b2e76f6b30 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=791908aad0adc6e4 server_cookie=5f707ad5a5606c5b in_seq=0 out_seq=0 2024-03-18T14:53:40.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.221+0000 7f3c617fa640 1 -- 172.21.15.76:0/3484986220 <== mds.2 v2:172.21.15.174:6836/239386414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b2e75dbd20 con 0x55b2e76f46d0 2024-03-18T14:53:40.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.221+0000 7f3c617fa640 10 client.25427 handle_command_reply: tid=0 2024-03-18T14:53:40.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.221+0000 7f3c82ddb740 1 client.25427 shutdown 2024-03-18T14:53:40.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.221+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3484986220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7670830 msgr2=0x55b2e7686290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.221+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3484986220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7670830 0x55b2e7686290 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3c7409b7f0 tx=0x7f3c74004a40 comp rx=0 tx=0).stop 2024-03-18T14:53:40.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.221+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3484986220 shutdown_connections 2024-03-18T14:53:40.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.221+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3484986220 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55b2e76f46d0 0x55b2e76f6b30 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.221+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3484986220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b2e7671150 0x55b2e7681720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.221+0000 7f3c617fa640 0 client.25427 ms_handle_reset on v2:172.21.15.174:6836/239386414 2024-03-18T14:53:40.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.221+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3484986220 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b2e7679d30 0x55b2e7681c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.221+0000 7f3c82ddb740 1 --2- 172.21.15.76:0/3484986220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2e7670830 0x55b2e7686290 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.221+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3484986220 >> 172.21.15.76:0/3484986220 conn(0x55b2e766af60 msgr2=0x55b2e7676c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.222+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3484986220 shutdown_connections 2024-03-18T14:53:40.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.222+0000 7f3c82ddb740 1 -- 172.21.15.76:0/3484986220 wait complete. 2024-03-18T14:53:40.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.222+0000 7f3c82ddb740 20 client.25427 trim_cache size 0 max 16384 2024-03-18T14:53:40.228 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-18T14:53:40.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.222+0000 7f3c801f3640 1 -- 172.21.15.76:0/1155513243 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3c640a2a80 msgr2=0x7f3c640a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.222+0000 7f3c801f3640 1 --2- 172.21.15.76:0/1155513243 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3c640a2a80 0x7f3c640a4f40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3c70005cc0 tx=0x7f3c70032e50 comp rx=0 tx=0).stop 2024-03-18T14:53:40.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.222+0000 7f3c801f3640 1 -- 172.21.15.76:0/1155513243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c780689e0 msgr2=0x7f3c78198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.222+0000 7f3c801f3640 1 --2- 172.21.15.76:0/1155513243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c780689e0 0x7f3c78198000 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400e9f0 tx=0x7f3c7400edf0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.222+0000 7f3c801f3640 1 -- 172.21.15.76:0/1155513243 shutdown_connections 2024-03-18T14:53:40.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.222+0000 7f3c801f3640 1 --2- 172.21.15.76:0/1155513243 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3c640a2a80 0x7f3c640a4f40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.222+0000 7f3c801f3640 1 --2- 172.21.15.76:0/1155513243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c780693a0 0x7f3c78198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.222+0000 7f3c801f3640 1 --2- 172.21.15.76:0/1155513243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c78107830 0x7f3c7819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.222+0000 7f3c801f3640 1 --2- 172.21.15.76:0/1155513243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c780689e0 0x7f3c78198000 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.222+0000 7f3c801f3640 1 -- 172.21.15.76:0/1155513243 >> 172.21.15.76:0/1155513243 conn(0x7f3c780777e0 msgr2=0x7f3c780fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.223+0000 7f3c801f3640 1 -- 172.21.15.76:0/1155513243 shutdown_connections 2024-03-18T14:53:40.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.223+0000 7f3c801f3640 1 -- 172.21.15.76:0/1155513243 wait complete. 2024-03-18T14:53:40.235 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T14:53:40.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.343+0000 7ff6df6c6640 1 -- 172.21.15.76:0/2404672608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d80692e0 msgr2=0x7ff6d810dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.343+0000 7ff6df6c6640 1 --2- 172.21.15.76:0/2404672608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d80692e0 0x7ff6d810dc40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009900 tx=0x7ff6c802bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:40.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.343+0000 7ff6df6c6640 1 -- 172.21.15.76:0/2404672608 shutdown_connections 2024-03-18T14:53:40.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.343+0000 7ff6df6c6640 1 --2- 172.21.15.76:0/2404672608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6d810e180 0x7ff6d8110570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.343+0000 7ff6df6c6640 1 --2- 172.21.15.76:0/2404672608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d80692e0 0x7ff6d810dc40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.343+0000 7ff6df6c6640 1 --2- 172.21.15.76:0/2404672608 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6d80689c0 0x7ff6d8068da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.343+0000 7ff6df6c6640 1 -- 172.21.15.76:0/2404672608 >> 172.21.15.76:0/2404672608 conn(0x7ff6d80777e0 msgr2=0x7ff6d8077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.343+0000 7ff6df6c6640 1 -- 172.21.15.76:0/2404672608 shutdown_connections 2024-03-18T14:53:40.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.343+0000 7ff6df6c6640 1 -- 172.21.15.76:0/2404672608 wait complete. 2024-03-18T14:53:40.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.344+0000 7ff6df6c6640 1 Processor -- start 2024-03-18T14:53:40.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.344+0000 7ff6df6c6640 1 -- start start 2024-03-18T14:53:40.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.344+0000 7ff6df6c6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6d80689c0 0x7ff6d819a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.344+0000 7ff6df6c6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6d80692e0 0x7ff6d819a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.344+0000 7ff6dcc61640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6d80692e0 0x7ff6d819a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.344+0000 7ff6dd462640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6d80689c0 0x7ff6d819a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.344+0000 7ff6dcc61640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6d80692e0 0x7ff6d819a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39328/0 (socket says 172.21.15.76:39328) 2024-03-18T14:53:40.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.344+0000 7ff6df6c6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d810e180 0x7ff6d819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.344+0000 7ff6dd462640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6d80689c0 0x7ff6d819a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:49072/0 (socket says 172.21.15.76:49072) 2024-03-18T14:53:40.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.344+0000 7ff6dcc61640 1 -- 172.21.15.76:0/2497386754 learned_addr learned my addr 172.21.15.76:0/2497386754 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:40.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.344+0000 7ff6df6c6640 1 -- 172.21.15.76:0/2497386754 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d8113d40 con 0x7ff6d80692e0 2024-03-18T14:53:40.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.344+0000 7ff6df6c6640 1 -- 172.21.15.76:0/2497386754 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d8113bc0 con 0x7ff6d810e180 2024-03-18T14:53:40.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.344+0000 7ff6df6c6640 1 -- 172.21.15.76:0/2497386754 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d8113ec0 con 0x7ff6d80689c0 2024-03-18T14:53:40.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.345+0000 7ff6ddc63640 1 --2- 172.21.15.76:0/2497386754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d810e180 0x7ff6d819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.345+0000 7ff6dd462640 1 -- 172.21.15.76:0/2497386754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d810e180 msgr2=0x7ff6d819ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.345+0000 7ff6dd462640 1 --2- 172.21.15.76:0/2497386754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d810e180 0x7ff6d819ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.345+0000 7ff6dd462640 1 -- 172.21.15.76:0/2497386754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6d80692e0 msgr2=0x7ff6d819a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.345+0000 7ff6dd462640 1 --2- 172.21.15.76:0/2497386754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6d80692e0 0x7ff6d819a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.345+0000 7ff6dd462640 1 -- 172.21.15.76:0/2497386754 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d819f2f0 con 0x7ff6d80689c0 2024-03-18T14:53:40.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.345+0000 7ff6dcc61640 1 --2- 172.21.15.76:0/2497386754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6d80692e0 0x7ff6d819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:40.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.345+0000 7ff6dd462640 1 --2- 172.21.15.76:0/2497386754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6d80689c0 0x7ff6d819a300 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400e9f0 tx=0x7ff6d400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.345+0000 7ff6ce7fc640 1 -- 172.21.15.76:0/2497386754 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d400ccc0 con 0x7ff6d80689c0 2024-03-18T14:53:40.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.345+0000 7ff6ce7fc640 1 -- 172.21.15.76:0/2497386754 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6d4005020 con 0x7ff6d80689c0 2024-03-18T14:53:40.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.345+0000 7ff6ce7fc640 1 -- 172.21.15.76:0/2497386754 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d40049e0 con 0x7ff6d80689c0 2024-03-18T14:53:40.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.345+0000 7ff6df6c6640 1 -- 172.21.15.76:0/2497386754 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d819f5e0 con 0x7ff6d80689c0 2024-03-18T14:53:40.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.345+0000 7ff6df6c6640 1 -- 172.21.15.76:0/2497386754 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6d81a6e70 con 0x7ff6d80689c0 2024-03-18T14:53:40.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.348+0000 7ff6e238d740 1 Processor -- start 2024-03-18T14:53:40.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.348+0000 7ff6e238d740 1 -- start start 2024-03-18T14:53:40.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.348+0000 7ff6e238d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4b830 0x561422c4bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.348+0000 7ff6e238d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4c150 0x561422c547f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.348+0000 7ff6e238d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c54d30 0x561422c57120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.348+0000 7ff6e238d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561422baf490 con 0x561422c54d30 2024-03-18T14:53:40.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.348+0000 7ff6e238d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561422b57c90 con 0x561422c4c150 2024-03-18T14:53:40.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.348+0000 7ff6e238d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561422bab530 con 0x561422c4b830 2024-03-18T14:53:40.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.348+0000 7ff6ce7fc640 1 -- 172.21.15.76:0/2497386754 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 23) ==== 137466+0+0 (secure 0 0 0) 0x7ff6d4005560 con 0x7ff6d80689c0 2024-03-18T14:53:40.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.349+0000 7ff6ddc63640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4b830 0x561422c4bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.349+0000 7ff6ddc63640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4b830 0x561422c4bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:49076/0 (socket says 172.21.15.76:49076) 2024-03-18T14:53:40.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.349+0000 7ff6ddc63640 1 -- 172.21.15.76:0/2891869490 learned_addr learned my addr 172.21.15.76:0/2891869490 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:40.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.349+0000 7ff6dd462640 1 --2- 172.21.15.76:0/2891869490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4c150 0x561422c547f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.349+0000 7ff6dcc61640 1 --2- 172.21.15.76:0/2891869490 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c54d30 0x561422c57120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.349+0000 7ff6ddc63640 1 -- 172.21.15.76:0/2891869490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4c150 msgr2=0x561422c547f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.349+0000 7ff6ddc63640 1 --2- 172.21.15.76:0/2891869490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4c150 0x561422c547f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.349+0000 7ff6ddc63640 1 -- 172.21.15.76:0/2891869490 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c54d30 msgr2=0x561422c57120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.349+0000 7ff6ddc63640 1 --2- 172.21.15.76:0/2891869490 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c54d30 0x561422c57120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.349+0000 7ff6ddc63640 1 -- 172.21.15.76:0/2891869490 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561422c12500 con 0x561422c4b830 2024-03-18T14:53:40.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.349+0000 7ff6ddc63640 1 --2- 172.21.15.76:0/2891869490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4b830 0x561422c4bc10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009f00 tx=0x7ff6d00107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=408d459ef3778420 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.349+0000 7ff6c37fe640 1 -- 172.21.15.76:0/2891869490 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0003b70 con 0x561422c4b830 2024-03-18T14:53:40.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.349+0000 7ff6c37fe640 1 -- 172.21.15.76:0/2891869490 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6d0003d10 con 0x561422c4b830 2024-03-18T14:53:40.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.349+0000 7ff6c37fe640 1 -- 172.21.15.76:0/2891869490 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0011ab0 con 0x561422c4b830 2024-03-18T14:53:40.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.349+0000 7ff6dd462640 1 --2- 172.21.15.76:0/2891869490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4c150 0x561422c547f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:53:40.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.350+0000 7ff6ce7fc640 1 --2- 172.21.15.76:0/2497386754 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6c40a2a80 0x7ff6c40a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.350+0000 7ff6dd462640 1 --2- 172.21.15.76:0/2497386754 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6c40a2a80 0x7ff6c40a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.350+0000 7ff6ce7fc640 1 -- 172.21.15.76:0/2497386754 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7ff6d40cefc0 con 0x7ff6d80689c0 2024-03-18T14:53:40.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.350+0000 7ff6e238d740 1 -- 172.21.15.76:0/2891869490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4b830 msgr2=0x561422c4bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.350+0000 7ff6e238d740 1 --2- 172.21.15.76:0/2891869490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4b830 0x561422c4bc10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009f00 tx=0x7ff6d00107a0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.350+0000 7ff6dd462640 1 --2- 172.21.15.76:0/2497386754 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6c40a2a80 0x7ff6c40a4f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff6d40cc6a0 tx=0x7ff6d40096e0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.350+0000 7ff6e238d740 1 -- 172.21.15.76:0/2891869490 shutdown_connections 2024-03-18T14:53:40.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.350+0000 7ff6e238d740 1 --2- 172.21.15.76:0/2891869490 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c54d30 0x561422c57120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.350+0000 7ff6e238d740 1 --2- 172.21.15.76:0/2891869490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4c150 0x561422c547f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.350+0000 7ff6e238d740 1 --2- 172.21.15.76:0/2891869490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4b830 0x561422c4bc10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.350+0000 7ff6e238d740 1 -- 172.21.15.76:0/2891869490 >> 172.21.15.76:0/2891869490 conn(0x561422c45f60 msgr2=0x561422c46340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.350+0000 7ff6e238d740 1 -- 172.21.15.76:0/2891869490 shutdown_connections 2024-03-18T14:53:40.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.351+0000 7ff6e238d740 1 -- 172.21.15.76:0/2891869490 wait complete. 2024-03-18T14:53:40.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.351+0000 7ff6e238d740 1 -- start start 2024-03-18T14:53:40.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.351+0000 7ff6e238d740 1 Processor -- start 2024-03-18T14:53:40.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.351+0000 7ff6e238d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c51d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.351+0000 7ff6e238d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4c150 0x561422c52290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.351+0000 7ff6e238d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c54d30 0x561422c527d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.351+0000 7ff6e238d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561422c57b20 con 0x561422c54d30 2024-03-18T14:53:40.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.351+0000 7ff6e238d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561422b6b230 con 0x561422c4b830 2024-03-18T14:53:40.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.351+0000 7ff6e238d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561422c57ca0 con 0x561422c4c150 2024-03-18T14:53:40.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.351+0000 7ff6e238d740 10 client.0 _ll_register_callbacks cb 0x561422ba7010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:40.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.351+0000 7ff6ddc63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c51d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.352+0000 7ff6dcc61640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4c150 0x561422c52290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.352+0000 7ff6ddc63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c51d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41180/0 (socket says 172.21.15.76:41180) 2024-03-18T14:53:40.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.352+0000 7ff6ddc63640 1 -- 172.21.15.76:0/3682966949 learned_addr learned my addr 172.21.15.76:0/3682966949 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:40.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.352+0000 7ff6ddc63640 1 --2- 172.21.15.76:0/3682966949 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c54d30 0x561422c527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.352+0000 7ff6ddc63640 1 -- 172.21.15.76:0/3682966949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4c150 msgr2=0x561422c52290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.352+0000 7ff6ddc63640 1 --2- 172.21.15.76:0/3682966949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4c150 0x561422c52290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.352+0000 7ff6ddc63640 1 -- 172.21.15.76:0/3682966949 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c54d30 msgr2=0x561422c527d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.352+0000 7ff6ddc63640 1 --2- 172.21.15.76:0/3682966949 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c54d30 0x561422c527d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.352+0000 7ff6ddc63640 1 -- 172.21.15.76:0/3682966949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561422b7a100 con 0x561422c4b830 2024-03-18T14:53:40.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.352+0000 7ff6dcc61640 1 --2- 172.21.15.76:0/3682966949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4c150 0x561422c52290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:40.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.352+0000 7ff6ddc63640 1 --2- 172.21.15.76:0/3682966949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c51d50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0003b90 tx=0x7ff6d0011610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82f4bf4d18a60521 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.352+0000 7ff6c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:53:40.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.352+0000 7ff6c0ff9640 1 -- 172.21.15.76:0/3682966949 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0025040 con 0x561422c4b830 2024-03-18T14:53:40.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.352+0000 7ff6c0ff9640 1 -- 172.21.15.76:0/3682966949 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7ff6d00040c0 con 0x561422c4b830 2024-03-18T14:53:40.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.352+0000 7ff6e238d740 1 -- 172.21.15.76:0/3682966949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6d0003d10 con 0x561422c4b830 2024-03-18T14:53:40.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.353+0000 7ff6c0ff9640 1 -- 172.21.15.76:0/3682966949 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff6d00042c0 con 0x561422c4b830 2024-03-18T14:53:40.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.353+0000 7ff6e238d740 10 client.25436 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:40.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.353+0000 7ff6e238d740 1 -- 172.21.15.76:0/3682966949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5614228c33f0 con 0x561422c4b830 2024-03-18T14:53:40.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.353+0000 7ff6c0ff9640 1 -- 172.21.15.76:0/3682966949 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6d0003d10 con 0x561422c4b830 2024-03-18T14:53:40.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.353+0000 7ff6c0ff9640 1 -- 172.21.15.76:0/3682966949 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7ff6d001bbb0 con 0x561422c4b830 2024-03-18T14:53:40.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.353+0000 7ff6e238d740 10 client.25436 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:40.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.353+0000 7ff6e238d740 10 client.25436 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T14:53:40.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.353+0000 7ff6e238d740 20 client.25436 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:40.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.353+0000 7ff6e238d740 1 --2- 172.21.15.76:0/3682966949 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x561422c64fa0 0x561422c67390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.353+0000 7ff6e238d740 4 client.25436 mds_command: new command op to 24490 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:53:40.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.353+0000 7ff6e238d740 1 -- 172.21.15.76:0/3682966949 --> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561422bbc9d0 con 0x561422c64fa0 2024-03-18T14:53:40.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.353+0000 7ff6dcc61640 1 --2- 172.21.15.76:0/3682966949 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x561422c64fa0 0x561422c67390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.354+0000 7ff6c0ff9640 10 client.25436 ms_handle_connect on v2:172.21.15.174:6838/399810652 2024-03-18T14:53:40.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.354+0000 7ff6dcc61640 1 --2- 172.21.15.76:0/3682966949 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x561422c64fa0 0x561422c67390 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3a1ea113f9488ab2 server_cookie=67a084b59850ef81 in_seq=0 out_seq=0 2024-03-18T14:53:40.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.354+0000 7ff6c0ff9640 1 -- 172.21.15.76:0/3682966949 <== mds.1 v2:172.21.15.174:6838/399810652 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561422bbc9d0 con 0x561422c64fa0 2024-03-18T14:53:40.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.354+0000 7ff6c0ff9640 10 client.25436 handle_command_reply: tid=0 2024-03-18T14:53:40.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.354+0000 7ff6e238d740 1 client.25436 shutdown 2024-03-18T14:53:40.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.354+0000 7ff6e238d740 1 -- 172.21.15.76:0/3682966949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 msgr2=0x561422c51d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.354+0000 7ff6e238d740 1 --2- 172.21.15.76:0/3682966949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c51d50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0003b90 tx=0x7ff6d0011610 comp rx=0 tx=0).stop 2024-03-18T14:53:40.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.354+0000 7ff6e238d740 1 -- 172.21.15.76:0/3682966949 shutdown_connections 2024-03-18T14:53:40.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.354+0000 7ff6e238d740 1 --2- 172.21.15.76:0/3682966949 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x561422c64fa0 0x561422c67390 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.354+0000 7ff6e238d740 1 --2- 172.21.15.76:0/3682966949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4c150 0x561422c52290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.354+0000 7ff6c0ff9640 0 client.25436 ms_handle_reset on v2:172.21.15.174:6838/399810652 2024-03-18T14:53:40.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.354+0000 7ff6e238d740 1 --2- 172.21.15.76:0/3682966949 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c54d30 0x561422c527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.354+0000 7ff6e238d740 1 --2- 172.21.15.76:0/3682966949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c51d50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.354+0000 7ff6e238d740 1 -- 172.21.15.76:0/3682966949 >> 172.21.15.76:0/3682966949 conn(0x561422c45f60 msgr2=0x561422c47cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.355+0000 7ff6e238d740 1 -- 172.21.15.76:0/3682966949 shutdown_connections 2024-03-18T14:53:40.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.355+0000 7ff6e238d740 1 -- 172.21.15.76:0/3682966949 wait complete. 2024-03-18T14:53:40.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.355+0000 7ff6e238d740 20 client.25436 trim_cache size 0 max 16384 2024-03-18T14:53:40.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.368+0000 7ff6e238d740 1 Processor -- start 2024-03-18T14:53:40.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.368+0000 7ff6e238d740 1 -- start start 2024-03-18T14:53:40.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.368+0000 7ff6e238d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c51630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6e238d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c4c150 0x561422c608c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6e238d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c54d30 0x561422c6ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6e238d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d81b1f40 con 0x561422c4c150 2024-03-18T14:53:40.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6e238d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d81af810 con 0x561422c4b830 2024-03-18T14:53:40.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6e238d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d81ad660 con 0x561422c54d30 2024-03-18T14:53:40.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6ddc63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c51630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6ddc63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c51630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41196/0 (socket says 172.21.15.76:41196) 2024-03-18T14:53:40.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6ddc63640 1 -- 172.21.15.76:0/3937456493 learned_addr learned my addr 172.21.15.76:0/3937456493 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:40.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6ddc63640 1 --2- 172.21.15.76:0/3937456493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c54d30 0x561422c6ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6dcc61640 1 --2- 172.21.15.76:0/3937456493 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c4c150 0x561422c608c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6ddc63640 1 -- 172.21.15.76:0/3937456493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c54d30 msgr2=0x561422c6ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6ddc63640 1 --2- 172.21.15.76:0/3937456493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c54d30 0x561422c6ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6ddc63640 1 -- 172.21.15.76:0/3937456493 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c4c150 msgr2=0x561422c608c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6ddc63640 1 --2- 172.21.15.76:0/3937456493 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c4c150 0x561422c608c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6ddc63640 1 -- 172.21.15.76:0/3937456493 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56142298d7d0 con 0x561422c4b830 2024-03-18T14:53:40.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6dcc61640 1 --2- 172.21.15.76:0/3937456493 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c4c150 0x561422c608c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:53:40.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6ddc63640 1 --2- 172.21.15.76:0/3937456493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c51630 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff6d001a4d0 tx=0x7ff6d0013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de8d333116069595 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6c2ffd640 1 -- 172.21.15.76:0/3937456493 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0009960 con 0x561422c4b830 2024-03-18T14:53:40.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6c2ffd640 1 -- 172.21.15.76:0/3937456493 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6d0009b00 con 0x561422c4b830 2024-03-18T14:53:40.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.369+0000 7ff6c2ffd640 1 -- 172.21.15.76:0/3937456493 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d001ab30 con 0x561422c4b830 2024-03-18T14:53:40.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.370+0000 7ff6e238d740 1 -- 172.21.15.76:0/3937456493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 msgr2=0x561422c51630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.370+0000 7ff6e238d740 1 --2- 172.21.15.76:0/3937456493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c51630 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff6d001a4d0 tx=0x7ff6d0013040 comp rx=0 tx=0).stop 2024-03-18T14:53:40.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.370+0000 7ff6e238d740 1 -- 172.21.15.76:0/3937456493 shutdown_connections 2024-03-18T14:53:40.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.370+0000 7ff6e238d740 1 --2- 172.21.15.76:0/3937456493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c54d30 0x561422c6ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.370+0000 7ff6e238d740 1 --2- 172.21.15.76:0/3937456493 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c4c150 0x561422c608c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.370+0000 7ff6e238d740 1 --2- 172.21.15.76:0/3937456493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c51630 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.370+0000 7ff6e238d740 1 -- 172.21.15.76:0/3937456493 >> 172.21.15.76:0/3937456493 conn(0x561422c45f60 msgr2=0x561422c51d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.370+0000 7ff6e238d740 1 -- 172.21.15.76:0/3937456493 shutdown_connections 2024-03-18T14:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.370+0000 7ff6e238d740 1 -- 172.21.15.76:0/3937456493 wait complete. 2024-03-18T14:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.370+0000 7ff6e238d740 1 -- start start 2024-03-18T14:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6e238d740 1 Processor -- start 2024-03-18T14:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6e238d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c4f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6e238d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4c150 0x561422c4fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6e238d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c54d30 0x561422c4dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6e238d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d81adcd0 con 0x561422c54d30 2024-03-18T14:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6e238d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d81af140 con 0x561422c4b830 2024-03-18T14:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6e238d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d81ae3a0 con 0x561422c4c150 2024-03-18T14:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6e238d740 10 client.0 _ll_register_callbacks cb 0x7ff6d4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6ddc63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c4f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6dcc61640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4c150 0x561422c4fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6ddc63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c4f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41202/0 (socket says 172.21.15.76:41202) 2024-03-18T14:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6ddc63640 1 -- 172.21.15.76:0/3673339616 learned_addr learned my addr 172.21.15.76:0/3673339616 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6ddc63640 1 --2- 172.21.15.76:0/3673339616 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c54d30 0x561422c4dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6ddc63640 1 -- 172.21.15.76:0/3673339616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4c150 msgr2=0x561422c4fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6ddc63640 1 --2- 172.21.15.76:0/3673339616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4c150 0x561422c4fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6ddc63640 1 -- 172.21.15.76:0/3673339616 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c54d30 msgr2=0x561422c4dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6ddc63640 1 --2- 172.21.15.76:0/3673339616 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c54d30 0x561422c4dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.371+0000 7ff6ddc63640 1 -- 172.21.15.76:0/3673339616 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561422c4e500 con 0x561422c4b830 2024-03-18T14:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.372+0000 7ff6ddc63640 1 --2- 172.21.15.76:0/3673339616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c4f510 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0011d40 tx=0x7ff6d0004120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4c71b8704a6c03c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.372+0000 7ff6c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.372+0000 7ff6c0ff9640 1 -- 172.21.15.76:0/3673339616 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0003cb0 con 0x561422c4b830 2024-03-18T14:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.372+0000 7ff6c0ff9640 1 -- 172.21.15.76:0/3673339616 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7ff6d0041040 con 0x561422c4b830 2024-03-18T14:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.372+0000 7ff6e238d740 1 -- 172.21.15.76:0/3673339616 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6d0009b00 con 0x561422c4b830 2024-03-18T14:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.372+0000 7ff6c0ff9640 1 -- 172.21.15.76:0/3673339616 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff6d0003e50 con 0x561422c4b830 2024-03-18T14:53:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.372+0000 7ff6e238d740 10 client.25442 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.372+0000 7ff6e238d740 1 -- 172.21.15.76:0/3673339616 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x561422cca7e0 con 0x561422c4b830 2024-03-18T14:53:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.372+0000 7ff6c0ff9640 1 -- 172.21.15.76:0/3673339616 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6d0004a00 con 0x561422c4b830 2024-03-18T14:53:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.372+0000 7ff6c0ff9640 1 -- 172.21.15.76:0/3673339616 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7ff6d001aba0 con 0x561422c4b830 2024-03-18T14:53:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.372+0000 7ff6e238d740 10 client.25442 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.372+0000 7ff6e238d740 10 client.25442 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T14:53:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.372+0000 7ff6e238d740 20 client.25442 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.372+0000 7ff6e238d740 1 --2- 172.21.15.76:0/3673339616 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x561422ccadf0 0x561422ccd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.372+0000 7ff6e238d740 4 client.25442 mds_command: new command op to 24490 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T14:53:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.372+0000 7ff6e238d740 1 -- 172.21.15.76:0/3673339616 --> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7ff6d0009b00 con 0x561422ccadf0 2024-03-18T14:53:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.373+0000 7ff6dcc61640 1 --2- 172.21.15.76:0/3673339616 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x561422ccadf0 0x561422ccd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.373+0000 7ff6c0ff9640 10 client.25442 ms_handle_connect on v2:172.21.15.174:6838/399810652 2024-03-18T14:53:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.373+0000 7ff6dcc61640 1 --2- 172.21.15.76:0/3673339616 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x561422ccadf0 0x561422ccd1e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dc518a646966e108 server_cookie=de1136e900168ea0 in_seq=0 out_seq=0 2024-03-18T14:53:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.373+0000 7ff6c0ff9640 1 -- 172.21.15.76:0/3673339616 <== mds.1 v2:172.21.15.174:6838/399810652 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7ff6d0009b00 con 0x561422ccadf0 2024-03-18T14:53:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.373+0000 7ff6c0ff9640 10 client.25442 handle_command_reply: tid=0 2024-03-18T14:53:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.373+0000 7ff6e238d740 1 client.25442 shutdown 2024-03-18T14:53:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.373+0000 7ff6e238d740 1 -- 172.21.15.76:0/3673339616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 msgr2=0x561422c4f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.373+0000 7ff6e238d740 1 --2- 172.21.15.76:0/3673339616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c4f510 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0011d40 tx=0x7ff6d0004120 comp rx=0 tx=0).stop 2024-03-18T14:53:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6e238d740 1 -- 172.21.15.76:0/3673339616 shutdown_connections 2024-03-18T14:53:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6e238d740 1 --2- 172.21.15.76:0/3673339616 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x561422ccadf0 0x561422ccd1e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6e238d740 1 --2- 172.21.15.76:0/3673339616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561422c4c150 0x561422c4fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6e238d740 1 --2- 172.21.15.76:0/3673339616 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561422c54d30 0x561422c4dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6e238d740 1 --2- 172.21.15.76:0/3673339616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561422c4b830 0x561422c4f510 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6e238d740 1 -- 172.21.15.76:0/3673339616 >> 172.21.15.76:0/3673339616 conn(0x561422c45f60 msgr2=0x561422c44fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6c0ff9640 0 client.25442 ms_handle_reset on v2:172.21.15.174:6838/399810652 2024-03-18T14:53:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6e238d740 1 -- 172.21.15.76:0/3673339616 shutdown_connections 2024-03-18T14:53:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6e238d740 1 -- 172.21.15.76:0/3673339616 wait complete. 2024-03-18T14:53:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6e238d740 20 client.25442 trim_cache size 0 max 16384 2024-03-18T14:53:40.380 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-18T14:53:40.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6df6c6640 1 -- 172.21.15.76:0/2497386754 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6c40a2a80 msgr2=0x7ff6c40a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6df6c6640 1 --2- 172.21.15.76:0/2497386754 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6c40a2a80 0x7ff6c40a4f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff6d40cc6a0 tx=0x7ff6d40096e0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6df6c6640 1 -- 172.21.15.76:0/2497386754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6d80689c0 msgr2=0x7ff6d819a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6df6c6640 1 --2- 172.21.15.76:0/2497386754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6d80689c0 0x7ff6d819a300 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400e9f0 tx=0x7ff6d400eeb0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6df6c6640 1 -- 172.21.15.76:0/2497386754 shutdown_connections 2024-03-18T14:53:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6df6c6640 1 --2- 172.21.15.76:0/2497386754 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6c40a2a80 0x7ff6c40a4f40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6df6c6640 1 --2- 172.21.15.76:0/2497386754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d810e180 0x7ff6d819ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6df6c6640 1 --2- 172.21.15.76:0/2497386754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6d80692e0 0x7ff6d819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6df6c6640 1 --2- 172.21.15.76:0/2497386754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6d80689c0 0x7ff6d819a300 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.374+0000 7ff6df6c6640 1 -- 172.21.15.76:0/2497386754 >> 172.21.15.76:0/2497386754 conn(0x7ff6d80777e0 msgr2=0x7ff6d810f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.375+0000 7ff6df6c6640 1 -- 172.21.15.76:0/2497386754 shutdown_connections 2024-03-18T14:53:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.375+0000 7ff6df6c6640 1 -- 172.21.15.76:0/2497386754 wait complete. 2024-03-18T14:53:40.386 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T14:53:40.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.493+0000 7f7e7a910640 1 -- 172.21.15.76:0/982112198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e74100640 msgr2=0x7f7e7410e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.493+0000 7f7e7a910640 1 --2- 172.21.15.76:0/982112198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e74100640 0x7f7e7410e600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009be0 tx=0x7f7e6c02bef0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.493+0000 7f7e7a910640 1 -- 172.21.15.76:0/982112198 shutdown_connections 2024-03-18T14:53:40.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.493+0000 7f7e7a910640 1 --2- 172.21.15.76:0/982112198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e740ffc80 0x7f7e74100100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.493+0000 7f7e7a910640 1 --2- 172.21.15.76:0/982112198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e74100640 0x7f7e7410e600 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.493+0000 7f7e7a910640 1 --2- 172.21.15.76:0/982112198 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e74107780 0x7f7e74107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.493+0000 7f7e7a910640 1 -- 172.21.15.76:0/982112198 >> 172.21.15.76:0/982112198 conn(0x7f7e74076c00 msgr2=0x7f7e740fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.493+0000 7f7e7a910640 1 -- 172.21.15.76:0/982112198 shutdown_connections 2024-03-18T14:53:40.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.493+0000 7f7e7a910640 1 -- 172.21.15.76:0/982112198 wait complete. 2024-03-18T14:53:40.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.493+0000 7f7e7a910640 1 Processor -- start 2024-03-18T14:53:40.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.493+0000 7f7e7a910640 1 -- start start 2024-03-18T14:53:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e7a910640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e740ffc80 0x7f7e74100e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e7a910640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e74100640 0x7f7e741013a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e7a910640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e74107780 0x7f7e741018e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e7a910640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7e74111b80 con 0x7f7e740ffc80 2024-03-18T14:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e7a910640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7e74111a00 con 0x7f7e74100640 2024-03-18T14:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e7a910640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7e74111d00 con 0x7f7e74107780 2024-03-18T14:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e73fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e740ffc80 0x7f7e74100e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e73fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e740ffc80 0x7f7e74100e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39388/0 (socket says 172.21.15.76:39388) 2024-03-18T14:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e73fff640 1 -- 172.21.15.76:0/2197811347 learned_addr learned my addr 172.21.15.76:0/2197811347 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e737fe640 1 --2- 172.21.15.76:0/2197811347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e74100640 0x7f7e741013a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e78ead640 1 --2- 172.21.15.76:0/2197811347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e74107780 0x7f7e741018e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e73fff640 1 -- 172.21.15.76:0/2197811347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e74107780 msgr2=0x7f7e741018e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e73fff640 1 --2- 172.21.15.76:0/2197811347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e74107780 0x7f7e741018e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e73fff640 1 -- 172.21.15.76:0/2197811347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e74100640 msgr2=0x7f7e741013a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e73fff640 1 --2- 172.21.15.76:0/2197811347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e74100640 0x7f7e741013a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e73fff640 1 -- 172.21.15.76:0/2197811347 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e741a6500 con 0x7f7e740ffc80 2024-03-18T14:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e737fe640 1 --2- 172.21.15.76:0/2197811347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e74100640 0x7f7e741013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.494+0000 7f7e78ead640 1 --2- 172.21.15.76:0/2197811347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e74107780 0x7f7e741018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.495+0000 7f7e73fff640 1 --2- 172.21.15.76:0/2197811347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e740ffc80 0x7f7e74100e60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400e9c0 tx=0x7f7e6400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.495+0000 7f7e717fa640 1 -- 172.21.15.76:0/2197811347 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400ccc0 con 0x7f7e740ffc80 2024-03-18T14:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.495+0000 7f7e717fa640 1 -- 172.21.15.76:0/2197811347 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e64005500 con 0x7f7e740ffc80 2024-03-18T14:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.495+0000 7f7e717fa640 1 -- 172.21.15.76:0/2197811347 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e64010430 con 0x7f7e740ffc80 2024-03-18T14:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.496+0000 7f7e7a910640 1 -- 172.21.15.76:0/2197811347 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e741a67f0 con 0x7f7e740ffc80 2024-03-18T14:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.496+0000 7f7e7a910640 1 -- 172.21.15.76:0/2197811347 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e741a6cc0 con 0x7f7e740ffc80 2024-03-18T14:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.498+0000 7f7e7d5dd740 1 Processor -- start 2024-03-18T14:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.498+0000 7f7e7d5dd740 1 -- start start 2024-03-18T14:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e7d5dd740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 0x558cc6a64c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e7d5dd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a65150 0x558cc6a6d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e7d5dd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a6dd30 0x558cc6a70120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e7d5dd740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x558cc69c8490 con 0x558cc6a64830 2024-03-18T14:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e7d5dd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558cc6970c90 con 0x558cc6a6dd30 2024-03-18T14:53:40.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e7d5dd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558cc69c4530 con 0x558cc6a65150 2024-03-18T14:53:40.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e78ead640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 0x558cc6a64c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e737fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a6dd30 0x558cc6a70120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e78ead640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 0x558cc6a64c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39394/0 (socket says 172.21.15.76:39394) 2024-03-18T14:53:40.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e78ead640 1 -- 172.21.15.76:0/2919096674 learned_addr learned my addr 172.21.15.76:0/2919096674 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:40.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e737fe640 1 -- 172.21.15.76:0/2919096674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a65150 msgr2=0x558cc6a6d7f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T14:53:40.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e717fa640 1 -- 172.21.15.76:0/2197811347 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 23) ==== 137466+0+0 (secure 0 0 0) 0x7f7e640046c0 con 0x7f7e740ffc80 2024-03-18T14:53:40.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e737fe640 1 --2- 172.21.15.76:0/2919096674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a65150 0x558cc6a6d7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e737fe640 1 -- 172.21.15.76:0/2919096674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 msgr2=0x558cc6a64c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e737fe640 1 --2- 172.21.15.76:0/2919096674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 0x558cc6a64c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e737fe640 1 -- 172.21.15.76:0/2919096674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cc6a2b500 con 0x558cc6a6dd30 2024-03-18T14:53:40.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e737fe640 1 --2- 172.21.15.76:0/2919096674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a6dd30 0x558cc6a70120 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009fa0 tx=0x7f7e680118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f1a3b760a0345b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.499+0000 7f7e5a7fc640 1 -- 172.21.15.76:0/2919096674 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e68004030 con 0x558cc6a6dd30 2024-03-18T14:53:40.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.500+0000 7f7e5a7fc640 1 -- 172.21.15.76:0/2919096674 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e68005500 con 0x558cc6a6dd30 2024-03-18T14:53:40.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.500+0000 7f7e5a7fc640 1 -- 172.21.15.76:0/2919096674 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e68004db0 con 0x558cc6a6dd30 2024-03-18T14:53:40.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.500+0000 7f7e717fa640 1 --2- 172.21.15.76:0/2197811347 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7e5c0a2a80 0x7f7e5c0a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.500+0000 7f7e78ead640 1 --2- 172.21.15.76:0/2197811347 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7e5c0a2a80 0x7f7e5c0a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.500+0000 7f7e717fa640 1 -- 172.21.15.76:0/2197811347 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f7e640cf2f0 con 0x7f7e740ffc80 2024-03-18T14:53:40.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.500+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/2919096674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a6dd30 msgr2=0x558cc6a70120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.500+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/2919096674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a6dd30 0x558cc6a70120 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009fa0 tx=0x7f7e680118d0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.501+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/2919096674 shutdown_connections 2024-03-18T14:53:40.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.501+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/2919096674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a65150 0x558cc6a6d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.501+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/2919096674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a6dd30 0x558cc6a70120 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.501+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/2919096674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 0x558cc6a64c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.501+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/2919096674 >> 172.21.15.76:0/2919096674 conn(0x558cc6a5ef60 msgr2=0x558cc6a5f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.501+0000 7f7e78ead640 1 --2- 172.21.15.76:0/2197811347 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7e5c0a2a80 0x7f7e5c0a4f40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0099a0 tx=0x7f7e6c004a40 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.501+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/2919096674 shutdown_connections 2024-03-18T14:53:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.501+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/2919096674 wait complete. 2024-03-18T14:53:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.501+0000 7f7e7d5dd740 1 -- start start 2024-03-18T14:53:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.502+0000 7f7e7d5dd740 1 Processor -- start 2024-03-18T14:53:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.502+0000 7f7e7d5dd740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 0x558cc6a6adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.502+0000 7f7e7d5dd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a65150 0x558cc6a6b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.502+0000 7f7e7d5dd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a6dd30 0x558cc6a6b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.502+0000 7f7e7d5dd740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x558cc6a70b20 con 0x558cc6a64830 2024-03-18T14:53:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.502+0000 7f7e737fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a65150 0x558cc6a6b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.502+0000 7f7e78ead640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a6dd30 0x558cc6a6b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.502+0000 7f7e737fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a65150 0x558cc6a6b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41246/0 (socket says 172.21.15.76:41246) 2024-03-18T14:53:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.502+0000 7f7e78ead640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a6dd30 0x558cc6a6b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:49136/0 (socket says 172.21.15.76:49136) 2024-03-18T14:53:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.502+0000 7f7e737fe640 1 -- 172.21.15.76:0/3456168581 learned_addr learned my addr 172.21.15.76:0/3456168581 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.502+0000 7f7e73fff640 1 --2- 172.21.15.76:0/3456168581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 0x558cc6a6adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.502+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/3456168581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558cc6984230 con 0x558cc6a65150 2024-03-18T14:53:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.502+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/3456168581 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558cc6a70ca0 con 0x558cc6a6dd30 2024-03-18T14:53:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.502+0000 7f7e7d5dd740 10 client.0 _ll_register_callbacks cb 0x558cc69c0010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.503+0000 7f7e78ead640 1 -- 172.21.15.76:0/3456168581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a65150 msgr2=0x558cc6a6b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.503+0000 7f7e78ead640 1 --2- 172.21.15.76:0/3456168581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a65150 0x558cc6a6b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.503+0000 7f7e78ead640 1 -- 172.21.15.76:0/3456168581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 msgr2=0x558cc6a6adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.503+0000 7f7e78ead640 1 --2- 172.21.15.76:0/3456168581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 0x558cc6a6adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.503+0000 7f7e78ead640 1 -- 172.21.15.76:0/3456168581 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cc6993100 con 0x558cc6a6dd30 2024-03-18T14:53:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.503+0000 7f7e73fff640 1 --2- 172.21.15.76:0/3456168581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 0x558cc6a6adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.503+0000 7f7e78ead640 1 --2- 172.21.15.76:0/3456168581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a6dd30 0x558cc6a6b830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c004af0 tx=0x7f7e6c02bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ee0673cc1b26ff5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.503+0000 7f7e537fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-18T14:53:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.503+0000 7f7e537fe640 1 -- 172.21.15.76:0/3456168581 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c051020 con 0x558cc6a6dd30 2024-03-18T14:53:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.503+0000 7f7e537fe640 1 -- 172.21.15.76:0/3456168581 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f7e6c033860 con 0x558cc6a6dd30 2024-03-18T14:53:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.503+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/3456168581 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e68005500 con 0x558cc6a6dd30 2024-03-18T14:53:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.503+0000 7f7e537fe640 1 -- 172.21.15.76:0/3456168581 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7e6c04d480 con 0x558cc6a6dd30 2024-03-18T14:53:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.503+0000 7f7e7d5dd740 10 client.25450 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.504+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/3456168581 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558cc66dc3f0 con 0x558cc6a6dd30 2024-03-18T14:53:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.504+0000 7f7e537fe640 1 -- 172.21.15.76:0/3456168581 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e68005500 con 0x558cc6a6dd30 2024-03-18T14:53:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.504+0000 7f7e537fe640 1 -- 172.21.15.76:0/3456168581 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f7e6c034630 con 0x558cc6a6dd30 2024-03-18T14:53:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.504+0000 7f7e7d5dd740 10 client.25450 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.504+0000 7f7e7d5dd740 10 client.25450 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.504+0000 7f7e7d5dd740 20 client.25450 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.504+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/3456168581 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558cc6a7e160 0x558cc6a80590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.504+0000 7f7e7d5dd740 4 client.25450 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:53:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.504+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/3456168581 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cc69d59d0 con 0x558cc6a7e160 2024-03-18T14:53:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.504+0000 7f7e73fff640 1 --2- 172.21.15.76:0/3456168581 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558cc6a7e160 0x558cc6a80590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.505+0000 7f7e73fff640 1 --2- 172.21.15.76:0/3456168581 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558cc6a7e160 0x558cc6a80590 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff165233a88836d3 server_cookie=1a05930f5fddcd53 in_seq=0 out_seq=0 2024-03-18T14:53:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.505+0000 7f7e537fe640 10 client.25450 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.506+0000 7f7e537fe640 1 -- 172.21.15.76:0/3456168581 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558cc69d59d0 con 0x558cc6a7e160 2024-03-18T14:53:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.506+0000 7f7e537fe640 10 client.25450 handle_command_reply: tid=0 2024-03-18T14:53:40.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.506+0000 7f7e7d5dd740 1 client.25450 shutdown 2024-03-18T14:53:40.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.506+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/3456168581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a6dd30 msgr2=0x558cc6a6b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.506+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/3456168581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a6dd30 0x558cc6a6b830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c004af0 tx=0x7f7e6c02bef0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.506+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/3456168581 shutdown_connections 2024-03-18T14:53:40.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.506+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/3456168581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a6dd30 0x558cc6a6b830 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.506+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/3456168581 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558cc6a7e160 0x558cc6a80590 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.506+0000 7f7e537fe640 0 client.25450 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:40.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.506+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/3456168581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a65150 0x558cc6a6b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.506+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/3456168581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 0x558cc6a6adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.506+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/3456168581 >> 172.21.15.76:0/3456168581 conn(0x558cc6a5ef60 msgr2=0x558cc6a60cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.506+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/3456168581 shutdown_connections 2024-03-18T14:53:40.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.506+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/3456168581 wait complete. 2024-03-18T14:53:40.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.506+0000 7f7e7d5dd740 20 client.25450 trim_cache size 0 max 16384 2024-03-18T14:53:40.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.532+0000 7f7e7d5dd740 1 Processor -- start 2024-03-18T14:53:40.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.532+0000 7f7e7d5dd740 1 -- start start 2024-03-18T14:53:40.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.532+0000 7f7e7d5dd740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 0x558cc6a6d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.532+0000 7f7e7d5dd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a65150 0x558cc6a798f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.532+0000 7f7e7d5dd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a6dd30 0x558cc6a85e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.532+0000 7f7e7d5dd740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7e741b1ec0 con 0x558cc6a64830 2024-03-18T14:53:40.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.532+0000 7f7e7d5dd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7e741af790 con 0x558cc6a65150 2024-03-18T14:53:40.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.532+0000 7f7e7d5dd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7e741ad600 con 0x558cc6a6dd30 2024-03-18T14:53:40.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.533+0000 7f7e73fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 0x558cc6a6d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.533+0000 7f7e73fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 0x558cc6a6d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39410/0 (socket says 172.21.15.76:39410) 2024-03-18T14:53:40.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.533+0000 7f7e73fff640 1 -- 172.21.15.76:0/2297093251 learned_addr learned my addr 172.21.15.76:0/2297093251 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:40.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.533+0000 7f7e78ead640 1 --2- 172.21.15.76:0/2297093251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a6dd30 0x558cc6a85e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.533+0000 7f7e737fe640 1 --2- 172.21.15.76:0/2297093251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a65150 0x558cc6a798f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.533+0000 7f7e78ead640 1 -- 172.21.15.76:0/2297093251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a65150 msgr2=0x558cc6a798f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.533+0000 7f7e78ead640 1 --2- 172.21.15.76:0/2297093251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a65150 0x558cc6a798f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.533+0000 7f7e78ead640 1 -- 172.21.15.76:0/2297093251 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 msgr2=0x558cc6a6d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.533+0000 7f7e78ead640 1 --2- 172.21.15.76:0/2297093251 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 0x558cc6a6d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.533+0000 7f7e78ead640 1 -- 172.21.15.76:0/2297093251 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cc67a67d0 con 0x558cc6a6dd30 2024-03-18T14:53:40.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.533+0000 7f7e78ead640 1 --2- 172.21.15.76:0/2297093251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a6dd30 0x558cc6a85e60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c02c430 tx=0x7f7e6c009ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=236ec4ebecca1637 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.533+0000 7f7e59ffb640 1 -- 172.21.15.76:0/2297093251 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c051070 con 0x558cc6a6dd30 2024-03-18T14:53:40.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.533+0000 7f7e59ffb640 1 -- 172.21.15.76:0/2297093251 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e6c065070 con 0x558cc6a6dd30 2024-03-18T14:53:40.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.533+0000 7f7e59ffb640 1 -- 172.21.15.76:0/2297093251 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c05c030 con 0x558cc6a6dd30 2024-03-18T14:53:40.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.534+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/2297093251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a6dd30 msgr2=0x558cc6a85e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.534+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/2297093251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a6dd30 0x558cc6a85e60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c02c430 tx=0x7f7e6c009ae0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.534+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/2297093251 shutdown_connections 2024-03-18T14:53:40.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.534+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/2297093251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a6dd30 0x558cc6a85e60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.534+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/2297093251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a65150 0x558cc6a798f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.534+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/2297093251 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a64830 0x558cc6a6d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.534+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/2297093251 >> 172.21.15.76:0/2297093251 conn(0x558cc6a5ef60 msgr2=0x558cc6a5d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.534+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/2297093251 shutdown_connections 2024-03-18T14:53:40.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.534+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/2297093251 wait complete. 2024-03-18T14:53:40.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.535+0000 7f7e7d5dd740 1 -- start start 2024-03-18T14:53:40.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.535+0000 7f7e7d5dd740 1 Processor -- start 2024-03-18T14:53:40.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.535+0000 7f7e7d5dd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a64830 0x558cc6a76310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.535+0000 7f7e7d5dd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a65150 0x558cc6a67790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.535+0000 7f7e7d5dd740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a6dd30 0x558cc6a67e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.535+0000 7f7e7d5dd740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7e741adc50 con 0x558cc6a6dd30 2024-03-18T14:53:40.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.535+0000 7f7e7d5dd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7e741af0c0 con 0x558cc6a64830 2024-03-18T14:53:40.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.535+0000 7f7e7d5dd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7e741ae320 con 0x558cc6a65150 2024-03-18T14:53:40.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.535+0000 7f7e7d5dd740 10 client.0 _ll_register_callbacks cb 0x7f7e6c044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:40.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.535+0000 7f7e73fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a64830 0x558cc6a76310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.535+0000 7f7e73fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a64830 0x558cc6a76310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41262/0 (socket says 172.21.15.76:41262) 2024-03-18T14:53:40.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.535+0000 7f7e737fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a65150 0x558cc6a67790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.535+0000 7f7e73fff640 1 -- 172.21.15.76:0/1550983107 learned_addr learned my addr 172.21.15.76:0/1550983107 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:40.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.535+0000 7f7e78ead640 1 --2- 172.21.15.76:0/1550983107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a6dd30 0x558cc6a67e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.536+0000 7f7e737fe640 1 -- 172.21.15.76:0/1550983107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a64830 msgr2=0x558cc6a76310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.536+0000 7f7e737fe640 1 --2- 172.21.15.76:0/1550983107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a64830 0x558cc6a76310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.536+0000 7f7e737fe640 1 -- 172.21.15.76:0/1550983107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a6dd30 msgr2=0x558cc6a67e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:40.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.536+0000 7f7e737fe640 1 --2- 172.21.15.76:0/1550983107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a6dd30 0x558cc6a67e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.536+0000 7f7e737fe640 1 -- 172.21.15.76:0/1550983107 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cc6ae3490 con 0x558cc6a65150 2024-03-18T14:53:40.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.536+0000 7f7e73fff640 1 --2- 172.21.15.76:0/1550983107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a64830 0x558cc6a76310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:40.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.536+0000 7f7e737fe640 1 --2- 172.21.15.76:0/1550983107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a65150 0x558cc6a67790 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7e68000c00 tx=0x7f7e68009c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9dc2cf064a6dadb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:40.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.536+0000 7f7e537fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-18T14:53:40.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.536+0000 7f7e537fe640 1 -- 172.21.15.76:0/1550983107 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6801d040 con 0x558cc6a65150 2024-03-18T14:53:40.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.536+0000 7f7e537fe640 1 -- 172.21.15.76:0/1550983107 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f7e68004390 con 0x558cc6a65150 2024-03-18T14:53:40.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.536+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/1550983107 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e6c065070 con 0x558cc6a65150 2024-03-18T14:53:40.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.536+0000 7f7e537fe640 1 -- 172.21.15.76:0/1550983107 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7e68034050 con 0x558cc6a65150 2024-03-18T14:53:40.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.536+0000 7f7e7d5dd740 10 client.25456 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:40.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.536+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/1550983107 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558cc6ae45b0 con 0x558cc6a65150 2024-03-18T14:53:40.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.537+0000 7f7e537fe640 1 -- 172.21.15.76:0/1550983107 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e6801e040 con 0x558cc6a65150 2024-03-18T14:53:40.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.538+0000 7f7e537fe640 1 -- 172.21.15.76:0/1550983107 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f7e680045b0 con 0x558cc6a65150 2024-03-18T14:53:40.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.538+0000 7f7e7d5dd740 10 client.25456 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:40.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.538+0000 7f7e7d5dd740 10 client.25456 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:40.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.538+0000 7f7e7d5dd740 20 client.25456 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:40.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.538+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/1550983107 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558cc6a7e160 0x558cc6ae6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:40.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.538+0000 7f7e7d5dd740 4 client.25456 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T14:53:40.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.538+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/1550983107 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f7e6c065070 con 0x558cc6a7e160 2024-03-18T14:53:40.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.538+0000 7f7e73fff640 1 --2- 172.21.15.76:0/1550983107 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558cc6a7e160 0x558cc6ae6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:40.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.538+0000 7f7e73fff640 1 --2- 172.21.15.76:0/1550983107 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558cc6a7e160 0x558cc6ae6b30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6cc35553655a872 server_cookie=e6e22fa9fbc9841b in_seq=0 out_seq=0 2024-03-18T14:53:40.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.538+0000 7f7e537fe640 10 client.25456 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:40.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.538+0000 7f7e537fe640 1 -- 172.21.15.76:0/1550983107 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f7e6c065070 con 0x558cc6a7e160 2024-03-18T14:53:40.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.538+0000 7f7e537fe640 10 client.25456 handle_command_reply: tid=0 2024-03-18T14:53:40.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.538+0000 7f7e7d5dd740 1 client.25456 shutdown 2024-03-18T14:53:40.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.539+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/1550983107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a65150 msgr2=0x558cc6a67790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.539+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/1550983107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a65150 0x558cc6a67790 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7e68000c00 tx=0x7f7e68009c90 comp rx=0 tx=0).stop 2024-03-18T14:53:40.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.539+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/1550983107 shutdown_connections 2024-03-18T14:53:40.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.539+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/1550983107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cc6a65150 0x558cc6a67790 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.539+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/1550983107 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558cc6a7e160 0x558cc6ae6b30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.539+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/1550983107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cc6a6dd30 0x558cc6a67e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.539+0000 7f7e7d5dd740 1 --2- 172.21.15.76:0/1550983107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cc6a64830 0x558cc6a76310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.539+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/1550983107 >> 172.21.15.76:0/1550983107 conn(0x558cc6a5ef60 msgr2=0x558cc6a6b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.539+0000 7f7e537fe640 0 client.25456 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:40.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.539+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/1550983107 shutdown_connections 2024-03-18T14:53:40.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.539+0000 7f7e7d5dd740 1 -- 172.21.15.76:0/1550983107 wait complete. 2024-03-18T14:53:40.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.539+0000 7f7e7d5dd740 20 client.25456 trim_cache size 0 max 16384 2024-03-18T14:53:40.544 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-18T14:53:40.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.539+0000 7f7e7a910640 1 -- 172.21.15.76:0/2197811347 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7e5c0a2a80 msgr2=0x7f7e5c0a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.539+0000 7f7e7a910640 1 --2- 172.21.15.76:0/2197811347 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7e5c0a2a80 0x7f7e5c0a4f40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0099a0 tx=0x7f7e6c004a40 comp rx=0 tx=0).stop 2024-03-18T14:53:40.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.539+0000 7f7e7a910640 1 -- 172.21.15.76:0/2197811347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e740ffc80 msgr2=0x7f7e74100e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:40.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.539+0000 7f7e7a910640 1 --2- 172.21.15.76:0/2197811347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e740ffc80 0x7f7e74100e60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400e9c0 tx=0x7f7e6400ee80 comp rx=0 tx=0).stop 2024-03-18T14:53:40.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.540+0000 7f7e7a910640 1 -- 172.21.15.76:0/2197811347 shutdown_connections 2024-03-18T14:53:40.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.540+0000 7f7e7a910640 1 --2- 172.21.15.76:0/2197811347 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7e5c0a2a80 0x7f7e5c0a4f40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.540+0000 7f7e7a910640 1 --2- 172.21.15.76:0/2197811347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e74107780 0x7f7e741018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.540+0000 7f7e7a910640 1 --2- 172.21.15.76:0/2197811347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e74100640 0x7f7e741013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.540+0000 7f7e7a910640 1 --2- 172.21.15.76:0/2197811347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e740ffc80 0x7f7e74100e60 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:40.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.540+0000 7f7e7a910640 1 -- 172.21.15.76:0/2197811347 >> 172.21.15.76:0/2197811347 conn(0x7f7e74076c00 msgr2=0x7f7e740fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:40.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.540+0000 7f7e7a910640 1 -- 172.21.15.76:0/2197811347 shutdown_connections 2024-03-18T14:53:40.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:40.540+0000 7f7e7a910640 1 -- 172.21.15.76:0/2197811347 wait complete. 2024-03-18T14:53:40.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:40 smithi076 ceph-mon[32205]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-18T14:53:40.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:40 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1301051792' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:53:40.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:40 smithi174 ceph-mon[36787]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-18T14:53:40.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:40 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1301051792' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:53:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:40 smithi077 ceph-mon[35865]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-18T14:53:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:40 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1301051792' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:53:41.552 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T14:53:41.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:53:41.552 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T14:53:41.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.660+0000 7f6920f1b640 1 -- 172.21.15.76:0/1080334744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f691c10e180 msgr2=0x7f691c110570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.660+0000 7f6920f1b640 1 --2- 172.21.15.76:0/1080334744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f691c10e180 0x7f691c110570 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6914009be0 tx=0x7f691402bef0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.661+0000 7f6920f1b640 1 -- 172.21.15.76:0/1080334744 shutdown_connections 2024-03-18T14:53:41.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.661+0000 7f6920f1b640 1 --2- 172.21.15.76:0/1080334744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f691c0692e0 0x7f691c10dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.661+0000 7f6920f1b640 1 --2- 172.21.15.76:0/1080334744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f691c10e180 0x7f691c110570 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.661+0000 7f6920f1b640 1 --2- 172.21.15.76:0/1080334744 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f691c0689c0 0x7f691c068da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.661+0000 7f6920f1b640 1 -- 172.21.15.76:0/1080334744 >> 172.21.15.76:0/1080334744 conn(0x7f691c0777e0 msgr2=0x7f691c077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:41.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.661+0000 7f6920f1b640 1 -- 172.21.15.76:0/1080334744 shutdown_connections 2024-03-18T14:53:41.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.661+0000 7f6920f1b640 1 -- 172.21.15.76:0/1080334744 wait complete. 2024-03-18T14:53:41.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.661+0000 7f6920f1b640 1 Processor -- start 2024-03-18T14:53:41.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.661+0000 7f6920f1b640 1 -- start start 2024-03-18T14:53:41.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f6920f1b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f691c0689c0 0x7f691c19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f6920f1b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f691c0692e0 0x7f691c19a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f6920f1b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f691c10e180 0x7f691c19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f6920f1b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f691c113d90 con 0x7f691c0692e0 2024-03-18T14:53:41.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f6920f1b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f691c113c10 con 0x7f691c0689c0 2024-03-18T14:53:41.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f6920f1b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f691c113f10 con 0x7f691c10e180 2024-03-18T14:53:41.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f691a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f691c0689c0 0x7f691c19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f691ad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f691c10e180 0x7f691c19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f691ad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f691c10e180 0x7f691c19eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:49168/0 (socket says 172.21.15.76:49168) 2024-03-18T14:53:41.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f691ad9d640 1 -- 172.21.15.76:0/284299835 learned_addr learned my addr 172.21.15.76:0/284299835 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:41.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f6919d9b640 1 --2- 172.21.15.76:0/284299835 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f691c0692e0 0x7f691c19a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f691a59c640 1 -- 172.21.15.76:0/284299835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f691c10e180 msgr2=0x7f691c19eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f691a59c640 1 --2- 172.21.15.76:0/284299835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f691c10e180 0x7f691c19eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f691a59c640 1 -- 172.21.15.76:0/284299835 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f691c0692e0 msgr2=0x7f691c19a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f691a59c640 1 --2- 172.21.15.76:0/284299835 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f691c0692e0 0x7f691c19a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f691a59c640 1 -- 172.21.15.76:0/284299835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f691c19f260 con 0x7f691c0689c0 2024-03-18T14:53:41.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f691ad9d640 1 --2- 172.21.15.76:0/284299835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f691c10e180 0x7f691c19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:41.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.662+0000 7f6919d9b640 1 --2- 172.21.15.76:0/284299835 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f691c0692e0 0x7f691c19a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:41.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.663+0000 7f691a59c640 1 --2- 172.21.15.76:0/284299835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f691c0689c0 0x7f691c19a270 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f690c005fd0 tx=0x7f690c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:41.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.663+0000 7f690b7fe640 1 -- 172.21.15.76:0/284299835 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690c00cd00 con 0x7f691c0689c0 2024-03-18T14:53:41.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.663+0000 7f690b7fe640 1 -- 172.21.15.76:0/284299835 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f690c0035f0 con 0x7f691c0689c0 2024-03-18T14:53:41.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.663+0000 7f6920f1b640 1 -- 172.21.15.76:0/284299835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f691c19f550 con 0x7f691c0689c0 2024-03-18T14:53:41.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.663+0000 7f690b7fe640 1 -- 172.21.15.76:0/284299835 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690c010670 con 0x7f691c0689c0 2024-03-18T14:53:41.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.663+0000 7f6920f1b640 1 -- 172.21.15.76:0/284299835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f691c1a6de0 con 0x7f691c0689c0 2024-03-18T14:53:41.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.665+0000 7f690b7fe640 1 -- 172.21.15.76:0/284299835 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 23) ==== 137466+0+0 (secure 0 0 0) 0x7f690c0037a0 con 0x7f691c0689c0 2024-03-18T14:53:41.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.666+0000 7f690b7fe640 1 --2- 172.21.15.76:0/284299835 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f69040a2a00 0x7f69040a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.666+0000 7f690b7fe640 1 -- 172.21.15.76:0/284299835 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f690c0cf180 con 0x7f691c0689c0 2024-03-18T14:53:41.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.666+0000 7f6923be3740 1 Processor -- start 2024-03-18T14:53:41.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.666+0000 7f6919d9b640 1 --2- 172.21.15.76:0/284299835 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f69040a2a00 0x7f69040a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f6923be3740 1 -- start start 2024-03-18T14:53:41.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f6919d9b640 1 --2- 172.21.15.76:0/284299835 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f69040a2a00 0x7f69040a4ec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f691c100100 tx=0x7f6910005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:41.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f6923be3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb62710 0x55d10fb61460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f6923be3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb61eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f6923be3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb6ada0 0x55d10fb6d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f6923be3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d10fac50d0 con 0x55d10fb6ada0 2024-03-18T14:53:41.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f6923be3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d10fa6dc70 con 0x55d10fb62710 2024-03-18T14:53:41.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f6923be3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d10fac1170 con 0x55d10fb61a30 2024-03-18T14:53:41.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f691a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb62710 0x55d10fb61460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f691ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb6ada0 0x55d10fb6d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f691a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb62710 0x55d10fb61460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41290/0 (socket says 172.21.15.76:41290) 2024-03-18T14:53:41.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f691a59c640 1 -- 172.21.15.76:0/1182116844 learned_addr learned my addr 172.21.15.76:0/1182116844 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:41.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f6919d9b640 1 --2- 172.21.15.76:0/1182116844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb61eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f6919d9b640 1 -- 172.21.15.76:0/1182116844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb62710 msgr2=0x55d10fb61460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:41.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f6919d9b640 1 --2- 172.21.15.76:0/1182116844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb62710 0x55d10fb61460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f6919d9b640 1 -- 172.21.15.76:0/1182116844 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb6ada0 msgr2=0x55d10fb6d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:41.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f6919d9b640 1 --2- 172.21.15.76:0/1182116844 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb6ada0 0x55d10fb6d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.667+0000 7f6919d9b640 1 -- 172.21.15.76:0/1182116844 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d10f8a17e0 con 0x55d10fb61a30 2024-03-18T14:53:41.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.668+0000 7f691a59c640 1 --2- 172.21.15.76:0/1182116844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb62710 0x55d10fb61460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:41.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.668+0000 7f6919d9b640 1 --2- 172.21.15.76:0/1182116844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb61eb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6910011be0 tx=0x7f6910014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=266ad7c7474ddf3a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:41.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.668+0000 7f6908ff9640 1 -- 172.21.15.76:0/1182116844 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691002b070 con 0x55d10fb61a30 2024-03-18T14:53:41.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.668+0000 7f6908ff9640 1 -- 172.21.15.76:0/1182116844 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f691000ec10 con 0x55d10fb61a30 2024-03-18T14:53:41.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.668+0000 7f6908ff9640 1 -- 172.21.15.76:0/1182116844 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6910026c90 con 0x55d10fb61a30 2024-03-18T14:53:41.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.669+0000 7f6923be3740 1 -- 172.21.15.76:0/1182116844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 msgr2=0x55d10fb61eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.669+0000 7f6923be3740 1 --2- 172.21.15.76:0/1182116844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb61eb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6910011be0 tx=0x7f6910014000 comp rx=0 tx=0).stop 2024-03-18T14:53:41.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.669+0000 7f6923be3740 1 -- 172.21.15.76:0/1182116844 shutdown_connections 2024-03-18T14:53:41.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.669+0000 7f6923be3740 1 --2- 172.21.15.76:0/1182116844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb61eb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.669+0000 7f6923be3740 1 --2- 172.21.15.76:0/1182116844 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb6ada0 0x55d10fb6d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.669+0000 7f6923be3740 1 --2- 172.21.15.76:0/1182116844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb62710 0x55d10fb61460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.669+0000 7f6923be3740 1 -- 172.21.15.76:0/1182116844 >> 172.21.15.76:0/1182116844 conn(0x55d10fb28c50 msgr2=0x55d10fb5c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:41.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.669+0000 7f6923be3740 1 -- 172.21.15.76:0/1182116844 shutdown_connections 2024-03-18T14:53:41.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.669+0000 7f6923be3740 1 -- 172.21.15.76:0/1182116844 wait complete. 2024-03-18T14:53:41.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.669+0000 7f6923be3740 1 -- start start 2024-03-18T14:53:41.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.669+0000 7f6923be3740 1 Processor -- start 2024-03-18T14:53:41.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f6923be3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb67dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f6923be3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb62710 0x55d10fb68310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f6923be3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb6ada0 0x55d10fb68850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f6923be3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d10fb6dc10 con 0x55d10fb6ada0 2024-03-18T14:53:41.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f6923be3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d10fa810d0 con 0x55d10fb62710 2024-03-18T14:53:41.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f6923be3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d10fb6dd90 con 0x55d10fb61a30 2024-03-18T14:53:41.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f6923be3740 10 client.0 _ll_register_callbacks cb 0x55d10fa4a760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:41.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f6919d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb62710 0x55d10fb68310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f691ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb6ada0 0x55d10fb68850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f691a59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb67dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f6919d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb62710 0x55d10fb68310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41298/0 (socket says 172.21.15.76:41298) 2024-03-18T14:53:41.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f6919d9b640 1 -- 172.21.15.76:0/3207392627 learned_addr learned my addr 172.21.15.76:0/3207392627 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:41.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f691a59c640 1 --2- 172.21.15.76:0/3207392627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb67dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:49188/0 (socket says 172.21.15.76:49188) 2024-03-18T14:53:41.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f6919d9b640 1 -- 172.21.15.76:0/3207392627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 msgr2=0x55d10fb67dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:41.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f6919d9b640 1 --2- 172.21.15.76:0/3207392627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb67dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f6919d9b640 1 -- 172.21.15.76:0/3207392627 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb6ada0 msgr2=0x55d10fb68850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:41.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f6919d9b640 1 --2- 172.21.15.76:0/3207392627 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb6ada0 0x55d10fb68850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.670+0000 7f6919d9b640 1 -- 172.21.15.76:0/3207392627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d10f901f70 con 0x55d10fb62710 2024-03-18T14:53:41.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.671+0000 7f6919d9b640 1 --2- 172.21.15.76:0/3207392627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb62710 0x55d10fb68310 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f691001ce00 tx=0x7f691000eed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b652521a53d2135e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:41.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.671+0000 7f6901ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:53:41.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.671+0000 7f6901ffb640 1 -- 172.21.15.76:0/3207392627 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6910034060 con 0x55d10fb62710 2024-03-18T14:53:41.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.671+0000 7f6901ffb640 1 -- 172.21.15.76:0/3207392627 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f6910026b70 con 0x55d10fb62710 2024-03-18T14:53:41.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.671+0000 7f6923be3740 1 -- 172.21.15.76:0/3207392627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f691000ec10 con 0x55d10fb62710 2024-03-18T14:53:41.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.671+0000 7f6901ffb640 1 -- 172.21.15.76:0/3207392627 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f691002b070 con 0x55d10fb62710 2024-03-18T14:53:41.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.671+0000 7f6923be3740 10 client.25472 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:41.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.671+0000 7f6923be3740 1 -- 172.21.15.76:0/3207392627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d10f799170 con 0x55d10fb62710 2024-03-18T14:53:41.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.671+0000 7f6901ffb640 1 -- 172.21.15.76:0/3207392627 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f691000ec10 con 0x55d10fb62710 2024-03-18T14:53:41.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.671+0000 7f6901ffb640 1 -- 172.21.15.76:0/3207392627 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f69100046c0 con 0x55d10fb62710 2024-03-18T14:53:41.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.671+0000 7f6923be3740 10 client.25472 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:41.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.671+0000 7f6923be3740 10 client.25472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:41.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.671+0000 7f6923be3740 20 client.25472 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:41.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.672+0000 7f6923be3740 1 --2- 172.21.15.76:0/3207392627 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55d10fb7ad70 0x55d10fb7d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.672+0000 7f6923be3740 4 client.25472 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:53:41.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.672+0000 7f6923be3740 1 -- 172.21.15.76:0/3207392627 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d10fad2610 con 0x55d10fb7ad70 2024-03-18T14:53:41.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.672+0000 7f691a59c640 1 --2- 172.21.15.76:0/3207392627 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55d10fb7ad70 0x55d10fb7d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.672+0000 7f6901ffb640 10 client.25472 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:41.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.672+0000 7f691a59c640 1 --2- 172.21.15.76:0/3207392627 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55d10fb7ad70 0x55d10fb7d160 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95ee72abdb9d8f0e server_cookie=42adc8adb3ee15a5 in_seq=0 out_seq=0 2024-03-18T14:53:41.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.673+0000 7f6901ffb640 1 -- 172.21.15.76:0/3207392627 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d10fad2610 con 0x55d10fb7ad70 2024-03-18T14:53:41.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.673+0000 7f6901ffb640 10 client.25472 handle_command_reply: tid=0 2024-03-18T14:53:41.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.673+0000 7f6923be3740 1 client.25472 shutdown 2024-03-18T14:53:41.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.673+0000 7f6923be3740 1 -- 172.21.15.76:0/3207392627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb62710 msgr2=0x55d10fb68310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.673+0000 7f6923be3740 1 --2- 172.21.15.76:0/3207392627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb62710 0x55d10fb68310 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f691001ce00 tx=0x7f691000eed0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.673+0000 7f6923be3740 1 -- 172.21.15.76:0/3207392627 shutdown_connections 2024-03-18T14:53:41.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.673+0000 7f6923be3740 1 --2- 172.21.15.76:0/3207392627 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55d10fb7ad70 0x55d10fb7d160 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.673+0000 7f6923be3740 1 --2- 172.21.15.76:0/3207392627 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb6ada0 0x55d10fb68850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.673+0000 7f6923be3740 1 --2- 172.21.15.76:0/3207392627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb62710 0x55d10fb68310 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.673+0000 7f6901ffb640 0 client.25472 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:41.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.673+0000 7f6923be3740 1 --2- 172.21.15.76:0/3207392627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb67dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.673+0000 7f6923be3740 1 -- 172.21.15.76:0/3207392627 >> 172.21.15.76:0/3207392627 conn(0x55d10fb28c50 msgr2=0x55d10fb5e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:41.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.673+0000 7f6923be3740 1 -- 172.21.15.76:0/3207392627 shutdown_connections 2024-03-18T14:53:41.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.673+0000 7f6923be3740 1 -- 172.21.15.76:0/3207392627 wait complete. 2024-03-18T14:53:41.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.673+0000 7f6923be3740 20 client.25472 trim_cache size 0 max 16384 2024-03-18T14:53:41.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f6923be3740 1 Processor -- start 2024-03-18T14:53:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f6923be3740 1 -- start start 2024-03-18T14:53:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f6923be3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb6a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f6923be3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb62710 0x55d10fb5c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f6923be3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb6ada0 0x55d10fb769f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f6923be3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f691c1adbd0 con 0x55d10fb62710 2024-03-18T14:53:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f6923be3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f691c1ab4a0 con 0x55d10fb6ada0 2024-03-18T14:53:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f6923be3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f691c1a92f0 con 0x55d10fb61a30 2024-03-18T14:53:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f691a59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb6a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f691ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb6ada0 0x55d10fb769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f691ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb6ada0 0x55d10fb769f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41312/0 (socket says 172.21.15.76:41312) 2024-03-18T14:53:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f691a59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb6a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:49198/0 (socket says 172.21.15.76:49198) 2024-03-18T14:53:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f691ad9d640 1 -- 172.21.15.76:0/1836694983 learned_addr learned my addr 172.21.15.76:0/1836694983 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:41.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f6919d9b640 1 --2- 172.21.15.76:0/1836694983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb62710 0x55d10fb5c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f691ad9d640 1 -- 172.21.15.76:0/1836694983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 msgr2=0x55d10fb6a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:41.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f691ad9d640 1 --2- 172.21.15.76:0/1836694983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb6a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f691ad9d640 1 -- 172.21.15.76:0/1836694983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb62710 msgr2=0x55d10fb5c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:41.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f691ad9d640 1 --2- 172.21.15.76:0/1836694983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb62710 0x55d10fb5c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.707+0000 7f691ad9d640 1 -- 172.21.15.76:0/1836694983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d10fb66c80 con 0x55d10fb6ada0 2024-03-18T14:53:41.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.708+0000 7f691ad9d640 1 --2- 172.21.15.76:0/1836694983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb6ada0 0x55d10fb769f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6914033420 tx=0x7f6914009280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d0734e594286282 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:41.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.708+0000 7f6903fff640 1 -- 172.21.15.76:0/1836694983 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6914045d80 con 0x55d10fb6ada0 2024-03-18T14:53:41.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.708+0000 7f6903fff640 1 -- 172.21.15.76:0/1836694983 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6914054070 con 0x55d10fb6ada0 2024-03-18T14:53:41.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.708+0000 7f6903fff640 1 -- 172.21.15.76:0/1836694983 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691402cb80 con 0x55d10fb6ada0 2024-03-18T14:53:41.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.709+0000 7f6923be3740 1 -- 172.21.15.76:0/1836694983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb6ada0 msgr2=0x55d10fb769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.709+0000 7f6923be3740 1 --2- 172.21.15.76:0/1836694983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb6ada0 0x55d10fb769f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6914033420 tx=0x7f6914009280 comp rx=0 tx=0).stop 2024-03-18T14:53:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.709+0000 7f6923be3740 1 -- 172.21.15.76:0/1836694983 shutdown_connections 2024-03-18T14:53:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.709+0000 7f6923be3740 1 --2- 172.21.15.76:0/1836694983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb6ada0 0x55d10fb769f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.709+0000 7f6923be3740 1 --2- 172.21.15.76:0/1836694983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb62710 0x55d10fb5c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.709+0000 7f6923be3740 1 --2- 172.21.15.76:0/1836694983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb6a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.709+0000 7f6923be3740 1 -- 172.21.15.76:0/1836694983 >> 172.21.15.76:0/1836694983 conn(0x55d10fb28c50 msgr2=0x55d10fb66570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.710+0000 7f6923be3740 1 -- 172.21.15.76:0/1836694983 shutdown_connections 2024-03-18T14:53:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.710+0000 7f6923be3740 1 -- 172.21.15.76:0/1836694983 wait complete. 2024-03-18T14:53:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.710+0000 7f6923be3740 1 -- start start 2024-03-18T14:53:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.710+0000 7f6923be3740 1 Processor -- start 2024-03-18T14:53:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.710+0000 7f6923be3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb6bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.710+0000 7f6923be3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb62710 0x55d10fb6c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.710+0000 7f6919d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb62710 0x55d10fb6c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f6919d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb62710 0x55d10fb6c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39474/0 (socket says 172.21.15.76:39474) 2024-03-18T14:53:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f6923be3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb6ada0 0x55d10fb71560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f691a59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb6bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f691a59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb6bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:49214/0 (socket says 172.21.15.76:49214) 2024-03-18T14:53:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f6919d9b640 1 -- 172.21.15.76:0/3442329931 learned_addr learned my addr 172.21.15.76:0/3442329931 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f6923be3740 1 -- 172.21.15.76:0/3442329931 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f691c1a9960 con 0x55d10fb62710 2024-03-18T14:53:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f6923be3740 1 -- 172.21.15.76:0/3442329931 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f691c1aadd0 con 0x55d10fb6ada0 2024-03-18T14:53:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f691ad9d640 1 --2- 172.21.15.76:0/3442329931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb6ada0 0x55d10fb71560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f6923be3740 1 -- 172.21.15.76:0/3442329931 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f691c1aa030 con 0x55d10fb61a30 2024-03-18T14:53:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f6923be3740 10 client.0 _ll_register_callbacks cb 0x7f6910029070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f6919d9b640 1 -- 172.21.15.76:0/3442329931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 msgr2=0x55d10fb6bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f6919d9b640 1 --2- 172.21.15.76:0/3442329931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb6bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f6919d9b640 1 -- 172.21.15.76:0/3442329931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb6ada0 msgr2=0x55d10fb71560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f6919d9b640 1 --2- 172.21.15.76:0/3442329931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb6ada0 0x55d10fb71560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f6919d9b640 1 -- 172.21.15.76:0/3442329931 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d10fb71aa0 con 0x55d10fb62710 2024-03-18T14:53:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f691ad9d640 1 --2- 172.21.15.76:0/3442329931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb6ada0 0x55d10fb71560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f691a59c640 1 --2- 172.21.15.76:0/3442329931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb6bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:53:41.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f6919d9b640 1 --2- 172.21.15.76:0/3442329931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb62710 0x55d10fb6c4b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6910004ac0 tx=0x7f6910026f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cad9109fb656024 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:41.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.711+0000 7f6901ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:53:41.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.712+0000 7f6901ffb640 1 -- 172.21.15.76:0/3442329931 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6910020070 con 0x55d10fb62710 2024-03-18T14:53:41.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.712+0000 7f6901ffb640 1 -- 172.21.15.76:0/3442329931 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f6910003940 con 0x55d10fb62710 2024-03-18T14:53:41.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.712+0000 7f6923be3740 1 -- 172.21.15.76:0/3442329931 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6914054070 con 0x55d10fb62710 2024-03-18T14:53:41.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.712+0000 7f6901ffb640 1 -- 172.21.15.76:0/3442329931 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f691004a050 con 0x55d10fb62710 2024-03-18T14:53:41.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.712+0000 7f6923be3740 10 client.15621 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:41.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.712+0000 7f6923be3740 1 -- 172.21.15.76:0/3442329931 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d10fbe1570 con 0x55d10fb62710 2024-03-18T14:53:41.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.712+0000 7f6901ffb640 1 -- 172.21.15.76:0/3442329931 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6910003b80 con 0x55d10fb62710 2024-03-18T14:53:41.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.713+0000 7f6901ffb640 1 -- 172.21.15.76:0/3442329931 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f6910041070 con 0x55d10fb62710 2024-03-18T14:53:41.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.713+0000 7f6923be3740 10 client.15621 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:41.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.713+0000 7f6923be3740 10 client.15621 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:41.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.713+0000 7f6923be3740 20 client.15621 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:41.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.713+0000 7f6923be3740 1 --2- 172.21.15.76:0/3442329931 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55d10fbe1ba0 0x55d10fbe3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.713+0000 7f6923be3740 4 client.15621 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T14:53:41.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.713+0000 7f6923be3740 1 -- 172.21.15.76:0/3442329931 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f6914054070 con 0x55d10fbe1ba0 2024-03-18T14:53:41.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.713+0000 7f691a59c640 1 --2- 172.21.15.76:0/3442329931 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55d10fbe1ba0 0x55d10fbe3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.714+0000 7f691a59c640 1 --2- 172.21.15.76:0/3442329931 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55d10fbe1ba0 0x55d10fbe3fc0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9663f2f89e3bcb23 server_cookie=fffb8bbf373d3c91 in_seq=0 out_seq=0 2024-03-18T14:53:41.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.714+0000 7f6901ffb640 10 client.15621 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:41.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.715+0000 7f6901ffb640 1 -- 172.21.15.76:0/3442329931 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f6914054070 con 0x55d10fbe1ba0 2024-03-18T14:53:41.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.715+0000 7f6901ffb640 10 client.15621 handle_command_reply: tid=0 2024-03-18T14:53:41.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.715+0000 7f6923be3740 1 client.15621 shutdown 2024-03-18T14:53:41.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.715+0000 7f6923be3740 1 -- 172.21.15.76:0/3442329931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb62710 msgr2=0x55d10fb6c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.715+0000 7f6923be3740 1 --2- 172.21.15.76:0/3442329931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb62710 0x55d10fb6c4b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6910004ac0 tx=0x7f6910026f20 comp rx=0 tx=0).stop 2024-03-18T14:53:41.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.715+0000 7f6923be3740 1 -- 172.21.15.76:0/3442329931 shutdown_connections 2024-03-18T14:53:41.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.715+0000 7f6923be3740 1 --2- 172.21.15.76:0/3442329931 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55d10fbe1ba0 0x55d10fbe3fc0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.715+0000 7f6923be3740 1 --2- 172.21.15.76:0/3442329931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d10fb6ada0 0x55d10fb71560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.715+0000 7f6901ffb640 0 client.15621 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:41.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.715+0000 7f6923be3740 1 --2- 172.21.15.76:0/3442329931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d10fb62710 0x55d10fb6c4b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.715+0000 7f6923be3740 1 --2- 172.21.15.76:0/3442329931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d10fb61a30 0x55d10fb6bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.715+0000 7f6923be3740 1 -- 172.21.15.76:0/3442329931 >> 172.21.15.76:0/3442329931 conn(0x55d10fb28c50 msgr2=0x55d10fb67d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:41.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.715+0000 7f6923be3740 1 -- 172.21.15.76:0/3442329931 shutdown_connections 2024-03-18T14:53:41.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.715+0000 7f6923be3740 1 -- 172.21.15.76:0/3442329931 wait complete. 2024-03-18T14:53:41.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.715+0000 7f6923be3740 20 client.15621 trim_cache size 0 max 16384 2024-03-18T14:53:41.721 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T14:53:41.721 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-18T14:53:41.721 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "bfe7c1d3-556c-48fc-a51b-409877ba3ce6", 2024-03-18T14:53:41.721 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-18T14:53:41.721 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T14:53:41.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.716+0000 7f6920f1b640 1 -- 172.21.15.76:0/284299835 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f69040a2a00 msgr2=0x7f69040a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.716+0000 7f6920f1b640 1 --2- 172.21.15.76:0/284299835 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f69040a2a00 0x7f69040a4ec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f691c100100 tx=0x7f6910005f90 comp rx=0 tx=0).stop 2024-03-18T14:53:41.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.716+0000 7f6920f1b640 1 -- 172.21.15.76:0/284299835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f691c0689c0 msgr2=0x7f691c19a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.716+0000 7f6920f1b640 1 --2- 172.21.15.76:0/284299835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f691c0689c0 0x7f691c19a270 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f690c005fd0 tx=0x7f690c00ee50 comp rx=0 tx=0).stop 2024-03-18T14:53:41.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.716+0000 7f6920f1b640 1 -- 172.21.15.76:0/284299835 shutdown_connections 2024-03-18T14:53:41.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.716+0000 7f6920f1b640 1 --2- 172.21.15.76:0/284299835 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f69040a2a00 0x7f69040a4ec0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.716+0000 7f6920f1b640 1 --2- 172.21.15.76:0/284299835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f691c10e180 0x7f691c19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.716+0000 7f6920f1b640 1 --2- 172.21.15.76:0/284299835 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f691c0692e0 0x7f691c19a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.716+0000 7f6920f1b640 1 --2- 172.21.15.76:0/284299835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f691c0689c0 0x7f691c19a270 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.716+0000 7f6920f1b640 1 -- 172.21.15.76:0/284299835 >> 172.21.15.76:0/284299835 conn(0x7f691c0777e0 msgr2=0x7f691c10f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:41.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.716+0000 7f6920f1b640 1 -- 172.21.15.76:0/284299835 shutdown_connections 2024-03-18T14:53:41.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.716+0000 7f6920f1b640 1 -- 172.21.15.76:0/284299835 wait complete. 2024-03-18T14:53:41.731 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T14:53:41.731 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T14:53:41.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.861+0000 7f701b9d4640 1 -- 172.21.15.76:0/2303292344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70140692e0 msgr2=0x7f701410dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.861+0000 7f701b9d4640 1 --2- 172.21.15.76:0/2303292344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70140692e0 0x7f701410dc40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7004009900 tx=0x7f700402bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:41.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.861+0000 7f701b9d4640 1 -- 172.21.15.76:0/2303292344 shutdown_connections 2024-03-18T14:53:41.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.861+0000 7f701b9d4640 1 --2- 172.21.15.76:0/2303292344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f701410e180 0x7f7014110570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.861+0000 7f701b9d4640 1 --2- 172.21.15.76:0/2303292344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70140692e0 0x7f701410dc40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.861+0000 7f701b9d4640 1 --2- 172.21.15.76:0/2303292344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70140689c0 0x7f7014068da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.861+0000 7f701b9d4640 1 -- 172.21.15.76:0/2303292344 >> 172.21.15.76:0/2303292344 conn(0x7f70140777e0 msgr2=0x7f7014077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:41.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.862+0000 7f701b9d4640 1 -- 172.21.15.76:0/2303292344 shutdown_connections 2024-03-18T14:53:41.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.862+0000 7f701b9d4640 1 -- 172.21.15.76:0/2303292344 wait complete. 2024-03-18T14:53:41.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.862+0000 7f701b9d4640 1 Processor -- start 2024-03-18T14:53:41.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.862+0000 7f701b9d4640 1 -- start start 2024-03-18T14:53:41.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f701b9d4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70140689c0 0x7f701419a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f701b9d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70140692e0 0x7f701419a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f701b9d4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f701410e180 0x7f701419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f701b9d4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7014113d40 con 0x7f70140689c0 2024-03-18T14:53:41.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f7018f6f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70140692e0 0x7f701419a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f7019770640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70140689c0 0x7f701419a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f7018f6f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70140692e0 0x7f701419a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41338/0 (socket says 172.21.15.76:41338) 2024-03-18T14:53:41.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f7018f6f640 1 -- 172.21.15.76:0/4013381329 learned_addr learned my addr 172.21.15.76:0/4013381329 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:41.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f701b9d4640 1 -- 172.21.15.76:0/4013381329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7014113bc0 con 0x7f70140692e0 2024-03-18T14:53:41.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f701b9d4640 1 -- 172.21.15.76:0/4013381329 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7014113ec0 con 0x7f701410e180 2024-03-18T14:53:41.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f7019f71640 1 --2- 172.21.15.76:0/4013381329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f701410e180 0x7f701419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f7019770640 1 -- 172.21.15.76:0/4013381329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f701410e180 msgr2=0x7f701419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f7019770640 1 --2- 172.21.15.76:0/4013381329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f701410e180 0x7f701419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f7019770640 1 -- 172.21.15.76:0/4013381329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70140692e0 msgr2=0x7f701419a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f7019770640 1 --2- 172.21.15.76:0/4013381329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70140692e0 0x7f701419a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f7019770640 1 -- 172.21.15.76:0/4013381329 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f701419f2f0 con 0x7f70140689c0 2024-03-18T14:53:41.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f7018f6f640 1 --2- 172.21.15.76:0/4013381329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70140692e0 0x7f701419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:41.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.863+0000 7f7019f71640 1 --2- 172.21.15.76:0/4013381329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f701410e180 0x7f701419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:41.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.864+0000 7f7019770640 1 --2- 172.21.15.76:0/4013381329 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70140689c0 0x7f701419a300 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f701000e9f0 tx=0x7f701000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:41.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.864+0000 7f700a7fc640 1 -- 172.21.15.76:0/4013381329 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701000ccc0 con 0x7f70140689c0 2024-03-18T14:53:41.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.864+0000 7f701b9d4640 1 -- 172.21.15.76:0/4013381329 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f701419f5e0 con 0x7f70140689c0 2024-03-18T14:53:41.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.864+0000 7f701b9d4640 1 -- 172.21.15.76:0/4013381329 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70141a6e70 con 0x7f70140689c0 2024-03-18T14:53:41.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.864+0000 7f700a7fc640 1 -- 172.21.15.76:0/4013381329 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7010005020 con 0x7f70140689c0 2024-03-18T14:53:41.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.865+0000 7f700a7fc640 1 -- 172.21.15.76:0/4013381329 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70100049e0 con 0x7f70140689c0 2024-03-18T14:53:41.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.867+0000 7f700a7fc640 1 -- 172.21.15.76:0/4013381329 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 23) ==== 137466+0+0 (secure 0 0 0) 0x7f7010004cc0 con 0x7f70140689c0 2024-03-18T14:53:41.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.868+0000 7f701e5c0740 1 Processor -- start 2024-03-18T14:53:41.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.868+0000 7f701e5c0740 1 -- start start 2024-03-18T14:53:41.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.868+0000 7f701e5c0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1c150 0x55f557e1c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.868+0000 7f700a7fc640 1 --2- 172.21.15.76:0/4013381329 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f70000a2a80 0x7f70000a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.868+0000 7f701e5c0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e1ca70 0x55f557e25110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.868+0000 7f701e5c0740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e25650 0x55f557e27a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.868+0000 7f7019f71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1c150 0x55f557e1c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.868+0000 7f7019f71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1c150 0x55f557e1c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41342/0 (socket says 172.21.15.76:41342) 2024-03-18T14:53:41.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.868+0000 7f7019f71640 1 -- 172.21.15.76:0/111460116 learned_addr learned my addr 172.21.15.76:0/111460116 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:41.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.868+0000 7f701e5c0740 1 -- 172.21.15.76:0/111460116 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f557d80500 con 0x55f557e25650 2024-03-18T14:53:41.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.868+0000 7f701e5c0740 1 -- 172.21.15.76:0/111460116 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f557d28be0 con 0x55f557e1c150 2024-03-18T14:53:41.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.868+0000 7f701e5c0740 1 -- 172.21.15.76:0/111460116 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f557d7c5a0 con 0x55f557e1ca70 2024-03-18T14:53:41.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.868+0000 7f7018f6f640 1 --2- 172.21.15.76:0/4013381329 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f70000a2a80 0x7f70000a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.868+0000 7f7019770640 1 --2- 172.21.15.76:0/111460116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e1ca70 0x55f557e25110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.868+0000 7f7019f71640 1 --2- 172.21.15.76:0/111460116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e25650 0x55f557e27a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.869+0000 7f7019f71640 1 -- 172.21.15.76:0/111460116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e1ca70 msgr2=0x55f557e25110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.869+0000 7f7019f71640 1 --2- 172.21.15.76:0/111460116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e1ca70 0x55f557e25110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.869+0000 7f7019f71640 1 -- 172.21.15.76:0/111460116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e25650 msgr2=0x55f557e27a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.869+0000 7f7019f71640 1 --2- 172.21.15.76:0/111460116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e25650 0x55f557e27a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.869+0000 7f7019f71640 1 -- 172.21.15.76:0/111460116 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f557d3c4b0 con 0x55f557e1c150 2024-03-18T14:53:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.869+0000 7f700a7fc640 1 -- 172.21.15.76:0/4013381329 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f70100cee10 con 0x7f70140689c0 2024-03-18T14:53:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.869+0000 7f7019f71640 1 --2- 172.21.15.76:0/111460116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1c150 0x55f557e1c530 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f700c011e70 tx=0x7f700c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a52067e5508aedc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.869+0000 7f7018f6f640 1 --2- 172.21.15.76:0/4013381329 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f70000a2a80 0x7f70000a4f40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f701410ee80 tx=0x7f70040058d0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.869+0000 7f6fff7fe640 1 -- 172.21.15.76:0/111460116 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f700c00fc50 con 0x55f557e1c150 2024-03-18T14:53:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.869+0000 7f6fff7fe640 1 -- 172.21.15.76:0/111460116 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f700c003e50 con 0x55f557e1c150 2024-03-18T14:53:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.869+0000 7f6fff7fe640 1 -- 172.21.15.76:0/111460116 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f700c0044e0 con 0x55f557e1c150 2024-03-18T14:53:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.870+0000 7f701e5c0740 1 -- 172.21.15.76:0/111460116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1c150 msgr2=0x55f557e1c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.870+0000 7f701e5c0740 1 --2- 172.21.15.76:0/111460116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1c150 0x55f557e1c530 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f700c011e70 tx=0x7f700c004ab0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.870+0000 7f701e5c0740 1 -- 172.21.15.76:0/111460116 shutdown_connections 2024-03-18T14:53:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.870+0000 7f701e5c0740 1 --2- 172.21.15.76:0/111460116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e1ca70 0x55f557e25110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.870+0000 7f701e5c0740 1 --2- 172.21.15.76:0/111460116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e25650 0x55f557e27a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.870+0000 7f701e5c0740 1 --2- 172.21.15.76:0/111460116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1c150 0x55f557e1c530 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.870+0000 7f701e5c0740 1 -- 172.21.15.76:0/111460116 >> 172.21.15.76:0/111460116 conn(0x55f557e16880 msgr2=0x55f557e16c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.870+0000 7f701e5c0740 1 -- 172.21.15.76:0/111460116 shutdown_connections 2024-03-18T14:53:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.870+0000 7f701e5c0740 1 -- 172.21.15.76:0/111460116 wait complete. 2024-03-18T14:53:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.870+0000 7f701e5c0740 1 -- start start 2024-03-18T14:53:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.870+0000 7f701e5c0740 1 Processor -- start 2024-03-18T14:53:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f701e5c0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e1c150 0x55f557e225a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f701e5c0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1ca70 0x55f557e22ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f701e5c0740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e25650 0x55f557e23020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f701e5c0740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f557e28440 con 0x55f557e25650 2024-03-18T14:53:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f701e5c0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f557d3c160 con 0x55f557e1ca70 2024-03-18T14:53:41.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f701e5c0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f557e285c0 con 0x55f557e1c150 2024-03-18T14:53:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f701e5c0740 10 client.0 _ll_register_callbacks cb 0x55f557d20150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f7019770640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e1c150 0x55f557e225a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f7019770640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e1c150 0x55f557e225a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:49270/0 (socket says 172.21.15.76:49270) 2024-03-18T14:53:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f7019770640 1 -- 172.21.15.76:0/2756512741 learned_addr learned my addr 172.21.15.76:0/2756512741 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f7018f6f640 1 --2- 172.21.15.76:0/2756512741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1ca70 0x55f557e22ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f7019f71640 1 --2- 172.21.15.76:0/2756512741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e25650 0x55f557e23020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f7019770640 1 -- 172.21.15.76:0/2756512741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1ca70 msgr2=0x55f557e22ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f7019770640 1 --2- 172.21.15.76:0/2756512741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1ca70 0x55f557e22ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f7019770640 1 -- 172.21.15.76:0/2756512741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e25650 msgr2=0x55f557e23020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f7019770640 1 --2- 172.21.15.76:0/2756512741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e25650 0x55f557e23020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f7019770640 1 -- 172.21.15.76:0/2756512741 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f557b5e7d0 con 0x55f557e1c150 2024-03-18T14:53:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f7018f6f640 1 --2- 172.21.15.76:0/2756512741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1ca70 0x55f557e22ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:53:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.871+0000 7f7019f71640 1 --2- 172.21.15.76:0/2756512741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e25650 0x55f557e23020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:53:41.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.872+0000 7f7019770640 1 --2- 172.21.15.76:0/2756512741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e1c150 0x55f557e225a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f701009b230 tx=0x7f701009ad80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d50ef0ebbb9dd341 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.872+0000 7f6ffcff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-18T14:53:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.872+0000 7f6ffcff9640 1 -- 172.21.15.76:0/2756512741 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701001c070 con 0x55f557e1c150 2024-03-18T14:53:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.872+0000 7f701e5c0740 1 -- 172.21.15.76:0/2756512741 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f700c003e50 con 0x55f557e1c150 2024-03-18T14:53:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.872+0000 7f6ffcff9640 1 -- 172.21.15.76:0/2756512741 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f70100995f0 con 0x55f557e1c150 2024-03-18T14:53:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.872+0000 7f6ffcff9640 1 -- 172.21.15.76:0/2756512741 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7010095e60 con 0x55f557e1c150 2024-03-18T14:53:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.872+0000 7f701e5c0740 10 client.25486 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.872+0000 7f701e5c0740 1 -- 172.21.15.76:0/2756512741 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f557a943f0 con 0x55f557e1c150 2024-03-18T14:53:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.872+0000 7f6ffcff9640 1 -- 172.21.15.76:0/2756512741 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70100a3a90 con 0x55f557e1c150 2024-03-18T14:53:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.872+0000 7f6ffcff9640 1 -- 172.21.15.76:0/2756512741 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f70100a3600 con 0x55f557e1c150 2024-03-18T14:53:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.873+0000 7f701e5c0740 10 client.25486 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.873+0000 7f701e5c0740 10 client.25486 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.873+0000 7f701e5c0740 20 client.25486 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.873+0000 7f701e5c0740 1 --2- 172.21.15.76:0/2756512741 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f557e35560 0x55f557e37950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.873+0000 7f701e5c0740 4 client.25486 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:53:41.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.873+0000 7f701e5c0740 1 -- 172.21.15.76:0/2756512741 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f700c003e50 con 0x55f557e35560 2024-03-18T14:53:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.873+0000 7f7018f6f640 1 --2- 172.21.15.76:0/2756512741 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f557e35560 0x55f557e37950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.873+0000 7f7018f6f640 1 --2- 172.21.15.76:0/2756512741 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f557e35560 0x55f557e37950 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c33dda8a96b9644b server_cookie=960d21fc18326e0 in_seq=0 out_seq=0 2024-03-18T14:53:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.873+0000 7f6ffcff9640 10 client.25486 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.874+0000 7f6ffcff9640 1 -- 172.21.15.76:0/2756512741 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f700c003e50 con 0x55f557e35560 2024-03-18T14:53:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.874+0000 7f6ffcff9640 10 client.25486 handle_command_reply: tid=0 2024-03-18T14:53:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.874+0000 7f701e5c0740 1 client.25486 shutdown 2024-03-18T14:53:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.874+0000 7f701e5c0740 1 -- 172.21.15.76:0/2756512741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e1c150 msgr2=0x55f557e225a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.874+0000 7f701e5c0740 1 --2- 172.21.15.76:0/2756512741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e1c150 0x55f557e225a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f701009b230 tx=0x7f701009ad80 comp rx=0 tx=0).stop 2024-03-18T14:53:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.874+0000 7f701e5c0740 1 -- 172.21.15.76:0/2756512741 shutdown_connections 2024-03-18T14:53:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.874+0000 7f701e5c0740 1 --2- 172.21.15.76:0/2756512741 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f557e35560 0x55f557e37950 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.874+0000 7f701e5c0740 1 --2- 172.21.15.76:0/2756512741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e25650 0x55f557e23020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.874+0000 7f701e5c0740 1 --2- 172.21.15.76:0/2756512741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1ca70 0x55f557e22ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.874+0000 7f6ffcff9640 0 client.25486 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.874+0000 7f701e5c0740 1 --2- 172.21.15.76:0/2756512741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e1c150 0x55f557e225a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.874+0000 7f701e5c0740 1 -- 172.21.15.76:0/2756512741 >> 172.21.15.76:0/2756512741 conn(0x55f557e16880 msgr2=0x55f557e185d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:41.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.874+0000 7f701e5c0740 1 -- 172.21.15.76:0/2756512741 shutdown_connections 2024-03-18T14:53:41.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.874+0000 7f701e5c0740 1 -- 172.21.15.76:0/2756512741 wait complete. 2024-03-18T14:53:41.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.874+0000 7f701e5c0740 20 client.25486 trim_cache size 0 max 16384 2024-03-18T14:53:41.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.891+0000 7f701e5c0740 1 Processor -- start 2024-03-18T14:53:41.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.891+0000 7f701e5c0740 1 -- start start 2024-03-18T14:53:41.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f701e5c0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1c150 0x55f557e215d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f701e5c0740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e1ca70 0x55f557e31240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f701e5c0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e25650 0x55f557e3d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f701e5c0740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff8010eb0 con 0x55f557e1ca70 2024-03-18T14:53:41.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f701e5c0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff800e780 con 0x55f557e1c150 2024-03-18T14:53:41.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f701e5c0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff800c5d0 con 0x55f557e25650 2024-03-18T14:53:41.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f7019770640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1c150 0x55f557e215d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f7018f6f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e1ca70 0x55f557e31240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f7019770640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1c150 0x55f557e215d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41362/0 (socket says 172.21.15.76:41362) 2024-03-18T14:53:41.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f7019770640 1 -- 172.21.15.76:0/2667101591 learned_addr learned my addr 172.21.15.76:0/2667101591 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:41.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f7019f71640 1 --2- 172.21.15.76:0/2667101591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e25650 0x55f557e3d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f7019770640 1 -- 172.21.15.76:0/2667101591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e25650 msgr2=0x55f557e3d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:41.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f7019770640 1 --2- 172.21.15.76:0/2667101591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e25650 0x55f557e3d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f7019770640 1 -- 172.21.15.76:0/2667101591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e1ca70 msgr2=0x55f557e31240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:41.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f7019770640 1 --2- 172.21.15.76:0/2667101591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e1ca70 0x55f557e31240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f7019770640 1 -- 172.21.15.76:0/2667101591 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f557e1c8a0 con 0x55f557e1c150 2024-03-18T14:53:41.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f7019770640 1 --2- 172.21.15.76:0/2667101591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1c150 0x55f557e215d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f701000e9c0 tx=0x7f701009f500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f983cf5693370494 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:41.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f6ffeffd640 1 -- 172.21.15.76:0/2667101591 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701007c070 con 0x55f557e1c150 2024-03-18T14:53:41.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f6ffeffd640 1 -- 172.21.15.76:0/2667101591 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70100cc9d0 con 0x55f557e1c150 2024-03-18T14:53:41.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.892+0000 7f6ffeffd640 1 -- 172.21.15.76:0/2667101591 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701001c070 con 0x55f557e1c150 2024-03-18T14:53:41.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.893+0000 7f701e5c0740 1 -- 172.21.15.76:0/2667101591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1c150 msgr2=0x55f557e215d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.893+0000 7f701e5c0740 1 --2- 172.21.15.76:0/2667101591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1c150 0x55f557e215d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f701000e9c0 tx=0x7f701009f500 comp rx=0 tx=0).stop 2024-03-18T14:53:41.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.893+0000 7f701e5c0740 1 -- 172.21.15.76:0/2667101591 shutdown_connections 2024-03-18T14:53:41.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.893+0000 7f701e5c0740 1 --2- 172.21.15.76:0/2667101591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e25650 0x55f557e3d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.893+0000 7f701e5c0740 1 --2- 172.21.15.76:0/2667101591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e1ca70 0x55f557e31240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.893+0000 7f701e5c0740 1 --2- 172.21.15.76:0/2667101591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1c150 0x55f557e215d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.893+0000 7f701e5c0740 1 -- 172.21.15.76:0/2667101591 >> 172.21.15.76:0/2667101591 conn(0x55f557e16880 msgr2=0x55f557e23cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:41.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.893+0000 7f701e5c0740 1 -- 172.21.15.76:0/2667101591 shutdown_connections 2024-03-18T14:53:41.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.893+0000 7f701e5c0740 1 -- 172.21.15.76:0/2667101591 wait complete. 2024-03-18T14:53:41.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.893+0000 7f701e5c0740 1 -- start start 2024-03-18T14:53:41.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.894+0000 7f701e5c0740 1 Processor -- start 2024-03-18T14:53:41.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.894+0000 7f701e5c0740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e1c150 0x55f557e2d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.894+0000 7f701e5c0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1ca70 0x55f557e2d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.894+0000 7f701e5c0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e25650 0x55f557e29880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.894+0000 7f7019770640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e1c150 0x55f557e2d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.894+0000 7f701e5c0740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff800cc40 con 0x55f557e1c150 2024-03-18T14:53:41.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.894+0000 7f7019770640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e1c150 0x55f557e2d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39538/0 (socket says 172.21.15.76:39538) 2024-03-18T14:53:41.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.894+0000 7f7019770640 1 -- 172.21.15.76:0/3613747242 learned_addr learned my addr 172.21.15.76:0/3613747242 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:41.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.894+0000 7f701e5c0740 1 -- 172.21.15.76:0/3613747242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff800e0b0 con 0x55f557e1ca70 2024-03-18T14:53:41.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.894+0000 7f701e5c0740 1 -- 172.21.15.76:0/3613747242 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff800d310 con 0x55f557e25650 2024-03-18T14:53:41.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.894+0000 7f7018f6f640 1 --2- 172.21.15.76:0/3613747242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1ca70 0x55f557e2d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.894+0000 7f7019f71640 1 --2- 172.21.15.76:0/3613747242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e25650 0x55f557e29880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.894+0000 7f701e5c0740 10 client.0 _ll_register_callbacks cb 0x7f70100046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:41.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.894+0000 7f7019770640 1 -- 172.21.15.76:0/3613747242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e25650 msgr2=0x55f557e29880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:41.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.894+0000 7f7019770640 1 --2- 172.21.15.76:0/3613747242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e25650 0x55f557e29880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f7019770640 1 -- 172.21.15.76:0/3613747242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1ca70 msgr2=0x55f557e2d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:41.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f7019770640 1 --2- 172.21.15.76:0/3613747242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1ca70 0x55f557e2d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f7019770640 1 -- 172.21.15.76:0/3613747242 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f557a540f0 con 0x55f557e1c150 2024-03-18T14:53:41.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f7018f6f640 1 --2- 172.21.15.76:0/3613747242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1ca70 0x55f557e2d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:41.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f7019f71640 1 --2- 172.21.15.76:0/3613747242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e25650 0x55f557e29880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:53:41.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f701e5c0740 1 -- 172.21.15.76:0/3613747242 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70100cc9d0 con 0x55f557e1c150 2024-03-18T14:53:41.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f7019770640 1 --2- 172.21.15.76:0/3613747242 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e1c150 0x55f557e2d3b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f70100995a0 tx=0x7f70100a39c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c101e245c381f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:41.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f6ffcff9640 10 client.15639 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:53:41.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f6ffcff9640 1 -- 172.21.15.76:0/3613747242 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701001c050 con 0x55f557e1c150 2024-03-18T14:53:41.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f6ffcff9640 1 -- 172.21.15.76:0/3613747242 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f70100a3600 con 0x55f557e1c150 2024-03-18T14:53:41.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f6ffcff9640 1 -- 172.21.15.76:0/3613747242 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f701007c070 con 0x55f557e1c150 2024-03-18T14:53:41.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f701e5c0740 10 client.15639 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:41.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f701e5c0740 1 -- 172.21.15.76:0/3613747242 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({fsmap=24}) -- 0x55f557d479d0 con 0x55f557e1c150 2024-03-18T14:53:41.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f6ffcff9640 1 -- 172.21.15.76:0/3613747242 <== mon.0 v2:172.21.15.76:3300/0 4 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f70100a3ce0 con 0x55f557e1c150 2024-03-18T14:53:41.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f701e5c0740 10 client.15639 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:41.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f701e5c0740 10 client.15639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:41.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f701e5c0740 20 client.15639 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:41.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.895+0000 7f701e5c0740 1 --2- 172.21.15.76:0/3613747242 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f557e9c160 0x55f557e9e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:41.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f701e5c0740 4 client.15639 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T14:53:41.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f701e5c0740 1 -- 172.21.15.76:0/3613747242 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f557e21c50 con 0x55f557e9c160 2024-03-18T14:53:41.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f7018f6f640 1 --2- 172.21.15.76:0/3613747242 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f557e9c160 0x55f557e9e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:41.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f6ffcff9640 10 client.15639 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:41.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f7018f6f640 1 --2- 172.21.15.76:0/3613747242 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f557e9c160 0x55f557e9e550 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52904db2b57a2862 server_cookie=fc35afc6e07b795d in_seq=0 out_seq=0 2024-03-18T14:53:41.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f6ffcff9640 1 -- 172.21.15.76:0/3613747242 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f557e21c50 con 0x55f557e9c160 2024-03-18T14:53:41.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f6ffcff9640 10 client.15639 handle_command_reply: tid=0 2024-03-18T14:53:41.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f701e5c0740 1 client.15639 shutdown 2024-03-18T14:53:41.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f701e5c0740 1 -- 172.21.15.76:0/3613747242 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e1c150 msgr2=0x55f557e2d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f701e5c0740 1 --2- 172.21.15.76:0/3613747242 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e1c150 0x55f557e2d3b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f70100995a0 tx=0x7f70100a39c0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f701e5c0740 1 -- 172.21.15.76:0/3613747242 shutdown_connections 2024-03-18T14:53:41.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f701e5c0740 1 --2- 172.21.15.76:0/3613747242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f557e25650 0x55f557e29880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f701e5c0740 1 --2- 172.21.15.76:0/3613747242 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f557e9c160 0x55f557e9e550 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f701e5c0740 1 --2- 172.21.15.76:0/3613747242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f557e1ca70 0x55f557e2d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f701e5c0740 1 --2- 172.21.15.76:0/3613747242 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f557e1c150 0x55f557e2d3b0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f701e5c0740 1 -- 172.21.15.76:0/3613747242 >> 172.21.15.76:0/3613747242 conn(0x55f557e16880 msgr2=0x55f557e21450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:41.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f6ffcff9640 0 client.15639 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:41.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.896+0000 7f701e5c0740 1 -- 172.21.15.76:0/3613747242 shutdown_connections 2024-03-18T14:53:41.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.897+0000 7f701e5c0740 1 -- 172.21.15.76:0/3613747242 wait complete. 2024-03-18T14:53:41.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.897+0000 7f701e5c0740 20 client.15639 trim_cache size 0 max 16384 2024-03-18T14:53:41.904 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T14:53:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T14:53:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-18T14:53:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: "bfe7c1d3-556c-48fc-a51b-409877ba3ce6": { 2024-03-18T14:53:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-18T14:53:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "bfe7c1d3-556c-48fc-a51b-409877ba3ce6", 2024-03-18T14:53:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-18T14:53:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T14:53:41.905 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T14:53:41.905 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T14:53:41.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.897+0000 7f701b9d4640 1 -- 172.21.15.76:0/4013381329 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f70000a2a80 msgr2=0x7f70000a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.897+0000 7f701b9d4640 1 --2- 172.21.15.76:0/4013381329 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f70000a2a80 0x7f70000a4f40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f701410ee80 tx=0x7f70040058d0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.897+0000 7f701b9d4640 1 -- 172.21.15.76:0/4013381329 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70140689c0 msgr2=0x7f701419a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:41.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.897+0000 7f701b9d4640 1 --2- 172.21.15.76:0/4013381329 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70140689c0 0x7f701419a300 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f701000e9f0 tx=0x7f701000eeb0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.897+0000 7f701b9d4640 1 -- 172.21.15.76:0/4013381329 shutdown_connections 2024-03-18T14:53:41.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.897+0000 7f701b9d4640 1 --2- 172.21.15.76:0/4013381329 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f70000a2a80 0x7f70000a4f40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.897+0000 7f701b9d4640 1 --2- 172.21.15.76:0/4013381329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f701410e180 0x7f701419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.897+0000 7f701b9d4640 1 --2- 172.21.15.76:0/4013381329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70140692e0 0x7f701419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.897+0000 7f701b9d4640 1 --2- 172.21.15.76:0/4013381329 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70140689c0 0x7f701419a300 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:41.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.897+0000 7f701b9d4640 1 -- 172.21.15.76:0/4013381329 >> 172.21.15.76:0/4013381329 conn(0x7f70140777e0 msgr2=0x7f701410f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:41.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.897+0000 7f701b9d4640 1 -- 172.21.15.76:0/4013381329 shutdown_connections 2024-03-18T14:53:41.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:53:41.897+0000 7f701b9d4640 1 -- 172.21.15.76:0/4013381329 wait complete. 2024-03-18T14:53:41.909 INFO:tasks.cephfs.filesystem:scrub status for tag:bfe7c1d3-556c-48fc-a51b-409877ba3ce6 - {'path': '/', 'tag': 'bfe7c1d3-556c-48fc-a51b-409877ba3ce6', 'options': 'recursive,force'} 2024-03-18T14:53:42.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:42 smithi076 ceph-mon[32205]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-18T14:53:42.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:42 smithi174 ceph-mon[36787]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-18T14:53:42.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:42 smithi077 ceph-mon[35865]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-18T14:53:43.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:43 smithi076 ceph-mon[32205]: scrub queued for path: / 2024-03-18T14:53:43.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:43 smithi076 ceph-mon[32205]: scrub summary: idle+waiting paths [/] 2024-03-18T14:53:43.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:43 smithi076 ceph-mon[32205]: scrub summary: active paths [/] 2024-03-18T14:53:43.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:43 smithi076 ceph-mon[32205]: scrub summary: idle+waiting paths [/] 2024-03-18T14:53:43.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:43 smithi076 ceph-mon[32205]: scrub summary: idle 2024-03-18T14:53:43.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:43 smithi174 ceph-mon[36787]: scrub queued for path: / 2024-03-18T14:53:43.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:43 smithi174 ceph-mon[36787]: scrub summary: idle+waiting paths [/] 2024-03-18T14:53:43.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:43 smithi174 ceph-mon[36787]: scrub summary: active paths [/] 2024-03-18T14:53:43.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:43 smithi174 ceph-mon[36787]: scrub summary: idle+waiting paths [/] 2024-03-18T14:53:43.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:43 smithi174 ceph-mon[36787]: scrub summary: idle 2024-03-18T14:53:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:43 smithi077 ceph-mon[35865]: scrub queued for path: / 2024-03-18T14:53:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:43 smithi077 ceph-mon[35865]: scrub summary: idle+waiting paths [/] 2024-03-18T14:53:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:43 smithi077 ceph-mon[35865]: scrub summary: active paths [/] 2024-03-18T14:53:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:43 smithi077 ceph-mon[35865]: scrub summary: idle+waiting paths [/] 2024-03-18T14:53:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:43 smithi077 ceph-mon[35865]: scrub summary: idle 2024-03-18T14:53:44.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:44 smithi076 ceph-mon[32205]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-03-18T14:53:44.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:44 smithi174 ceph-mon[36787]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-03-18T14:53:44.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:44 smithi077 ceph-mon[35865]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-03-18T14:53:45.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:45 smithi076 ceph-mon[32205]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-18T14:53:45.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:45 smithi174 ceph-mon[36787]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-18T14:53:45.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:45 smithi077 ceph-mon[35865]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-18T14:53:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.158+0000 7f09f159a640 1 -- 172.21.15.76:0/540960936 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e4005220 con 0x7f09f41077b0 2024-03-18T14:53:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.158+0000 7f09f8bd4640 1 -- 172.21.15.76:0/540960936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09f41077b0 msgr2=0x7f09f4107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.158+0000 7f09f8bd4640 1 --2- 172.21.15.76:0/540960936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09f41077b0 0x7f09f4107b90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f09e4009930 tx=0x7f09e402bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.158+0000 7f09f8bd4640 1 -- 172.21.15.76:0/540960936 shutdown_connections 2024-03-18T14:53:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.158+0000 7f09f8bd4640 1 --2- 172.21.15.76:0/540960936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09f4069820 0x7f09f410e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.158+0000 7f09f8bd4640 1 --2- 172.21.15.76:0/540960936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09f4068e60 0x7f09f40692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.158+0000 7f09f8bd4640 1 --2- 172.21.15.76:0/540960936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09f41077b0 0x7f09f4107b90 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.158+0000 7f09f8bd4640 1 -- 172.21.15.76:0/540960936 >> 172.21.15.76:0/540960936 conn(0x7f09f4077530 msgr2=0x7f09f4077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.158+0000 7f09f8bd4640 1 -- 172.21.15.76:0/540960936 shutdown_connections 2024-03-18T14:53:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.158+0000 7f09f8bd4640 1 -- 172.21.15.76:0/540960936 wait complete. 2024-03-18T14:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.159+0000 7f09f8bd4640 1 Processor -- start 2024-03-18T14:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.159+0000 7f09f8bd4640 1 -- start start 2024-03-18T14:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.159+0000 7f09f8bd4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09f4068e60 0x7f09f4197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.159+0000 7f09f8bd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09f4069820 0x7f09f41984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.159+0000 7f09f8bd4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09f41077b0 0x7f09f419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.160+0000 7f09f259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09f4068e60 0x7f09f4197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.160+0000 7f09f259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09f4068e60 0x7f09f4197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:49308/0 (socket says 172.21.15.76:49308) 2024-03-18T14:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.160+0000 7f09f1d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09f4069820 0x7f09f41984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.160+0000 7f09f1d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09f4069820 0x7f09f41984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41400/0 (socket says 172.21.15.76:41400) 2024-03-18T14:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.160+0000 7f09f259c640 1 -- 172.21.15.76:0/3881737366 learned_addr learned my addr 172.21.15.76:0/3881737366 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.160+0000 7f09f8bd4640 1 -- 172.21.15.76:0/3881737366 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f09f4111c10 con 0x7f09f41077b0 2024-03-18T14:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.160+0000 7f09f8bd4640 1 -- 172.21.15.76:0/3881737366 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f09f4111a90 con 0x7f09f4069820 2024-03-18T14:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.160+0000 7f09f8bd4640 1 -- 172.21.15.76:0/3881737366 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f09f4111d90 con 0x7f09f4068e60 2024-03-18T14:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.160+0000 7f09f2d9d640 1 --2- 172.21.15.76:0/3881737366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09f41077b0 0x7f09f419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.160+0000 7f09f1d9b640 1 -- 172.21.15.76:0/3881737366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09f4068e60 msgr2=0x7f09f4197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.160+0000 7f09f1d9b640 1 --2- 172.21.15.76:0/3881737366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09f4068e60 0x7f09f4197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.160+0000 7f09f1d9b640 1 -- 172.21.15.76:0/3881737366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09f41077b0 msgr2=0x7f09f419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.160+0000 7f09f1d9b640 1 --2- 172.21.15.76:0/3881737366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09f41077b0 0x7f09f419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.160+0000 7f09f1d9b640 1 -- 172.21.15.76:0/3881737366 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f419cfa0 con 0x7f09f4069820 2024-03-18T14:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.160+0000 7f09f1d9b640 1 --2- 172.21.15.76:0/3881737366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09f4069820 0x7f09f41984f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f09e800e9f0 tx=0x7f09e800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.161+0000 7f09e37fe640 1 -- 172.21.15.76:0/3881737366 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e800cd80 con 0x7f09f4069820 2024-03-18T14:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.161+0000 7f09e37fe640 1 -- 172.21.15.76:0/3881737366 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09e80035f0 con 0x7f09f4069820 2024-03-18T14:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.161+0000 7f09f8bd4640 1 -- 172.21.15.76:0/3881737366 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f419d290 con 0x7f09f4069820 2024-03-18T14:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.161+0000 7f09e37fe640 1 -- 172.21.15.76:0/3881737366 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e8010690 con 0x7f09f4069820 2024-03-18T14:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.161+0000 7f09f8bd4640 1 -- 172.21.15.76:0/3881737366 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f41a4b20 con 0x7f09f4069820 2024-03-18T14:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.163+0000 7f09fb7bf740 1 Processor -- start 2024-03-18T14:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.163+0000 7f09fb7bf740 1 -- start start 2024-03-18T14:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.164+0000 7f09fb7bf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 0x55eedf2f7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.164+0000 7f09fb7bf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7a50 0x55eedf3002e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.164+0000 7f09fb7bf740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf300820 0x55eedf302c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.164+0000 7f09fb7bf740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55eedf25ba40 con 0x55eedf300820 2024-03-18T14:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.164+0000 7f09fb7bf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55eedf204920 con 0x55eedf2f7130 2024-03-18T14:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.164+0000 7f09fb7bf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55eedf257ae0 con 0x55eedf2f7a50 2024-03-18T14:53:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.164+0000 7f09f2d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 0x55eedf2f7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.164+0000 7f09f259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7a50 0x55eedf3002e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.164+0000 7f09f2d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 0x55eedf2f7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41404/0 (socket says 172.21.15.76:41404) 2024-03-18T14:53:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.164+0000 7f09f259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7a50 0x55eedf3002e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:49314/0 (socket says 172.21.15.76:49314) 2024-03-18T14:53:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.164+0000 7f09f2d9d640 1 -- 172.21.15.76:0/372928261 learned_addr learned my addr 172.21.15.76:0/372928261 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.164+0000 7f09f2d9d640 1 -- 172.21.15.76:0/372928261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7a50 msgr2=0x55eedf3002e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.164+0000 7f09f2d9d640 1 --2- 172.21.15.76:0/372928261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7a50 0x55eedf3002e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.164+0000 7f09f2d9d640 1 -- 172.21.15.76:0/372928261 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf300820 msgr2=0x55eedf302c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T14:53:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.164+0000 7f09f2d9d640 1 --2- 172.21.15.76:0/372928261 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf300820 0x55eedf302c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.164+0000 7f09f2d9d640 1 -- 172.21.15.76:0/372928261 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eedf21c460 con 0x55eedf2f7130 2024-03-18T14:53:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.164+0000 7f09f2d9d640 1 --2- 172.21.15.76:0/372928261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 0x55eedf2f7510 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009f40 tx=0x7f09ec013c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bce0b5b615f3b930 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.165+0000 7f09e37fe640 1 -- 172.21.15.76:0/3881737366 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 23) ==== 137466+0+0 (secure 0 0 0) 0x7f09e8004020 con 0x7f09f4069820 2024-03-18T14:53:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.165+0000 7f09e0ff9640 1 -- 172.21.15.76:0/372928261 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ec003a30 con 0x55eedf2f7130 2024-03-18T14:53:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.165+0000 7f09e0ff9640 1 -- 172.21.15.76:0/372928261 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09ec013e50 con 0x55eedf2f7130 2024-03-18T14:53:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.165+0000 7f09e0ff9640 1 -- 172.21.15.76:0/372928261 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ec0046a0 con 0x55eedf2f7130 2024-03-18T14:53:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.165+0000 7f09e37fe640 1 --2- 172.21.15.76:0/3881737366 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f09dc0a2ab0 0x7f09dc0a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.165+0000 7f09fb7bf740 1 -- 172.21.15.76:0/372928261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 msgr2=0x55eedf2f7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.165+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/372928261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 0x55eedf2f7510 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009f40 tx=0x7f09ec013c60 comp rx=0 tx=0).stop 2024-03-18T14:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.165+0000 7f09fb7bf740 1 -- 172.21.15.76:0/372928261 shutdown_connections 2024-03-18T14:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.166+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/372928261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7a50 0x55eedf3002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.166+0000 7f09e37fe640 1 -- 172.21.15.76:0/3881737366 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f09e80cf250 con 0x7f09f4069820 2024-03-18T14:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.166+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/372928261 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf300820 0x55eedf302c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.166+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/372928261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 0x55eedf2f7510 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.166+0000 7f09fb7bf740 1 -- 172.21.15.76:0/372928261 >> 172.21.15.76:0/372928261 conn(0x55eedf2f1a50 msgr2=0x55eedf2f1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.166+0000 7f09f259c640 1 --2- 172.21.15.76:0/3881737366 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f09dc0a2ab0 0x7f09dc0a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.166+0000 7f09fb7bf740 1 -- 172.21.15.76:0/372928261 shutdown_connections 2024-03-18T14:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.166+0000 7f09fb7bf740 1 -- 172.21.15.76:0/372928261 wait complete. 2024-03-18T14:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.166+0000 7f09fb7bf740 1 -- start start 2024-03-18T14:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.166+0000 7f09f259c640 1 --2- 172.21.15.76:0/3881737366 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f09dc0a2ab0 0x7f09dc0a4f70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f09e40096c0 tx=0x7f09e4000bc0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.166+0000 7f09fb7bf740 1 Processor -- start 2024-03-18T14:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09fb7bf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7130 0x55eedf2fd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09fb7bf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7a50 0x55eedf2fde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09fb7bf740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf300820 0x55eedf2fe3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09fb7bf740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55eedf303450 con 0x55eedf300820 2024-03-18T14:53:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09fb7bf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55eedf217d70 con 0x55eedf2f7a50 2024-03-18T14:53:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09fb7bf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55eedf3035d0 con 0x55eedf2f7130 2024-03-18T14:53:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09f259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7130 0x55eedf2fd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09f1d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7a50 0x55eedf2fde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09f1d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7a50 0x55eedf2fde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41418/0 (socket says 172.21.15.76:41418) 2024-03-18T14:53:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09fb7bf740 10 client.0 _ll_register_callbacks cb 0x55eedef6fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09f259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7130 0x55eedf2fd940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:49318/0 (socket says 172.21.15.76:49318) 2024-03-18T14:53:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09f1d9b640 1 -- 172.21.15.76:0/4264192258 learned_addr learned my addr 172.21.15.76:0/4264192258 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09f2d9d640 1 --2- 172.21.15.76:0/4264192258 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf300820 0x55eedf2fe3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09f1d9b640 1 -- 172.21.15.76:0/4264192258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7130 msgr2=0x55eedf2fd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09f1d9b640 1 --2- 172.21.15.76:0/4264192258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7130 0x55eedf2fd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09f1d9b640 1 -- 172.21.15.76:0/4264192258 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf300820 msgr2=0x55eedf2fe3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:46.173 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09f1d9b640 1 --2- 172.21.15.76:0/4264192258 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf300820 0x55eedf2fe3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.173 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09f1d9b640 1 -- 172.21.15.76:0/4264192258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eedef70210 con 0x55eedf2f7a50 2024-03-18T14:53:46.173 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09f2d9d640 1 --2- 172.21.15.76:0/4264192258 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf300820 0x55eedf2fe3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:53:46.173 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09f1d9b640 1 --2- 172.21.15.76:0/4264192258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7a50 0x55eedf2fde80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f09e80cc9c0 tx=0x7f09e809be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67434ae8c8ca52df server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:46.173 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:53:46.173 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09d9ffb640 1 -- 172.21.15.76:0/4264192258 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e809f8e0 con 0x55eedf2f7a50 2024-03-18T14:53:46.173 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09fb7bf740 1 -- 172.21.15.76:0/4264192258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09ec013e50 con 0x55eedf2f7a50 2024-03-18T14:53:46.173 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.167+0000 7f09d9ffb640 1 -- 172.21.15.76:0/4264192258 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f09e8004310 con 0x55eedf2f7a50 2024-03-18T14:53:46.173 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.168+0000 7f09d9ffb640 1 -- 172.21.15.76:0/4264192258 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f09e80b8020 con 0x55eedf2f7a50 2024-03-18T14:53:46.173 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.168+0000 7f09fb7bf740 10 client.25508 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:46.173 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.168+0000 7f09fb7bf740 1 -- 172.21.15.76:0/4264192258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55eedf03c620 con 0x55eedf2f7a50 2024-03-18T14:53:46.173 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.168+0000 7f09d9ffb640 1 -- 172.21.15.76:0/4264192258 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09ec013e50 con 0x55eedf2f7a50 2024-03-18T14:53:46.173 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.168+0000 7f09d9ffb640 1 -- 172.21.15.76:0/4264192258 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f09e80a3aa0 con 0x55eedf2f7a50 2024-03-18T14:53:46.173 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.168+0000 7f09fb7bf740 10 client.25508 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.168+0000 7f09fb7bf740 10 client.25508 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.168+0000 7f09fb7bf740 20 client.25508 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.168+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/4264192258 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55eedf310b40 0x55eedf312f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.168+0000 7f09fb7bf740 4 client.25508 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.168+0000 7f09fb7bf740 1 -- 172.21.15.76:0/4264192258 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eedf268f80 con 0x55eedf310b40 2024-03-18T14:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.168+0000 7f09f259c640 1 --2- 172.21.15.76:0/4264192258 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55eedf310b40 0x55eedf312f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.169+0000 7f09d9ffb640 10 client.25508 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.169+0000 7f09f259c640 1 --2- 172.21.15.76:0/4264192258 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55eedf310b40 0x55eedf312f30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55812cdfd14ef70e server_cookie=1ac44b2eaf3725b2 in_seq=0 out_seq=0 2024-03-18T14:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.169+0000 7f09d9ffb640 1 -- 172.21.15.76:0/4264192258 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55eedf268f80 con 0x55eedf310b40 2024-03-18T14:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.169+0000 7f09d9ffb640 10 client.25508 handle_command_reply: tid=0 2024-03-18T14:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.169+0000 7f09fb7bf740 1 client.25508 shutdown 2024-03-18T14:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.169+0000 7f09fb7bf740 1 -- 172.21.15.76:0/4264192258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7a50 msgr2=0x55eedf2fde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.169+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/4264192258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7a50 0x55eedf2fde80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f09e80cc9c0 tx=0x7f09e809be90 comp rx=0 tx=0).stop 2024-03-18T14:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.169+0000 7f09fb7bf740 1 -- 172.21.15.76:0/4264192258 shutdown_connections 2024-03-18T14:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.169+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/4264192258 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55eedf310b40 0x55eedf312f30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.169+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/4264192258 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf300820 0x55eedf2fe3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.169+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/4264192258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7a50 0x55eedf2fde80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.169+0000 7f09d9ffb640 0 client.25508 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.169+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/4264192258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7130 0x55eedf2fd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.169+0000 7f09fb7bf740 1 -- 172.21.15.76:0/4264192258 >> 172.21.15.76:0/4264192258 conn(0x55eedf2f1a50 msgr2=0x55eedf2f2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.170+0000 7f09fb7bf740 1 -- 172.21.15.76:0/4264192258 shutdown_connections 2024-03-18T14:53:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.170+0000 7f09fb7bf740 1 -- 172.21.15.76:0/4264192258 wait complete. 2024-03-18T14:53:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.170+0000 7f09fb7bf740 20 client.25508 trim_cache size 0 max 16384 2024-03-18T14:53:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09fb7bf740 1 Processor -- start 2024-03-18T14:53:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09fb7bf740 1 -- start start 2024-03-18T14:53:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09fb7bf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 0x55eedf300070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09fb7bf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7a50 0x55eedf30c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09fb7bf740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf300820 0x55eedf2f3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09fb7bf740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f09f41ad9a0 con 0x55eedf300820 2024-03-18T14:53:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09fb7bf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f09f41ab270 con 0x55eedf2f7130 2024-03-18T14:53:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09fb7bf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f09f41a90c0 con 0x55eedf2f7a50 2024-03-18T14:53:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09f2d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf300820 0x55eedf2f3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09f1d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7a50 0x55eedf30c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09f259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 0x55eedf300070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09f1d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7a50 0x55eedf30c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:49326/0 (socket says 172.21.15.76:49326) 2024-03-18T14:53:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09f259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 0x55eedf300070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:41424/0 (socket says 172.21.15.76:41424) 2024-03-18T14:53:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09f1d9b640 1 -- 172.21.15.76:0/3738426055 learned_addr learned my addr 172.21.15.76:0/3738426055 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09f1d9b640 1 -- 172.21.15.76:0/3738426055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 msgr2=0x55eedf300070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09f1d9b640 1 --2- 172.21.15.76:0/3738426055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 0x55eedf300070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09f1d9b640 1 -- 172.21.15.76:0/3738426055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf300820 msgr2=0x55eedf2f3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09f1d9b640 1 --2- 172.21.15.76:0/3738426055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf300820 0x55eedf2f3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09f1d9b640 1 -- 172.21.15.76:0/3738426055 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eedf30f6e0 con 0x55eedf2f7a50 2024-03-18T14:53:46.188 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.185+0000 7f09f259c640 1 --2- 172.21.15.76:0/3738426055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 0x55eedf300070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:46.188 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.186+0000 7f09f1d9b640 1 --2- 172.21.15.76:0/3738426055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7a50 0x55eedf30c1f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f09e80ccff0 tx=0x7f09e8097b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67ce5a2c99cd0c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:46.188 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.186+0000 7f09dbfff640 1 -- 172.21.15.76:0/3738426055 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e809b2d0 con 0x55eedf2f7a50 2024-03-18T14:53:46.188 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.186+0000 7f09dbfff640 1 -- 172.21.15.76:0/3738426055 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09e80a5070 con 0x55eedf2f7a50 2024-03-18T14:53:46.188 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.186+0000 7f09dbfff640 1 -- 172.21.15.76:0/3738426055 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e8004310 con 0x55eedf2f7a50 2024-03-18T14:53:46.191 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.187+0000 7f09fb7bf740 1 -- 172.21.15.76:0/3738426055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7a50 msgr2=0x55eedf30c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:46.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.187+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/3738426055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7a50 0x55eedf30c1f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f09e80ccff0 tx=0x7f09e8097b60 comp rx=0 tx=0).stop 2024-03-18T14:53:46.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.187+0000 7f09fb7bf740 1 -- 172.21.15.76:0/3738426055 shutdown_connections 2024-03-18T14:53:46.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.187+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/3738426055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf2f7a50 0x55eedf30c1f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.187+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/3738426055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf300820 0x55eedf2f3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.187+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/3738426055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 0x55eedf300070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.187+0000 7f09fb7bf740 1 -- 172.21.15.76:0/3738426055 >> 172.21.15.76:0/3738426055 conn(0x55eedf2f1a50 msgr2=0x55eedf2f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:46.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.187+0000 7f09fb7bf740 1 -- 172.21.15.76:0/3738426055 shutdown_connections 2024-03-18T14:53:46.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.187+0000 7f09fb7bf740 1 -- 172.21.15.76:0/3738426055 wait complete. 2024-03-18T14:53:46.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.187+0000 7f09fb7bf740 1 -- start start 2024-03-18T14:53:46.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.187+0000 7f09fb7bf740 1 Processor -- start 2024-03-18T14:53:46.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09fb7bf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 0x55eedf304520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09fb7bf740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf2f7a50 0x55eedf304a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09fb7bf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf300820 0x55eedf37e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09fb7bf740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f09f41a9730 con 0x55eedf2f7a50 2024-03-18T14:53:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09fb7bf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f09f41aaba0 con 0x55eedf2f7130 2024-03-18T14:53:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09fb7bf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f09f41a9e00 con 0x55eedf300820 2024-03-18T14:53:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09f1d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf2f7a50 0x55eedf304a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09f259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 0x55eedf304520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09f1d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf2f7a50 0x55eedf304a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39594/0 (socket says 172.21.15.76:39594) 2024-03-18T14:53:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09f1d9b640 1 -- 172.21.15.76:0/3224113311 learned_addr learned my addr 172.21.15.76:0/3224113311 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09fb7bf740 10 client.0 _ll_register_callbacks cb 0x7f09e80cc7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09f2d9d640 1 --2- 172.21.15.76:0/3224113311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf300820 0x55eedf37e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09f1d9b640 1 -- 172.21.15.76:0/3224113311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf300820 msgr2=0x55eedf37e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09f1d9b640 1 --2- 172.21.15.76:0/3224113311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf300820 0x55eedf37e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09f1d9b640 1 -- 172.21.15.76:0/3224113311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 msgr2=0x55eedf304520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09f1d9b640 1 --2- 172.21.15.76:0/3224113311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 0x55eedf304520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09f1d9b640 1 -- 172.21.15.76:0/3224113311 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eedf37eba0 con 0x55eedf2f7a50 2024-03-18T14:53:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09f259c640 1 --2- 172.21.15.76:0/3224113311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 0x55eedf304520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09f1d9b640 1 --2- 172.21.15.76:0/3224113311 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf2f7a50 0x55eedf304a60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f09e809b080 tx=0x7f09e8097a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=645de0820d071ac6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.188+0000 7f09d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:53:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.189+0000 7f09d9ffb640 1 -- 172.21.15.76:0/3224113311 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e809b2d0 con 0x55eedf2f7a50 2024-03-18T14:53:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.189+0000 7f09d9ffb640 1 -- 172.21.15.76:0/3224113311 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f09e80ccbe0 con 0x55eedf2f7a50 2024-03-18T14:53:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.189+0000 7f09fb7bf740 1 -- 172.21.15.76:0/3224113311 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09e80a5070 con 0x55eedf2f7a50 2024-03-18T14:53:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.189+0000 7f09d9ffb640 1 -- 172.21.15.76:0/3224113311 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f09e809f8e0 con 0x55eedf2f7a50 2024-03-18T14:53:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.189+0000 7f09fb7bf740 10 client.15654 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.189+0000 7f09fb7bf740 1 -- 172.21.15.76:0/3224113311 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55eedf37fd50 con 0x55eedf2f7a50 2024-03-18T14:53:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.189+0000 7f09d9ffb640 1 -- 172.21.15.76:0/3224113311 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09e80a5070 con 0x55eedf2f7a50 2024-03-18T14:53:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.189+0000 7f09d9ffb640 1 -- 172.21.15.76:0/3224113311 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f09e809a650 con 0x55eedf2f7a50 2024-03-18T14:53:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.190+0000 7f09fb7bf740 10 client.15654 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.190+0000 7f09fb7bf740 10 client.15654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.190+0000 7f09fb7bf740 20 client.15654 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.190+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/3224113311 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55eedf3800e0 0x55eedf382500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.190+0000 7f09fb7bf740 4 client.15654 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:53:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.190+0000 7f09fb7bf740 1 -- 172.21.15.76:0/3224113311 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eedf2632d0 con 0x55eedf3800e0 2024-03-18T14:53:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.190+0000 7f09f259c640 1 --2- 172.21.15.76:0/3224113311 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55eedf3800e0 0x55eedf382500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.190+0000 7f09d9ffb640 10 client.15654 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.190+0000 7f09f259c640 1 --2- 172.21.15.76:0/3224113311 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55eedf3800e0 0x55eedf382500 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8a77a14d3e577cb server_cookie=3615c66e46d6a238 in_seq=0 out_seq=0 2024-03-18T14:53:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.193+0000 7f09d9ffb640 1 -- 172.21.15.76:0/3224113311 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55eedf2632d0 con 0x55eedf3800e0 2024-03-18T14:53:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.193+0000 7f09d9ffb640 10 client.15654 handle_command_reply: tid=0 2024-03-18T14:53:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.193+0000 7f09fb7bf740 1 client.15654 shutdown 2024-03-18T14:53:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.193+0000 7f09fb7bf740 1 -- 172.21.15.76:0/3224113311 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf2f7a50 msgr2=0x55eedf304a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.193+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/3224113311 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf2f7a50 0x55eedf304a60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f09e809b080 tx=0x7f09e8097a30 comp rx=0 tx=0).stop 2024-03-18T14:53:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.194+0000 7f09fb7bf740 1 -- 172.21.15.76:0/3224113311 shutdown_connections 2024-03-18T14:53:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.194+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/3224113311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eedf300820 0x55eedf37e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.194+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/3224113311 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55eedf3800e0 0x55eedf382500 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.194+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/3224113311 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eedf2f7a50 0x55eedf304a60 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.194+0000 7f09d9ffb640 0 client.15654 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.194+0000 7f09fb7bf740 1 --2- 172.21.15.76:0/3224113311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eedf2f7130 0x55eedf304520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.194+0000 7f09fb7bf740 1 -- 172.21.15.76:0/3224113311 >> 172.21.15.76:0/3224113311 conn(0x55eedf2f1a50 msgr2=0x55eedf2f36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.194+0000 7f09fb7bf740 1 -- 172.21.15.76:0/3224113311 shutdown_connections 2024-03-18T14:53:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.194+0000 7f09fb7bf740 1 -- 172.21.15.76:0/3224113311 wait complete. 2024-03-18T14:53:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.194+0000 7f09fb7bf740 20 client.15654 trim_cache size 0 max 16384 2024-03-18T14:53:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:53:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:53:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:53:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:53:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.195+0000 7f09f8bd4640 1 -- 172.21.15.76:0/3881737366 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f09dc0a2ab0 msgr2=0x7f09dc0a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.195+0000 7f09f8bd4640 1 --2- 172.21.15.76:0/3881737366 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f09dc0a2ab0 0x7f09dc0a4f70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f09e40096c0 tx=0x7f09e4000bc0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.195+0000 7f09f8bd4640 1 -- 172.21.15.76:0/3881737366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09f4069820 msgr2=0x7f09f41984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.195+0000 7f09f8bd4640 1 --2- 172.21.15.76:0/3881737366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09f4069820 0x7f09f41984f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f09e800e9f0 tx=0x7f09e800eeb0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.195+0000 7f09f8bd4640 1 -- 172.21.15.76:0/3881737366 shutdown_connections 2024-03-18T14:53:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.195+0000 7f09f8bd4640 1 --2- 172.21.15.76:0/3881737366 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f09dc0a2ab0 0x7f09dc0a4f70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.195+0000 7f09f8bd4640 1 --2- 172.21.15.76:0/3881737366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09f41077b0 0x7f09f419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.195+0000 7f09f8bd4640 1 --2- 172.21.15.76:0/3881737366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09f4069820 0x7f09f41984f0 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f09e800e9f0 tx=0x7f09e800eeb0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.195+0000 7f09f8bd4640 1 --2- 172.21.15.76:0/3881737366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09f4068e60 0x7f09f4197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.195+0000 7f09f8bd4640 1 -- 172.21.15.76:0/3881737366 >> 172.21.15.76:0/3881737366 conn(0x7f09f4077530 msgr2=0x7f09f40fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.195+0000 7f09f8bd4640 1 -- 172.21.15.76:0/3881737366 shutdown_connections 2024-03-18T14:53:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:46.195+0000 7f09f8bd4640 1 -- 172.21.15.76:0/3881737366 wait complete. 2024-03-18T14:53:48.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:48 smithi076 ceph-mon[32205]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-18T14:53:48.943 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:48 smithi077 ceph-mon[35865]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-18T14:53:48.957 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:48 smithi174 ceph-mon[36787]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-18T14:53:50.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:50 smithi076 ceph-mon[32205]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-18T14:53:50.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:50 smithi174 ceph-mon[36787]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-18T14:53:50.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:50 smithi077 ceph-mon[35865]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-18T14:53:52.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:52 smithi076 ceph-mon[32205]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T14:53:52.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:52 smithi174 ceph-mon[36787]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T14:53:52.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:52 smithi077 ceph-mon[35865]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T14:53:54.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:54 smithi076 ceph-mon[32205]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-18T14:53:54.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:54 smithi174 ceph-mon[36787]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-18T14:53:54.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:54 smithi077 ceph-mon[35865]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-18T14:53:56.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:56 smithi076 ceph-mon[32205]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-18T14:53:56.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:56 smithi174 ceph-mon[36787]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-18T14:53:56.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:56 smithi077 ceph-mon[35865]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-18T14:53:58.536 INFO:tasks.workunit.client.0.smithi076.stderr:Note: switching to '4e19f704afd9957ea657ab858f9ecf76401fef6f'. 2024-03-18T14:53:58.536 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-18T14:53:58.536 INFO:tasks.workunit.client.0.smithi076.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-18T14:53:58.536 INFO:tasks.workunit.client.0.smithi076.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-18T14:53:58.536 INFO:tasks.workunit.client.0.smithi076.stderr:state without impacting any branches by switching back to a branch. 2024-03-18T14:53:58.536 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-18T14:53:58.537 INFO:tasks.workunit.client.0.smithi076.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-18T14:53:58.537 INFO:tasks.workunit.client.0.smithi076.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-18T14:53:58.537 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-18T14:53:58.537 INFO:tasks.workunit.client.0.smithi076.stderr: git switch -c 2024-03-18T14:53:58.537 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-18T14:53:58.537 INFO:tasks.workunit.client.0.smithi076.stderr:Or undo this operation with: 2024-03-18T14:53:58.537 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-18T14:53:58.537 INFO:tasks.workunit.client.0.smithi076.stderr: git switch - 2024-03-18T14:53:58.537 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-18T14:53:58.537 INFO:tasks.workunit.client.0.smithi076.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-18T14:53:58.537 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-18T14:53:58.537 INFO:tasks.workunit.client.0.smithi076.stderr:HEAD is now at 4e19f704afd ignorelist mon down 2024-03-18T14:53:58.541 DEBUG:teuthology.orchestra.run.smithi076:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-18T14:53:58.567 INFO:tasks.workunit.client.0.smithi076.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-18T14:53:58.570 INFO:tasks.workunit.client.0.smithi076.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T14:53:58.570 INFO:tasks.workunit.client.0.smithi076.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-18T14:53:58.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:53:58 smithi076 ceph-mon[32205]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-18T14:53:58.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:53:58 smithi174 ceph-mon[36787]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-18T14:53:58.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:53:58 smithi077 ceph-mon[35865]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-18T14:53:59.292 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.289+0000 7f3639cbf640 1 -- 172.21.15.76:0/110755540 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3634105680 msgr2=0x7f3634105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.289+0000 7f3639cbf640 1 --2- 172.21.15.76:0/110755540 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3634105680 0x7f3634105a60 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3624009900 tx=0x7f362402bc30 comp rx=0 tx=0).stop 2024-03-18T14:53:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.289+0000 7f3639cbf640 1 -- 172.21.15.76:0/110755540 shutdown_connections 2024-03-18T14:53:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.289+0000 7f3639cbf640 1 --2- 172.21.15.76:0/110755540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3634100930 0x7f3634100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.289+0000 7f3639cbf640 1 --2- 172.21.15.76:0/110755540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36341012f0 0x7f363410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.289+0000 7f3639cbf640 1 --2- 172.21.15.76:0/110755540 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3634105680 0x7f3634105a60 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.289+0000 7f3639cbf640 1 -- 172.21.15.76:0/110755540 >> 172.21.15.76:0/110755540 conn(0x7f3634077530 msgr2=0x7f3634077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.289+0000 7f3639cbf640 1 -- 172.21.15.76:0/110755540 shutdown_connections 2024-03-18T14:53:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.290+0000 7f3639cbf640 1 -- 172.21.15.76:0/110755540 wait complete. 2024-03-18T14:53:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.290+0000 7f3639cbf640 1 Processor -- start 2024-03-18T14:53:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.290+0000 7f3639cbf640 1 -- start start 2024-03-18T14:53:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.290+0000 7f3639cbf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3634100930 0x7f363419a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.290+0000 7f3639cbf640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36341012f0 0x7f363419a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.290+0000 7f3639cbf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3634105680 0x7f363419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f3639cbf640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3634111cc0 con 0x7f36341012f0 2024-03-18T14:53:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f3639cbf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3634111b40 con 0x7f3634105680 2024-03-18T14:53:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f3639cbf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3634111e40 con 0x7f3634100930 2024-03-18T14:53:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f362affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36341012f0 0x7f363419a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f36337fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3634100930 0x7f363419a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f3633fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3634105680 0x7f363419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f36337fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3634100930 0x7f363419a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:52756/0 (socket says 172.21.15.76:52756) 2024-03-18T14:53:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f362affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36341012f0 0x7f363419a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54728/0 (socket says 172.21.15.76:54728) 2024-03-18T14:53:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f36337fe640 1 -- 172.21.15.76:0/1425318618 learned_addr learned my addr 172.21.15.76:0/1425318618 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f3633fff640 1 -- 172.21.15.76:0/1425318618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3634100930 msgr2=0x7f363419a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f3633fff640 1 --2- 172.21.15.76:0/1425318618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3634100930 0x7f363419a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f3633fff640 1 -- 172.21.15.76:0/1425318618 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36341012f0 msgr2=0x7f363419a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f3633fff640 1 --2- 172.21.15.76:0/1425318618 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36341012f0 0x7f363419a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f3633fff640 1 -- 172.21.15.76:0/1425318618 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f363419f290 con 0x7f3634105680 2024-03-18T14:53:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f362affd640 1 --2- 172.21.15.76:0/1425318618 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36341012f0 0x7f363419a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f3633fff640 1 --2- 172.21.15.76:0/1425318618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3634105680 0x7f363419eb60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f362c00ebc0 tx=0x7f362c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f36317fa640 1 -- 172.21.15.76:0/1425318618 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362c003b70 con 0x7f3634105680 2024-03-18T14:53:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f3639cbf640 1 -- 172.21.15.76:0/1425318618 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f363419f580 con 0x7f3634105680 2024-03-18T14:53:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f36317fa640 1 -- 172.21.15.76:0/1425318618 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f362c003d10 con 0x7f3634105680 2024-03-18T14:53:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f36317fa640 1 -- 172.21.15.76:0/1425318618 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362c010720 con 0x7f3634105680 2024-03-18T14:53:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.291+0000 7f3639cbf640 1 -- 172.21.15.76:0/1425318618 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36341a6e10 con 0x7f3634105680 2024-03-18T14:53:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.293+0000 7f36317fa640 1 -- 172.21.15.76:0/1425318618 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 23) ==== 137466+0+0 (secure 0 0 0) 0x7f362c004020 con 0x7f3634105680 2024-03-18T14:53:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.294+0000 7f36317fa640 1 --2- 172.21.15.76:0/1425318618 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f36200a2ab0 0x7f36200a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.294+0000 7f363c981740 1 Processor -- start 2024-03-18T14:53:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.294+0000 7f36337fe640 1 --2- 172.21.15.76:0/1425318618 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f36200a2ab0 0x7f36200a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.294+0000 7f36317fa640 1 -- 172.21.15.76:0/1425318618 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f362c0cf500 con 0x7f3634105680 2024-03-18T14:53:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.294+0000 7f363c981740 1 -- start start 2024-03-18T14:53:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.294+0000 7f363c981740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10130 0x557bf7c10510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.294+0000 7f363c981740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c192e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.294+0000 7f363c981740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c19820 0x557bf7c1bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.294+0000 7f363c981740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557bf7b74a40 con 0x557bf7c10a50 2024-03-18T14:53:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.294+0000 7f363c981740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x557bf7b1d920 con 0x557bf7c19820 2024-03-18T14:53:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.294+0000 7f363c981740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557bf7b70ae0 con 0x557bf7c10130 2024-03-18T14:53:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.295+0000 7f36337fe640 1 --2- 172.21.15.76:0/1425318618 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f36200a2ab0 0x7f36200a4f70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3624000c00 tx=0x7f36240058d0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.295+0000 7f362affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c192e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.295+0000 7f36337fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10130 0x557bf7c10510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.295+0000 7f362affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c192e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54736/0 (socket says 172.21.15.76:54736) 2024-03-18T14:53:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.295+0000 7f362affd640 1 -- 172.21.15.76:0/1823632500 learned_addr learned my addr 172.21.15.76:0/1823632500 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.295+0000 7f3633fff640 1 --2- 172.21.15.76:0/1823632500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c19820 0x557bf7c1bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.295+0000 7f36337fe640 1 -- 172.21.15.76:0/1823632500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c19820 msgr2=0x557bf7c1bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.295+0000 7f36337fe640 1 --2- 172.21.15.76:0/1823632500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c19820 0x557bf7c1bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.295+0000 7f36337fe640 1 -- 172.21.15.76:0/1823632500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 msgr2=0x557bf7c192e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.295+0000 7f36337fe640 1 --2- 172.21.15.76:0/1823632500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c192e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.295+0000 7f36337fe640 1 -- 172.21.15.76:0/1823632500 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bf7b35460 con 0x557bf7c10130 2024-03-18T14:53:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.295+0000 7f362affd640 1 --2- 172.21.15.76:0/1823632500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c192e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.295+0000 7f3633fff640 1 --2- 172.21.15.76:0/1823632500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c19820 0x557bf7c1bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.295+0000 7f36337fe640 1 --2- 172.21.15.76:0/1823632500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10130 0x557bf7c10510 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f362402efd0 tx=0x7f362403f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d32f179924b38656 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.295+0000 7f3629ffb640 1 -- 172.21.15.76:0/1823632500 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362402cc70 con 0x557bf7c10130 2024-03-18T14:53:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.295+0000 7f3629ffb640 1 -- 172.21.15.76:0/1823632500 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3624050070 con 0x557bf7c10130 2024-03-18T14:53:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.295+0000 7f3629ffb640 1 -- 172.21.15.76:0/1823632500 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36240035f0 con 0x557bf7c10130 2024-03-18T14:53:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.296+0000 7f363c981740 1 -- 172.21.15.76:0/1823632500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10130 msgr2=0x557bf7c10510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.296+0000 7f363c981740 1 --2- 172.21.15.76:0/1823632500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10130 0x557bf7c10510 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f362402efd0 tx=0x7f362403f000 comp rx=0 tx=0).stop 2024-03-18T14:53:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.296+0000 7f363c981740 1 -- 172.21.15.76:0/1823632500 shutdown_connections 2024-03-18T14:53:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.296+0000 7f363c981740 1 --2- 172.21.15.76:0/1823632500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c19820 0x557bf7c1bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.296+0000 7f363c981740 1 --2- 172.21.15.76:0/1823632500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c192e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.296+0000 7f363c981740 1 --2- 172.21.15.76:0/1823632500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10130 0x557bf7c10510 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.296+0000 7f363c981740 1 -- 172.21.15.76:0/1823632500 >> 172.21.15.76:0/1823632500 conn(0x557bf7c0aa50 msgr2=0x557bf7c0ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.296+0000 7f363c981740 1 -- 172.21.15.76:0/1823632500 shutdown_connections 2024-03-18T14:53:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.296+0000 7f363c981740 1 -- 172.21.15.76:0/1823632500 wait complete. 2024-03-18T14:53:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.296+0000 7f363c981740 1 -- start start 2024-03-18T14:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f363c981740 1 Processor -- start 2024-03-18T14:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f363c981740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10130 0x557bf7c167e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f363c981740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c16d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f363c981740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c19820 0x557bf7c17260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f363c981740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557bf7c1c3f0 con 0x557bf7c10a50 2024-03-18T14:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f363c981740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x557bf7b30d70 con 0x557bf7c19820 2024-03-18T14:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f363c981740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557bf7c1c570 con 0x557bf7c10130 2024-03-18T14:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f363c981740 10 client.0 _ll_register_callbacks cb 0x557bf7888a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f362affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c16d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f362affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c16d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54748/0 (socket says 172.21.15.76:54748) 2024-03-18T14:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f36337fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10130 0x557bf7c167e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f362affd640 1 -- 172.21.15.76:0/2178650534 learned_addr learned my addr 172.21.15.76:0/2178650534 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f3633fff640 1 --2- 172.21.15.76:0/2178650534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c19820 0x557bf7c17260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f362affd640 1 -- 172.21.15.76:0/2178650534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10130 msgr2=0x557bf7c167e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f362affd640 1 --2- 172.21.15.76:0/2178650534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10130 0x557bf7c167e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f362affd640 1 -- 172.21.15.76:0/2178650534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c19820 msgr2=0x557bf7c17260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f362affd640 1 --2- 172.21.15.76:0/2178650534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c19820 0x557bf7c17260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f362affd640 1 -- 172.21.15.76:0/2178650534 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bf7889210 con 0x557bf7c10a50 2024-03-18T14:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f36337fe640 1 --2- 172.21.15.76:0/2178650534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10130 0x557bf7c167e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f362affd640 1 --2- 172.21.15.76:0/2178650534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c16d20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f361c012d50 tx=0x7f361c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dedaac825190c58 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.297+0000 7f3612ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.298+0000 7f3612ffd640 1 -- 172.21.15.76:0/2178650534 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361c012110 con 0x557bf7c10a50 2024-03-18T14:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.298+0000 7f363c981740 1 -- 172.21.15.76:0/2178650534 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3624050070 con 0x557bf7c10a50 2024-03-18T14:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.298+0000 7f3612ffd640 1 -- 172.21.15.76:0/2178650534 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f361c004050 con 0x557bf7c10a50 2024-03-18T14:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.298+0000 7f3612ffd640 1 -- 172.21.15.76:0/2178650534 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f361c0122b0 con 0x557bf7c10a50 2024-03-18T14:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.298+0000 7f363c981740 10 client.15666 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.298+0000 7f363c981740 1 -- 172.21.15.76:0/2178650534 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557bf7955620 con 0x557bf7c10a50 2024-03-18T14:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.298+0000 7f3612ffd640 1 -- 172.21.15.76:0/2178650534 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3624050070 con 0x557bf7c10a50 2024-03-18T14:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.298+0000 7f3612ffd640 1 -- 172.21.15.76:0/2178650534 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f361c025430 con 0x557bf7c10a50 2024-03-18T14:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.298+0000 7f363c981740 10 client.15666 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.298+0000 7f363c981740 10 client.15666 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.298+0000 7f363c981740 20 client.15666 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.298+0000 7f363c981740 1 --2- 172.21.15.76:0/2178650534 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557bf7c298f0 0x557bf7c2bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.298+0000 7f363c981740 4 client.15666 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.298+0000 7f363c981740 1 -- 172.21.15.76:0/2178650534 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bf7b81f80 con 0x557bf7c298f0 2024-03-18T14:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.298+0000 7f36337fe640 1 --2- 172.21.15.76:0/2178650534 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557bf7c298f0 0x557bf7c2bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.299+0000 7f3612ffd640 10 client.15666 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.299+0000 7f36337fe640 1 --2- 172.21.15.76:0/2178650534 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557bf7c298f0 0x557bf7c2bd40 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c26520c77104a669 server_cookie=7e108c66e8a31454 in_seq=0 out_seq=0 2024-03-18T14:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.299+0000 7f3612ffd640 1 -- 172.21.15.76:0/2178650534 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557bf7b81f80 con 0x557bf7c298f0 2024-03-18T14:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.299+0000 7f3612ffd640 10 client.15666 handle_command_reply: tid=0 2024-03-18T14:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.299+0000 7f363c981740 1 client.15666 shutdown 2024-03-18T14:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.300+0000 7f363c981740 1 -- 172.21.15.76:0/2178650534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 msgr2=0x557bf7c16d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.300+0000 7f363c981740 1 --2- 172.21.15.76:0/2178650534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c16d20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f361c012d50 tx=0x7f361c00f790 comp rx=0 tx=0).stop 2024-03-18T14:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.300+0000 7f363c981740 1 -- 172.21.15.76:0/2178650534 shutdown_connections 2024-03-18T14:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.300+0000 7f363c981740 1 --2- 172.21.15.76:0/2178650534 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557bf7c298f0 0x557bf7c2bd40 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.300+0000 7f363c981740 1 --2- 172.21.15.76:0/2178650534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c19820 0x557bf7c17260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.300+0000 7f3612ffd640 0 client.15666 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.300+0000 7f363c981740 1 --2- 172.21.15.76:0/2178650534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c16d20 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.300+0000 7f363c981740 1 --2- 172.21.15.76:0/2178650534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10130 0x557bf7c167e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.300+0000 7f363c981740 1 -- 172.21.15.76:0/2178650534 >> 172.21.15.76:0/2178650534 conn(0x557bf7c0aa50 msgr2=0x557bf7c0c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.300+0000 7f363c981740 1 -- 172.21.15.76:0/2178650534 shutdown_connections 2024-03-18T14:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.300+0000 7f363c981740 1 -- 172.21.15.76:0/2178650534 wait complete. 2024-03-18T14:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.300+0000 7f363c981740 20 client.15666 trim_cache size 0 max 16384 2024-03-18T14:53:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.314+0000 7f363c981740 1 Processor -- start 2024-03-18T14:53:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.314+0000 7f363c981740 1 -- start start 2024-03-18T14:53:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.314+0000 7f363c981740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c10130 0x557bf7c25190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.314+0000 7f363c981740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10a50 0x557bf7c31650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.314+0000 7f363c981740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c19820 0x557bf7c167a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.314+0000 7f363c981740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f36341b1d90 con 0x557bf7c19820 2024-03-18T14:53:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.314+0000 7f363c981740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f36341af660 con 0x557bf7c10130 2024-03-18T14:53:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.314+0000 7f363c981740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f36341ad4b0 con 0x557bf7c10a50 2024-03-18T14:53:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.315+0000 7f3633fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c19820 0x557bf7c167a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.321 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.315+0000 7f3633fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c19820 0x557bf7c167a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54762/0 (socket says 172.21.15.76:54762) 2024-03-18T14:53:59.321 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.315+0000 7f3633fff640 1 -- 172.21.15.76:0/916192182 learned_addr learned my addr 172.21.15.76:0/916192182 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:59.321 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.315+0000 7f36337fe640 1 --2- 172.21.15.76:0/916192182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c10130 0x557bf7c25190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.321 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.315+0000 7f362affd640 1 --2- 172.21.15.76:0/916192182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10a50 0x557bf7c31650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.321 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.315+0000 7f3633fff640 1 -- 172.21.15.76:0/916192182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10a50 msgr2=0x557bf7c31650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:59.321 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.315+0000 7f3633fff640 1 --2- 172.21.15.76:0/916192182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10a50 0x557bf7c31650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.321 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.315+0000 7f3633fff640 1 -- 172.21.15.76:0/916192182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c10130 msgr2=0x557bf7c25190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:59.321 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.315+0000 7f3633fff640 1 --2- 172.21.15.76:0/916192182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c10130 0x557bf7c25190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.321 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.315+0000 7f3633fff640 1 -- 172.21.15.76:0/916192182 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bf7c14d30 con 0x557bf7c19820 2024-03-18T14:53:59.321 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.315+0000 7f362affd640 1 --2- 172.21.15.76:0/916192182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10a50 0x557bf7c31650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:53:59.321 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.315+0000 7f3633fff640 1 --2- 172.21.15.76:0/916192182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c19820 0x557bf7c167a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f362c098410 tx=0x7f362c0098f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e952fe42ea433c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:59.321 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.315+0000 7f36297fa640 1 -- 172.21.15.76:0/916192182 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362c09fb90 con 0x557bf7c19820 2024-03-18T14:53:59.321 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.315+0000 7f36297fa640 1 -- 172.21.15.76:0/916192182 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f362c0a41e0 con 0x557bf7c19820 2024-03-18T14:53:59.322 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.315+0000 7f36297fa640 1 -- 172.21.15.76:0/916192182 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362c0cf0b0 con 0x557bf7c19820 2024-03-18T14:53:59.322 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 1 -- 172.21.15.76:0/916192182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c19820 msgr2=0x557bf7c167a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:59.322 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 1 --2- 172.21.15.76:0/916192182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c19820 0x557bf7c167a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f362c098410 tx=0x7f362c0098f0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.322 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 1 -- 172.21.15.76:0/916192182 shutdown_connections 2024-03-18T14:53:59.322 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 1 --2- 172.21.15.76:0/916192182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10a50 0x557bf7c31650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.322 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 1 --2- 172.21.15.76:0/916192182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c19820 0x557bf7c167a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.322 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 1 --2- 172.21.15.76:0/916192182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c10130 0x557bf7c25190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.322 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 1 -- 172.21.15.76:0/916192182 >> 172.21.15.76:0/916192182 conn(0x557bf7c0aa50 msgr2=0x557bf7c150b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:59.322 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 1 -- 172.21.15.76:0/916192182 shutdown_connections 2024-03-18T14:53:59.322 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 1 -- 172.21.15.76:0/916192182 wait complete. 2024-03-18T14:53:59.322 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 1 -- start start 2024-03-18T14:53:59.322 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 1 Processor -- start 2024-03-18T14:53:59.322 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10130 0x557bf7c21b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.322 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c1cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.323 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c19820 0x557bf7c1d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.323 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f36341adb20 con 0x557bf7c10a50 2024-03-18T14:53:59.323 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f36341aef90 con 0x557bf7c19820 2024-03-18T14:53:59.323 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f36341ae1f0 con 0x557bf7c10130 2024-03-18T14:53:59.323 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.316+0000 7f363c981740 10 client.0 _ll_register_callbacks cb 0x7f361c009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:53:59.323 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f362affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c1cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.323 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f362affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c1cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54770/0 (socket says 172.21.15.76:54770) 2024-03-18T14:53:59.323 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f3633fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c19820 0x557bf7c1d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.323 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f362affd640 1 -- 172.21.15.76:0/1204785525 learned_addr learned my addr 172.21.15.76:0/1204785525 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:53:59.323 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f36337fe640 1 --2- 172.21.15.76:0/1204785525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10130 0x557bf7c21b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.323 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f3633fff640 1 -- 172.21.15.76:0/1204785525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10130 msgr2=0x557bf7c21b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:59.323 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f3633fff640 1 --2- 172.21.15.76:0/1204785525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10130 0x557bf7c21b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.323 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f3633fff640 1 -- 172.21.15.76:0/1204785525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 msgr2=0x557bf7c1cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f3633fff640 1 --2- 172.21.15.76:0/1204785525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c1cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f3633fff640 1 -- 172.21.15.76:0/1204785525 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bf7c1dd00 con 0x557bf7c19820 2024-03-18T14:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f362affd640 1 --2- 172.21.15.76:0/1204785525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c1cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f3633fff640 1 --2- 172.21.15.76:0/1204785525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c19820 0x557bf7c1d500 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f362c09b9d0 tx=0x7f362c0ccbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb0b4a935870dcc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f3612ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f3612ffd640 1 -- 172.21.15.76:0/1204785525 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362c0b8030 con 0x557bf7c19820 2024-03-18T14:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f363c981740 1 -- 172.21.15.76:0/1204785525 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f362c0a41e0 con 0x557bf7c19820 2024-03-18T14:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f3612ffd640 1 -- 172.21.15.76:0/1204785525 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f362c0043d0 con 0x557bf7c19820 2024-03-18T14:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f3612ffd640 1 -- 172.21.15.76:0/1204785525 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f362c0ba2d0 con 0x557bf7c19820 2024-03-18T14:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f363c981740 10 client.25532 fetch_fsmap learned FSMap version 24 2024-03-18T14:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f363c981740 1 -- 172.21.15.76:0/1204785525 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557bf7c8fd60 con 0x557bf7c19820 2024-03-18T14:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.317+0000 7f3612ffd640 1 -- 172.21.15.76:0/1204785525 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f362c0a41e0 con 0x557bf7c19820 2024-03-18T14:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.318+0000 7f3612ffd640 1 -- 172.21.15.76:0/1204785525 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f362c09ae90 con 0x557bf7c19820 2024-03-18T14:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.318+0000 7f363c981740 10 client.25532 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.318+0000 7f363c981740 10 client.25532 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.318+0000 7f363c981740 20 client.25532 populate_metadata read hostname 'smithi076' 2024-03-18T14:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.318+0000 7f363c981740 1 --2- 172.21.15.76:0/1204785525 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557bf7c90360 0x557bf7c92780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.318+0000 7f363c981740 4 client.25532 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.318+0000 7f363c981740 1 -- 172.21.15.76:0/1204785525 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557bf7b7c2d0 con 0x557bf7c90360 2024-03-18T14:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.318+0000 7f36337fe640 1 --2- 172.21.15.76:0/1204785525 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557bf7c90360 0x557bf7c92780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.318+0000 7f36337fe640 1 --2- 172.21.15.76:0/1204785525 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557bf7c90360 0x557bf7c92780 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f2e4f2e35e9172c server_cookie=8221f9b8809de9dc in_seq=0 out_seq=0 2024-03-18T14:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.318+0000 7f3612ffd640 10 client.25532 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:59.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.324+0000 7f3612ffd640 1 -- 172.21.15.76:0/1204785525 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557bf7b7c2d0 con 0x557bf7c90360 2024-03-18T14:53:59.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.324+0000 7f3612ffd640 10 client.25532 handle_command_reply: tid=0 2024-03-18T14:53:59.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.324+0000 7f363c981740 1 client.25532 shutdown 2024-03-18T14:53:59.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.324+0000 7f363c981740 1 -- 172.21.15.76:0/1204785525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c19820 msgr2=0x557bf7c1d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:59.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.324+0000 7f363c981740 1 --2- 172.21.15.76:0/1204785525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c19820 0x557bf7c1d500 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f362c09b9d0 tx=0x7f362c0ccbf0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.324+0000 7f363c981740 1 -- 172.21.15.76:0/1204785525 shutdown_connections 2024-03-18T14:53:59.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.324+0000 7f363c981740 1 --2- 172.21.15.76:0/1204785525 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557bf7c90360 0x557bf7c92780 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.324+0000 7f363c981740 1 --2- 172.21.15.76:0/1204785525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bf7c19820 0x557bf7c1d500 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.324+0000 7f363c981740 1 --2- 172.21.15.76:0/1204785525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf7c10a50 0x557bf7c1cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.324+0000 7f363c981740 1 --2- 172.21.15.76:0/1204785525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf7c10130 0x557bf7c21b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.324+0000 7f3612ffd640 0 client.25532 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:53:59.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.324+0000 7f363c981740 1 -- 172.21.15.76:0/1204785525 >> 172.21.15.76:0/1204785525 conn(0x557bf7c0aa50 msgr2=0x557bf7c09320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:59.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.324+0000 7f363c981740 1 -- 172.21.15.76:0/1204785525 shutdown_connections 2024-03-18T14:53:59.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.324+0000 7f363c981740 1 -- 172.21.15.76:0/1204785525 wait complete. 2024-03-18T14:53:59.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.324+0000 7f363c981740 20 client.25532 trim_cache size 0 max 16384 2024-03-18T14:53:59.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:53:59.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:53:59.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:53:59.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:53:59.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.325+0000 7f3639cbf640 1 -- 172.21.15.76:0/1425318618 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f36200a2ab0 msgr2=0x7f36200a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:59.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.325+0000 7f3639cbf640 1 --2- 172.21.15.76:0/1425318618 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f36200a2ab0 0x7f36200a4f70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3624000c00 tx=0x7f36240058d0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.325+0000 7f3639cbf640 1 -- 172.21.15.76:0/1425318618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3634105680 msgr2=0x7f363419eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:53:59.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.325+0000 7f3639cbf640 1 --2- 172.21.15.76:0/1425318618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3634105680 0x7f363419eb60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f362c00ebc0 tx=0x7f362c00c6a0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.325+0000 7f3639cbf640 1 -- 172.21.15.76:0/1425318618 shutdown_connections 2024-03-18T14:53:59.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.325+0000 7f3639cbf640 1 --2- 172.21.15.76:0/1425318618 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f36200a2ab0 0x7f36200a4f70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.325+0000 7f3639cbf640 1 --2- 172.21.15.76:0/1425318618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3634105680 0x7f363419eb60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.325+0000 7f3639cbf640 1 --2- 172.21.15.76:0/1425318618 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36341012f0 0x7f363419a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.325+0000 7f3639cbf640 1 --2- 172.21.15.76:0/1425318618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3634100930 0x7f363419a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:53:59.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.325+0000 7f3639cbf640 1 -- 172.21.15.76:0/1425318618 >> 172.21.15.76:0/1425318618 conn(0x7f3634077530 msgr2=0x7f363410b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:53:59.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.325+0000 7f3639cbf640 1 -- 172.21.15.76:0/1425318618 shutdown_connections 2024-03-18T14:53:59.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:53:59.325+0000 7f3639cbf640 1 -- 172.21.15.76:0/1425318618 wait complete. 2024-03-18T14:53:59.350 INFO:tasks.workunit.client.0.smithi076.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-18T14:53:59.419 INFO:tasks.workunit.client.0.smithi076.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-18T14:53:59.459 INFO:tasks.workunit.client.0.smithi076.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T14:53:59.461 INFO:tasks.workunit.client.0.smithi076.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T14:53:59.461 INFO:tasks.workunit.client.0.smithi076.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-18T14:53:59.499 INFO:tasks.workunit.client.0.smithi076.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T14:53:59.503 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T14:53:59.503 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-18T14:53:59.560 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-03-18T14:53:59.561 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-03-18T14:53:59.562 DEBUG:teuthology.orchestra.run.smithi076:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e19f704afd9957ea657ab858f9ecf76401fef6f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-03-18T14:53:59.621 INFO:tasks.workunit.client.0.smithi076.stderr:+ dbench 1 2024-03-18T14:53:59.626 INFO:tasks.workunit.client.0.smithi076.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-18T14:53:59.626 INFO:tasks.workunit.client.0.smithi076.stdout: 2024-03-18T14:53:59.626 INFO:tasks.workunit.client.0.smithi076.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-18T14:53:59.630 INFO:tasks.workunit.client.0.smithi076.stdout:0 of 1 processes prepared for launch 0 sec 2024-03-18T14:53:59.730 INFO:tasks.workunit.client.0.smithi076.stdout:1 of 1 processes prepared for launch 0 sec 2024-03-18T14:53:59.730 INFO:tasks.workunit.client.0.smithi076.stdout:releasing clients 2024-03-18T14:54:00.411 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:54:00 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:54:00.033+0000 7f8f49380700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:54:00.411 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:54:00 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:54:00.033+0000 7f8f49380700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:54:00.411 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:54:00 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:54:00.033+0000 7f8f49380700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:54:00.411 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:54:00 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:54:00.033+0000 7f8f49380700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:54:00.411 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 14:54:00 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T14:54:00.033+0000 7f8f49380700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T14:54:00.730 INFO:tasks.workunit.client.0.smithi076.stdout: 1 2416 28.10 MB/sec warmup 1 sec latency 21.699 ms 2024-03-18T14:54:00.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:00 smithi076 ceph-mon[32205]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-18T14:54:00.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:00 smithi174 ceph-mon[36787]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-18T14:54:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:00 smithi077 ceph-mon[35865]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-18T14:54:01.730 INFO:tasks.workunit.client.0.smithi076.stdout: 1 5065 22.59 MB/sec warmup 2 sec latency 17.257 ms 2024-03-18T14:54:02.731 INFO:tasks.workunit.client.0.smithi076.stdout: 1 8025 20.00 MB/sec warmup 3 sec latency 16.981 ms 2024-03-18T14:54:02.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:02 smithi076 ceph-mon[32205]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-18T14:54:02.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:02 smithi174 ceph-mon[36787]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-18T14:54:02.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:02 smithi077 ceph-mon[35865]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-18T14:54:03.730 INFO:tasks.workunit.client.0.smithi076.stdout: 1 11103 19.58 MB/sec warmup 4 sec latency 21.391 ms 2024-03-18T14:54:03.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:03 smithi076 ceph-mon[32205]: mgrmap e24: x(active, since 45s), standbys: z, y 2024-03-18T14:54:03.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:03 smithi076 ceph-mon[32205]: pgmap v26: 97 pgs: 97 active+clean; 3.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 230 KiB/s wr, 22 op/s 2024-03-18T14:54:03.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:03 smithi174 ceph-mon[36787]: mgrmap e24: x(active, since 45s), standbys: z, y 2024-03-18T14:54:03.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:03 smithi174 ceph-mon[36787]: pgmap v26: 97 pgs: 97 active+clean; 3.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 230 KiB/s wr, 22 op/s 2024-03-18T14:54:03.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:03 smithi077 ceph-mon[35865]: mgrmap e24: x(active, since 45s), standbys: z, y 2024-03-18T14:54:03.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:03 smithi077 ceph-mon[35865]: pgmap v26: 97 pgs: 97 active+clean; 3.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 230 KiB/s wr, 22 op/s 2024-03-18T14:54:04.731 INFO:tasks.workunit.client.0.smithi076.stdout: 1 14460 19.42 MB/sec warmup 5 sec latency 15.648 ms 2024-03-18T14:54:05.731 INFO:tasks.workunit.client.0.smithi076.stdout: 1 17737 19.15 MB/sec warmup 6 sec latency 24.280 ms 2024-03-18T14:54:06.731 INFO:tasks.workunit.client.0.smithi076.stdout: 1 20385 17.81 MB/sec warmup 7 sec latency 28.583 ms 2024-03-18T14:54:06.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:06 smithi076 ceph-mon[32205]: pgmap v27: 97 pgs: 97 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 670 KiB/s wr, 105 op/s 2024-03-18T14:54:06.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:06 smithi174 ceph-mon[36787]: pgmap v27: 97 pgs: 97 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 670 KiB/s wr, 105 op/s 2024-03-18T14:54:06.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:06 smithi077 ceph-mon[35865]: pgmap v27: 97 pgs: 97 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 670 KiB/s wr, 105 op/s 2024-03-18T14:54:07.731 INFO:tasks.workunit.client.0.smithi076.stdout: 1 23135 17.64 MB/sec warmup 8 sec latency 13.951 ms 2024-03-18T14:54:08.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 25811 17.30 MB/sec warmup 9 sec latency 14.996 ms 2024-03-18T14:54:08.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:08 smithi076 ceph-mon[32205]: pgmap v28: 97 pgs: 97 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 669 KiB/s wr, 103 op/s 2024-03-18T14:54:08.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:08 smithi174 ceph-mon[36787]: pgmap v28: 97 pgs: 97 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 669 KiB/s wr, 103 op/s 2024-03-18T14:54:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:08 smithi077 ceph-mon[35865]: pgmap v28: 97 pgs: 97 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 669 KiB/s wr, 103 op/s 2024-03-18T14:54:09.731 INFO:tasks.workunit.client.0.smithi076.stdout: 1 28849 17.40 MB/sec warmup 10 sec latency 24.801 ms 2024-03-18T14:54:10.731 INFO:tasks.workunit.client.0.smithi076.stdout: 1 32102 17.43 MB/sec warmup 11 sec latency 15.082 ms 2024-03-18T14:54:10.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:10 smithi076 ceph-mon[32205]: pgmap v29: 97 pgs: 97 active+clean; 16 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.4 MiB/s wr, 189 op/s 2024-03-18T14:54:10.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:10 smithi174 ceph-mon[36787]: pgmap v29: 97 pgs: 97 active+clean; 16 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.4 MiB/s wr, 189 op/s 2024-03-18T14:54:10.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:10 smithi077 ceph-mon[35865]: pgmap v29: 97 pgs: 97 active+clean; 16 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.4 MiB/s wr, 189 op/s 2024-03-18T14:54:11.731 INFO:tasks.workunit.client.0.smithi076.stdout: 1 34554 16.76 MB/sec warmup 12 sec latency 14.405 ms 2024-03-18T14:54:11.910 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T14:54:11.910 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T14:54:12.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.018+0000 7f4c8720b640 1 -- 172.21.15.76:0/1519013801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c80069d50 msgr2=0x7f4c8006a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.018+0000 7f4c8720b640 1 --2- 172.21.15.76:0/1519013801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c80069d50 0x7f4c8006a1d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009930 tx=0x7f4c7402bc30 comp rx=0 tx=0).stop 2024-03-18T14:54:12.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.019+0000 7f4c8720b640 1 -- 172.21.15.76:0/1519013801 shutdown_connections 2024-03-18T14:54:12.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.019+0000 7f4c8720b640 1 --2- 172.21.15.76:0/1519013801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c8006a710 0x7f4c8010e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.019+0000 7f4c8720b640 1 --2- 172.21.15.76:0/1519013801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c80069d50 0x7f4c8006a1d0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.019+0000 7f4c8720b640 1 --2- 172.21.15.76:0/1519013801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c80107780 0x7f4c80107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.019+0000 7f4c8720b640 1 -- 172.21.15.76:0/1519013801 >> 172.21.15.76:0/1519013801 conn(0x7f4c800776c0 msgr2=0x7f4c80077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.019+0000 7f4c8720b640 1 -- 172.21.15.76:0/1519013801 shutdown_connections 2024-03-18T14:54:12.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.019+0000 7f4c8720b640 1 -- 172.21.15.76:0/1519013801 wait complete. 2024-03-18T14:54:12.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.019+0000 7f4c8720b640 1 Processor -- start 2024-03-18T14:54:12.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.019+0000 7f4c8720b640 1 -- start start 2024-03-18T14:54:12.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c8720b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c80069d50 0x7f4c80100a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c8720b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c8006a710 0x7f4c80100fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c8720b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c80107780 0x7f4c80101510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c8720b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4c80111b90 con 0x7f4c80069d50 2024-03-18T14:54:12.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c8720b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4c80111a10 con 0x7f4c80107780 2024-03-18T14:54:12.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c8720b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4c80111d10 con 0x7f4c8006a710 2024-03-18T14:54:12.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c84fa7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c80069d50 0x7f4c80100a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c84fa7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c80069d50 0x7f4c80100a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44792/0 (socket says 172.21.15.76:44792) 2024-03-18T14:54:12.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c84fa7640 1 -- 172.21.15.76:0/454511788 learned_addr learned my addr 172.21.15.76:0/454511788 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c857a8640 1 --2- 172.21.15.76:0/454511788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c80107780 0x7f4c80101510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c7ffff640 1 --2- 172.21.15.76:0/454511788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c8006a710 0x7f4c80100fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c84fa7640 1 -- 172.21.15.76:0/454511788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c8006a710 msgr2=0x7f4c80100fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c84fa7640 1 --2- 172.21.15.76:0/454511788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c8006a710 0x7f4c80100fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c84fa7640 1 -- 172.21.15.76:0/454511788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c80107780 msgr2=0x7f4c80101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c84fa7640 1 --2- 172.21.15.76:0/454511788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c80107780 0x7f4c80101510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c84fa7640 1 -- 172.21.15.76:0/454511788 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c801a65b0 con 0x7f4c80069d50 2024-03-18T14:54:12.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c84fa7640 1 --2- 172.21.15.76:0/454511788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c80069d50 0x7f4c80100a90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4c7000e9f0 tx=0x7f4c7000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c7dffb640 1 -- 172.21.15.76:0/454511788 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7000cd80 con 0x7f4c80069d50 2024-03-18T14:54:12.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c7dffb640 1 -- 172.21.15.76:0/454511788 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c70010910 con 0x7f4c80069d50 2024-03-18T14:54:12.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c8720b640 1 -- 172.21.15.76:0/454511788 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c801a68a0 con 0x7f4c80069d50 2024-03-18T14:54:12.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.020+0000 7f4c7dffb640 1 -- 172.21.15.76:0/454511788 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c70018990 con 0x7f4c80069d50 2024-03-18T14:54:12.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.021+0000 7f4c8720b640 1 -- 172.21.15.76:0/454511788 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c801a6dc0 con 0x7f4c80069d50 2024-03-18T14:54:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.023+0000 7f4c7dffb640 1 -- 172.21.15.76:0/454511788 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f4c70010ac0 con 0x7f4c80069d50 2024-03-18T14:54:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.023+0000 7f4c89df5740 1 Processor -- start 2024-03-18T14:54:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.023+0000 7f4c7dffb640 1 --2- 172.21.15.76:0/454511788 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f4c680a2a90 0x7f4c680a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.023+0000 7f4c89df5740 1 -- start start 2024-03-18T14:54:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.023+0000 7f4c857a8640 1 --2- 172.21.15.76:0/454511788 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f4c680a2a90 0x7f4c680a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.023+0000 7f4c7dffb640 1 -- 172.21.15.76:0/454511788 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f4c700cf3a0 con 0x7f4c80069d50 2024-03-18T14:54:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.023+0000 7f4c89df5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9150 0x55f74f0d9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.023+0000 7f4c89df5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d9a70 0x55f74f0e2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.023+0000 7f4c89df5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0e2650 0x55f74f0e4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.023+0000 7f4c89df5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f74f03d500 con 0x55f74f0d9150 2024-03-18T14:54:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.023+0000 7f4c89df5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f74efe5be0 con 0x55f74f0d9a70 2024-03-18T14:54:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.023+0000 7f4c89df5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f74f0395a0 con 0x55f74f0e2650 2024-03-18T14:54:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.024+0000 7f4c84fa7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9150 0x55f74f0d9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.024+0000 7f4c84fa7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9150 0x55f74f0d9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44794/0 (socket says 172.21.15.76:44794) 2024-03-18T14:54:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.024+0000 7f4c84fa7640 1 -- 172.21.15.76:0/1255755427 learned_addr learned my addr 172.21.15.76:0/1255755427 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.024+0000 7f4c7ffff640 1 --2- 172.21.15.76:0/1255755427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d9a70 0x55f74f0e2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.024+0000 7f4c857a8640 1 --2- 172.21.15.76:0/1255755427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0e2650 0x55f74f0e4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.024+0000 7f4c84fa7640 1 -- 172.21.15.76:0/1255755427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0e2650 msgr2=0x55f74f0e4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.024+0000 7f4c84fa7640 1 --2- 172.21.15.76:0/1255755427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0e2650 0x55f74f0e4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.024+0000 7f4c84fa7640 1 -- 172.21.15.76:0/1255755427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d9a70 msgr2=0x55f74f0e2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.024+0000 7f4c84fa7640 1 --2- 172.21.15.76:0/1255755427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d9a70 0x55f74f0e2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.024+0000 7f4c84fa7640 1 -- 172.21.15.76:0/1255755427 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f74eff94b0 con 0x55f74f0d9150 2024-03-18T14:54:12.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.024+0000 7f4c7ffff640 1 --2- 172.21.15.76:0/1255755427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d9a70 0x55f74f0e2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.024+0000 7f4c857a8640 1 --2- 172.21.15.76:0/1255755427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0e2650 0x55f74f0e4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:54:12.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.024+0000 7f4c857a8640 1 --2- 172.21.15.76:0/454511788 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f4c680a2a90 0x7f4c680a4f50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4c78005fd0 tx=0x7f4c78009f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.024+0000 7f4c84fa7640 1 --2- 172.21.15.76:0/1255755427 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9150 0x55f74f0d9530 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4c7000c6a0 tx=0x7f4c7009b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27c7b908b36b820f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.024+0000 7f4c66ffd640 1 -- 172.21.15.76:0/1255755427 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c70099bb0 con 0x55f74f0d9150 2024-03-18T14:54:12.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.024+0000 7f4c66ffd640 1 -- 172.21.15.76:0/1255755427 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c70097f20 con 0x55f74f0d9150 2024-03-18T14:54:12.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.024+0000 7f4c66ffd640 1 -- 172.21.15.76:0/1255755427 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c70018b30 con 0x55f74f0d9150 2024-03-18T14:54:12.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.025+0000 7f4c89df5740 1 -- 172.21.15.76:0/1255755427 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9150 msgr2=0x55f74f0d9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.025+0000 7f4c89df5740 1 --2- 172.21.15.76:0/1255755427 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9150 0x55f74f0d9530 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4c7000c6a0 tx=0x7f4c7009b970 comp rx=0 tx=0).stop 2024-03-18T14:54:12.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.025+0000 7f4c89df5740 1 -- 172.21.15.76:0/1255755427 shutdown_connections 2024-03-18T14:54:12.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.025+0000 7f4c89df5740 1 --2- 172.21.15.76:0/1255755427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0e2650 0x55f74f0e4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.025+0000 7f4c89df5740 1 --2- 172.21.15.76:0/1255755427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d9a70 0x55f74f0e2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.025+0000 7f4c89df5740 1 --2- 172.21.15.76:0/1255755427 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9150 0x55f74f0d9530 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.025+0000 7f4c89df5740 1 -- 172.21.15.76:0/1255755427 >> 172.21.15.76:0/1255755427 conn(0x55f74f0d3880 msgr2=0x55f74f0d3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.025+0000 7f4c89df5740 1 -- 172.21.15.76:0/1255755427 shutdown_connections 2024-03-18T14:54:12.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.025+0000 7f4c89df5740 1 -- 172.21.15.76:0/1255755427 wait complete. 2024-03-18T14:54:12.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.026+0000 7f4c89df5740 1 -- start start 2024-03-18T14:54:12.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.026+0000 7f4c89df5740 1 Processor -- start 2024-03-18T14:54:12.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.026+0000 7f4c89df5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9150 0x55f74f0df680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.026+0000 7f4c89df5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0d9a70 0x55f74f0dfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.026+0000 7f4c89df5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0e2650 0x55f74f0e0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.026+0000 7f4c89df5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f74f0e53e0 con 0x55f74f0d9150 2024-03-18T14:54:12.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.026+0000 7f4c89df5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f74eff9160 con 0x55f74f0e2650 2024-03-18T14:54:12.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.026+0000 7f4c89df5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f74f0e5560 con 0x55f74f0d9a70 2024-03-18T14:54:12.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.026+0000 7f4c89df5740 10 client.0 _ll_register_callbacks cb 0x55f74efdd150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:12.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.026+0000 7f4c84fa7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9150 0x55f74f0df680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.026+0000 7f4c857a8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0e2650 0x55f74f0e0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.026+0000 7f4c84fa7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9150 0x55f74f0df680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44804/0 (socket says 172.21.15.76:44804) 2024-03-18T14:54:12.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.026+0000 7f4c84fa7640 1 -- 172.21.15.76:0/2659593900 learned_addr learned my addr 172.21.15.76:0/2659593900 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.026+0000 7f4c7ffff640 1 --2- 172.21.15.76:0/2659593900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0d9a70 0x55f74f0dfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c84fa7640 1 -- 172.21.15.76:0/2659593900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0d9a70 msgr2=0x55f74f0dfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c84fa7640 1 --2- 172.21.15.76:0/2659593900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0d9a70 0x55f74f0dfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c84fa7640 1 -- 172.21.15.76:0/2659593900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0e2650 msgr2=0x55f74f0e0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c84fa7640 1 --2- 172.21.15.76:0/2659593900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0e2650 0x55f74f0e0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c84fa7640 1 -- 172.21.15.76:0/2659593900 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f74ee1b7d0 con 0x55f74f0d9150 2024-03-18T14:54:12.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c857a8640 1 --2- 172.21.15.76:0/2659593900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0e2650 0x55f74f0e0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c84fa7640 1 --2- 172.21.15.76:0/2659593900 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9150 0x55f74f0df680 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4c700a3a70 tx=0x7f4c700ccca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf2bcba4cbe1b1b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:12.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c5ffff640 1 -- 172.21.15.76:0/2659593900 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c700ba060 con 0x55f74f0d9150 2024-03-18T14:54:12.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c5ffff640 1 -- 172.21.15.76:0/2659593900 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f4c700860f0 con 0x55f74f0d9150 2024-03-18T14:54:12.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c89df5740 1 -- 172.21.15.76:0/2659593900 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c70097f20 con 0x55f74f0d9150 2024-03-18T14:54:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c5ffff640 1 -- 172.21.15.76:0/2659593900 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4c70018b30 con 0x55f74f0d9150 2024-03-18T14:54:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c89df5740 10 client.15690 fetch_fsmap learned FSMap version 24 2024-03-18T14:54:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c89df5740 1 -- 172.21.15.76:0/2659593900 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f74ed513f0 con 0x55f74f0d9150 2024-03-18T14:54:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c5ffff640 1 -- 172.21.15.76:0/2659593900 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c70097f20 con 0x55f74f0d9150 2024-03-18T14:54:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c5ffff640 1 -- 172.21.15.76:0/2659593900 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f4c7009ad00 con 0x55f74f0d9150 2024-03-18T14:54:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c89df5740 10 client.15690 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:54:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c89df5740 10 client.15690 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c89df5740 20 client.15690 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c89df5740 1 --2- 172.21.15.76:0/2659593900 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f74f0f2470 0x55f74f0f4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c89df5740 4 client.15690 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:54:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.027+0000 7f4c89df5740 1 -- 172.21.15.76:0/2659593900 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f74f04aa40 con 0x55f74f0f2470 2024-03-18T14:54:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.028+0000 7f4c7ffff640 1 --2- 172.21.15.76:0/2659593900 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f74f0f2470 0x55f74f0f4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.028+0000 7f4c7ffff640 1 --2- 172.21.15.76:0/2659593900 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f74f0f2470 0x55f74f0f4860 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e88c0d1ee33e5bc server_cookie=d1e15d9afe0ec7a in_seq=0 out_seq=0 2024-03-18T14:54:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.028+0000 7f4c5ffff640 10 client.15690 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.029+0000 7f4c5ffff640 1 -- 172.21.15.76:0/2659593900 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f74f04aa40 con 0x55f74f0f2470 2024-03-18T14:54:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.029+0000 7f4c5ffff640 10 client.15690 handle_command_reply: tid=0 2024-03-18T14:54:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.029+0000 7f4c89df5740 1 client.15690 shutdown 2024-03-18T14:54:12.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.030+0000 7f4c89df5740 1 -- 172.21.15.76:0/2659593900 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9150 msgr2=0x55f74f0df680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.030+0000 7f4c89df5740 1 --2- 172.21.15.76:0/2659593900 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9150 0x55f74f0df680 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4c700a3a70 tx=0x7f4c700ccca0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.030+0000 7f4c89df5740 1 -- 172.21.15.76:0/2659593900 shutdown_connections 2024-03-18T14:54:12.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.030+0000 7f4c89df5740 1 --2- 172.21.15.76:0/2659593900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0d9a70 0x55f74f0dfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.030+0000 7f4c89df5740 1 --2- 172.21.15.76:0/2659593900 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f74f0f2470 0x55f74f0f4860 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.030+0000 7f4c89df5740 1 --2- 172.21.15.76:0/2659593900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0e2650 0x55f74f0e0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.030+0000 7f4c89df5740 1 --2- 172.21.15.76:0/2659593900 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9150 0x55f74f0df680 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.030+0000 7f4c5ffff640 0 client.15690 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:12.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.030+0000 7f4c89df5740 1 -- 172.21.15.76:0/2659593900 >> 172.21.15.76:0/2659593900 conn(0x55f74f0d3880 msgr2=0x55f74f0d55d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.030+0000 7f4c89df5740 1 -- 172.21.15.76:0/2659593900 shutdown_connections 2024-03-18T14:54:12.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.030+0000 7f4c89df5740 1 -- 172.21.15.76:0/2659593900 wait complete. 2024-03-18T14:54:12.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.030+0000 7f4c89df5740 20 client.15690 trim_cache size 0 max 16384 2024-03-18T14:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c89df5740 1 Processor -- start 2024-03-18T14:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c89df5740 1 -- start start 2024-03-18T14:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c89df5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d3880 0x55f74f0e1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c89df5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0d9150 0x55f74f0ed7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c89df5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9a70 0x55f74f0f7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c89df5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4c801b1e40 con 0x55f74f0d9a70 2024-03-18T14:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c89df5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4c801af710 con 0x55f74f0d3880 2024-03-18T14:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c89df5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4c801ad560 con 0x55f74f0d9150 2024-03-18T14:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c857a8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9a70 0x55f74f0f7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c84fa7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d3880 0x55f74f0e1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c7ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0d9150 0x55f74f0ed7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c857a8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9a70 0x55f74f0f7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44820/0 (socket says 172.21.15.76:44820) 2024-03-18T14:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c857a8640 1 -- 172.21.15.76:0/3067052739 learned_addr learned my addr 172.21.15.76:0/3067052739 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c84fa7640 1 -- 172.21.15.76:0/3067052739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0d9150 msgr2=0x55f74f0ed7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c84fa7640 1 --2- 172.21.15.76:0/3067052739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0d9150 0x55f74f0ed7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c84fa7640 1 -- 172.21.15.76:0/3067052739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9a70 msgr2=0x55f74f0f7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c84fa7640 1 --2- 172.21.15.76:0/3067052739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9a70 0x55f74f0f7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c84fa7640 1 -- 172.21.15.76:0/3067052739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f74ed110f0 con 0x55f74f0d3880 2024-03-18T14:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c857a8640 1 --2- 172.21.15.76:0/3067052739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9a70 0x55f74f0f7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c7ffff640 1 --2- 172.21.15.76:0/3067052739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0d9150 0x55f74f0ed7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.048+0000 7f4c84fa7640 1 --2- 172.21.15.76:0/3067052739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d3880 0x55f74f0e1db0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4c70098280 tx=0x7f4c70099dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70fe8d81ec0029ba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.049+0000 7f4c667fc640 1 -- 172.21.15.76:0/3067052739 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c70097970 con 0x55f74f0d3880 2024-03-18T14:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.049+0000 7f4c667fc640 1 -- 172.21.15.76:0/3067052739 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c700cc900 con 0x55f74f0d3880 2024-03-18T14:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.049+0000 7f4c667fc640 1 -- 172.21.15.76:0/3067052739 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c70018cd0 con 0x55f74f0d3880 2024-03-18T14:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.049+0000 7f4c89df5740 1 -- 172.21.15.76:0/3067052739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d3880 msgr2=0x55f74f0e1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.049+0000 7f4c89df5740 1 --2- 172.21.15.76:0/3067052739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d3880 0x55f74f0e1db0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4c70098280 tx=0x7f4c70099dc0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.049+0000 7f4c89df5740 1 -- 172.21.15.76:0/3067052739 shutdown_connections 2024-03-18T14:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.049+0000 7f4c89df5740 1 --2- 172.21.15.76:0/3067052739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0d9150 0x55f74f0ed7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.049+0000 7f4c89df5740 1 --2- 172.21.15.76:0/3067052739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9a70 0x55f74f0f7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.049+0000 7f4c89df5740 1 --2- 172.21.15.76:0/3067052739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d3880 0x55f74f0e1db0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.049+0000 7f4c89df5740 1 -- 172.21.15.76:0/3067052739 >> 172.21.15.76:0/3067052739 conn(0x55f74f0db050 msgr2=0x55f74f0dfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.049+0000 7f4c89df5740 1 -- 172.21.15.76:0/3067052739 shutdown_connections 2024-03-18T14:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.049+0000 7f4c89df5740 1 -- 172.21.15.76:0/3067052739 wait complete. 2024-03-18T14:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.049+0000 7f4c89df5740 1 -- start start 2024-03-18T14:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c89df5740 1 Processor -- start 2024-03-18T14:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c89df5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d3880 0x55f74f0f7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c89df5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0d9150 0x55f74f0f7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c89df5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9a70 0x55f74f0f5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c89df5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4c801adbd0 con 0x55f74f0d9a70 2024-03-18T14:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c89df5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4c801af040 con 0x55f74f0d3880 2024-03-18T14:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c89df5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4c801ae2a0 con 0x55f74f0d9150 2024-03-18T14:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c89df5740 10 client.0 _ll_register_callbacks cb 0x7f4c780093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c857a8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9a70 0x55f74f0f5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c84fa7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d3880 0x55f74f0f7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c857a8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9a70 0x55f74f0f5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44822/0 (socket says 172.21.15.76:44822) 2024-03-18T14:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c857a8640 1 -- 172.21.15.76:0/3810576988 learned_addr learned my addr 172.21.15.76:0/3810576988 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c84fa7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d3880 0x55f74f0f7530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:38664/0 (socket says 172.21.15.76:38664) 2024-03-18T14:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c7ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0d9150 0x55f74f0f7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c857a8640 1 -- 172.21.15.76:0/3810576988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0d9150 msgr2=0x55f74f0f7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c857a8640 1 --2- 172.21.15.76:0/3810576988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0d9150 0x55f74f0f7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c857a8640 1 -- 172.21.15.76:0/3810576988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d3880 msgr2=0x55f74f0f7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c857a8640 1 --2- 172.21.15.76:0/3810576988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d3880 0x55f74f0f7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c857a8640 1 -- 172.21.15.76:0/3810576988 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f74f0049d0 con 0x55f74f0d9a70 2024-03-18T14:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.050+0000 7f4c84fa7640 1 --2- 172.21.15.76:0/3810576988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d3880 0x55f74f0f7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.051+0000 7f4c857a8640 1 --2- 172.21.15.76:0/3810576988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9a70 0x55f74f0f5a90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4c78011ca0 tx=0x7f4c78004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b89289305b08c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.051+0000 7f4c5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.051+0000 7f4c5ffff640 1 -- 172.21.15.76:0/3810576988 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c78017040 con 0x55f74f0d9a70 2024-03-18T14:54:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.051+0000 7f4c5ffff640 1 -- 172.21.15.76:0/3810576988 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f4c78009040 con 0x55f74f0d9a70 2024-03-18T14:54:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.051+0000 7f4c89df5740 1 -- 172.21.15.76:0/3810576988 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c700cc900 con 0x55f74f0d9a70 2024-03-18T14:54:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.051+0000 7f4c5ffff640 1 -- 172.21.15.76:0/3810576988 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4c7801dd20 con 0x55f74f0d9a70 2024-03-18T14:54:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.051+0000 7f4c89df5740 10 client.15696 fetch_fsmap learned FSMap version 24 2024-03-18T14:54:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.051+0000 7f4c89df5740 1 -- 172.21.15.76:0/3810576988 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f74f158e60 con 0x55f74f0d9a70 2024-03-18T14:54:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.051+0000 7f4c5ffff640 1 -- 172.21.15.76:0/3810576988 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c700cc900 con 0x55f74f0d9a70 2024-03-18T14:54:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.051+0000 7f4c5ffff640 1 -- 172.21.15.76:0/3810576988 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f4c78005860 con 0x55f74f0d9a70 2024-03-18T14:54:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.051+0000 7f4c89df5740 10 client.15696 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:54:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.051+0000 7f4c89df5740 10 client.15696 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.051+0000 7f4c89df5740 20 client.15696 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.051+0000 7f4c89df5740 1 --2- 172.21.15.76:0/3810576988 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f74f0e2650 0x55f74f15b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.051+0000 7f4c89df5740 4 client.15696 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T14:54:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.051+0000 7f4c89df5740 1 -- 172.21.15.76:0/3810576988 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f74f044d90 con 0x55f74f0e2650 2024-03-18T14:54:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.051+0000 7f4c84fa7640 1 --2- 172.21.15.76:0/3810576988 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f74f0e2650 0x55f74f15b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.052+0000 7f4c5ffff640 10 client.15696 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.052+0000 7f4c84fa7640 1 --2- 172.21.15.76:0/3810576988 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f74f0e2650 0x55f74f15b6a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b33a20848d73f8d5 server_cookie=7dee60e3e27e7753 in_seq=0 out_seq=0 2024-03-18T14:54:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.053+0000 7f4c5ffff640 1 -- 172.21.15.76:0/3810576988 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f74f044d90 con 0x55f74f0e2650 2024-03-18T14:54:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.053+0000 7f4c5ffff640 10 client.15696 handle_command_reply: tid=0 2024-03-18T14:54:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.053+0000 7f4c89df5740 1 client.15696 shutdown 2024-03-18T14:54:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.053+0000 7f4c89df5740 1 -- 172.21.15.76:0/3810576988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9a70 msgr2=0x55f74f0f5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.053+0000 7f4c89df5740 1 --2- 172.21.15.76:0/3810576988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9a70 0x55f74f0f5a90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4c78011ca0 tx=0x7f4c78004ab0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.053+0000 7f4c89df5740 1 -- 172.21.15.76:0/3810576988 shutdown_connections 2024-03-18T14:54:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.053+0000 7f4c89df5740 1 --2- 172.21.15.76:0/3810576988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f74f0d9150 0x55f74f0f7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.053+0000 7f4c89df5740 1 --2- 172.21.15.76:0/3810576988 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f74f0e2650 0x55f74f15b6a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.053+0000 7f4c89df5740 1 --2- 172.21.15.76:0/3810576988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f74f0d9a70 0x55f74f0f5a90 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.053+0000 7f4c89df5740 1 --2- 172.21.15.76:0/3810576988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f74f0d3880 0x55f74f0f7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.053+0000 7f4c5ffff640 0 client.15696 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.053+0000 7f4c89df5740 1 -- 172.21.15.76:0/3810576988 >> 172.21.15.76:0/3810576988 conn(0x55f74f0db050 msgr2=0x55f74f0d4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.053+0000 7f4c89df5740 1 -- 172.21.15.76:0/3810576988 shutdown_connections 2024-03-18T14:54:12.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.053+0000 7f4c89df5740 1 -- 172.21.15.76:0/3810576988 wait complete. 2024-03-18T14:54:12.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.053+0000 7f4c89df5740 20 client.15696 trim_cache size 0 max 16384 2024-03-18T14:54:12.060 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T14:54:12.060 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-18T14:54:12.060 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-18T14:54:12.060 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T14:54:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.054+0000 7f4c8720b640 1 -- 172.21.15.76:0/454511788 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f4c680a2a90 msgr2=0x7f4c680a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.054+0000 7f4c8720b640 1 --2- 172.21.15.76:0/454511788 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f4c680a2a90 0x7f4c680a4f50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4c78005fd0 tx=0x7f4c78009f90 comp rx=0 tx=0).stop 2024-03-18T14:54:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.054+0000 7f4c8720b640 1 -- 172.21.15.76:0/454511788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c80069d50 msgr2=0x7f4c80100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.054+0000 7f4c8720b640 1 --2- 172.21.15.76:0/454511788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c80069d50 0x7f4c80100a90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4c7000e9f0 tx=0x7f4c7000eeb0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.054+0000 7f4c8720b640 1 -- 172.21.15.76:0/454511788 shutdown_connections 2024-03-18T14:54:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.054+0000 7f4c8720b640 1 --2- 172.21.15.76:0/454511788 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f4c680a2a90 0x7f4c680a4f50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.054+0000 7f4c8720b640 1 --2- 172.21.15.76:0/454511788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c8006a710 0x7f4c80100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.054+0000 7f4c8720b640 1 --2- 172.21.15.76:0/454511788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c80107780 0x7f4c80101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.054+0000 7f4c8720b640 1 --2- 172.21.15.76:0/454511788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c80069d50 0x7f4c80100a90 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.054+0000 7f4c8720b640 1 -- 172.21.15.76:0/454511788 >> 172.21.15.76:0/454511788 conn(0x7f4c800776c0 msgr2=0x7f4c800ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.054+0000 7f4c8720b640 1 -- 172.21.15.76:0/454511788 shutdown_connections 2024-03-18T14:54:12.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.054+0000 7f4c8720b640 1 -- 172.21.15.76:0/454511788 wait complete. 2024-03-18T14:54:12.065 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T14:54:12.066 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:54:12.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.178+0000 7f700c913640 1 -- 172.21.15.76:0/3161312635 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f700806a520 msgr2=0x7f700810e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.178+0000 7f700c913640 1 --2- 172.21.15.76:0/3161312635 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f700806a520 0x7f700810e690 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7000009cd0 tx=0x7f700002c0f0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.179+0000 7f700c913640 1 -- 172.21.15.76:0/3161312635 shutdown_connections 2024-03-18T14:54:12.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.179+0000 7f700c913640 1 --2- 172.21.15.76:0/3161312635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7008069ba0 0x7f7008069fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.179+0000 7f700c913640 1 --2- 172.21.15.76:0/3161312635 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f700806a520 0x7f700810e690 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.179+0000 7f700c913640 1 --2- 172.21.15.76:0/3161312635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70081077d0 0x7f7008107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.179+0000 7f700c913640 1 -- 172.21.15.76:0/3161312635 >> 172.21.15.76:0/3161312635 conn(0x7f7008077810 msgr2=0x7f7008077c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.179+0000 7f700c913640 1 -- 172.21.15.76:0/3161312635 shutdown_connections 2024-03-18T14:54:12.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.179+0000 7f700c913640 1 -- 172.21.15.76:0/3161312635 wait complete. 2024-03-18T14:54:12.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.179+0000 7f700c913640 1 Processor -- start 2024-03-18T14:54:12.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.180+0000 7f700c913640 1 -- start start 2024-03-18T14:54:12.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.180+0000 7f700c913640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7008069ba0 0x7f7008198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.180+0000 7f700c913640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f700806a520 0x7f7008198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.180+0000 7f700c913640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70081077d0 0x7f700819c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.180+0000 7f700c913640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7008111bd0 con 0x7f70081077d0 2024-03-18T14:54:12.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.180+0000 7f700c913640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7008111a50 con 0x7f7008069ba0 2024-03-18T14:54:12.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.180+0000 7f700c913640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7008111d50 con 0x7f700806a520 2024-03-18T14:54:12.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.180+0000 7f7006d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70081077d0 0x7f700819c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.180+0000 7f700659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7008069ba0 0x7f7008198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.180+0000 7f7005d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f700806a520 0x7f7008198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.180+0000 7f700659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7008069ba0 0x7f7008198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:38676/0 (socket says 172.21.15.76:38676) 2024-03-18T14:54:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.180+0000 7f700659c640 1 -- 172.21.15.76:0/2375080819 learned_addr learned my addr 172.21.15.76:0/2375080819 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.180+0000 7f7006d9d640 1 -- 172.21.15.76:0/2375080819 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f700806a520 msgr2=0x7f7008198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.180+0000 7f7006d9d640 1 --2- 172.21.15.76:0/2375080819 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f700806a520 0x7f7008198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.180+0000 7f7006d9d640 1 -- 172.21.15.76:0/2375080819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7008069ba0 msgr2=0x7f7008198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.180+0000 7f7006d9d640 1 --2- 172.21.15.76:0/2375080819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7008069ba0 0x7f7008198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.180+0000 7f7006d9d640 1 -- 172.21.15.76:0/2375080819 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f700819d100 con 0x7f70081077d0 2024-03-18T14:54:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.181+0000 7f7006d9d640 1 --2- 172.21.15.76:0/2375080819 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70081077d0 0x7f700819c9d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7000009bf0 tx=0x7f7000032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.181+0000 7f6ff77fe640 1 -- 172.21.15.76:0/2375080819 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7000041020 con 0x7f70081077d0 2024-03-18T14:54:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.181+0000 7f6ff77fe640 1 -- 172.21.15.76:0/2375080819 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7000032e80 con 0x7f70081077d0 2024-03-18T14:54:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.181+0000 7f700c913640 1 -- 172.21.15.76:0/2375080819 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f700819d390 con 0x7f70081077d0 2024-03-18T14:54:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.181+0000 7f6ff77fe640 1 -- 172.21.15.76:0/2375080819 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7000039040 con 0x7f70081077d0 2024-03-18T14:54:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.181+0000 7f700c913640 1 -- 172.21.15.76:0/2375080819 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70081a4c80 con 0x7f70081077d0 2024-03-18T14:54:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.182+0000 7f700c913640 1 -- 172.21.15.76:0/2375080819 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7008102f00 con 0x7f70081077d0 2024-03-18T14:54:12.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.184+0000 7f6ff77fe640 1 -- 172.21.15.76:0/2375080819 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f7000044020 con 0x7f70081077d0 2024-03-18T14:54:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.184+0000 7f6ff77fe640 1 --2- 172.21.15.76:0/2375080819 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f6ff00a2a90 0x7f6ff00a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.185+0000 7f6ff77fe640 1 -- 172.21.15.76:0/2375080819 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f70000f0b00 con 0x7f70081077d0 2024-03-18T14:54:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.185+0000 7f700659c640 1 --2- 172.21.15.76:0/2375080819 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f6ff00a2a90 0x7f6ff00a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.186+0000 7f700659c640 1 --2- 172.21.15.76:0/2375080819 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f6ff00a2a90 0x7f6ff00a4f50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009c10 tx=0x7f6ff8005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.192+0000 7f6ff77fe640 1 -- 172.21.15.76:0/2375080819 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f700004b020 con 0x7f70081077d0 2024-03-18T14:54:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.417+0000 7fe78ef76640 1 -- 172.21.15.76:0/1005710127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe788100930 msgr2=0x7fe788100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.417+0000 7fe78ef76640 1 --2- 172.21.15.76:0/1005710127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe788100930 0x7fe788100db0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe774009900 tx=0x7fe77402bc30 comp rx=0 tx=0).stop 2024-03-18T14:54:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.417+0000 7fe78ef76640 1 -- 172.21.15.76:0/1005710127 shutdown_connections 2024-03-18T14:54:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.417+0000 7fe78ef76640 1 --2- 172.21.15.76:0/1005710127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7881012f0 0x7fe78810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.417+0000 7fe78ef76640 1 --2- 172.21.15.76:0/1005710127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe788100930 0x7fe788100db0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.417+0000 7fe78ef76640 1 --2- 172.21.15.76:0/1005710127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe788105680 0x7fe788105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.417+0000 7fe78ef76640 1 -- 172.21.15.76:0/1005710127 >> 172.21.15.76:0/1005710127 conn(0x7fe788077530 msgr2=0x7fe788077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.417+0000 7fe78ef76640 1 -- 172.21.15.76:0/1005710127 shutdown_connections 2024-03-18T14:54:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.417+0000 7fe78ef76640 1 -- 172.21.15.76:0/1005710127 wait complete. 2024-03-18T14:54:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe78ef76640 1 Processor -- start 2024-03-18T14:54:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe78ef76640 1 -- start start 2024-03-18T14:54:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe78ef76640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe788100930 0x7fe788197040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe78ef76640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7881012f0 0x7fe788197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe78ef76640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe788105680 0x7fe78819b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe78ef76640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe788110ba0 con 0x7fe788100930 2024-03-18T14:54:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe78ef76640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe788110a20 con 0x7fe7881012f0 2024-03-18T14:54:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe78ef76640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe788110d20 con 0x7fe788105680 2024-03-18T14:54:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe78cd12640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe788100930 0x7fe788197040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe787fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7881012f0 0x7fe788197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe787fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7881012f0 0x7fe788197580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:38686/0 (socket says 172.21.15.76:38686) 2024-03-18T14:54:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe787fff640 1 -- 172.21.15.76:0/3701775626 learned_addr learned my addr 172.21.15.76:0/3701775626 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe78d513640 1 --2- 172.21.15.76:0/3701775626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe788105680 0x7fe78819b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe78cd12640 1 -- 172.21.15.76:0/3701775626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe788105680 msgr2=0x7fe78819b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe78cd12640 1 --2- 172.21.15.76:0/3701775626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe788105680 0x7fe78819b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe78cd12640 1 -- 172.21.15.76:0/3701775626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7881012f0 msgr2=0x7fe788197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe78cd12640 1 --2- 172.21.15.76:0/3701775626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7881012f0 0x7fe788197580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe78cd12640 1 -- 172.21.15.76:0/3701775626 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe78819bfe0 con 0x7fe788100930 2024-03-18T14:54:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe787fff640 1 --2- 172.21.15.76:0/3701775626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7881012f0 0x7fe788197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.418+0000 7fe78d513640 1 --2- 172.21.15.76:0/3701775626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe788105680 0x7fe78819b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.419+0000 7fe78cd12640 1 --2- 172.21.15.76:0/3701775626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe788100930 0x7fe788197040 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe77c00e9f0 tx=0x7fe77c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.419+0000 7fe785ffb640 1 -- 172.21.15.76:0/3701775626 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77c00ccc0 con 0x7fe788100930 2024-03-18T14:54:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.419+0000 7fe785ffb640 1 -- 172.21.15.76:0/3701775626 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe77c010910 con 0x7fe788100930 2024-03-18T14:54:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.419+0000 7fe785ffb640 1 -- 172.21.15.76:0/3701775626 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77c018960 con 0x7fe788100930 2024-03-18T14:54:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.419+0000 7fe78ef76640 1 -- 172.21.15.76:0/3701775626 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe78819c2d0 con 0x7fe788100930 2024-03-18T14:54:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.419+0000 7fe78ef76640 1 -- 172.21.15.76:0/3701775626 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe788068e60 con 0x7fe788100930 2024-03-18T14:54:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.421+0000 7fe785ffb640 1 -- 172.21.15.76:0/3701775626 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7fe77c010ac0 con 0x7fe788100930 2024-03-18T14:54:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe791c33740 1 Processor -- start 2024-03-18T14:54:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe791c33740 1 -- start start 2024-03-18T14:54:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe791c33740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd94a130 0x55cedd94a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe791c33740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94aa50 0x55cedd9532e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe785ffb640 1 --2- 172.21.15.76:0/3701775626 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe7780a2bb0 0x7fe7780a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe791c33740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd953820 0x55cedd955c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe791c33740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55cedd8aea40 con 0x55cedd94aa50 2024-03-18T14:54:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe791c33740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cedd857920 con 0x55cedd953820 2024-03-18T14:54:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe791c33740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cedd8aaae0 con 0x55cedd94a130 2024-03-18T14:54:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe78cd12640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94aa50 0x55cedd9532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe78d513640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd94a130 0x55cedd94a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe78cd12640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94aa50 0x55cedd9532e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44874/0 (socket says 172.21.15.76:44874) 2024-03-18T14:54:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe78cd12640 1 -- 172.21.15.76:0/595251080 learned_addr learned my addr 172.21.15.76:0/595251080 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe787fff640 1 --2- 172.21.15.76:0/3701775626 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe7780a2bb0 0x7fe7780a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe78d513640 1 --2- 172.21.15.76:0/595251080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd953820 0x55cedd955c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe785ffb640 1 -- 172.21.15.76:0/3701775626 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fe77c0cf380 con 0x7fe788100930 2024-03-18T14:54:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe78cd12640 1 -- 172.21.15.76:0/595251080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd94a130 msgr2=0x55cedd94a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe78cd12640 1 --2- 172.21.15.76:0/595251080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd94a130 0x55cedd94a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe78cd12640 1 -- 172.21.15.76:0/595251080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd953820 msgr2=0x55cedd955c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe78cd12640 1 --2- 172.21.15.76:0/595251080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd953820 0x55cedd955c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe78cd12640 1 -- 172.21.15.76:0/595251080 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cedd86f460 con 0x55cedd94aa50 2024-03-18T14:54:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.422+0000 7fe78d513640 1 --2- 172.21.15.76:0/595251080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd94a130 0x55cedd94a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.423+0000 7fe78cd12640 1 --2- 172.21.15.76:0/595251080 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94aa50 0x55cedd9532e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe77c0cd120 tx=0x7fe77c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eed5711e429c294 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.423+0000 7fe787fff640 1 --2- 172.21.15.76:0/3701775626 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe7780a2bb0 0x7fe7780a5070 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe774000c00 tx=0x7fe77403d040 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.423+0000 7fe772ffd640 1 -- 172.21.15.76:0/595251080 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77c0a3f80 con 0x55cedd94aa50 2024-03-18T14:54:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.423+0000 7fe772ffd640 1 -- 172.21.15.76:0/595251080 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe77c0960e0 con 0x55cedd94aa50 2024-03-18T14:54:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.423+0000 7fe772ffd640 1 -- 172.21.15.76:0/595251080 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77c0ccee0 con 0x55cedd94aa50 2024-03-18T14:54:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.423+0000 7fe791c33740 1 -- 172.21.15.76:0/595251080 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94aa50 msgr2=0x55cedd9532e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.423+0000 7fe791c33740 1 --2- 172.21.15.76:0/595251080 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94aa50 0x55cedd9532e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe77c0cd120 tx=0x7fe77c00cf60 comp rx=0 tx=0).stop 2024-03-18T14:54:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.423+0000 7fe791c33740 1 -- 172.21.15.76:0/595251080 shutdown_connections 2024-03-18T14:54:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.424+0000 7fe791c33740 1 --2- 172.21.15.76:0/595251080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd953820 0x55cedd955c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.424+0000 7fe791c33740 1 --2- 172.21.15.76:0/595251080 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94aa50 0x55cedd9532e0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.424+0000 7fe791c33740 1 --2- 172.21.15.76:0/595251080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd94a130 0x55cedd94a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.424+0000 7fe791c33740 1 -- 172.21.15.76:0/595251080 >> 172.21.15.76:0/595251080 conn(0x55cedd944a50 msgr2=0x55cedd944e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.424+0000 7fe791c33740 1 -- 172.21.15.76:0/595251080 shutdown_connections 2024-03-18T14:54:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.424+0000 7fe791c33740 1 -- 172.21.15.76:0/595251080 wait complete. 2024-03-18T14:54:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.424+0000 7fe791c33740 1 -- start start 2024-03-18T14:54:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.424+0000 7fe791c33740 1 Processor -- start 2024-03-18T14:54:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.424+0000 7fe791c33740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd94a130 0x55cedd950940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.424+0000 7fe791c33740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94aa50 0x55cedd950e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.424+0000 7fe791c33740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd953820 0x55cedd9513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.424+0000 7fe791c33740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55cedd9563f0 con 0x55cedd94aa50 2024-03-18T14:54:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.424+0000 7fe791c33740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cedd86ad70 con 0x55cedd94a130 2024-03-18T14:54:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.424+0000 7fe791c33740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cedd956570 con 0x55cedd953820 2024-03-18T14:54:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.424+0000 7fe791c33740 10 client.0 _ll_register_callbacks cb 0x55cedd5c2a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.424+0000 7fe787fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94aa50 0x55cedd950e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.424+0000 7fe78cd12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd94a130 0x55cedd950940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.425+0000 7fe787fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94aa50 0x55cedd950e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44876/0 (socket says 172.21.15.76:44876) 2024-03-18T14:54:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.425+0000 7fe787fff640 1 -- 172.21.15.76:0/1672208914 learned_addr learned my addr 172.21.15.76:0/1672208914 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.425+0000 7fe78d513640 1 --2- 172.21.15.76:0/1672208914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd953820 0x55cedd9513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.425+0000 7fe78cd12640 1 -- 172.21.15.76:0/1672208914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd953820 msgr2=0x55cedd9513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.425+0000 7fe78cd12640 1 --2- 172.21.15.76:0/1672208914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd953820 0x55cedd9513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.425+0000 7fe78cd12640 1 -- 172.21.15.76:0/1672208914 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94aa50 msgr2=0x55cedd950e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.425+0000 7fe78cd12640 1 --2- 172.21.15.76:0/1672208914 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94aa50 0x55cedd950e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.425+0000 7fe78cd12640 1 -- 172.21.15.76:0/1672208914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cedd5c3210 con 0x55cedd94a130 2024-03-18T14:54:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.425+0000 7fe787fff640 1 --2- 172.21.15.76:0/1672208914 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94aa50 0x55cedd950e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.425+0000 7fe78cd12640 1 --2- 172.21.15.76:0/1672208914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd94a130 0x55cedd950940 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x55cedd94afe0 tx=0x7fe77c097c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cc65a7bd1a3fb03 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.425+0000 7fe767fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:54:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.425+0000 7fe767fff640 1 -- 172.21.15.76:0/1672208914 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77c0a3f80 con 0x55cedd94a130 2024-03-18T14:54:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.425+0000 7fe791c33740 1 -- 172.21.15.76:0/1672208914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe77c0960e0 con 0x55cedd94a130 2024-03-18T14:54:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.425+0000 7fe767fff640 1 -- 172.21.15.76:0/1672208914 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fe77c0ccc80 con 0x55cedd94a130 2024-03-18T14:54:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.425+0000 7fe767fff640 1 -- 172.21.15.76:0/1672208914 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe77c0ccec0 con 0x55cedd94a130 2024-03-18T14:54:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.425+0000 7fe791c33740 10 client.25574 fetch_fsmap learned FSMap version 24 2024-03-18T14:54:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.425+0000 7fe791c33740 1 -- 172.21.15.76:0/1672208914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55cedd68f620 con 0x55cedd94a130 2024-03-18T14:54:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.426+0000 7fe767fff640 1 -- 172.21.15.76:0/1672208914 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe77c0960e0 con 0x55cedd94a130 2024-03-18T14:54:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.426+0000 7fe767fff640 1 -- 172.21.15.76:0/1672208914 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fe77c018c00 con 0x55cedd94a130 2024-03-18T14:54:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.426+0000 7fe791c33740 10 client.25574 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:54:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.426+0000 7fe791c33740 10 client.25574 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.426+0000 7fe791c33740 20 client.25574 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.426+0000 7fe791c33740 1 --2- 172.21.15.76:0/1672208914 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55cedd9638c0 0x55cedd965cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.426+0000 7fe791c33740 4 client.25574 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:54:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.426+0000 7fe791c33740 1 -- 172.21.15.76:0/1672208914 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cedd8bbf80 con 0x55cedd9638c0 2024-03-18T14:54:12.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.426+0000 7f700c913640 1 -- 172.21.15.76:0/2375080819 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7008109ce0 con 0x7f70081077d0 2024-03-18T14:54:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.426+0000 7fe787fff640 1 --2- 172.21.15.76:0/1672208914 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55cedd9638c0 0x55cedd965cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.426+0000 7fe767fff640 10 client.25574 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.426+0000 7fe787fff640 1 --2- 172.21.15.76:0/1672208914 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55cedd9638c0 0x55cedd965cb0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0c13d78da542702 server_cookie=3c17feeaa662b1df in_seq=0 out_seq=0 2024-03-18T14:54:12.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.427+0000 7f6ff77fe640 1 -- 172.21.15.76:0/2375080819 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11863 (secure 0 0 0) 0x7f70000bd770 con 0x7f70081077d0 2024-03-18T14:54:12.434 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:54:12.434 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/2801069369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":2801069369},{"type":"v1","addr":"172.21.15.76:6839","nonce":2801069369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2645229232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2645229232},{"type":"v1","addr":"172.21.15.76:6841","nonce":2645229232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T14:51:01.293009+0000","modified":"2024-03-18T14:53:02.728776+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24490,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/3551464902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3551464902},{"type":"v1","addr":"172.21.15.77:6835","nonce":3551464902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6839/399810652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":399810652},{"type":"v1","addr":"172.21.15.174:6839","nonce":399810652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1685041985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1685041985},{"type":"v1","addr":"172.21.15.77:6839","nonce":1685041985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524,24463,24490]},"id":1}]} 2024-03-18T14:54:12.435 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 24 2024-03-18T14:54:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.429+0000 7f700c913640 1 -- 172.21.15.76:0/2375080819 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f6ff00a2a90 msgr2=0x7f6ff00a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.429+0000 7f700c913640 1 --2- 172.21.15.76:0/2375080819 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f6ff00a2a90 0x7f6ff00a4f50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009c10 tx=0x7f6ff8005f90 comp rx=0 tx=0).stop 2024-03-18T14:54:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.429+0000 7f700c913640 1 -- 172.21.15.76:0/2375080819 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70081077d0 msgr2=0x7f700819c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.429+0000 7f700c913640 1 --2- 172.21.15.76:0/2375080819 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70081077d0 0x7f700819c9d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7000009bf0 tx=0x7f7000032c90 comp rx=0 tx=0).stop 2024-03-18T14:54:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.429+0000 7f700c913640 1 -- 172.21.15.76:0/2375080819 shutdown_connections 2024-03-18T14:54:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.429+0000 7f700c913640 1 --2- 172.21.15.76:0/2375080819 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f6ff00a2a90 0x7f6ff00a4f50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.429+0000 7f700c913640 1 --2- 172.21.15.76:0/2375080819 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f700806a520 0x7f7008198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.429+0000 7f700c913640 1 --2- 172.21.15.76:0/2375080819 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70081077d0 0x7f700819c9d0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.429+0000 7f700c913640 1 --2- 172.21.15.76:0/2375080819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7008069ba0 0x7f7008198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.429+0000 7f700c913640 1 -- 172.21.15.76:0/2375080819 >> 172.21.15.76:0/2375080819 conn(0x7f7008077810 msgr2=0x7f70080fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.429+0000 7f700c913640 1 -- 172.21.15.76:0/2375080819 shutdown_connections 2024-03-18T14:54:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.429+0000 7f700c913640 1 -- 172.21.15.76:0/2375080819 wait complete. 2024-03-18T14:54:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.429+0000 7fe767fff640 1 -- 172.21.15.76:0/1672208914 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cedd8bbf80 con 0x55cedd9638c0 2024-03-18T14:54:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.429+0000 7fe767fff640 10 client.25574 handle_command_reply: tid=0 2024-03-18T14:54:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.429+0000 7fe791c33740 1 client.25574 shutdown 2024-03-18T14:54:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.430+0000 7fe791c33740 1 -- 172.21.15.76:0/1672208914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd94a130 msgr2=0x55cedd950940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.430+0000 7fe791c33740 1 --2- 172.21.15.76:0/1672208914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd94a130 0x55cedd950940 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x55cedd94afe0 tx=0x7fe77c097c60 comp rx=0 tx=0).stop 2024-03-18T14:54:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.430+0000 7fe791c33740 1 -- 172.21.15.76:0/1672208914 shutdown_connections 2024-03-18T14:54:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.430+0000 7fe791c33740 1 --2- 172.21.15.76:0/1672208914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd953820 0x55cedd9513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.430+0000 7fe791c33740 1 --2- 172.21.15.76:0/1672208914 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55cedd9638c0 0x55cedd965cb0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.430+0000 7fe791c33740 1 --2- 172.21.15.76:0/1672208914 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94aa50 0x55cedd950e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.430+0000 7fe767fff640 0 client.25574 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.430+0000 7fe791c33740 1 --2- 172.21.15.76:0/1672208914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd94a130 0x55cedd950940 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.430+0000 7fe791c33740 1 -- 172.21.15.76:0/1672208914 >> 172.21.15.76:0/1672208914 conn(0x55cedd944a50 msgr2=0x55cedd9467a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.430+0000 7fe791c33740 1 -- 172.21.15.76:0/1672208914 shutdown_connections 2024-03-18T14:54:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.430+0000 7fe791c33740 1 -- 172.21.15.76:0/1672208914 wait complete. 2024-03-18T14:54:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.430+0000 7fe791c33740 20 client.25574 trim_cache size 0 max 16384 2024-03-18T14:54:12.440 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T14:54:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.444+0000 7fe791c33740 1 Processor -- start 2024-03-18T14:54:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.444+0000 7fe791c33740 1 -- start start 2024-03-18T14:54:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.445+0000 7fe791c33740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94a130 0x55cedd953070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.445+0000 7fe791c33740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd94aa50 0x55cedd968cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.445+0000 7fe791c33740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd953820 0x55cedd94dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.445+0000 7fe791c33740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe7881adaa0 con 0x55cedd94a130 2024-03-18T14:54:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.445+0000 7fe791c33740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe7881ab370 con 0x55cedd94aa50 2024-03-18T14:54:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.445+0000 7fe791c33740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe7881a91c0 con 0x55cedd953820 2024-03-18T14:54:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.445+0000 7fe78d513640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd953820 0x55cedd94dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.445+0000 7fe78d513640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd953820 0x55cedd94dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:53394/0 (socket says 172.21.15.76:53394) 2024-03-18T14:54:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.445+0000 7fe78cd12640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94a130 0x55cedd953070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.445+0000 7fe78d513640 1 -- 172.21.15.76:0/1565871991 learned_addr learned my addr 172.21.15.76:0/1565871991 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.445+0000 7fe787fff640 1 --2- 172.21.15.76:0/1565871991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd94aa50 0x55cedd968cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.445+0000 7fe78d513640 1 -- 172.21.15.76:0/1565871991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd94aa50 msgr2=0x55cedd968cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.445+0000 7fe78d513640 1 --2- 172.21.15.76:0/1565871991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd94aa50 0x55cedd968cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.445+0000 7fe78d513640 1 -- 172.21.15.76:0/1565871991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94a130 msgr2=0x55cedd953070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.445+0000 7fe78d513640 1 --2- 172.21.15.76:0/1565871991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94a130 0x55cedd953070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.445+0000 7fe78d513640 1 -- 172.21.15.76:0/1565871991 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cedd9601f0 con 0x55cedd953820 2024-03-18T14:54:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.445+0000 7fe78d513640 1 --2- 172.21.15.76:0/1565871991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd953820 0x55cedd94dbc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe780011e50 tx=0x7fe780009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb3ceff0ae693073 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.446+0000 7fe7727fc640 1 -- 172.21.15.76:0/1565871991 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780003e00 con 0x55cedd953820 2024-03-18T14:54:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.446+0000 7fe7727fc640 1 -- 172.21.15.76:0/1565871991 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe780011030 con 0x55cedd953820 2024-03-18T14:54:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.446+0000 7fe7727fc640 1 -- 172.21.15.76:0/1565871991 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7800051a0 con 0x55cedd953820 2024-03-18T14:54:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.447+0000 7fe791c33740 1 -- 172.21.15.76:0/1565871991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd953820 msgr2=0x55cedd94dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.447+0000 7fe791c33740 1 --2- 172.21.15.76:0/1565871991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd953820 0x55cedd94dbc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe780011e50 tx=0x7fe780009500 comp rx=0 tx=0).stop 2024-03-18T14:54:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.447+0000 7fe791c33740 1 -- 172.21.15.76:0/1565871991 shutdown_connections 2024-03-18T14:54:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.447+0000 7fe791c33740 1 --2- 172.21.15.76:0/1565871991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd953820 0x55cedd94dbc0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.447+0000 7fe791c33740 1 --2- 172.21.15.76:0/1565871991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd94aa50 0x55cedd968cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.447+0000 7fe791c33740 1 --2- 172.21.15.76:0/1565871991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd94a130 0x55cedd953070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.447+0000 7fe791c33740 1 -- 172.21.15.76:0/1565871991 >> 172.21.15.76:0/1565871991 conn(0x55cedd944a50 msgr2=0x55cedd95f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.447+0000 7fe791c33740 1 -- 172.21.15.76:0/1565871991 shutdown_connections 2024-03-18T14:54:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.447+0000 7fe791c33740 1 -- 172.21.15.76:0/1565871991 wait complete. 2024-03-18T14:54:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.447+0000 7fe791c33740 1 -- start start 2024-03-18T14:54:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.447+0000 7fe791c33740 1 Processor -- start 2024-03-18T14:54:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.447+0000 7fe791c33740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd94a130 0x55cedd9665b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.447+0000 7fe791c33740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd94aa50 0x55cedd966af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.447+0000 7fe791c33740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd953820 0x55cedd967030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.447+0000 7fe791c33740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe7881a9830 con 0x55cedd953820 2024-03-18T14:54:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.447+0000 7fe791c33740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe7881aaca0 con 0x55cedd94aa50 2024-03-18T14:54:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.447+0000 7fe791c33740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe7881a9f00 con 0x55cedd94a130 2024-03-18T14:54:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe791c33740 10 client.0 _ll_register_callbacks cb 0x7fe780011930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe78d513640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd953820 0x55cedd967030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe78cd12640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd94a130 0x55cedd9665b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe787fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd94aa50 0x55cedd966af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe78d513640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd953820 0x55cedd967030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44890/0 (socket says 172.21.15.76:44890) 2024-03-18T14:54:12.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe78cd12640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd94a130 0x55cedd9665b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:53402/0 (socket says 172.21.15.76:53402) 2024-03-18T14:54:12.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe78d513640 1 -- 172.21.15.76:0/1266069775 learned_addr learned my addr 172.21.15.76:0/1266069775 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe78d513640 1 -- 172.21.15.76:0/1266069775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd94a130 msgr2=0x55cedd9665b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe78d513640 1 --2- 172.21.15.76:0/1266069775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd94a130 0x55cedd9665b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe78d513640 1 -- 172.21.15.76:0/1266069775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd94aa50 msgr2=0x55cedd966af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe78d513640 1 --2- 172.21.15.76:0/1266069775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd94aa50 0x55cedd966af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe78d513640 1 -- 172.21.15.76:0/1266069775 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cedd9c8860 con 0x55cedd953820 2024-03-18T14:54:12.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe78d513640 1 --2- 172.21.15.76:0/1266069775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd953820 0x55cedd967030 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe7800048c0 tx=0x7fe780011cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2738e19b71e474fb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe767fff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:12.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe767fff640 1 -- 172.21.15.76:0/1266069775 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78002d020 con 0x55cedd953820 2024-03-18T14:54:12.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe767fff640 1 -- 172.21.15.76:0/1266069775 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fe7800112a0 con 0x55cedd953820 2024-03-18T14:54:12.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe791c33740 1 -- 172.21.15.76:0/1266069775 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe780011030 con 0x55cedd953820 2024-03-18T14:54:12.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.448+0000 7fe767fff640 1 -- 172.21.15.76:0/1266069775 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe780004c50 con 0x55cedd953820 2024-03-18T14:54:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.449+0000 7fe791c33740 10 client.15720 fetch_fsmap learned FSMap version 24 2024-03-18T14:54:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.449+0000 7fe791c33740 1 -- 172.21.15.76:0/1266069775 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55cedd9c9b10 con 0x55cedd953820 2024-03-18T14:54:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.449+0000 7fe767fff640 1 -- 172.21.15.76:0/1266069775 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe780011030 con 0x55cedd953820 2024-03-18T14:54:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.449+0000 7fe767fff640 1 -- 172.21.15.76:0/1266069775 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fe780029450 con 0x55cedd953820 2024-03-18T14:54:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.449+0000 7fe791c33740 10 client.15720 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:54:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.449+0000 7fe791c33740 10 client.15720 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.449+0000 7fe791c33740 20 client.15720 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.449+0000 7fe791c33740 1 --2- 172.21.15.76:0/1266069775 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55cedd9ca250 0x55cedd9cc670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.449+0000 7fe791c33740 4 client.15720 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:54:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.449+0000 7fe791c33740 1 -- 172.21.15.76:0/1266069775 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cedd8b62d0 con 0x55cedd9ca250 2024-03-18T14:54:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.449+0000 7fe78cd12640 1 --2- 172.21.15.76:0/1266069775 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55cedd9ca250 0x55cedd9cc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.449+0000 7fe767fff640 10 client.15720 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.449+0000 7fe78cd12640 1 --2- 172.21.15.76:0/1266069775 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55cedd9ca250 0x55cedd9cc670 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea9bc61887698f5e server_cookie=b7b9e0ac39b60103 in_seq=0 out_seq=0 2024-03-18T14:54:12.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.578+0000 7f97d52c0640 1 -- 172.21.15.76:0/159322973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97c8098ef0 msgr2=0x7f97c80992d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.578+0000 7f97d52c0640 1 --2- 172.21.15.76:0/159322973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97c8098ef0 0x7f97c80992d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f97c4009900 tx=0x7f97c402bc60 comp rx=0 tx=0).stop 2024-03-18T14:54:12.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.578+0000 7f97d52c0640 1 -- 172.21.15.76:0/159322973 shutdown_connections 2024-03-18T14:54:12.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.578+0000 7f97d52c0640 1 --2- 172.21.15.76:0/159322973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97c80946c0 0x7f97c8094b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.578+0000 7f97d52c0640 1 --2- 172.21.15.76:0/159322973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97c80047c0 0x7f97c8094180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.578+0000 7f97d52c0640 1 --2- 172.21.15.76:0/159322973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97c8098ef0 0x7f97c80992d0 secure :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f97c4009900 tx=0x7f97c402bc60 comp rx=0 tx=0).stop 2024-03-18T14:54:12.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.578+0000 7f97d52c0640 1 -- 172.21.15.76:0/159322973 >> 172.21.15.76:0/159322973 conn(0x7f97c8009a70 msgr2=0x7f97c8009e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.579+0000 7f97d52c0640 1 -- 172.21.15.76:0/159322973 shutdown_connections 2024-03-18T14:54:12.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.579+0000 7f97d52c0640 1 -- 172.21.15.76:0/159322973 wait complete. 2024-03-18T14:54:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.580+0000 7f97d52c0640 1 Processor -- start 2024-03-18T14:54:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.580+0000 7f97d52c0640 1 -- start start 2024-03-18T14:54:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.580+0000 7f97d52c0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97c80047c0 0x7f97c809d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.580+0000 7f97d52c0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97c80946c0 0x7f97c809d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.580+0000 7f97d52c0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97c813ca90 0x7f97c813ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.580+0000 7f97d52c0640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f97c80a4110 con 0x7f97c80047c0 2024-03-18T14:54:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.580+0000 7f97d52c0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f97c80a1e50 con 0x7f97c80946c0 2024-03-18T14:54:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.580+0000 7f97d52c0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f97c80a4290 con 0x7f97c813ca90 2024-03-18T14:54:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.580+0000 7f97cf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97c80946c0 0x7f97c809d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.581+0000 7f97cffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97c80047c0 0x7f97c809d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.581+0000 7f97d4abf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97c813ca90 0x7f97c813ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.581+0000 7f97cf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97c80946c0 0x7f97c809d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:38734/0 (socket says 172.21.15.76:38734) 2024-03-18T14:54:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.581+0000 7f97cf7fe640 1 -- 172.21.15.76:0/3893610446 learned_addr learned my addr 172.21.15.76:0/3893610446 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.581+0000 7f97cffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97c80047c0 0x7f97c809d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44904/0 (socket says 172.21.15.76:44904) 2024-03-18T14:54:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.581+0000 7f97cf7fe640 1 -- 172.21.15.76:0/3893610446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97c813ca90 msgr2=0x7f97c813ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.581+0000 7f97cf7fe640 1 --2- 172.21.15.76:0/3893610446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97c813ca90 0x7f97c813ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.581+0000 7f97cf7fe640 1 -- 172.21.15.76:0/3893610446 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97c80047c0 msgr2=0x7f97c809d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.581+0000 7f97cf7fe640 1 --2- 172.21.15.76:0/3893610446 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97c80047c0 0x7f97c809d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.581+0000 7f97cf7fe640 1 -- 172.21.15.76:0/3893610446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c813f4f0 con 0x7f97c80946c0 2024-03-18T14:54:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.581+0000 7f97cf7fe640 1 --2- 172.21.15.76:0/3893610446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97c80946c0 0x7f97c809d900 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00b450 tx=0x7f97bc00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.581+0000 7f97cd7fa640 1 -- 172.21.15.76:0/3893610446 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc015020 con 0x7f97c80946c0 2024-03-18T14:54:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.581+0000 7f97cd7fa640 1 -- 172.21.15.76:0/3893610446 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97bc011d70 con 0x7f97c80946c0 2024-03-18T14:54:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.581+0000 7f97d52c0640 1 -- 172.21.15.76:0/3893610446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c813f7e0 con 0x7f97c80946c0 2024-03-18T14:54:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.582+0000 7f97cd7fa640 1 -- 172.21.15.76:0/3893610446 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc00f340 con 0x7f97c80946c0 2024-03-18T14:54:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.582+0000 7f97d52c0640 1 -- 172.21.15.76:0/3893610446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c813fd50 con 0x7f97c80946c0 2024-03-18T14:54:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97cd7fa640 1 -- 172.21.15.76:0/3893610446 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f97bc018020 con 0x7f97c80946c0 2024-03-18T14:54:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97d91ec740 1 Processor -- start 2024-03-18T14:54:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97d91ec740 1 -- start start 2024-03-18T14:54:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97d91ec740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 0x55ce1e918e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97d91ec740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e919340 0x55ce1e9219e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97d91ec740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e921f20 0x55ce1e924310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97d91ec740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ce1e87c490 con 0x55ce1e921f20 2024-03-18T14:54:12.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97d91ec740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ce1e824c90 con 0x55ce1e919340 2024-03-18T14:54:12.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97d91ec740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ce1e878530 con 0x55ce1e918a20 2024-03-18T14:54:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97cffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e919340 0x55ce1e9219e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97cf7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e921f20 0x55ce1e924310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97d4abf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 0x55ce1e918e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97cffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e919340 0x55ce1e9219e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:38742/0 (socket says 172.21.15.76:38742) 2024-03-18T14:54:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97cffff640 1 -- 172.21.15.76:0/2482063160 learned_addr learned my addr 172.21.15.76:0/2482063160 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97cd7fa640 1 --2- 172.21.15.76:0/3893610446 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f97c00a2b10 0x7f97c00a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97cffff640 1 -- 172.21.15.76:0/2482063160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 msgr2=0x55ce1e918e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97cffff640 1 --2- 172.21.15.76:0/2482063160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 0x55ce1e918e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97cffff640 1 -- 172.21.15.76:0/2482063160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e921f20 msgr2=0x55ce1e924310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97cffff640 1 --2- 172.21.15.76:0/2482063160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e921f20 0x55ce1e924310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.585+0000 7f97cffff640 1 -- 172.21.15.76:0/2482063160 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce1e8df500 con 0x55ce1e919340 2024-03-18T14:54:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.586+0000 7f97d4abf640 1 --2- 172.21.15.76:0/3893610446 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f97c00a2b10 0x7f97c00a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.586+0000 7f97cd7fa640 1 -- 172.21.15.76:0/3893610446 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f97bc0ce950 con 0x7f97c80946c0 2024-03-18T14:54:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.586+0000 7f97cffff640 1 --2- 172.21.15.76:0/2482063160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e919340 0x55ce1e9219e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f97c402c270 tx=0x7f97c402c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d5f2afe30840f71 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.586+0000 7f97ba7fc640 1 -- 172.21.15.76:0/2482063160 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c4042070 con 0x55ce1e919340 2024-03-18T14:54:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.586+0000 7f97ba7fc640 1 -- 172.21.15.76:0/2482063160 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97c40042c0 con 0x55ce1e919340 2024-03-18T14:54:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.586+0000 7f97ba7fc640 1 -- 172.21.15.76:0/2482063160 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c4005260 con 0x55ce1e919340 2024-03-18T14:54:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.587+0000 7f97d91ec740 1 -- 172.21.15.76:0/2482063160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e919340 msgr2=0x55ce1e9219e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.587+0000 7f97d91ec740 1 --2- 172.21.15.76:0/2482063160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e919340 0x55ce1e9219e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f97c402c270 tx=0x7f97c402c680 comp rx=0 tx=0).stop 2024-03-18T14:54:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.587+0000 7f97d4abf640 1 --2- 172.21.15.76:0/3893610446 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f97c00a2b10 0x7f97c00a4fd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f97d00502f0 tx=0x7f97d00673e0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.587+0000 7f97d91ec740 1 -- 172.21.15.76:0/2482063160 shutdown_connections 2024-03-18T14:54:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.587+0000 7f97d91ec740 1 --2- 172.21.15.76:0/2482063160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e921f20 0x55ce1e924310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.587+0000 7f97d91ec740 1 --2- 172.21.15.76:0/2482063160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e919340 0x55ce1e9219e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.587+0000 7f97d91ec740 1 --2- 172.21.15.76:0/2482063160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 0x55ce1e918e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.587+0000 7f97d91ec740 1 -- 172.21.15.76:0/2482063160 >> 172.21.15.76:0/2482063160 conn(0x55ce1e913150 msgr2=0x55ce1e913530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.587+0000 7f97d91ec740 1 -- 172.21.15.76:0/2482063160 shutdown_connections 2024-03-18T14:54:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.587+0000 7f97d91ec740 1 -- 172.21.15.76:0/2482063160 wait complete. 2024-03-18T14:54:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.587+0000 7f97d91ec740 1 -- start start 2024-03-18T14:54:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.587+0000 7f97d91ec740 1 Processor -- start 2024-03-18T14:54:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.587+0000 7f97d91ec740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e918a20 0x55ce1e91ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97d91ec740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e919340 0x55ce1e91f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97d91ec740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e921f20 0x55ce1e91f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97d91ec740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ce1e924cb0 con 0x55ce1e921f20 2024-03-18T14:54:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97d91ec740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ce1e838230 con 0x55ce1e918a20 2024-03-18T14:54:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97d91ec740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ce1e924e30 con 0x55ce1e919340 2024-03-18T14:54:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97cffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e918a20 0x55ce1e91ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97d4abf640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e921f20 0x55ce1e91f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97d91ec740 10 client.0 _ll_register_callbacks cb 0x55ce1e874010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97cffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e918a20 0x55ce1e91ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:38754/0 (socket says 172.21.15.76:38754) 2024-03-18T14:54:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97cffff640 1 -- 172.21.15.76:0/448849140 learned_addr learned my addr 172.21.15.76:0/448849140 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97cf7fe640 1 --2- 172.21.15.76:0/448849140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e919340 0x55ce1e91f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97cffff640 1 -- 172.21.15.76:0/448849140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e919340 msgr2=0x55ce1e91f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97cffff640 1 --2- 172.21.15.76:0/448849140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e919340 0x55ce1e91f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97cffff640 1 -- 172.21.15.76:0/448849140 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e921f20 msgr2=0x55ce1e91f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97cffff640 1 --2- 172.21.15.76:0/448849140 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e921f20 0x55ce1e91f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97cffff640 1 -- 172.21.15.76:0/448849140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce1e847100 con 0x55ce1e918a20 2024-03-18T14:54:12.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97cffff640 1 --2- 172.21.15.76:0/448849140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e918a20 0x55ce1e91ee50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f97c4004bb0 tx=0x7f97c4004dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b164102e7b373c17 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:54:12.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97af7fe640 1 -- 172.21.15.76:0/448849140 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c404f060 con 0x55ce1e918a20 2024-03-18T14:54:12.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97af7fe640 1 -- 172.21.15.76:0/448849140 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f97c402cac0 con 0x55ce1e918a20 2024-03-18T14:54:12.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97d91ec740 1 -- 172.21.15.76:0/448849140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97c40042c0 con 0x55ce1e918a20 2024-03-18T14:54:12.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97af7fe640 1 -- 172.21.15.76:0/448849140 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f97c4042070 con 0x55ce1e918a20 2024-03-18T14:54:12.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97d91ec740 10 client.25589 fetch_fsmap learned FSMap version 24 2024-03-18T14:54:12.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.588+0000 7f97d91ec740 1 -- 172.21.15.76:0/448849140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ce1e5903f0 con 0x55ce1e918a20 2024-03-18T14:54:12.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.589+0000 7f97af7fe640 1 -- 172.21.15.76:0/448849140 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97c40042c0 con 0x55ce1e918a20 2024-03-18T14:54:12.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.589+0000 7f97af7fe640 1 -- 172.21.15.76:0/448849140 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f97c4003a30 con 0x55ce1e918a20 2024-03-18T14:54:12.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.589+0000 7f97d91ec740 10 client.25589 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:54:12.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.589+0000 7f97d91ec740 10 client.25589 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T14:54:12.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.589+0000 7f97d91ec740 20 client.25589 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:12.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.589+0000 7f97d91ec740 1 --2- 172.21.15.76:0/448849140 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55ce1e932010 0x55ce1e934400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.589+0000 7f97d91ec740 4 client.25589 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:54:12.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.589+0000 7f97d91ec740 1 -- 172.21.15.76:0/448849140 --> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce1e8899d0 con 0x55ce1e932010 2024-03-18T14:54:12.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.589+0000 7f97cf7fe640 1 --2- 172.21.15.76:0/448849140 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55ce1e932010 0x55ce1e934400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.589+0000 7f97af7fe640 10 client.25589 ms_handle_connect on v2:172.21.15.174:6836/239386414 2024-03-18T14:54:12.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.589+0000 7f97cf7fe640 1 --2- 172.21.15.76:0/448849140 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55ce1e932010 0x55ce1e934400 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=141cc9f88bace19d server_cookie=516690e883ce36cd in_seq=0 out_seq=0 2024-03-18T14:54:12.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.590+0000 7f97af7fe640 1 -- 172.21.15.76:0/448849140 <== mds.2 v2:172.21.15.174:6836/239386414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ce1e8899d0 con 0x55ce1e932010 2024-03-18T14:54:12.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.590+0000 7f97af7fe640 10 client.25589 handle_command_reply: tid=0 2024-03-18T14:54:12.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.590+0000 7f97d91ec740 1 client.25589 shutdown 2024-03-18T14:54:12.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.590+0000 7f97d91ec740 1 -- 172.21.15.76:0/448849140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e918a20 msgr2=0x55ce1e91ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.590+0000 7f97d91ec740 1 --2- 172.21.15.76:0/448849140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e918a20 0x55ce1e91ee50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f97c4004bb0 tx=0x7f97c4004dc0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.590+0000 7f97d91ec740 1 -- 172.21.15.76:0/448849140 shutdown_connections 2024-03-18T14:54:12.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.590+0000 7f97d91ec740 1 --2- 172.21.15.76:0/448849140 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55ce1e932010 0x55ce1e934400 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.590+0000 7f97d91ec740 1 --2- 172.21.15.76:0/448849140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e919340 0x55ce1e91f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.590+0000 7f97af7fe640 0 client.25589 ms_handle_reset on v2:172.21.15.174:6836/239386414 2024-03-18T14:54:12.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.590+0000 7f97d91ec740 1 --2- 172.21.15.76:0/448849140 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e921f20 0x55ce1e91f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.590+0000 7f97d91ec740 1 --2- 172.21.15.76:0/448849140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e918a20 0x55ce1e91ee50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.590+0000 7f97d91ec740 1 -- 172.21.15.76:0/448849140 >> 172.21.15.76:0/448849140 conn(0x55ce1e913150 msgr2=0x55ce1e914ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.590+0000 7f97d91ec740 1 -- 172.21.15.76:0/448849140 shutdown_connections 2024-03-18T14:54:12.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.590+0000 7f97d91ec740 1 -- 172.21.15.76:0/448849140 wait complete. 2024-03-18T14:54:12.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.591+0000 7f97d91ec740 20 client.25589 trim_cache size 0 max 16384 2024-03-18T14:54:12.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.605+0000 7f97d91ec740 1 Processor -- start 2024-03-18T14:54:12.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.605+0000 7f97d91ec740 1 -- start start 2024-03-18T14:54:12.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.605+0000 7f97d91ec740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 0x55ce1e921580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.605+0000 7f97d91ec740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e919340 0x55ce1e92dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.605+0000 7f97d91ec740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e921f20 0x55ce1e939a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.605+0000 7f97d91ec740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f97c814ad30 con 0x55ce1e919340 2024-03-18T14:54:12.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.605+0000 7f97d91ec740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f97c8148600 con 0x55ce1e921f20 2024-03-18T14:54:12.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.605+0000 7f97d91ec740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f97c8146450 con 0x55ce1e918a20 2024-03-18T14:54:12.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.605+0000 7f97cffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 0x55ce1e921580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.605+0000 7f97cf7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e919340 0x55ce1e92dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.605+0000 7f97d4abf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e921f20 0x55ce1e939a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.605+0000 7f97cffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 0x55ce1e921580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:53428/0 (socket says 172.21.15.76:53428) 2024-03-18T14:54:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.605+0000 7f97cffff640 1 -- 172.21.15.76:0/3201061372 learned_addr learned my addr 172.21.15.76:0/3201061372 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.606+0000 7f97cffff640 1 -- 172.21.15.76:0/3201061372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e921f20 msgr2=0x55ce1e939a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.606+0000 7f97cffff640 1 --2- 172.21.15.76:0/3201061372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e921f20 0x55ce1e939a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.606+0000 7f97cffff640 1 -- 172.21.15.76:0/3201061372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e919340 msgr2=0x55ce1e92dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.606+0000 7f97cffff640 1 --2- 172.21.15.76:0/3201061372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e919340 0x55ce1e92dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.606+0000 7f97cffff640 1 -- 172.21.15.76:0/3201061372 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce1e65a7d0 con 0x55ce1e918a20 2024-03-18T14:54:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.606+0000 7f97cf7fe640 1 --2- 172.21.15.76:0/3201061372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e919340 0x55ce1e92dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.606+0000 7f97cffff640 1 --2- 172.21.15.76:0/3201061372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 0x55ce1e921580 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f97c402bca0 tx=0x7f97c4005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab01fe65f3c47816 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.606+0000 7f97b9ffb640 1 -- 172.21.15.76:0/3201061372 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c404f040 con 0x55ce1e918a20 2024-03-18T14:54:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.606+0000 7f97b9ffb640 1 -- 172.21.15.76:0/3201061372 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97c4004be0 con 0x55ce1e918a20 2024-03-18T14:54:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.606+0000 7f97b9ffb640 1 -- 172.21.15.76:0/3201061372 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c402cac0 con 0x55ce1e918a20 2024-03-18T14:54:12.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.607+0000 7f97d91ec740 1 -- 172.21.15.76:0/3201061372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 msgr2=0x55ce1e921580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.607+0000 7f97d91ec740 1 --2- 172.21.15.76:0/3201061372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 0x55ce1e921580 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f97c402bca0 tx=0x7f97c4005fd0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.607+0000 7f97d91ec740 1 -- 172.21.15.76:0/3201061372 shutdown_connections 2024-03-18T14:54:12.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.607+0000 7f97d91ec740 1 --2- 172.21.15.76:0/3201061372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e921f20 0x55ce1e939a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.607+0000 7f97d91ec740 1 --2- 172.21.15.76:0/3201061372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e919340 0x55ce1e92dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.607+0000 7f97d91ec740 1 --2- 172.21.15.76:0/3201061372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 0x55ce1e921580 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.607+0000 7f97d91ec740 1 -- 172.21.15.76:0/3201061372 >> 172.21.15.76:0/3201061372 conn(0x55ce1e913150 msgr2=0x55ce1e912140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.607+0000 7f97d91ec740 1 -- 172.21.15.76:0/3201061372 shutdown_connections 2024-03-18T14:54:12.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.607+0000 7f97d91ec740 1 -- 172.21.15.76:0/3201061372 wait complete. 2024-03-18T14:54:12.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.607+0000 7f97d91ec740 1 -- start start 2024-03-18T14:54:12.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97d91ec740 1 Processor -- start 2024-03-18T14:54:12.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97d91ec740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 0x55ce1e92a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97d91ec740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e919340 0x55ce1e925780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97d91ec740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e921f20 0x55ce1e925d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97d91ec740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f97c8146ac0 con 0x55ce1e921f20 2024-03-18T14:54:12.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97d91ec740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f97c8147f30 con 0x55ce1e919340 2024-03-18T14:54:12.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97d91ec740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f97c8147190 con 0x55ce1e918a20 2024-03-18T14:54:12.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97d91ec740 10 client.0 _ll_register_callbacks cb 0x7f97bc0d43e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:12.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97cffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 0x55ce1e92a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97d4abf640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e921f20 0x55ce1e925d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97cffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 0x55ce1e92a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:53436/0 (socket says 172.21.15.76:53436) 2024-03-18T14:54:12.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97cffff640 1 -- 172.21.15.76:0/620346291 learned_addr learned my addr 172.21.15.76:0/620346291 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97cf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e919340 0x55ce1e925780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97cffff640 1 -- 172.21.15.76:0/620346291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e919340 msgr2=0x55ce1e925780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97cffff640 1 --2- 172.21.15.76:0/620346291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e919340 0x55ce1e925780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97cffff640 1 -- 172.21.15.76:0/620346291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e921f20 msgr2=0x55ce1e925d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97cffff640 1 --2- 172.21.15.76:0/620346291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e921f20 0x55ce1e925d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97cffff640 1 -- 172.21.15.76:0/620346291 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce1e926530 con 0x55ce1e918a20 2024-03-18T14:54:12.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.608+0000 7f97d4abf640 1 --2- 172.21.15.76:0/620346291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e921f20 0x55ce1e925d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.609+0000 7f97cffff640 1 --2- 172.21.15.76:0/620346291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 0x55ce1e92a410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f97c402c1a0 tx=0x7f97c4003fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2d57d01d0b09b26 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.609+0000 7f97af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-18T14:54:12.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.609+0000 7f97af7fe640 1 -- 172.21.15.76:0/620346291 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c404f040 con 0x55ce1e918a20 2024-03-18T14:54:12.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.609+0000 7f97d91ec740 1 -- 172.21.15.76:0/620346291 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97c4004be0 con 0x55ce1e918a20 2024-03-18T14:54:12.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.609+0000 7f97af7fe640 1 -- 172.21.15.76:0/620346291 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f97c4005190 con 0x55ce1e918a20 2024-03-18T14:54:12.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.609+0000 7f97af7fe640 1 -- 172.21.15.76:0/620346291 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f97c406d050 con 0x55ce1e918a20 2024-03-18T14:54:12.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.609+0000 7f97d91ec740 10 client.25588 fetch_fsmap learned FSMap version 24 2024-03-18T14:54:12.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.609+0000 7f97d91ec740 1 -- 172.21.15.76:0/620346291 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ce1e998100 con 0x55ce1e918a20 2024-03-18T14:54:12.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.609+0000 7f97af7fe640 1 -- 172.21.15.76:0/620346291 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97c4004be0 con 0x55ce1e918a20 2024-03-18T14:54:12.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.610+0000 7f97af7fe640 1 -- 172.21.15.76:0/620346291 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f97c405c070 con 0x55ce1e918a20 2024-03-18T14:54:12.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.610+0000 7f97d91ec740 10 client.25588 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:54:12.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.610+0000 7f97d91ec740 10 client.25588 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T14:54:12.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.610+0000 7f97d91ec740 20 client.25588 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:12.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.610+0000 7f97d91ec740 1 --2- 172.21.15.76:0/620346291 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55ce1e932010 0x55ce1e99a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.610+0000 7f97d91ec740 4 client.25588 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T14:54:12.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.610+0000 7f97d91ec740 1 -- 172.21.15.76:0/620346291 --> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ce1e883d20 con 0x55ce1e932010 2024-03-18T14:54:12.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.610+0000 7f97cf7fe640 1 --2- 172.21.15.76:0/620346291 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55ce1e932010 0x55ce1e99a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.610+0000 7f97af7fe640 10 client.25588 ms_handle_connect on v2:172.21.15.174:6836/239386414 2024-03-18T14:54:12.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.610+0000 7f97cf7fe640 1 --2- 172.21.15.76:0/620346291 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55ce1e932010 0x55ce1e99a770 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=77a220ef8bb9d6b0 server_cookie=99edd383b1fe1998 in_seq=0 out_seq=0 2024-03-18T14:54:12.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.611+0000 7f97af7fe640 1 -- 172.21.15.76:0/620346291 <== mds.2 v2:172.21.15.174:6836/239386414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ce1e883d20 con 0x55ce1e932010 2024-03-18T14:54:12.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.611+0000 7f97af7fe640 10 client.25588 handle_command_reply: tid=0 2024-03-18T14:54:12.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.611+0000 7f97d91ec740 1 client.25588 shutdown 2024-03-18T14:54:12.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.611+0000 7f97d91ec740 1 -- 172.21.15.76:0/620346291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 msgr2=0x55ce1e92a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.611+0000 7f97d91ec740 1 --2- 172.21.15.76:0/620346291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 0x55ce1e92a410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f97c402c1a0 tx=0x7f97c4003fa0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.611+0000 7f97d91ec740 1 -- 172.21.15.76:0/620346291 shutdown_connections 2024-03-18T14:54:12.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.611+0000 7f97d91ec740 1 --2- 172.21.15.76:0/620346291 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55ce1e932010 0x55ce1e99a770 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.611+0000 7f97d91ec740 1 --2- 172.21.15.76:0/620346291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce1e921f20 0x55ce1e925d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.611+0000 7f97af7fe640 0 client.25588 ms_handle_reset on v2:172.21.15.174:6836/239386414 2024-03-18T14:54:12.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.611+0000 7f97d91ec740 1 --2- 172.21.15.76:0/620346291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce1e919340 0x55ce1e925780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.611+0000 7f97d91ec740 1 --2- 172.21.15.76:0/620346291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce1e918a20 0x55ce1e92a410 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.611+0000 7f97d91ec740 1 -- 172.21.15.76:0/620346291 >> 172.21.15.76:0/620346291 conn(0x55ce1e913150 msgr2=0x55ce1e91edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.611+0000 7f97d91ec740 1 -- 172.21.15.76:0/620346291 shutdown_connections 2024-03-18T14:54:12.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.611+0000 7f97d91ec740 1 -- 172.21.15.76:0/620346291 wait complete. 2024-03-18T14:54:12.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.611+0000 7f97d91ec740 20 client.25588 trim_cache size 0 max 16384 2024-03-18T14:54:12.621 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-18T14:54:12.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.612+0000 7f97d52c0640 1 -- 172.21.15.76:0/3893610446 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f97c00a2b10 msgr2=0x7f97c00a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.612+0000 7f97d52c0640 1 --2- 172.21.15.76:0/3893610446 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f97c00a2b10 0x7f97c00a4fd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f97d00502f0 tx=0x7f97d00673e0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.612+0000 7f97d52c0640 1 -- 172.21.15.76:0/3893610446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97c80946c0 msgr2=0x7f97c809d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.612+0000 7f97d52c0640 1 --2- 172.21.15.76:0/3893610446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97c80946c0 0x7f97c809d900 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00b450 tx=0x7f97bc00b910 comp rx=0 tx=0).stop 2024-03-18T14:54:12.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.612+0000 7f97d52c0640 1 -- 172.21.15.76:0/3893610446 shutdown_connections 2024-03-18T14:54:12.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.612+0000 7f97d52c0640 1 --2- 172.21.15.76:0/3893610446 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f97c00a2b10 0x7f97c00a4fd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.612+0000 7f97d52c0640 1 --2- 172.21.15.76:0/3893610446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97c813ca90 0x7f97c813ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.612+0000 7f97d52c0640 1 --2- 172.21.15.76:0/3893610446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97c80946c0 0x7f97c809d900 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.612+0000 7f97d52c0640 1 --2- 172.21.15.76:0/3893610446 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97c80047c0 0x7f97c809d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.612+0000 7f97d52c0640 1 -- 172.21.15.76:0/3893610446 >> 172.21.15.76:0/3893610446 conn(0x7f97c8009a70 msgr2=0x7f97c8093170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.612+0000 7f97d52c0640 1 -- 172.21.15.76:0/3893610446 shutdown_connections 2024-03-18T14:54:12.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.612+0000 7f97d52c0640 1 -- 172.21.15.76:0/3893610446 wait complete. 2024-03-18T14:54:12.624 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T14:54:12.689 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:12 smithi076 ceph-mon[32205]: pgmap v30: 97 pgs: 97 active+clean; 16 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.4 MiB/s wr, 186 op/s 2024-03-18T14:54:12.689 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:12 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2375080819' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:54:12.731 INFO:tasks.workunit.client.0.smithi076.stdout: 1 36489 16.39 MB/sec warmup 13 sec latency 65.523 ms 2024-03-18T14:54:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.737+0000 7fe767fff640 1 -- 172.21.15.76:0/1266069775 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cedd8b62d0 con 0x55cedd9ca250 2024-03-18T14:54:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.737+0000 7fe767fff640 10 client.15720 handle_command_reply: tid=0 2024-03-18T14:54:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.737+0000 7fe791c33740 1 client.15720 shutdown 2024-03-18T14:54:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.738+0000 7fe791c33740 1 -- 172.21.15.76:0/1266069775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd953820 msgr2=0x55cedd967030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.738+0000 7fe791c33740 1 --2- 172.21.15.76:0/1266069775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd953820 0x55cedd967030 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe7800048c0 tx=0x7fe780011cc0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.738+0000 7fe791c33740 1 -- 172.21.15.76:0/1266069775 shutdown_connections 2024-03-18T14:54:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.738+0000 7fe791c33740 1 --2- 172.21.15.76:0/1266069775 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55cedd9ca250 0x55cedd9cc670 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.738+0000 7fe791c33740 1 --2- 172.21.15.76:0/1266069775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cedd953820 0x55cedd967030 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.738+0000 7fe791c33740 1 --2- 172.21.15.76:0/1266069775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cedd94aa50 0x55cedd966af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.738+0000 7fe767fff640 0 client.15720 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.738+0000 7fe791c33740 1 --2- 172.21.15.76:0/1266069775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cedd94a130 0x55cedd9665b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.738+0000 7fe791c33740 1 -- 172.21.15.76:0/1266069775 >> 172.21.15.76:0/1266069775 conn(0x55cedd944a50 msgr2=0x55cedd943bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.738+0000 7fe791c33740 1 -- 172.21.15.76:0/1266069775 shutdown_connections 2024-03-18T14:54:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.738+0000 7fe791c33740 1 -- 172.21.15.76:0/1266069775 wait complete. 2024-03-18T14:54:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.738+0000 7fe791c33740 20 client.15720 trim_cache size 0 max 16384 2024-03-18T14:54:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:54:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:54:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:54:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:54:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.738+0000 7fe78ef76640 1 -- 172.21.15.76:0/3701775626 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe7780a2bb0 msgr2=0x7fe7780a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.738+0000 7fe78ef76640 1 --2- 172.21.15.76:0/3701775626 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe7780a2bb0 0x7fe7780a5070 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe774000c00 tx=0x7fe77403d040 comp rx=0 tx=0).stop 2024-03-18T14:54:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.738+0000 7fe78ef76640 1 -- 172.21.15.76:0/3701775626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe788100930 msgr2=0x7fe788197040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.738+0000 7fe78ef76640 1 --2- 172.21.15.76:0/3701775626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe788100930 0x7fe788197040 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe77c00e9f0 tx=0x7fe77c00eeb0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.739+0000 7fe78ef76640 1 -- 172.21.15.76:0/3701775626 shutdown_connections 2024-03-18T14:54:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.739+0000 7fe78ef76640 1 --2- 172.21.15.76:0/3701775626 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe7780a2bb0 0x7fe7780a5070 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.739+0000 7fe78ef76640 1 --2- 172.21.15.76:0/3701775626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe788105680 0x7fe78819b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.739+0000 7fe78ef76640 1 --2- 172.21.15.76:0/3701775626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7881012f0 0x7fe788197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.739+0000 7fe78ef76640 1 --2- 172.21.15.76:0/3701775626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe788100930 0x7fe788197040 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.739+0000 7fe78ef76640 1 -- 172.21.15.76:0/3701775626 >> 172.21.15.76:0/3701775626 conn(0x7fe788077530 msgr2=0x7fe7880ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.739+0000 7fe78ef76640 1 -- 172.21.15.76:0/3701775626 shutdown_connections 2024-03-18T14:54:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:12.739+0000 7fe78ef76640 1 -- 172.21.15.76:0/3701775626 wait complete. 2024-03-18T14:54:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.783+0000 7fee494dc640 1 -- 172.21.15.76:0/892157364 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee441079e0 msgr2=0x7fee44107dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.783+0000 7fee494dc640 1 --2- 172.21.15.76:0/892157364 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee441079e0 0x7fee44107dc0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fee34009930 tx=0x7fee3402bc30 comp rx=0 tx=0).stop 2024-03-18T14:54:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.783+0000 7fee494dc640 1 -- 172.21.15.76:0/892157364 shutdown_connections 2024-03-18T14:54:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.783+0000 7fee494dc640 1 --2- 172.21.15.76:0/892157364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee441000a0 0x7fee44100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.783+0000 7fee494dc640 1 --2- 172.21.15.76:0/892157364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee44100a60 0x7fee4410e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.783+0000 7fee494dc640 1 --2- 172.21.15.76:0/892157364 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee441079e0 0x7fee44107dc0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.783+0000 7fee494dc640 1 -- 172.21.15.76:0/892157364 >> 172.21.15.76:0/892157364 conn(0x7fee44075a20 msgr2=0x7fee44075e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.783+0000 7fee494dc640 1 -- 172.21.15.76:0/892157364 shutdown_connections 2024-03-18T14:54:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.783+0000 7fee494dc640 1 -- 172.21.15.76:0/892157364 wait complete. 2024-03-18T14:54:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.783+0000 7fee494dc640 1 Processor -- start 2024-03-18T14:54:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee494dc640 1 -- start start 2024-03-18T14:54:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee494dc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee441000a0 0x7fee4419a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee494dc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee44100a60 0x7fee4419a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee494dc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee441079e0 0x7fee4419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee494dc640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fee44111c40 con 0x7fee44100a60 2024-03-18T14:54:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee494dc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fee44111ac0 con 0x7fee441079e0 2024-03-18T14:54:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee494dc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fee44111dc0 con 0x7fee441000a0 2024-03-18T14:54:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee427fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee44100a60 0x7fee4419a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee427fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee44100a60 0x7fee4419a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44950/0 (socket says 172.21.15.76:44950) 2024-03-18T14:54:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee427fc640 1 -- 172.21.15.76:0/1922893438 learned_addr learned my addr 172.21.15.76:0/1922893438 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee42ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee441000a0 0x7fee4419a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee437fe640 1 --2- 172.21.15.76:0/1922893438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee441079e0 0x7fee4419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee427fc640 1 -- 172.21.15.76:0/1922893438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee441000a0 msgr2=0x7fee4419a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee427fc640 1 --2- 172.21.15.76:0/1922893438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee441000a0 0x7fee4419a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee427fc640 1 -- 172.21.15.76:0/1922893438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee441079e0 msgr2=0x7fee4419ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee427fc640 1 --2- 172.21.15.76:0/1922893438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee441079e0 0x7fee4419ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee427fc640 1 -- 172.21.15.76:0/1922893438 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee4419f370 con 0x7fee44100a60 2024-03-18T14:54:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.784+0000 7fee427fc640 1 --2- 172.21.15.76:0/1922893438 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee44100a60 0x7fee4419a8c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fee3800e9f0 tx=0x7fee3800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.785+0000 7fee2bfff640 1 -- 172.21.15.76:0/1922893438 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3800cd80 con 0x7fee44100a60 2024-03-18T14:54:12.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.785+0000 7fee494dc640 1 -- 172.21.15.76:0/1922893438 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee4419f660 con 0x7fee44100a60 2024-03-18T14:54:12.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.785+0000 7fee2bfff640 1 -- 172.21.15.76:0/1922893438 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee38005500 con 0x7fee44100a60 2024-03-18T14:54:12.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.785+0000 7fee2bfff640 1 -- 172.21.15.76:0/1922893438 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee38010690 con 0x7fee44100a60 2024-03-18T14:54:12.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.785+0000 7fee494dc640 1 -- 172.21.15.76:0/1922893438 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee441a6ef0 con 0x7fee44100a60 2024-03-18T14:54:12.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.787+0000 7fee2bfff640 1 -- 172.21.15.76:0/1922893438 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7fee380046c0 con 0x7fee44100a60 2024-03-18T14:54:12.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.787+0000 7fee4c1a7740 1 Processor -- start 2024-03-18T14:54:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.787+0000 7fee4c1a7740 1 -- start start 2024-03-18T14:54:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee4c1a7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef96830 0x55d89ef96c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee4c1a7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef97150 0x55d89ef9f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee4c1a7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef9fd30 0x55d89efa2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee4c1a7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d89eefa490 con 0x55d89ef9fd30 2024-03-18T14:54:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee4c1a7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d89eea2c90 con 0x55d89ef96830 2024-03-18T14:54:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee4c1a7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d89eef6530 con 0x55d89ef97150 2024-03-18T14:54:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee2bfff640 1 --2- 172.21.15.76:0/1922893438 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fee2c0a2a90 0x7fee2c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee437fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef96830 0x55d89ef96c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee42ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef97150 0x55d89ef9f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee427fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef9fd30 0x55d89efa2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee437fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef96830 0x55d89ef96c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:38802/0 (socket says 172.21.15.76:38802) 2024-03-18T14:54:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee437fe640 1 -- 172.21.15.76:0/3691164549 learned_addr learned my addr 172.21.15.76:0/3691164549 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee437fe640 1 --2- 172.21.15.76:0/1922893438 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fee2c0a2a90 0x7fee2c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee2bfff640 1 -- 172.21.15.76:0/1922893438 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fee380cf4c0 con 0x7fee44100a60 2024-03-18T14:54:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee42ffd640 1 -- 172.21.15.76:0/3691164549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef96830 msgr2=0x55d89ef96c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee42ffd640 1 --2- 172.21.15.76:0/3691164549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef96830 0x55d89ef96c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee42ffd640 1 -- 172.21.15.76:0/3691164549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef9fd30 msgr2=0x55d89efa2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee42ffd640 1 --2- 172.21.15.76:0/3691164549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef9fd30 0x55d89efa2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee42ffd640 1 -- 172.21.15.76:0/3691164549 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d89ef5d500 con 0x55d89ef97150 2024-03-18T14:54:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee437fe640 1 --2- 172.21.15.76:0/3691164549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef96830 0x55d89ef96c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee437fe640 1 --2- 172.21.15.76:0/1922893438 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fee2c0a2a90 0x7fee2c0a4f50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fee3c005c40 tx=0x7fee3c009500 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee42ffd640 1 --2- 172.21.15.76:0/3691164549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef97150 0x55d89ef9f7f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fee34009230 tx=0x7fee34009610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a85b4eb763d2835 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.788+0000 7fee297fa640 1 -- 172.21.15.76:0/3691164549 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee34042070 con 0x55d89ef97150 2024-03-18T14:54:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.789+0000 7fee297fa640 1 -- 172.21.15.76:0/3691164549 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee3402c690 con 0x55d89ef97150 2024-03-18T14:54:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.789+0000 7fee297fa640 1 -- 172.21.15.76:0/3691164549 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee34004b20 con 0x55d89ef97150 2024-03-18T14:54:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.789+0000 7fee4c1a7740 1 -- 172.21.15.76:0/3691164549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef97150 msgr2=0x55d89ef9f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.789+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/3691164549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef97150 0x55d89ef9f7f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fee34009230 tx=0x7fee34009610 comp rx=0 tx=0).stop 2024-03-18T14:54:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.789+0000 7fee4c1a7740 1 -- 172.21.15.76:0/3691164549 shutdown_connections 2024-03-18T14:54:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.789+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/3691164549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef97150 0x55d89ef9f7f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.789+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/3691164549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef9fd30 0x55d89efa2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.789+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/3691164549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef96830 0x55d89ef96c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.789+0000 7fee4c1a7740 1 -- 172.21.15.76:0/3691164549 >> 172.21.15.76:0/3691164549 conn(0x55d89ef90f60 msgr2=0x55d89ef91340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.789+0000 7fee4c1a7740 1 -- 172.21.15.76:0/3691164549 shutdown_connections 2024-03-18T14:54:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.789+0000 7fee4c1a7740 1 -- 172.21.15.76:0/3691164549 wait complete. 2024-03-18T14:54:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee4c1a7740 1 -- start start 2024-03-18T14:54:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee4c1a7740 1 Processor -- start 2024-03-18T14:54:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee4c1a7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef96830 0x55d89ef9cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee4c1a7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef97150 0x55d89ef9d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee4c1a7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef9fd30 0x55d89ef9d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee4c1a7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d89efa2ac0 con 0x55d89ef97150 2024-03-18T14:54:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee4c1a7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d89eeb6230 con 0x55d89ef9fd30 2024-03-18T14:54:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee4c1a7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d89efa2c40 con 0x55d89ef96830 2024-03-18T14:54:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee4c1a7740 10 client.0 _ll_register_callbacks cb 0x55d89eef2010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee427fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef97150 0x55d89ef9d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee427fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef97150 0x55d89ef9d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44960/0 (socket says 172.21.15.76:44960) 2024-03-18T14:54:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee427fc640 1 -- 172.21.15.76:0/1644921559 learned_addr learned my addr 172.21.15.76:0/1644921559 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee42ffd640 1 --2- 172.21.15.76:0/1644921559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef96830 0x55d89ef9cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee437fe640 1 --2- 172.21.15.76:0/1644921559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef9fd30 0x55d89ef9d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee427fc640 1 -- 172.21.15.76:0/1644921559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef96830 msgr2=0x55d89ef9cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee427fc640 1 --2- 172.21.15.76:0/1644921559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef96830 0x55d89ef9cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee427fc640 1 -- 172.21.15.76:0/1644921559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef9fd30 msgr2=0x55d89ef9d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee427fc640 1 --2- 172.21.15.76:0/1644921559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef9fd30 0x55d89ef9d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee427fc640 1 -- 172.21.15.76:0/1644921559 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d89eec5100 con 0x55d89ef97150 2024-03-18T14:54:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee427fc640 1 --2- 172.21.15.76:0/1644921559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef97150 0x55d89ef9d310 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fee38097d80 tx=0x7fee3806a7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c29509f3832fd2f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.790+0000 7fee26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.791+0000 7fee26ff5640 1 -- 172.21.15.76:0/1644921559 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee38084e60 con 0x55d89ef97150 2024-03-18T14:54:12.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.791+0000 7fee26ff5640 1 -- 172.21.15.76:0/1644921559 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fee380667c0 con 0x55d89ef97150 2024-03-18T14:54:12.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.791+0000 7fee4c1a7740 1 -- 172.21.15.76:0/1644921559 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d89ef079d0 con 0x55d89ef97150 2024-03-18T14:54:12.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.791+0000 7fee26ff5640 1 -- 172.21.15.76:0/1644921559 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fee3806a030 con 0x55d89ef97150 2024-03-18T14:54:12.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.791+0000 7fee4c1a7740 10 client.15750 fetch_fsmap learned FSMap version 24 2024-03-18T14:54:12.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.791+0000 7fee4c1a7740 1 -- 172.21.15.76:0/1644921559 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d89ec0e3f0 con 0x55d89ef97150 2024-03-18T14:54:12.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.791+0000 7fee26ff5640 1 -- 172.21.15.76:0/1644921559 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d89ef079d0 con 0x55d89ef97150 2024-03-18T14:54:12.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.791+0000 7fee26ff5640 1 -- 172.21.15.76:0/1644921559 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fee380a3cc0 con 0x55d89ef97150 2024-03-18T14:54:12.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.791+0000 7fee4c1a7740 10 client.15750 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:54:12.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.791+0000 7fee4c1a7740 10 client.15750 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T14:54:12.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.791+0000 7fee4c1a7740 20 client.15750 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:12.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.791+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/1644921559 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x55d89efb0100 0x55d89efb2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.791+0000 7fee4c1a7740 4 client.15750 mds_command: new command op to 24490 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:54:12.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.791+0000 7fee4c1a7740 1 -- 172.21.15.76:0/1644921559 --> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d89ef01d20 con 0x55d89efb0100 2024-03-18T14:54:12.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.792+0000 7fee42ffd640 1 --2- 172.21.15.76:0/1644921559 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x55d89efb0100 0x55d89efb2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.792+0000 7fee42ffd640 1 --2- 172.21.15.76:0/1644921559 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x55d89efb0100 0x55d89efb2530 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9e847b739d7a79ad server_cookie=821d10de7c0b0cd3 in_seq=0 out_seq=0 2024-03-18T14:54:12.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.792+0000 7fee26ff5640 10 client.15750 ms_handle_connect on v2:172.21.15.174:6838/399810652 2024-03-18T14:54:12.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.792+0000 7fee26ff5640 1 -- 172.21.15.76:0/1644921559 <== mds.1 v2:172.21.15.174:6838/399810652 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d89ef01d20 con 0x55d89efb0100 2024-03-18T14:54:12.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.792+0000 7fee26ff5640 10 client.15750 handle_command_reply: tid=0 2024-03-18T14:54:12.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.793+0000 7fee4c1a7740 1 client.15750 shutdown 2024-03-18T14:54:12.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.793+0000 7fee4c1a7740 1 -- 172.21.15.76:0/1644921559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef97150 msgr2=0x55d89ef9d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.793+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/1644921559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef97150 0x55d89ef9d310 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fee38097d80 tx=0x7fee3806a7f0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.793+0000 7fee4c1a7740 1 -- 172.21.15.76:0/1644921559 shutdown_connections 2024-03-18T14:54:12.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.793+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/1644921559 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x55d89efb0100 0x55d89efb2530 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.793+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/1644921559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef9fd30 0x55d89ef9d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.793+0000 7fee26ff5640 0 client.15750 ms_handle_reset on v2:172.21.15.174:6838/399810652 2024-03-18T14:54:12.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.793+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/1644921559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef97150 0x55d89ef9d310 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.793+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/1644921559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef96830 0x55d89ef9cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.793+0000 7fee4c1a7740 1 -- 172.21.15.76:0/1644921559 >> 172.21.15.76:0/1644921559 conn(0x55d89ef90f60 msgr2=0x55d89ef92cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.793+0000 7fee4c1a7740 1 -- 172.21.15.76:0/1644921559 shutdown_connections 2024-03-18T14:54:12.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.793+0000 7fee4c1a7740 1 -- 172.21.15.76:0/1644921559 wait complete. 2024-03-18T14:54:12.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.793+0000 7fee4c1a7740 20 client.15750 trim_cache size 0 max 16384 2024-03-18T14:54:12.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.806+0000 7fee4c1a7740 1 Processor -- start 2024-03-18T14:54:12.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.806+0000 7fee4c1a7740 1 -- start start 2024-03-18T14:54:12.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.806+0000 7fee4c1a7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef96830 0x55d89ef9f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.806+0000 7fee4c1a7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef97150 0x55d89efab860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.806+0000 7fee4c1a7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef9fd30 0x55d89efb7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.806+0000 7fee4c1a7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fee441b1e20 con 0x55d89ef9fd30 2024-03-18T14:54:12.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.806+0000 7fee4c1a7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fee441af6f0 con 0x55d89ef97150 2024-03-18T14:54:12.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.806+0000 7fee4c1a7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fee441ad540 con 0x55d89ef96830 2024-03-18T14:54:12.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.807+0000 7fee437fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef9fd30 0x55d89efb7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.807+0000 7fee437fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef9fd30 0x55d89efb7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44966/0 (socket says 172.21.15.76:44966) 2024-03-18T14:54:12.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.807+0000 7fee437fe640 1 -- 172.21.15.76:0/2281788822 learned_addr learned my addr 172.21.15.76:0/2281788822 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.807+0000 7fee427fc640 1 --2- 172.21.15.76:0/2281788822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef97150 0x55d89efab860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.807+0000 7fee42ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef96830 0x55d89ef9f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.807+0000 7fee437fe640 1 -- 172.21.15.76:0/2281788822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef96830 msgr2=0x55d89ef9f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.807+0000 7fee437fe640 1 --2- 172.21.15.76:0/2281788822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef96830 0x55d89ef9f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.807+0000 7fee437fe640 1 -- 172.21.15.76:0/2281788822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef97150 msgr2=0x55d89efab860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.807+0000 7fee437fe640 1 --2- 172.21.15.76:0/2281788822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef97150 0x55d89efab860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.807+0000 7fee437fe640 1 -- 172.21.15.76:0/2281788822 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d89ecd87d0 con 0x55d89ef9fd30 2024-03-18T14:54:12.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.807+0000 7fee427fc640 1 --2- 172.21.15.76:0/2281788822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef97150 0x55d89efab860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.807+0000 7fee437fe640 1 --2- 172.21.15.76:0/2281788822 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef9fd30 0x55d89efb7db0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009f30 tx=0x7fee3c013070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12f352b77daa24a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.807+0000 7fee28ff9640 1 -- 172.21.15.76:0/2281788822 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c02f070 con 0x55d89ef9fd30 2024-03-18T14:54:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.807+0000 7fee28ff9640 1 -- 172.21.15.76:0/2281788822 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee3c00fcc0 con 0x55d89ef9fd30 2024-03-18T14:54:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.807+0000 7fee28ff9640 1 -- 172.21.15.76:0/2281788822 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c02add0 con 0x55d89ef9fd30 2024-03-18T14:54:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.808+0000 7fee4c1a7740 1 -- 172.21.15.76:0/2281788822 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef9fd30 msgr2=0x55d89efb7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.808+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/2281788822 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef9fd30 0x55d89efb7db0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009f30 tx=0x7fee3c013070 comp rx=0 tx=0).stop 2024-03-18T14:54:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.808+0000 7fee4c1a7740 1 -- 172.21.15.76:0/2281788822 shutdown_connections 2024-03-18T14:54:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.808+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/2281788822 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef9fd30 0x55d89efb7db0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.808+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/2281788822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef97150 0x55d89efab860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.808+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/2281788822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef96830 0x55d89ef9f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.808+0000 7fee4c1a7740 1 -- 172.21.15.76:0/2281788822 >> 172.21.15.76:0/2281788822 conn(0x55d89ef90f60 msgr2=0x55d89ef9cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.808+0000 7fee4c1a7740 1 -- 172.21.15.76:0/2281788822 shutdown_connections 2024-03-18T14:54:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.808+0000 7fee4c1a7740 1 -- 172.21.15.76:0/2281788822 wait complete. 2024-03-18T14:54:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.808+0000 7fee4c1a7740 1 -- start start 2024-03-18T14:54:12.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.808+0000 7fee4c1a7740 1 Processor -- start 2024-03-18T14:54:12.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee4c1a7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef96830 0x55d89efa3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee4c1a7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef97150 0x55d89efa3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee4c1a7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef9fd30 0x55d89efa42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee4c1a7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fee441adbb0 con 0x55d89ef97150 2024-03-18T14:54:12.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee4c1a7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fee441af020 con 0x55d89ef96830 2024-03-18T14:54:12.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee4c1a7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fee441ae280 con 0x55d89ef9fd30 2024-03-18T14:54:12.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee4c1a7740 10 client.0 _ll_register_callbacks cb 0x7fee38009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:12.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee427fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef97150 0x55d89efa3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee42ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef96830 0x55d89efa3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee427fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef97150 0x55d89efa3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44968/0 (socket says 172.21.15.76:44968) 2024-03-18T14:54:12.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee427fc640 1 -- 172.21.15.76:0/1490902322 learned_addr learned my addr 172.21.15.76:0/1490902322 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee437fe640 1 --2- 172.21.15.76:0/1490902322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef9fd30 0x55d89efa42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee42ffd640 1 -- 172.21.15.76:0/1490902322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef9fd30 msgr2=0x55d89efa42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee42ffd640 1 --2- 172.21.15.76:0/1490902322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef9fd30 0x55d89efa42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee42ffd640 1 -- 172.21.15.76:0/1490902322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef97150 msgr2=0x55d89efa3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee42ffd640 1 --2- 172.21.15.76:0/1490902322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef97150 0x55d89efa3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee42ffd640 1 -- 172.21.15.76:0/1490902322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d89f015490 con 0x55d89ef96830 2024-03-18T14:54:12.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee427fc640 1 --2- 172.21.15.76:0/1490902322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef97150 0x55d89efa3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee42ffd640 1 --2- 172.21.15.76:0/1490902322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef96830 0x55d89efa3870 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fee34005ab0 tx=0x7fee34009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d273edbf282ede92 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:54:12.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee26ff5640 1 -- 172.21.15.76:0/1490902322 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee34042040 con 0x55d89ef96830 2024-03-18T14:54:12.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee4c1a7740 1 -- 172.21.15.76:0/1490902322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee3c00fcc0 con 0x55d89ef96830 2024-03-18T14:54:12.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.809+0000 7fee26ff5640 1 -- 172.21.15.76:0/1490902322 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fee3404eca0 con 0x55d89ef96830 2024-03-18T14:54:12.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.810+0000 7fee26ff5640 1 -- 172.21.15.76:0/1490902322 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fee3405d050 con 0x55d89ef96830 2024-03-18T14:54:12.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.810+0000 7fee4c1a7740 10 client.25613 fetch_fsmap learned FSMap version 24 2024-03-18T14:54:12.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.810+0000 7fee4c1a7740 1 -- 172.21.15.76:0/1490902322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d89f0167b0 con 0x55d89ef96830 2024-03-18T14:54:12.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.810+0000 7fee26ff5640 1 -- 172.21.15.76:0/1490902322 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee3c00fcc0 con 0x55d89ef96830 2024-03-18T14:54:12.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.810+0000 7fee26ff5640 1 -- 172.21.15.76:0/1490902322 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fee340047e0 con 0x55d89ef96830 2024-03-18T14:54:12.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.810+0000 7fee4c1a7740 10 client.25613 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:54:12.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.810+0000 7fee4c1a7740 10 client.25613 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T14:54:12.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.810+0000 7fee4c1a7740 20 client.25613 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:12.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.810+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/1490902322 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x55d89f016bb0 0x55d89f018fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.810+0000 7fee4c1a7740 4 client.25613 mds_command: new command op to 24490 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T14:54:12.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.810+0000 7fee4c1a7740 1 -- 172.21.15.76:0/1490902322 --> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d89ee7f6a0 con 0x55d89f016bb0 2024-03-18T14:54:12.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.810+0000 7fee427fc640 1 --2- 172.21.15.76:0/1490902322 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x55d89f016bb0 0x55d89f018fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee427fc640 1 --2- 172.21.15.76:0/1490902322 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x55d89f016bb0 0x55d89f018fa0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4a2c55a1a0a3b7dd server_cookie=85d9a97992a67e76 in_seq=0 out_seq=0 2024-03-18T14:54:12.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee26ff5640 10 client.25613 ms_handle_connect on v2:172.21.15.174:6838/399810652 2024-03-18T14:54:12.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee26ff5640 1 -- 172.21.15.76:0/1490902322 <== mds.1 v2:172.21.15.174:6838/399810652 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d89ee7f6a0 con 0x55d89f016bb0 2024-03-18T14:54:12.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee26ff5640 10 client.25613 handle_command_reply: tid=0 2024-03-18T14:54:12.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee4c1a7740 1 client.25613 shutdown 2024-03-18T14:54:12.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee4c1a7740 1 -- 172.21.15.76:0/1490902322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef96830 msgr2=0x55d89efa3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/1490902322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef96830 0x55d89efa3870 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fee34005ab0 tx=0x7fee34009680 comp rx=0 tx=0).stop 2024-03-18T14:54:12.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee4c1a7740 1 -- 172.21.15.76:0/1490902322 shutdown_connections 2024-03-18T14:54:12.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/1490902322 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x55d89f016bb0 0x55d89f018fa0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/1490902322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d89ef9fd30 0x55d89efa42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee26ff5640 0 client.25613 ms_handle_reset on v2:172.21.15.174:6838/399810652 2024-03-18T14:54:12.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/1490902322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d89ef97150 0x55d89efa3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee4c1a7740 1 --2- 172.21.15.76:0/1490902322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d89ef96830 0x55d89efa3870 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee4c1a7740 1 -- 172.21.15.76:0/1490902322 >> 172.21.15.76:0/1490902322 conn(0x55d89ef90f60 msgr2=0x55d89ef9d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee4c1a7740 1 -- 172.21.15.76:0/1490902322 shutdown_connections 2024-03-18T14:54:12.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee4c1a7740 1 -- 172.21.15.76:0/1490902322 wait complete. 2024-03-18T14:54:12.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee4c1a7740 20 client.25613 trim_cache size 0 max 16384 2024-03-18T14:54:12.822 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-18T14:54:12.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee494dc640 1 -- 172.21.15.76:0/1922893438 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fee2c0a2a90 msgr2=0x7fee2c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee494dc640 1 --2- 172.21.15.76:0/1922893438 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fee2c0a2a90 0x7fee2c0a4f50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fee3c005c40 tx=0x7fee3c009500 comp rx=0 tx=0).stop 2024-03-18T14:54:12.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee494dc640 1 -- 172.21.15.76:0/1922893438 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee44100a60 msgr2=0x7fee4419a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.811+0000 7fee494dc640 1 --2- 172.21.15.76:0/1922893438 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee44100a60 0x7fee4419a8c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fee3800e9f0 tx=0x7fee3800edf0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.812+0000 7fee494dc640 1 -- 172.21.15.76:0/1922893438 shutdown_connections 2024-03-18T14:54:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.812+0000 7fee494dc640 1 --2- 172.21.15.76:0/1922893438 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fee2c0a2a90 0x7fee2c0a4f50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.812+0000 7fee494dc640 1 --2- 172.21.15.76:0/1922893438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee441079e0 0x7fee4419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.812+0000 7fee494dc640 1 --2- 172.21.15.76:0/1922893438 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee44100a60 0x7fee4419a8c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.812+0000 7fee494dc640 1 --2- 172.21.15.76:0/1922893438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee441000a0 0x7fee4419a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.812+0000 7fee494dc640 1 -- 172.21.15.76:0/1922893438 >> 172.21.15.76:0/1922893438 conn(0x7fee44075a20 msgr2=0x7fee440fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.812+0000 7fee494dc640 1 -- 172.21.15.76:0/1922893438 shutdown_connections 2024-03-18T14:54:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.812+0000 7fee494dc640 1 -- 172.21.15.76:0/1922893438 wait complete. 2024-03-18T14:54:12.824 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T14:54:12.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:12 smithi174 ceph-mon[36787]: pgmap v30: 97 pgs: 97 active+clean; 16 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.4 MiB/s wr, 186 op/s 2024-03-18T14:54:12.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:12 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2375080819' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:54:12.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.974+0000 7fc97a0f1640 1 -- 172.21.15.76:0/2286784807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc974107780 msgr2=0x7fc974107b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.974+0000 7fc97a0f1640 1 --2- 172.21.15.76:0/2286784807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc974107780 0x7fc974107b60 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc964009930 tx=0x7fc96402bc30 comp rx=0 tx=0).stop 2024-03-18T14:54:12.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.975+0000 7fc97a0f1640 1 -- 172.21.15.76:0/2286784807 shutdown_connections 2024-03-18T14:54:12.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.975+0000 7fc97a0f1640 1 --2- 172.21.15.76:0/2286784807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc974102130 0x7fc97410e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.975+0000 7fc97a0f1640 1 --2- 172.21.15.76:0/2286784807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc974101770 0x7fc974101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.975+0000 7fc97a0f1640 1 --2- 172.21.15.76:0/2286784807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc974107780 0x7fc974107b60 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.975+0000 7fc97a0f1640 1 -- 172.21.15.76:0/2286784807 >> 172.21.15.76:0/2286784807 conn(0x7fc974078be0 msgr2=0x7fc974078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.975+0000 7fc97a0f1640 1 -- 172.21.15.76:0/2286784807 shutdown_connections 2024-03-18T14:54:12.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.975+0000 7fc97a0f1640 1 -- 172.21.15.76:0/2286784807 wait complete. 2024-03-18T14:54:12.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.975+0000 7fc97a0f1640 1 Processor -- start 2024-03-18T14:54:12.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.975+0000 7fc97a0f1640 1 -- start start 2024-03-18T14:54:12.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.975+0000 7fc97a0f1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc974101770 0x7fc974198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc97a0f1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc974102130 0x7fc974198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc97a0f1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc974107780 0x7fc97419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc97a0f1640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc974111b90 con 0x7fc974101770 2024-03-18T14:54:12.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc97a0f1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc974111a10 con 0x7fc974107780 2024-03-18T14:54:12.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc97a0f1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc974111d10 con 0x7fc974102130 2024-03-18T14:54:12.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc9737fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc974101770 0x7fc974198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc9737fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc974101770 0x7fc974198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45000/0 (socket says 172.21.15.76:45000) 2024-03-18T14:54:12.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc973fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc974107780 0x7fc97419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc9737fe640 1 -- 172.21.15.76:0/1277118460 learned_addr learned my addr 172.21.15.76:0/1277118460 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc972ffd640 1 --2- 172.21.15.76:0/1277118460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc974102130 0x7fc974198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc9737fe640 1 -- 172.21.15.76:0/1277118460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc974102130 msgr2=0x7fc974198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc9737fe640 1 --2- 172.21.15.76:0/1277118460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc974102130 0x7fc974198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc9737fe640 1 -- 172.21.15.76:0/1277118460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc974107780 msgr2=0x7fc97419c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc9737fe640 1 --2- 172.21.15.76:0/1277118460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc974107780 0x7fc97419c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc9737fe640 1 -- 172.21.15.76:0/1277118460 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc97419d040 con 0x7fc974101770 2024-03-18T14:54:12.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc973fff640 1 --2- 172.21.15.76:0/1277118460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc974107780 0x7fc97419c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc9737fe640 1 --2- 172.21.15.76:0/1277118460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc974101770 0x7fc974198050 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc964009230 tx=0x7fc96402cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc970ff9640 1 -- 172.21.15.76:0/1277118460 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc964041070 con 0x7fc974101770 2024-03-18T14:54:12.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc97a0f1640 1 -- 172.21.15.76:0/1277118460 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc97419d2d0 con 0x7fc974101770 2024-03-18T14:54:12.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc970ff9640 1 -- 172.21.15.76:0/1277118460 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc964004450 con 0x7fc974101770 2024-03-18T14:54:12.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc970ff9640 1 -- 172.21.15.76:0/1277118460 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc964003e50 con 0x7fc974101770 2024-03-18T14:54:12.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.976+0000 7fc97a0f1640 1 -- 172.21.15.76:0/1277118460 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9741a4bc0 con 0x7fc974101770 2024-03-18T14:54:12.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:12 smithi077 ceph-mon[35865]: pgmap v30: 97 pgs: 97 active+clean; 16 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.4 MiB/s wr, 186 op/s 2024-03-18T14:54:12.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:12 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2375080819' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:54:12.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.979+0000 7fc970ff9640 1 -- 172.21.15.76:0/1277118460 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7fc964044070 con 0x7fc974101770 2024-03-18T14:54:12.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.979+0000 7fc97cdb0740 1 Processor -- start 2024-03-18T14:54:12.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.979+0000 7fc97cdb0740 1 -- start start 2024-03-18T14:54:12.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.979+0000 7fc97cdb0740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d60f7830 0x5612d60f7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.979+0000 7fc97cdb0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d61007f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.979+0000 7fc97cdb0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d6100d30 0x5612d6103120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.979+0000 7fc97cdb0740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5612d605b490 con 0x5612d60f7830 2024-03-18T14:54:12.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.979+0000 7fc97cdb0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5612d6003c90 con 0x5612d60f8150 2024-03-18T14:54:12.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.979+0000 7fc97cdb0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5612d6057530 con 0x5612d6100d30 2024-03-18T14:54:12.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.979+0000 7fc970ff9640 1 --2- 172.21.15.76:0/1277118460 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc9600a2c30 0x7fc9600a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.979+0000 7fc973fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d60f7830 0x5612d60f7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.979+0000 7fc973fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d60f7830 0x5612d60f7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45002/0 (socket says 172.21.15.76:45002) 2024-03-18T14:54:12.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.979+0000 7fc973fff640 1 -- 172.21.15.76:0/152917589 learned_addr learned my addr 172.21.15.76:0/152917589 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.980+0000 7fc972ffd640 1 --2- 172.21.15.76:0/152917589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d6100d30 0x5612d6103120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.980+0000 7fc9737fe640 1 --2- 172.21.15.76:0/152917589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d61007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.980+0000 7fc973fff640 1 --2- 172.21.15.76:0/1277118460 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc9600a2c30 0x7fc9600a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.980+0000 7fc970ff9640 1 -- 172.21.15.76:0/1277118460 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fc9640f0890 con 0x7fc974101770 2024-03-18T14:54:12.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.980+0000 7fc973fff640 1 -- 172.21.15.76:0/152917589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d6100d30 msgr2=0x5612d6103120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.980+0000 7fc973fff640 1 --2- 172.21.15.76:0/152917589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d6100d30 0x5612d6103120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.980+0000 7fc973fff640 1 -- 172.21.15.76:0/152917589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 msgr2=0x5612d61007f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.980+0000 7fc973fff640 1 --2- 172.21.15.76:0/152917589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d61007f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.980+0000 7fc973fff640 1 -- 172.21.15.76:0/152917589 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612d60be500 con 0x5612d60f7830 2024-03-18T14:54:12.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.980+0000 7fc972ffd640 1 --2- 172.21.15.76:0/152917589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d6100d30 0x5612d6103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.980+0000 7fc973fff640 1 --2- 172.21.15.76:0/152917589 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d60f7830 0x5612d60f7c10 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009fd0 tx=0x7fc96c00f7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a08cc9aef3e949d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.980+0000 7fc973fff640 1 --2- 172.21.15.76:0/1277118460 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc9600a2c30 0x7fc9600a50f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc96c01e000 tx=0x7fc96c020000 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.980+0000 7fc969ffb640 1 -- 172.21.15.76:0/152917589 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c013860 con 0x5612d60f7830 2024-03-18T14:54:12.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.980+0000 7fc969ffb640 1 -- 172.21.15.76:0/152917589 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc96c023020 con 0x5612d60f7830 2024-03-18T14:54:12.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.980+0000 7fc969ffb640 1 -- 172.21.15.76:0/152917589 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c01fd80 con 0x5612d60f7830 2024-03-18T14:54:12.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.981+0000 7fc97cdb0740 1 -- 172.21.15.76:0/152917589 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d60f7830 msgr2=0x5612d60f7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.981+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/152917589 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d60f7830 0x5612d60f7c10 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009fd0 tx=0x7fc96c00f7d0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.981+0000 7fc97cdb0740 1 -- 172.21.15.76:0/152917589 shutdown_connections 2024-03-18T14:54:12.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.981+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/152917589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d6100d30 0x5612d6103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.981+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/152917589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d61007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.981+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/152917589 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d60f7830 0x5612d60f7c10 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.981+0000 7fc97cdb0740 1 -- 172.21.15.76:0/152917589 >> 172.21.15.76:0/152917589 conn(0x5612d60f1f60 msgr2=0x5612d60f2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.981+0000 7fc97cdb0740 1 -- 172.21.15.76:0/152917589 shutdown_connections 2024-03-18T14:54:12.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.981+0000 7fc97cdb0740 1 -- 172.21.15.76:0/152917589 wait complete. 2024-03-18T14:54:12.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.982+0000 7fc97cdb0740 1 -- start start 2024-03-18T14:54:12.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.982+0000 7fc97cdb0740 1 Processor -- start 2024-03-18T14:54:12.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.982+0000 7fc97cdb0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d60f7830 0x5612d60fdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.982+0000 7fc97cdb0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d60fe320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.982+0000 7fc97cdb0740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d6100d30 0x5612d60fe860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.982+0000 7fc97cdb0740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5612d6103ac0 con 0x5612d6100d30 2024-03-18T14:54:12.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.982+0000 7fc97cdb0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5612d6017230 con 0x5612d60f8150 2024-03-18T14:54:12.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.982+0000 7fc97cdb0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5612d6103c40 con 0x5612d60f7830 2024-03-18T14:54:12.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.982+0000 7fc97cdb0740 10 client.0 _ll_register_callbacks cb 0x5612d6053010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:12.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.982+0000 7fc973fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d6100d30 0x5612d60fe860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.982+0000 7fc973fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d6100d30 0x5612d60fe860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45006/0 (socket says 172.21.15.76:45006) 2024-03-18T14:54:12.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.982+0000 7fc972ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d60fe320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.982+0000 7fc973fff640 1 -- 172.21.15.76:0/703528663 learned_addr learned my addr 172.21.15.76:0/703528663 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:12.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.982+0000 7fc9737fe640 1 --2- 172.21.15.76:0/703528663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d60f7830 0x5612d60fdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc973fff640 1 -- 172.21.15.76:0/703528663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d60f7830 msgr2=0x5612d60fdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc973fff640 1 --2- 172.21.15.76:0/703528663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d60f7830 0x5612d60fdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc973fff640 1 -- 172.21.15.76:0/703528663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 msgr2=0x5612d60fe320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:12.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc973fff640 1 --2- 172.21.15.76:0/703528663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d60fe320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc973fff640 1 -- 172.21.15.76:0/703528663 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612d6026100 con 0x5612d6100d30 2024-03-18T14:54:12.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc972ffd640 1 --2- 172.21.15.76:0/703528663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d60fe320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc9737fe640 1 --2- 172.21.15.76:0/703528663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d60f7830 0x5612d60fdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:12.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc973fff640 1 --2- 172.21.15.76:0/703528663 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d6100d30 0x5612d60fe860 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0121d0 tx=0x7fc96c012200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3515f9961712a11c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:12.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc952ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:12.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc952ffd640 1 -- 172.21.15.76:0/703528663 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c013a20 con 0x5612d6100d30 2024-03-18T14:54:12.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc952ffd640 1 -- 172.21.15.76:0/703528663 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fc96c01f480 con 0x5612d6100d30 2024-03-18T14:54:12.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc97cdb0740 1 -- 172.21.15.76:0/703528663 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc96c023020 con 0x5612d6100d30 2024-03-18T14:54:12.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc952ffd640 1 -- 172.21.15.76:0/703528663 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc96c01f680 con 0x5612d6100d30 2024-03-18T14:54:12.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc97cdb0740 10 client.15768 fetch_fsmap learned FSMap version 24 2024-03-18T14:54:12.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc97cdb0740 1 -- 172.21.15.76:0/703528663 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5612d5d6f3f0 con 0x5612d6100d30 2024-03-18T14:54:12.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc952ffd640 1 -- 172.21.15.76:0/703528663 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc96c023020 con 0x5612d6100d30 2024-03-18T14:54:12.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc952ffd640 1 -- 172.21.15.76:0/703528663 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fc96c02f070 con 0x5612d6100d30 2024-03-18T14:54:12.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc97cdb0740 10 client.15768 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:54:12.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc97cdb0740 10 client.15768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:12.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc97cdb0740 20 client.15768 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:12.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/703528663 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5612d6111030 0x5612d6113420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc97cdb0740 4 client.15768 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:54:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.983+0000 7fc97cdb0740 1 -- 172.21.15.76:0/703528663 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612d60689d0 con 0x5612d6111030 2024-03-18T14:54:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.984+0000 7fc9737fe640 1 --2- 172.21.15.76:0/703528663 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5612d6111030 0x5612d6113420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.984+0000 7fc9737fe640 1 --2- 172.21.15.76:0/703528663 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5612d6111030 0x5612d6113420 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eabaf4c0854d040e server_cookie=388497f8b0191640 in_seq=0 out_seq=0 2024-03-18T14:54:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.984+0000 7fc952ffd640 10 client.15768 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.985+0000 7fc952ffd640 1 -- 172.21.15.76:0/703528663 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5612d60689d0 con 0x5612d6111030 2024-03-18T14:54:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.985+0000 7fc952ffd640 10 client.15768 handle_command_reply: tid=0 2024-03-18T14:54:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.985+0000 7fc97cdb0740 1 client.15768 shutdown 2024-03-18T14:54:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.985+0000 7fc97cdb0740 1 -- 172.21.15.76:0/703528663 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d6100d30 msgr2=0x5612d60fe860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.985+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/703528663 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d6100d30 0x5612d60fe860 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0121d0 tx=0x7fc96c012200 comp rx=0 tx=0).stop 2024-03-18T14:54:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.985+0000 7fc97cdb0740 1 -- 172.21.15.76:0/703528663 shutdown_connections 2024-03-18T14:54:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.985+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/703528663 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5612d6111030 0x5612d6113420 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.985+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/703528663 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d6100d30 0x5612d60fe860 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.985+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/703528663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d60fe320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.985+0000 7fc952ffd640 0 client.15768 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.985+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/703528663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d60f7830 0x5612d60fdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.985+0000 7fc97cdb0740 1 -- 172.21.15.76:0/703528663 >> 172.21.15.76:0/703528663 conn(0x5612d60f1f60 msgr2=0x5612d60f3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.985+0000 7fc97cdb0740 1 -- 172.21.15.76:0/703528663 shutdown_connections 2024-03-18T14:54:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.985+0000 7fc97cdb0740 1 -- 172.21.15.76:0/703528663 wait complete. 2024-03-18T14:54:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:12.985+0000 7fc97cdb0740 20 client.15768 trim_cache size 0 max 16384 2024-03-18T14:54:13.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.000+0000 7fc97cdb0740 1 Processor -- start 2024-03-18T14:54:13.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.000+0000 7fc97cdb0740 1 -- start start 2024-03-18T14:54:13.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.000+0000 7fc97cdb0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d60f7830 0x5612d6100510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:13.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.000+0000 7fc97cdb0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d610c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:13.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.000+0000 7fc97cdb0740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d6100d30 0x5612d6118c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:13.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.000+0000 7fc97cdb0740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc9741afcb0 con 0x5612d6100d30 2024-03-18T14:54:13.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.000+0000 7fc97cdb0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc9741ad580 con 0x5612d60f8150 2024-03-18T14:54:13.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.000+0000 7fc97cdb0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc9741ab3d0 con 0x5612d60f7830 2024-03-18T14:54:13.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.000+0000 7fc973fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d6100d30 0x5612d6118c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:13.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.000+0000 7fc9737fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d60f7830 0x5612d6100510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:13.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.000+0000 7fc972ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d610c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:13.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.000+0000 7fc9737fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d60f7830 0x5612d6100510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:53544/0 (socket says 172.21.15.76:53544) 2024-03-18T14:54:13.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.000+0000 7fc972ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d610c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:38888/0 (socket says 172.21.15.76:38888) 2024-03-18T14:54:13.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.000+0000 7fc973fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d6100d30 0x5612d6118c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45022/0 (socket says 172.21.15.76:45022) 2024-03-18T14:54:13.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.000+0000 7fc972ffd640 1 -- 172.21.15.76:0/3796530779 learned_addr learned my addr 172.21.15.76:0/3796530779 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:13.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.001+0000 7fc973fff640 1 -- 172.21.15.76:0/3796530779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d60f7830 msgr2=0x5612d6100510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:13.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.001+0000 7fc973fff640 1 --2- 172.21.15.76:0/3796530779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d60f7830 0x5612d6100510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:13.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.001+0000 7fc973fff640 1 -- 172.21.15.76:0/3796530779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 msgr2=0x5612d610c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:13.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.001+0000 7fc973fff640 1 --2- 172.21.15.76:0/3796530779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d610c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:13.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.001+0000 7fc973fff640 1 -- 172.21.15.76:0/3796530779 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612d5e397d0 con 0x5612d6100d30 2024-03-18T14:54:13.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.001+0000 7fc972ffd640 1 --2- 172.21.15.76:0/3796530779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d610c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:13.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.001+0000 7fc973fff640 1 --2- 172.21.15.76:0/3796530779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d6100d30 0x5612d6118c20 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc96c005770 tx=0x7fc96c009720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8df3a4e6b607523 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:13.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.001+0000 7fc9697fa640 1 -- 172.21.15.76:0/3796530779 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c03e070 con 0x5612d6100d30 2024-03-18T14:54:13.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.001+0000 7fc9697fa640 1 -- 172.21.15.76:0/3796530779 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc96c023070 con 0x5612d6100d30 2024-03-18T14:54:13.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.001+0000 7fc9697fa640 1 -- 172.21.15.76:0/3796530779 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c0057f0 con 0x5612d6100d30 2024-03-18T14:54:13.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.002+0000 7fc97cdb0740 1 -- 172.21.15.76:0/3796530779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d6100d30 msgr2=0x5612d6118c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:13.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.002+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/3796530779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d6100d30 0x5612d6118c20 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc96c005770 tx=0x7fc96c009720 comp rx=0 tx=0).stop 2024-03-18T14:54:13.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.002+0000 7fc97cdb0740 1 -- 172.21.15.76:0/3796530779 shutdown_connections 2024-03-18T14:54:13.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.002+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/3796530779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d6100d30 0x5612d6118c20 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:13.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.002+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/3796530779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d610c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:13.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.002+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/3796530779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d60f7830 0x5612d6100510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:13.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.002+0000 7fc97cdb0740 1 -- 172.21.15.76:0/3796530779 >> 172.21.15.76:0/3796530779 conn(0x5612d60f1f60 msgr2=0x5612d60fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:13.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.002+0000 7fc97cdb0740 1 -- 172.21.15.76:0/3796530779 shutdown_connections 2024-03-18T14:54:13.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.002+0000 7fc97cdb0740 1 -- 172.21.15.76:0/3796530779 wait complete. 2024-03-18T14:54:13.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.002+0000 7fc97cdb0740 1 -- start start 2024-03-18T14:54:13.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.002+0000 7fc97cdb0740 1 Processor -- start 2024-03-18T14:54:13.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc97cdb0740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d60f7830 0x5612d6108ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:13.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc97cdb0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d61044b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:13.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc97cdb0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d6100d30 0x5612d61049f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:13.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc97cdb0740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc9741aba40 con 0x5612d60f7830 2024-03-18T14:54:13.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc97cdb0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc9741aceb0 con 0x5612d60f8150 2024-03-18T14:54:13.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc97cdb0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc9741ac110 con 0x5612d6100d30 2024-03-18T14:54:13.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc97cdb0740 10 client.0 _ll_register_callbacks cb 0x7fc9640b8ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:13.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc9737fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d60f7830 0x5612d6108ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:13.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc9737fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d60f7830 0x5612d6108ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45036/0 (socket says 172.21.15.76:45036) 2024-03-18T14:54:13.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc9737fe640 1 -- 172.21.15.76:0/430394886 learned_addr learned my addr 172.21.15.76:0/430394886 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:13.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc972ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d61044b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:13.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc973fff640 1 --2- 172.21.15.76:0/430394886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d6100d30 0x5612d61049f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:13.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc9737fe640 1 -- 172.21.15.76:0/430394886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d6100d30 msgr2=0x5612d61049f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:13.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc9737fe640 1 --2- 172.21.15.76:0/430394886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d6100d30 0x5612d61049f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:13.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc9737fe640 1 -- 172.21.15.76:0/430394886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 msgr2=0x5612d61044b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:13.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc9737fe640 1 --2- 172.21.15.76:0/430394886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d61044b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:13.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc9737fe640 1 -- 172.21.15.76:0/430394886 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612d6104f30 con 0x5612d60f7830 2024-03-18T14:54:13.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc972ffd640 1 --2- 172.21.15.76:0/430394886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d61044b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:13.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc9737fe640 1 --2- 172.21.15.76:0/430394886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d60f7830 0x5612d6108ee0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc9640b9880 tx=0x7fc96402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d43fbdcff8aad9e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:13.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc952ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:13.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc952ffd640 1 -- 172.21.15.76:0/430394886 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96404b040 con 0x5612d60f7830 2024-03-18T14:54:13.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc952ffd640 1 -- 172.21.15.76:0/430394886 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fc9640bb1b0 con 0x5612d60f7830 2024-03-18T14:54:13.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.003+0000 7fc97cdb0740 1 -- 172.21.15.76:0/430394886 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc96c023070 con 0x5612d60f7830 2024-03-18T14:54:13.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.004+0000 7fc952ffd640 1 -- 172.21.15.76:0/430394886 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc9640bceb0 con 0x5612d60f7830 2024-03-18T14:54:13.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.004+0000 7fc97cdb0740 10 client.15774 fetch_fsmap learned FSMap version 24 2024-03-18T14:54:13.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.004+0000 7fc97cdb0740 1 -- 172.21.15.76:0/430394886 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5612d6177030 con 0x5612d60f7830 2024-03-18T14:54:13.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.004+0000 7fc952ffd640 1 -- 172.21.15.76:0/430394886 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc96c023070 con 0x5612d60f7830 2024-03-18T14:54:13.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.004+0000 7fc952ffd640 1 -- 172.21.15.76:0/430394886 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fc9640ee190 con 0x5612d60f7830 2024-03-18T14:54:13.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.004+0000 7fc97cdb0740 10 client.15774 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T14:54:13.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.004+0000 7fc97cdb0740 10 client.15774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:13.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.004+0000 7fc97cdb0740 20 client.15774 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:13.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.004+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/430394886 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5612d6177760 0x5612d6179b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:13.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.004+0000 7fc97cdb0740 4 client.15774 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T14:54:13.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.004+0000 7fc97cdb0740 1 -- 172.21.15.76:0/430394886 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5612d6062d20 con 0x5612d6177760 2024-03-18T14:54:13.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.004+0000 7fc972ffd640 1 --2- 172.21.15.76:0/430394886 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5612d6177760 0x5612d6179b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:13.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.004+0000 7fc952ffd640 10 client.15774 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:13.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.004+0000 7fc972ffd640 1 --2- 172.21.15.76:0/430394886 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5612d6177760 0x5612d6179b80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e7eaa26e1b0ee45 server_cookie=cceb56c2a96ffe68 in_seq=0 out_seq=0 2024-03-18T14:54:13.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.007+0000 7fc952ffd640 1 -- 172.21.15.76:0/430394886 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5612d6062d20 con 0x5612d6177760 2024-03-18T14:54:13.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.007+0000 7fc952ffd640 10 client.15774 handle_command_reply: tid=0 2024-03-18T14:54:13.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.007+0000 7fc97cdb0740 1 client.15774 shutdown 2024-03-18T14:54:13.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.007+0000 7fc97cdb0740 1 -- 172.21.15.76:0/430394886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d60f7830 msgr2=0x5612d6108ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:13.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.007+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/430394886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d60f7830 0x5612d6108ee0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc9640b9880 tx=0x7fc96402bc30 comp rx=0 tx=0).stop 2024-03-18T14:54:13.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.007+0000 7fc97cdb0740 1 -- 172.21.15.76:0/430394886 shutdown_connections 2024-03-18T14:54:13.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.007+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/430394886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612d6100d30 0x5612d61049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:13.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.007+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/430394886 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5612d6177760 0x5612d6179b80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:13.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.007+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/430394886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612d60f8150 0x5612d61044b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:13.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.007+0000 7fc952ffd640 0 client.15774 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:13.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.007+0000 7fc97cdb0740 1 --2- 172.21.15.76:0/430394886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612d60f7830 0x5612d6108ee0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:13.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.007+0000 7fc97cdb0740 1 -- 172.21.15.76:0/430394886 >> 172.21.15.76:0/430394886 conn(0x5612d60f1f60 msgr2=0x5612d60f0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:13.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.007+0000 7fc97cdb0740 1 -- 172.21.15.76:0/430394886 shutdown_connections 2024-03-18T14:54:13.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.007+0000 7fc97cdb0740 1 -- 172.21.15.76:0/430394886 wait complete. 2024-03-18T14:54:13.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.007+0000 7fc97cdb0740 20 client.15774 trim_cache size 0 max 16384 2024-03-18T14:54:13.015 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-18T14:54:13.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.008+0000 7fc97a0f1640 1 -- 172.21.15.76:0/1277118460 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc9600a2c30 msgr2=0x7fc9600a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:13.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.008+0000 7fc97a0f1640 1 --2- 172.21.15.76:0/1277118460 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc9600a2c30 0x7fc9600a50f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc96c01e000 tx=0x7fc96c020000 comp rx=0 tx=0).stop 2024-03-18T14:54:13.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.008+0000 7fc97a0f1640 1 -- 172.21.15.76:0/1277118460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc974101770 msgr2=0x7fc974198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:13.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.008+0000 7fc97a0f1640 1 --2- 172.21.15.76:0/1277118460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc974101770 0x7fc974198050 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc964009230 tx=0x7fc96402cb30 comp rx=0 tx=0).stop 2024-03-18T14:54:13.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.008+0000 7fc97a0f1640 1 -- 172.21.15.76:0/1277118460 shutdown_connections 2024-03-18T14:54:13.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.008+0000 7fc97a0f1640 1 --2- 172.21.15.76:0/1277118460 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc9600a2c30 0x7fc9600a50f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:13.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.008+0000 7fc97a0f1640 1 --2- 172.21.15.76:0/1277118460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc974102130 0x7fc974198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:13.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.008+0000 7fc97a0f1640 1 --2- 172.21.15.76:0/1277118460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc974107780 0x7fc97419c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:13.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.008+0000 7fc97a0f1640 1 --2- 172.21.15.76:0/1277118460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc974101770 0x7fc974198050 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:13.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.008+0000 7fc97a0f1640 1 -- 172.21.15.76:0/1277118460 >> 172.21.15.76:0/1277118460 conn(0x7fc974078be0 msgr2=0x7fc9741040c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:13.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.008+0000 7fc97a0f1640 1 -- 172.21.15.76:0/1277118460 shutdown_connections 2024-03-18T14:54:13.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:13.008+0000 7fc97a0f1640 1 -- 172.21.15.76:0/1277118460 wait complete. 2024-03-18T14:54:13.477 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:54:13 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setuser ceph since I am not root 2024-03-18T14:54:13.477 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:54:13 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setgroup ceph since I am not root 2024-03-18T14:54:13.477 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:54:13 smithi077 ceph-mds[65404]: -- 172.21.15.77:0/1154666548 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563b6f6b3a40 con 0x563b6f687000 2024-03-18T14:54:13.477 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:54:13 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: starting mds.e at 2024-03-18T14:54:13.731 INFO:tasks.workunit.client.0.smithi076.stdout: 1 39263 16.50 MB/sec warmup 14 sec latency 162.282 ms 2024-03-18T14:54:13.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:13 smithi076 ceph-mon[32205]: daemon mds.e restarted 2024-03-18T14:54:13.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:13 smithi076 ceph-mon[32205]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T14:54:13.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:13 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.77:6834/1541266751,v1:172.21.15.77:6835/1541266751] up:boot 2024-03-18T14:54:13.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:13 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:54:13.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:13 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:54:13.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:13 smithi174 ceph-mon[36787]: daemon mds.e restarted 2024-03-18T14:54:13.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:13 smithi174 ceph-mon[36787]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T14:54:13.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:13 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.77:6834/1541266751,v1:172.21.15.77:6835/1541266751] up:boot 2024-03-18T14:54:13.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:13 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:54:13.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:13 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:54:13.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:13 smithi077 ceph-mon[35865]: daemon mds.e restarted 2024-03-18T14:54:13.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:13 smithi077 ceph-mon[35865]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T14:54:13.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:13 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.77:6834/1541266751,v1:172.21.15.77:6835/1541266751] up:boot 2024-03-18T14:54:13.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:13 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:54:13.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:13 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:54:14.021 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T14:54:14.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:54:14.021 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T14:54:14.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.131+0000 7f7d39fb2640 1 -- 172.21.15.76:0/1748465680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d340689c0 msgr2=0x7f7d34068e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.131+0000 7f7d39fb2640 1 --2- 172.21.15.76:0/1748465680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d340689c0 0x7f7d34068e40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009930 tx=0x7f7d2802bc30 comp rx=0 tx=0).stop 2024-03-18T14:54:14.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.132+0000 7f7d39fb2640 1 -- 172.21.15.76:0/1748465680 shutdown_connections 2024-03-18T14:54:14.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.132+0000 7f7d39fb2640 1 --2- 172.21.15.76:0/1748465680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d34069380 0x7f7d3410e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.132+0000 7f7d39fb2640 1 --2- 172.21.15.76:0/1748465680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d340689c0 0x7f7d34068e40 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.132+0000 7f7d39fb2640 1 --2- 172.21.15.76:0/1748465680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d34107760 0x7f7d34107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.132+0000 7f7d39fb2640 1 -- 172.21.15.76:0/1748465680 >> 172.21.15.76:0/1748465680 conn(0x7f7d340777e0 msgr2=0x7f7d34077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:14.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.132+0000 7f7d39fb2640 1 -- 172.21.15.76:0/1748465680 shutdown_connections 2024-03-18T14:54:14.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.132+0000 7f7d39fb2640 1 -- 172.21.15.76:0/1748465680 wait complete. 2024-03-18T14:54:14.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.132+0000 7f7d39fb2640 1 Processor -- start 2024-03-18T14:54:14.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.132+0000 7f7d39fb2640 1 -- start start 2024-03-18T14:54:14.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.132+0000 7f7d39fb2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d340689c0 0x7f7d34198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.132+0000 7f7d39fb2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d34069380 0x7f7d34198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.132+0000 7f7d39fb2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d34107760 0x7f7d3419c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.132+0000 7f7d39fb2640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7d34111b70 con 0x7f7d34069380 2024-03-18T14:54:14.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.132+0000 7f7d39fb2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7d341119f0 con 0x7f7d34107760 2024-03-18T14:54:14.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.132+0000 7f7d39fb2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7d34111cf0 con 0x7f7d340689c0 2024-03-18T14:54:14.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d32ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d34069380 0x7f7d34198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d337fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d340689c0 0x7f7d34198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d32ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d34069380 0x7f7d34198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45048/0 (socket says 172.21.15.76:45048) 2024-03-18T14:54:14.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d32ffd640 1 -- 172.21.15.76:0/465020195 learned_addr learned my addr 172.21.15.76:0/465020195 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:14.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d33fff640 1 --2- 172.21.15.76:0/465020195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d34107760 0x7f7d3419c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d32ffd640 1 -- 172.21.15.76:0/465020195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d340689c0 msgr2=0x7f7d34198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d32ffd640 1 --2- 172.21.15.76:0/465020195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d340689c0 0x7f7d34198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d32ffd640 1 -- 172.21.15.76:0/465020195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d34107760 msgr2=0x7f7d3419c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d32ffd640 1 --2- 172.21.15.76:0/465020195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d34107760 0x7f7d3419c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d32ffd640 1 -- 172.21.15.76:0/465020195 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3419d100 con 0x7f7d34069380 2024-03-18T14:54:14.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d337fe640 1 --2- 172.21.15.76:0/465020195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d340689c0 0x7f7d34198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:14.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d33fff640 1 --2- 172.21.15.76:0/465020195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d34107760 0x7f7d3419c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:14.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d32ffd640 1 --2- 172.21.15.76:0/465020195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d34069380 0x7f7d34198650 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009230 tx=0x7f7d2802cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:14.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d30ff9640 1 -- 172.21.15.76:0/465020195 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d28041070 con 0x7f7d34069380 2024-03-18T14:54:14.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d39fb2640 1 -- 172.21.15.76:0/465020195 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d3419d390 con 0x7f7d34069380 2024-03-18T14:54:14.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d30ff9640 1 -- 172.21.15.76:0/465020195 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d28004450 con 0x7f7d34069380 2024-03-18T14:54:14.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d30ff9640 1 -- 172.21.15.76:0/465020195 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d28003e50 con 0x7f7d34069380 2024-03-18T14:54:14.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.133+0000 7f7d39fb2640 1 -- 172.21.15.76:0/465020195 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d341a4c80 con 0x7f7d34069380 2024-03-18T14:54:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.135+0000 7f7d30ff9640 1 -- 172.21.15.76:0/465020195 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f7d28044070 con 0x7f7d34069380 2024-03-18T14:54:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d30ff9640 1 --2- 172.21.15.76:0/465020195 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7d1c0a2c30 0x7f7d1c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d3cb9f740 1 Processor -- start 2024-03-18T14:54:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d3cb9f740 1 -- start start 2024-03-18T14:54:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d337fe640 1 --2- 172.21.15.76:0/465020195 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7d1c0a2c30 0x7f7d1c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d30ff9640 1 -- 172.21.15.76:0/465020195 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f7d280f0880 con 0x7f7d34069380 2024-03-18T14:54:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d3cb9f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8cd710 0x558cdc8cc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d3cb9f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cca30 0x558cdc8cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d3cb9f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 0x558cdc8dc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d3cb9f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x558cdc8300d0 con 0x558cdc8da1b0 2024-03-18T14:54:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d3cb9f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558cdc7d8c70 con 0x558cdc8cca30 2024-03-18T14:54:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d3cb9f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558cdc82c170 con 0x558cdc8cd710 2024-03-18T14:54:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d33fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 0x558cdc8dc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d33fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 0x558cdc8dc5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45052/0 (socket says 172.21.15.76:45052) 2024-03-18T14:54:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d32ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cca30 0x558cdc8cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d33fff640 1 -- 172.21.15.76:0/485307283 learned_addr learned my addr 172.21.15.76:0/485307283 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d337fe640 1 --2- 172.21.15.76:0/485307283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8cd710 0x558cdc8cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d337fe640 1 --2- 172.21.15.76:0/465020195 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7d1c0a2c30 0x7f7d1c0a50f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009f50 tx=0x7f7d24011810 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d32ffd640 1 -- 172.21.15.76:0/485307283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8cd710 msgr2=0x558cdc8cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d32ffd640 1 --2- 172.21.15.76:0/485307283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8cd710 0x558cdc8cc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d32ffd640 1 -- 172.21.15.76:0/485307283 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 msgr2=0x558cdc8dc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d32ffd640 1 --2- 172.21.15.76:0/485307283 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 0x558cdc8dc5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.136+0000 7f7d32ffd640 1 -- 172.21.15.76:0/485307283 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cdc60c7e0 con 0x558cdc8cca30 2024-03-18T14:54:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.137+0000 7f7d32ffd640 1 --2- 172.21.15.76:0/485307283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cca30 0x558cdc8cceb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7d280edfb0 tx=0x7f7d28009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3808f82cc9c9c3f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.137+0000 7f7d19ffb640 1 -- 172.21.15.76:0/485307283 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2804b040 con 0x558cdc8cca30 2024-03-18T14:54:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.137+0000 7f7d19ffb640 1 -- 172.21.15.76:0/485307283 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d280c0070 con 0x558cdc8cca30 2024-03-18T14:54:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.137+0000 7f7d19ffb640 1 -- 172.21.15.76:0/485307283 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d280bb230 con 0x558cdc8cca30 2024-03-18T14:54:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.137+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/485307283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cca30 msgr2=0x558cdc8cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.137+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/485307283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cca30 0x558cdc8cceb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7d280edfb0 tx=0x7f7d28009260 comp rx=0 tx=0).stop 2024-03-18T14:54:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.137+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/485307283 shutdown_connections 2024-03-18T14:54:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.137+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/485307283 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 0x558cdc8dc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.137+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/485307283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cca30 0x558cdc8cceb0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.137+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/485307283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8cd710 0x558cdc8cc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.137+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/485307283 >> 172.21.15.76:0/485307283 conn(0x558cdc893c50 msgr2=0x558cdc8c7480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:14.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.138+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/485307283 shutdown_connections 2024-03-18T14:54:14.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.138+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/485307283 wait complete. 2024-03-18T14:54:14.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.138+0000 7f7d3cb9f740 1 -- start start 2024-03-18T14:54:14.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.138+0000 7f7d3cb9f740 1 Processor -- start 2024-03-18T14:54:14.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.138+0000 7f7d3cb9f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8cca30 0x558cdc8d7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.138+0000 7f7d3cb9f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cd710 0x558cdc8d7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.138+0000 7f7d3cb9f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8da1b0 0x558cdc8d7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.138+0000 7f7d3cb9f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x558cdc8dcde0 con 0x558cdc8cca30 2024-03-18T14:54:14.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.138+0000 7f7d3cb9f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558cdc7ec0d0 con 0x558cdc8cd710 2024-03-18T14:54:14.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.138+0000 7f7d3cb9f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558cdc8dcf60 con 0x558cdc8da1b0 2024-03-18T14:54:14.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.138+0000 7f7d3cb9f740 10 client.0 _ll_register_callbacks cb 0x558cdc7b5760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:14.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.138+0000 7f7d337fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8cca30 0x558cdc8d7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.138+0000 7f7d337fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8cca30 0x558cdc8d7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45062/0 (socket says 172.21.15.76:45062) 2024-03-18T14:54:14.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.138+0000 7f7d337fe640 1 -- 172.21.15.76:0/2960860165 learned_addr learned my addr 172.21.15.76:0/2960860165 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:14.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.138+0000 7f7d32ffd640 1 --2- 172.21.15.76:0/2960860165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cd710 0x558cdc8d7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.138+0000 7f7d33fff640 1 --2- 172.21.15.76:0/2960860165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8da1b0 0x558cdc8d7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d337fe640 1 -- 172.21.15.76:0/2960860165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8da1b0 msgr2=0x558cdc8d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d337fe640 1 --2- 172.21.15.76:0/2960860165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8da1b0 0x558cdc8d7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d337fe640 1 -- 172.21.15.76:0/2960860165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cd710 msgr2=0x558cdc8d7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d337fe640 1 --2- 172.21.15.76:0/2960860165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cd710 0x558cdc8d7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d337fe640 1 -- 172.21.15.76:0/2960860165 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cdc66cf70 con 0x558cdc8cca30 2024-03-18T14:54:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d337fe640 1 --2- 172.21.15.76:0/2960860165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8cca30 0x558cdc8d7140 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7d2401dbd0 tx=0x7f7d24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=871a875a20f08424 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d12ffd640 1 -- 172.21.15.76:0/2960860165 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d24017040 con 0x558cdc8cca30 2024-03-18T14:54:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d12ffd640 1 -- 172.21.15.76:0/2960860165 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f7d24009040 con 0x558cdc8cca30 2024-03-18T14:54:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/2960860165 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d280c0070 con 0x558cdc8cca30 2024-03-18T14:54:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d12ffd640 1 -- 172.21.15.76:0/2960860165 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7d2402b020 con 0x558cdc8cca30 2024-03-18T14:54:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d3cb9f740 10 client.15792 fetch_fsmap learned FSMap version 25 2024-03-18T14:54:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/2960860165 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558cdc504170 con 0x558cdc8cca30 2024-03-18T14:54:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d12ffd640 1 -- 172.21.15.76:0/2960860165 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d280c0070 con 0x558cdc8cca30 2024-03-18T14:54:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d12ffd640 1 -- 172.21.15.76:0/2960860165 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7d24003980 con 0x558cdc8cca30 2024-03-18T14:54:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d3cb9f740 10 client.15792 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T14:54:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d3cb9f740 10 client.15792 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d3cb9f740 20 client.15792 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.139+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/2960860165 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558cdc8e9f90 0x558cdc8ec410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.140+0000 7f7d3cb9f740 4 client.15792 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:54:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.140+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/2960860165 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cdc83d610 con 0x558cdc8e9f90 2024-03-18T14:54:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.140+0000 7f7d32ffd640 1 --2- 172.21.15.76:0/2960860165 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558cdc8e9f90 0x558cdc8ec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.140+0000 7f7d12ffd640 10 client.15792 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.140+0000 7f7d32ffd640 1 --2- 172.21.15.76:0/2960860165 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558cdc8e9f90 0x558cdc8ec410 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c87dfa366b800bd server_cookie=453da75d90788726 in_seq=0 out_seq=0 2024-03-18T14:54:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.141+0000 7f7d12ffd640 1 -- 172.21.15.76:0/2960860165 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558cdc83d610 con 0x558cdc8e9f90 2024-03-18T14:54:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.141+0000 7f7d12ffd640 10 client.15792 handle_command_reply: tid=0 2024-03-18T14:54:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.141+0000 7f7d3cb9f740 1 client.15792 shutdown 2024-03-18T14:54:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.141+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/2960860165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8cca30 msgr2=0x558cdc8d7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.141+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/2960860165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8cca30 0x558cdc8d7140 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7d2401dbd0 tx=0x7f7d24004730 comp rx=0 tx=0).stop 2024-03-18T14:54:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.141+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/2960860165 shutdown_connections 2024-03-18T14:54:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.141+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/2960860165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8da1b0 0x558cdc8d7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.141+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/2960860165 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558cdc8e9f90 0x558cdc8ec410 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.141+0000 7f7d12ffd640 0 client.15792 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.141+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/2960860165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cd710 0x558cdc8d7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.141+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/2960860165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8cca30 0x558cdc8d7140 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.141+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/2960860165 >> 172.21.15.76:0/2960860165 conn(0x558cdc893c50 msgr2=0x558cdc8c91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.141+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/2960860165 shutdown_connections 2024-03-18T14:54:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.141+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/2960860165 wait complete. 2024-03-18T14:54:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.141+0000 7f7d3cb9f740 20 client.15792 trim_cache size 0 max 16384 2024-03-18T14:54:14.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.158+0000 7f7d3cb9f740 1 Processor -- start 2024-03-18T14:54:14.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.158+0000 7f7d3cb9f740 1 -- start start 2024-03-18T14:54:14.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.158+0000 7f7d3cb9f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cca30 0x558cdc8d9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.158+0000 7f7d3cb9f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8cd710 0x558cdc8d6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.158+0000 7f7d3cb9f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 0x558cdc8e5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.158+0000 7f7d3cb9f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7d341adb10 con 0x558cdc8da1b0 2024-03-18T14:54:14.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.158+0000 7f7d3cb9f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7d341ab380 con 0x558cdc8cca30 2024-03-18T14:54:14.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.158+0000 7f7d3cb9f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7d341a91f0 con 0x558cdc8cd710 2024-03-18T14:54:14.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.158+0000 7f7d33fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 0x558cdc8e5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.158+0000 7f7d337fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cca30 0x558cdc8d9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.159+0000 7f7d33fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 0x558cdc8e5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45066/0 (socket says 172.21.15.76:45066) 2024-03-18T14:54:14.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.159+0000 7f7d33fff640 1 -- 172.21.15.76:0/3468582111 learned_addr learned my addr 172.21.15.76:0/3468582111 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:14.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.159+0000 7f7d337fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cca30 0x558cdc8d9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:38936/0 (socket says 172.21.15.76:38936) 2024-03-18T14:54:14.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.159+0000 7f7d32ffd640 1 --2- 172.21.15.76:0/3468582111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8cd710 0x558cdc8d6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.159+0000 7f7d33fff640 1 -- 172.21.15.76:0/3468582111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8cd710 msgr2=0x558cdc8d6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:14.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.159+0000 7f7d33fff640 1 --2- 172.21.15.76:0/3468582111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8cd710 0x558cdc8d6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.159+0000 7f7d33fff640 1 -- 172.21.15.76:0/3468582111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cca30 msgr2=0x558cdc8d9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:14.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.159+0000 7f7d33fff640 1 --2- 172.21.15.76:0/3468582111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cca30 0x558cdc8d9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.159+0000 7f7d33fff640 1 -- 172.21.15.76:0/3468582111 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cdc8cba60 con 0x558cdc8da1b0 2024-03-18T14:54:14.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.159+0000 7f7d337fe640 1 --2- 172.21.15.76:0/3468582111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cca30 0x558cdc8d9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:14.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.159+0000 7f7d33fff640 1 --2- 172.21.15.76:0/3468582111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 0x558cdc8e5a60 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c004170 tx=0x7f7d2c009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf210a0667e53a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:14.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.159+0000 7f7d197fa640 1 -- 172.21.15.76:0/3468582111 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c01f030 con 0x558cdc8da1b0 2024-03-18T14:54:14.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.159+0000 7f7d197fa640 1 -- 172.21.15.76:0/3468582111 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d2c0047b0 con 0x558cdc8da1b0 2024-03-18T14:54:14.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.159+0000 7f7d197fa640 1 -- 172.21.15.76:0/3468582111 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c005520 con 0x558cdc8da1b0 2024-03-18T14:54:14.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.160+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/3468582111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 msgr2=0x558cdc8e5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.160+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/3468582111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 0x558cdc8e5a60 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c004170 tx=0x7f7d2c009500 comp rx=0 tx=0).stop 2024-03-18T14:54:14.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.160+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/3468582111 shutdown_connections 2024-03-18T14:54:14.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.160+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/3468582111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8cd710 0x558cdc8d6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.160+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/3468582111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 0x558cdc8e5a60 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.160+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/3468582111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cca30 0x558cdc8d9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.160+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/3468582111 >> 172.21.15.76:0/3468582111 conn(0x558cdc893c50 msgr2=0x558cdc8c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:14.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.160+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/3468582111 shutdown_connections 2024-03-18T14:54:14.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.160+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/3468582111 wait complete. 2024-03-18T14:54:14.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.160+0000 7f7d3cb9f740 1 -- start start 2024-03-18T14:54:14.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.160+0000 7f7d3cb9f740 1 Processor -- start 2024-03-18T14:54:14.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.160+0000 7f7d3cb9f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8cca30 0x558cdc8e22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.160+0000 7f7d3cb9f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cd710 0x558cdc8d33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.160+0000 7f7d3cb9f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 0x558cdc8d38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.160+0000 7f7d3cb9f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7d341a9840 con 0x558cdc8da1b0 2024-03-18T14:54:14.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.160+0000 7f7d3cb9f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7d341aacb0 con 0x558cdc8cd710 2024-03-18T14:54:14.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.160+0000 7f7d3cb9f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7d341a9f10 con 0x558cdc8cca30 2024-03-18T14:54:14.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d3cb9f740 10 client.0 _ll_register_callbacks cb 0x7f7d2401d5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:14.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d33fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 0x558cdc8d38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d33fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 0x558cdc8d38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45076/0 (socket says 172.21.15.76:45076) 2024-03-18T14:54:14.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d33fff640 1 -- 172.21.15.76:0/2738616199 learned_addr learned my addr 172.21.15.76:0/2738616199 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:14.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d337fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8cca30 0x558cdc8e22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d32ffd640 1 --2- 172.21.15.76:0/2738616199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cd710 0x558cdc8d33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d33fff640 1 -- 172.21.15.76:0/2738616199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8cca30 msgr2=0x558cdc8e22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:14.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d33fff640 1 --2- 172.21.15.76:0/2738616199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8cca30 0x558cdc8e22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d33fff640 1 -- 172.21.15.76:0/2738616199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cd710 msgr2=0x558cdc8d33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:14.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d33fff640 1 --2- 172.21.15.76:0/2738616199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cd710 0x558cdc8d33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d33fff640 1 -- 172.21.15.76:0/2738616199 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cdc8d40f0 con 0x558cdc8da1b0 2024-03-18T14:54:14.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d32ffd640 1 --2- 172.21.15.76:0/2738616199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cd710 0x558cdc8d33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:14.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d337fe640 1 --2- 172.21.15.76:0/2738616199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8cca30 0x558cdc8e22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:14.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d33fff640 1 --2- 172.21.15.76:0/2738616199 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 0x558cdc8d38f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c011fd0 tx=0x7f7d2c004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f242eb43c5bf984b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:14.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:14.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d12ffd640 1 -- 172.21.15.76:0/2738616199 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c005670 con 0x558cdc8da1b0 2024-03-18T14:54:14.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d12ffd640 1 -- 172.21.15.76:0/2738616199 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f7d2c005810 con 0x558cdc8da1b0 2024-03-18T14:54:14.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/2738616199 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d2c0047b0 con 0x558cdc8da1b0 2024-03-18T14:54:14.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d12ffd640 1 -- 172.21.15.76:0/2738616199 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7d2c033020 con 0x558cdc8da1b0 2024-03-18T14:54:14.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d3cb9f740 10 client.15798 fetch_fsmap learned FSMap version 25 2024-03-18T14:54:14.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/2738616199 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558cdc951320 con 0x558cdc8da1b0 2024-03-18T14:54:14.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.161+0000 7f7d12ffd640 1 -- 172.21.15.76:0/2738616199 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d2c0047b0 con 0x558cdc8da1b0 2024-03-18T14:54:14.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.162+0000 7f7d12ffd640 1 -- 172.21.15.76:0/2738616199 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7d2c038b70 con 0x558cdc8da1b0 2024-03-18T14:54:14.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.162+0000 7f7d3cb9f740 10 client.15798 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T14:54:14.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.162+0000 7f7d3cb9f740 10 client.15798 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:14.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.162+0000 7f7d3cb9f740 20 client.15798 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:14.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.162+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/2738616199 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558cdc951a10 0x558cdc953e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.162+0000 7f7d3cb9f740 4 client.15798 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T14:54:14.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.162+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/2738616199 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558cdc837960 con 0x558cdc951a10 2024-03-18T14:54:14.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.162+0000 7f7d337fe640 1 --2- 172.21.15.76:0/2738616199 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558cdc951a10 0x558cdc953e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.162+0000 7f7d337fe640 1 --2- 172.21.15.76:0/2738616199 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558cdc951a10 0x558cdc953e30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d529927fe41865b server_cookie=474d231e9546120d in_seq=0 out_seq=0 2024-03-18T14:54:14.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.162+0000 7f7d12ffd640 10 client.15798 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:14.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.163+0000 7f7d12ffd640 1 -- 172.21.15.76:0/2738616199 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558cdc837960 con 0x558cdc951a10 2024-03-18T14:54:14.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.163+0000 7f7d12ffd640 10 client.15798 handle_command_reply: tid=0 2024-03-18T14:54:14.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.163+0000 7f7d3cb9f740 1 client.15798 shutdown 2024-03-18T14:54:14.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.163+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/2738616199 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 msgr2=0x558cdc8d38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.163+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/2738616199 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 0x558cdc8d38f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c011fd0 tx=0x7f7d2c004c70 comp rx=0 tx=0).stop 2024-03-18T14:54:14.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.163+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/2738616199 shutdown_connections 2024-03-18T14:54:14.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.163+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/2738616199 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558cdc951a10 0x558cdc953e30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.163+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/2738616199 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558cdc8da1b0 0x558cdc8d38f0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.163+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/2738616199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558cdc8cd710 0x558cdc8d33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.163+0000 7f7d3cb9f740 1 --2- 172.21.15.76:0/2738616199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558cdc8cca30 0x558cdc8e22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.163+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/2738616199 >> 172.21.15.76:0/2738616199 conn(0x558cdc893c50 msgr2=0x558cdc8d7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:14.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.163+0000 7f7d12ffd640 0 client.15798 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:14.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.163+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/2738616199 shutdown_connections 2024-03-18T14:54:14.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.163+0000 7f7d3cb9f740 1 -- 172.21.15.76:0/2738616199 wait complete. 2024-03-18T14:54:14.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.163+0000 7f7d3cb9f740 20 client.15798 trim_cache size 0 max 16384 2024-03-18T14:54:14.170 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T14:54:14.171 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-18T14:54:14.171 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "ca273113-2b4e-48a4-ad4e-eb064c6dbd50", 2024-03-18T14:54:14.171 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-18T14:54:14.171 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T14:54:14.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.164+0000 7f7d39fb2640 1 -- 172.21.15.76:0/465020195 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7d1c0a2c30 msgr2=0x7f7d1c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.164+0000 7f7d39fb2640 1 --2- 172.21.15.76:0/465020195 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7d1c0a2c30 0x7f7d1c0a50f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009f50 tx=0x7f7d24011810 comp rx=0 tx=0).stop 2024-03-18T14:54:14.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.164+0000 7f7d39fb2640 1 -- 172.21.15.76:0/465020195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d34069380 msgr2=0x7f7d34198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.164+0000 7f7d39fb2640 1 --2- 172.21.15.76:0/465020195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d34069380 0x7f7d34198650 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009230 tx=0x7f7d2802cb30 comp rx=0 tx=0).stop 2024-03-18T14:54:14.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.164+0000 7f7d39fb2640 1 -- 172.21.15.76:0/465020195 shutdown_connections 2024-03-18T14:54:14.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.164+0000 7f7d39fb2640 1 --2- 172.21.15.76:0/465020195 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7d1c0a2c30 0x7f7d1c0a50f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.164+0000 7f7d39fb2640 1 --2- 172.21.15.76:0/465020195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d34107760 0x7f7d3419c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.164+0000 7f7d39fb2640 1 --2- 172.21.15.76:0/465020195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d34069380 0x7f7d34198650 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.164+0000 7f7d39fb2640 1 --2- 172.21.15.76:0/465020195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d340689c0 0x7f7d34198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.164+0000 7f7d39fb2640 1 -- 172.21.15.76:0/465020195 >> 172.21.15.76:0/465020195 conn(0x7f7d340777e0 msgr2=0x7f7d340fe2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:14.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.164+0000 7f7d39fb2640 1 -- 172.21.15.76:0/465020195 shutdown_connections 2024-03-18T14:54:14.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.164+0000 7f7d39fb2640 1 -- 172.21.15.76:0/465020195 wait complete. 2024-03-18T14:54:14.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T14:54:14.175 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T14:54:14.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.283+0000 7f78f3962640 1 -- 172.21.15.76:0/228742755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78ec101050 msgr2=0x7f78ec10d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.283+0000 7f78f3962640 1 --2- 172.21.15.76:0/228742755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78ec101050 0x7f78ec10d560 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009cd0 tx=0x7f78e402c0f0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.283+0000 7f78f3962640 1 -- 172.21.15.76:0/228742755 shutdown_connections 2024-03-18T14:54:14.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.283+0000 7f78f3962640 1 --2- 172.21.15.76:0/228742755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78ec101050 0x7f78ec10d560 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.283+0000 7f78f3962640 1 --2- 172.21.15.76:0/228742755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78ec100690 0x7f78ec100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.283+0000 7f78f3962640 1 --2- 172.21.15.76:0/228742755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78ec1066a0 0x7f78ec106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.283+0000 7f78f3962640 1 -- 172.21.15.76:0/228742755 >> 172.21.15.76:0/228742755 conn(0x7f78ec076a60 msgr2=0x7f78ec076e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:14.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.283+0000 7f78f3962640 1 -- 172.21.15.76:0/228742755 shutdown_connections 2024-03-18T14:54:14.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.283+0000 7f78f3962640 1 -- 172.21.15.76:0/228742755 wait complete. 2024-03-18T14:54:14.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f3962640 1 Processor -- start 2024-03-18T14:54:14.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f3962640 1 -- start start 2024-03-18T14:54:14.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f3962640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78ec100690 0x7f78ec1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f3962640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78ec101050 0x7f78ec1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f3962640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78ec1066a0 0x7f78ec19c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f3962640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f78ec110ab0 con 0x7f78ec101050 2024-03-18T14:54:14.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f3962640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f78ec110930 con 0x7f78ec100690 2024-03-18T14:54:14.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f3962640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f78ec110c30 con 0x7f78ec1066a0 2024-03-18T14:54:14.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f0efd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78ec101050 0x7f78ec1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f0efd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78ec101050 0x7f78ec1985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45102/0 (socket says 172.21.15.76:45102) 2024-03-18T14:54:14.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f0efd640 1 -- 172.21.15.76:0/576997424 learned_addr learned my addr 172.21.15.76:0/576997424 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:14.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f16fe640 1 --2- 172.21.15.76:0/576997424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78ec100690 0x7f78ec1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f1eff640 1 --2- 172.21.15.76:0/576997424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78ec1066a0 0x7f78ec19c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f0efd640 1 -- 172.21.15.76:0/576997424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78ec1066a0 msgr2=0x7f78ec19c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f0efd640 1 --2- 172.21.15.76:0/576997424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78ec1066a0 0x7f78ec19c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f0efd640 1 -- 172.21.15.76:0/576997424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78ec100690 msgr2=0x7f78ec1980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f0efd640 1 --2- 172.21.15.76:0/576997424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78ec100690 0x7f78ec1980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.284+0000 7f78f0efd640 1 -- 172.21.15.76:0/576997424 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ec19d0a0 con 0x7f78ec101050 2024-03-18T14:54:14.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.285+0000 7f78f0efd640 1 --2- 172.21.15.76:0/576997424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78ec101050 0x7f78ec1985f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f78dc00ea60 tx=0x7f78dc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:14.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.285+0000 7f78e27fc640 1 -- 172.21.15.76:0/576997424 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc00cde0 con 0x7f78ec101050 2024-03-18T14:54:14.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.285+0000 7f78e27fc640 1 -- 172.21.15.76:0/576997424 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78dc0035f0 con 0x7f78ec101050 2024-03-18T14:54:14.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.285+0000 7f78f3962640 1 -- 172.21.15.76:0/576997424 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78ec19d390 con 0x7f78ec101050 2024-03-18T14:54:14.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.285+0000 7f78e27fc640 1 -- 172.21.15.76:0/576997424 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc010670 con 0x7f78ec101050 2024-03-18T14:54:14.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.285+0000 7f78f3962640 1 -- 172.21.15.76:0/576997424 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78ec1a4c20 con 0x7f78ec101050 2024-03-18T14:54:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.287+0000 7f78e27fc640 1 -- 172.21.15.76:0/576997424 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f78dc0037a0 con 0x7f78ec101050 2024-03-18T14:54:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.287+0000 7f78f662f740 1 Processor -- start 2024-03-18T14:54:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.287+0000 7f78f662f740 1 -- start start 2024-03-18T14:54:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f662f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031560 0x55ff56031940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f662f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff56031e80 0x55ff5603a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f662f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff5603aa60 0x55ff5603ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f662f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ff55f91500 con 0x55ff5603aa60 2024-03-18T14:54:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f662f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ff55f39be0 con 0x55ff56031560 2024-03-18T14:54:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f662f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ff55f8d5a0 con 0x55ff56031e80 2024-03-18T14:54:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78e27fc640 1 --2- 172.21.15.76:0/576997424 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f78d80a2a90 0x7f78d80a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f1eff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031560 0x55ff56031940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f0efd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff5603aa60 0x55ff5603ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f1eff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031560 0x55ff56031940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:38966/0 (socket says 172.21.15.76:38966) 2024-03-18T14:54:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f16fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff56031e80 0x55ff5603a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f1eff640 1 -- 172.21.15.76:0/3635654923 learned_addr learned my addr 172.21.15.76:0/3635654923 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f1eff640 1 --2- 172.21.15.76:0/576997424 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f78d80a2a90 0x7f78d80a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f16fe640 1 -- 172.21.15.76:0/3635654923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031560 msgr2=0x55ff56031940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f16fe640 1 --2- 172.21.15.76:0/3635654923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031560 0x55ff56031940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f16fe640 1 -- 172.21.15.76:0/3635654923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff5603aa60 msgr2=0x55ff5603ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f16fe640 1 --2- 172.21.15.76:0/3635654923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff5603aa60 0x55ff5603ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78e27fc640 1 -- 172.21.15.76:0/576997424 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f78dc0cf3c0 con 0x7f78ec101050 2024-03-18T14:54:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f16fe640 1 -- 172.21.15.76:0/3635654923 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff55f4d4b0 con 0x55ff56031e80 2024-03-18T14:54:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f16fe640 1 --2- 172.21.15.76:0/3635654923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff56031e80 0x55ff5603a520 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009c00 tx=0x7f78e8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d3a87c826679b90 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78f1eff640 1 --2- 172.21.15.76:0/576997424 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f78d80a2a90 0x7f78d80a4f50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009430 tx=0x7f78e402c0f0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.288+0000 7f78d77fe640 1 -- 172.21.15.76:0/3635654923 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e8004050 con 0x55ff56031e80 2024-03-18T14:54:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.289+0000 7f78d77fe640 1 -- 172.21.15.76:0/3635654923 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78e8005500 con 0x55ff56031e80 2024-03-18T14:54:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.289+0000 7f78d77fe640 1 -- 172.21.15.76:0/3635654923 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e8004d60 con 0x55ff56031e80 2024-03-18T14:54:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.289+0000 7f78f662f740 1 -- 172.21.15.76:0/3635654923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff56031e80 msgr2=0x55ff5603a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.289+0000 7f78f662f740 1 --2- 172.21.15.76:0/3635654923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff56031e80 0x55ff5603a520 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009c00 tx=0x7f78e8011810 comp rx=0 tx=0).stop 2024-03-18T14:54:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.289+0000 7f78f662f740 1 -- 172.21.15.76:0/3635654923 shutdown_connections 2024-03-18T14:54:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.289+0000 7f78f662f740 1 --2- 172.21.15.76:0/3635654923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff56031e80 0x55ff5603a520 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.289+0000 7f78f662f740 1 --2- 172.21.15.76:0/3635654923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff5603aa60 0x55ff5603ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.289+0000 7f78f662f740 1 --2- 172.21.15.76:0/3635654923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031560 0x55ff56031940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.289+0000 7f78f662f740 1 -- 172.21.15.76:0/3635654923 >> 172.21.15.76:0/3635654923 conn(0x55ff5602bc90 msgr2=0x55ff5602c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.289+0000 7f78f662f740 1 -- 172.21.15.76:0/3635654923 shutdown_connections 2024-03-18T14:54:14.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.289+0000 7f78f662f740 1 -- 172.21.15.76:0/3635654923 wait complete. 2024-03-18T14:54:14.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.290+0000 7f78f662f740 1 -- start start 2024-03-18T14:54:14.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.290+0000 7f78f662f740 1 Processor -- start 2024-03-18T14:54:14.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.290+0000 7f78f662f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff56031560 0x55ff56037a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.290+0000 7f78f662f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031e80 0x55ff56037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.290+0000 7f78f662f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff5603aa60 0x55ff56038500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.290+0000 7f78f662f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ff5603d850 con 0x55ff5603aa60 2024-03-18T14:54:14.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.290+0000 7f78f662f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ff55f4d160 con 0x55ff56031e80 2024-03-18T14:54:14.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.290+0000 7f78f662f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ff5603d9d0 con 0x55ff56031560 2024-03-18T14:54:14.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.290+0000 7f78f662f740 10 client.0 _ll_register_callbacks cb 0x55ff55f31150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:14.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.290+0000 7f78f1eff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff5603aa60 0x55ff56038500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.290+0000 7f78f1eff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff5603aa60 0x55ff56038500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45124/0 (socket says 172.21.15.76:45124) 2024-03-18T14:54:14.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.290+0000 7f78f1eff640 1 -- 172.21.15.76:0/4090739909 learned_addr learned my addr 172.21.15.76:0/4090739909 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:14.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.290+0000 7f78f16fe640 1 --2- 172.21.15.76:0/4090739909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff56031560 0x55ff56037a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.290+0000 7f78f0efd640 1 --2- 172.21.15.76:0/4090739909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031e80 0x55ff56037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78f1eff640 1 -- 172.21.15.76:0/4090739909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff56031560 msgr2=0x55ff56037a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78f1eff640 1 --2- 172.21.15.76:0/4090739909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff56031560 0x55ff56037a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78f1eff640 1 -- 172.21.15.76:0/4090739909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031e80 msgr2=0x55ff56037fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78f1eff640 1 --2- 172.21.15.76:0/4090739909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031e80 0x55ff56037fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78f1eff640 1 -- 172.21.15.76:0/4090739909 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff55d6f7d0 con 0x55ff5603aa60 2024-03-18T14:54:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78f1eff640 1 --2- 172.21.15.76:0/4090739909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff5603aa60 0x55ff56038500 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f78e4037040 tx=0x7f78e40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58eb50521e229cc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78d4ff9640 1 -- 172.21.15.76:0/4090739909 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e4053060 con 0x55ff5603aa60 2024-03-18T14:54:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78f662f740 1 -- 172.21.15.76:0/4090739909 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78e8005500 con 0x55ff5603aa60 2024-03-18T14:54:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78d4ff9640 1 -- 172.21.15.76:0/4090739909 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f78e4004040 con 0x55ff5603aa60 2024-03-18T14:54:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78d4ff9640 1 -- 172.21.15.76:0/4090739909 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f78e4047070 con 0x55ff5603aa60 2024-03-18T14:54:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78f662f740 10 client.15810 fetch_fsmap learned FSMap version 25 2024-03-18T14:54:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78f662f740 1 -- 172.21.15.76:0/4090739909 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ff55ca53f0 con 0x55ff5603aa60 2024-03-18T14:54:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78d4ff9640 1 -- 172.21.15.76:0/4090739909 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78e8005500 con 0x55ff5603aa60 2024-03-18T14:54:14.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78d4ff9640 1 -- 172.21.15.76:0/4090739909 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f78e403f070 con 0x55ff5603aa60 2024-03-18T14:54:14.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78f662f740 10 client.15810 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T14:54:14.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78f662f740 10 client.15810 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:14.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78f662f740 20 client.15810 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:14.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78f662f740 1 --2- 172.21.15.76:0/4090739909 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ff5604aa30 0x55ff5604ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78f662f740 4 client.15810 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:54:14.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.291+0000 7f78f662f740 1 -- 172.21.15.76:0/4090739909 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff55f9ea40 con 0x55ff5604aa30 2024-03-18T14:54:14.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.292+0000 7f78f16fe640 1 --2- 172.21.15.76:0/4090739909 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ff5604aa30 0x55ff5604ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.292+0000 7f78f16fe640 1 --2- 172.21.15.76:0/4090739909 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ff5604aa30 0x55ff5604ceb0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=737cc182391775ee server_cookie=6184775af96105e in_seq=0 out_seq=0 2024-03-18T14:54:14.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.292+0000 7f78d4ff9640 10 client.15810 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:14.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.293+0000 7f78d4ff9640 1 -- 172.21.15.76:0/4090739909 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ff55f9ea40 con 0x55ff5604aa30 2024-03-18T14:54:14.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.293+0000 7f78d4ff9640 10 client.15810 handle_command_reply: tid=0 2024-03-18T14:54:14.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.293+0000 7f78f662f740 1 client.15810 shutdown 2024-03-18T14:54:14.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.293+0000 7f78f662f740 1 -- 172.21.15.76:0/4090739909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff5603aa60 msgr2=0x55ff56038500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.293+0000 7f78f662f740 1 --2- 172.21.15.76:0/4090739909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff5603aa60 0x55ff56038500 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f78e4037040 tx=0x7f78e40095c0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.293+0000 7f78f662f740 1 -- 172.21.15.76:0/4090739909 shutdown_connections 2024-03-18T14:54:14.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.293+0000 7f78f662f740 1 --2- 172.21.15.76:0/4090739909 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ff5604aa30 0x55ff5604ceb0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.293+0000 7f78f662f740 1 --2- 172.21.15.76:0/4090739909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff5603aa60 0x55ff56038500 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.293+0000 7f78f662f740 1 --2- 172.21.15.76:0/4090739909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031e80 0x55ff56037fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.293+0000 7f78d4ff9640 0 client.15810 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:14.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.293+0000 7f78f662f740 1 --2- 172.21.15.76:0/4090739909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff56031560 0x55ff56037a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.293+0000 7f78f662f740 1 -- 172.21.15.76:0/4090739909 >> 172.21.15.76:0/4090739909 conn(0x55ff5602bc90 msgr2=0x55ff5602d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:14.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.293+0000 7f78f662f740 1 -- 172.21.15.76:0/4090739909 shutdown_connections 2024-03-18T14:54:14.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.293+0000 7f78f662f740 1 -- 172.21.15.76:0/4090739909 wait complete. 2024-03-18T14:54:14.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.293+0000 7f78f662f740 20 client.15810 trim_cache size 0 max 16384 2024-03-18T14:54:14.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f662f740 1 Processor -- start 2024-03-18T14:54:14.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f662f740 1 -- start start 2024-03-18T14:54:14.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f662f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff56031560 0x55ff5603a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f662f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031e80 0x55ff560465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f662f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff5603aa60 0x55ff56052780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f662f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f78ec1adb40 con 0x55ff56031560 2024-03-18T14:54:14.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f662f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f78ec1ab3b0 con 0x55ff56031e80 2024-03-18T14:54:14.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f662f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f78ec1a9220 con 0x55ff5603aa60 2024-03-18T14:54:14.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f16fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff56031560 0x55ff5603a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f16fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff56031560 0x55ff5603a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45140/0 (socket says 172.21.15.76:45140) 2024-03-18T14:54:14.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f16fe640 1 -- 172.21.15.76:0/1383768377 learned_addr learned my addr 172.21.15.76:0/1383768377 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:14.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f1eff640 1 --2- 172.21.15.76:0/1383768377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff5603aa60 0x55ff56052780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f0efd640 1 --2- 172.21.15.76:0/1383768377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031e80 0x55ff560465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f16fe640 1 -- 172.21.15.76:0/1383768377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff5603aa60 msgr2=0x55ff56052780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:14.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f16fe640 1 --2- 172.21.15.76:0/1383768377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff5603aa60 0x55ff56052780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f16fe640 1 -- 172.21.15.76:0/1383768377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031e80 msgr2=0x55ff560465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:14.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f16fe640 1 --2- 172.21.15.76:0/1383768377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031e80 0x55ff560465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f16fe640 1 -- 172.21.15.76:0/1383768377 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff55c650f0 con 0x55ff56031560 2024-03-18T14:54:14.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f0efd640 1 --2- 172.21.15.76:0/1383768377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031e80 0x55ff560465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:54:14.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78f16fe640 1 --2- 172.21.15.76:0/1383768377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff56031560 0x55ff5603a1b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f78e8004910 tx=0x7f78e8005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a9eb740c9e2838 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:14.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.311+0000 7f78d6ffd640 1 -- 172.21.15.76:0/1383768377 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e8031070 con 0x55ff56031560 2024-03-18T14:54:14.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.312+0000 7f78d6ffd640 1 -- 172.21.15.76:0/1383768377 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78e802c370 con 0x55ff56031560 2024-03-18T14:54:14.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.312+0000 7f78d6ffd640 1 -- 172.21.15.76:0/1383768377 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e8019760 con 0x55ff56031560 2024-03-18T14:54:14.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.312+0000 7f78f662f740 1 -- 172.21.15.76:0/1383768377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff56031560 msgr2=0x55ff5603a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.312+0000 7f78f662f740 1 --2- 172.21.15.76:0/1383768377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff56031560 0x55ff5603a1b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f78e8004910 tx=0x7f78e8005d30 comp rx=0 tx=0).stop 2024-03-18T14:54:14.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.312+0000 7f78f662f740 1 -- 172.21.15.76:0/1383768377 shutdown_connections 2024-03-18T14:54:14.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.312+0000 7f78f662f740 1 --2- 172.21.15.76:0/1383768377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff5603aa60 0x55ff56052780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.312+0000 7f78f662f740 1 --2- 172.21.15.76:0/1383768377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031e80 0x55ff560465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.312+0000 7f78f662f740 1 --2- 172.21.15.76:0/1383768377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff56031560 0x55ff5603a1b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.312+0000 7f78f662f740 1 -- 172.21.15.76:0/1383768377 >> 172.21.15.76:0/1383768377 conn(0x55ff5602bc90 msgr2=0x55ff56026900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:14.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.312+0000 7f78f662f740 1 -- 172.21.15.76:0/1383768377 shutdown_connections 2024-03-18T14:54:14.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.312+0000 7f78f662f740 1 -- 172.21.15.76:0/1383768377 wait complete. 2024-03-18T14:54:14.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.312+0000 7f78f662f740 1 -- start start 2024-03-18T14:54:14.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f662f740 1 Processor -- start 2024-03-18T14:54:14.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f662f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff56031560 0x55ff5603e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f662f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031e80 0x55ff5603edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f662f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff5603aa60 0x55ff560afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f662f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f78ec1a9870 con 0x55ff56031560 2024-03-18T14:54:14.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f662f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f78ec1aace0 con 0x55ff56031e80 2024-03-18T14:54:14.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f662f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f78ec1a9f40 con 0x55ff5603aa60 2024-03-18T14:54:14.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f662f740 10 client.0 _ll_register_callbacks cb 0x7f78e8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:14.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f16fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff56031560 0x55ff5603e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f16fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff56031560 0x55ff5603e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45154/0 (socket says 172.21.15.76:45154) 2024-03-18T14:54:14.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f16fe640 1 -- 172.21.15.76:0/1172387158 learned_addr learned my addr 172.21.15.76:0/1172387158 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:14.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f0efd640 1 --2- 172.21.15.76:0/1172387158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031e80 0x55ff5603edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f1eff640 1 --2- 172.21.15.76:0/1172387158 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff5603aa60 0x55ff560afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f16fe640 1 -- 172.21.15.76:0/1172387158 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff5603aa60 msgr2=0x55ff560afdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:14.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f16fe640 1 --2- 172.21.15.76:0/1172387158 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff5603aa60 0x55ff560afdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f16fe640 1 -- 172.21.15.76:0/1172387158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031e80 msgr2=0x55ff5603edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:14.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f16fe640 1 --2- 172.21.15.76:0/1172387158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031e80 0x55ff5603edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f16fe640 1 -- 172.21.15.76:0/1172387158 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff55f589d0 con 0x55ff56031560 2024-03-18T14:54:14.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f1eff640 1 --2- 172.21.15.76:0/1172387158 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff5603aa60 0x55ff560afdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:54:14.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78f16fe640 1 --2- 172.21.15.76:0/1172387158 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff56031560 0x55ff5603e870 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f78e8011d10 tx=0x7f78e8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d82fd835f97b8fda server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:14.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.313+0000 7f78d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:14.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.314+0000 7f78d4ff9640 1 -- 172.21.15.76:0/1172387158 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e8031070 con 0x55ff56031560 2024-03-18T14:54:14.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.314+0000 7f78d4ff9640 1 -- 172.21.15.76:0/1172387158 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f78e8004240 con 0x55ff56031560 2024-03-18T14:54:14.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.314+0000 7f78f662f740 1 -- 172.21.15.76:0/1172387158 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78e802c370 con 0x55ff56031560 2024-03-18T14:54:14.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.314+0000 7f78d4ff9640 1 -- 172.21.15.76:0/1172387158 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f78e8004440 con 0x55ff56031560 2024-03-18T14:54:14.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.314+0000 7f78f662f740 10 client.15816 fetch_fsmap learned FSMap version 25 2024-03-18T14:54:14.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.314+0000 7f78f662f740 1 -- 172.21.15.76:0/1172387158 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ff560b1230 con 0x55ff56031560 2024-03-18T14:54:14.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.314+0000 7f78d4ff9640 1 -- 172.21.15.76:0/1172387158 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78e802c370 con 0x55ff56031560 2024-03-18T14:54:14.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.314+0000 7f78d4ff9640 1 -- 172.21.15.76:0/1172387158 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f78e802c6e0 con 0x55ff56031560 2024-03-18T14:54:14.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.314+0000 7f78f662f740 10 client.15816 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T14:54:14.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.314+0000 7f78f662f740 10 client.15816 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:14.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.314+0000 7f78f662f740 20 client.15816 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:14.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.314+0000 7f78f662f740 1 --2- 172.21.15.76:0/1172387158 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ff560b17b0 0x55ff560b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:14.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.314+0000 7f78f662f740 4 client.15816 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T14:54:14.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.314+0000 7f78f662f740 1 -- 172.21.15.76:0/1172387158 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ff55f98d90 con 0x55ff560b17b0 2024-03-18T14:54:14.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.314+0000 7f78f0efd640 1 --2- 172.21.15.76:0/1172387158 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ff560b17b0 0x55ff560b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:14.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.315+0000 7f78d4ff9640 10 client.15816 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:14.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.315+0000 7f78f0efd640 1 --2- 172.21.15.76:0/1172387158 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ff560b17b0 0x55ff560b3bd0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff674da5b09c3c1a server_cookie=948ccf8afeea8319 in_seq=0 out_seq=0 2024-03-18T14:54:14.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.315+0000 7f78d4ff9640 1 -- 172.21.15.76:0/1172387158 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ff55f98d90 con 0x55ff560b17b0 2024-03-18T14:54:14.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.315+0000 7f78d4ff9640 10 client.15816 handle_command_reply: tid=0 2024-03-18T14:54:14.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.315+0000 7f78f662f740 1 client.15816 shutdown 2024-03-18T14:54:14.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.315+0000 7f78f662f740 1 -- 172.21.15.76:0/1172387158 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff56031560 msgr2=0x55ff5603e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.315+0000 7f78f662f740 1 --2- 172.21.15.76:0/1172387158 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff56031560 0x55ff5603e870 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f78e8011d10 tx=0x7f78e8009780 comp rx=0 tx=0).stop 2024-03-18T14:54:14.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.315+0000 7f78f662f740 1 -- 172.21.15.76:0/1172387158 shutdown_connections 2024-03-18T14:54:14.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.315+0000 7f78f662f740 1 --2- 172.21.15.76:0/1172387158 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff5603aa60 0x55ff560afdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.315+0000 7f78f662f740 1 --2- 172.21.15.76:0/1172387158 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ff560b17b0 0x55ff560b3bd0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.315+0000 7f78f662f740 1 --2- 172.21.15.76:0/1172387158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff56031e80 0x55ff5603edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.315+0000 7f78f662f740 1 --2- 172.21.15.76:0/1172387158 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ff56031560 0x55ff5603e870 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.315+0000 7f78d4ff9640 0 client.15816 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:14.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.315+0000 7f78f662f740 1 -- 172.21.15.76:0/1172387158 >> 172.21.15.76:0/1172387158 conn(0x55ff5602bc90 msgr2=0x55ff56037a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:14.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.315+0000 7f78f662f740 1 -- 172.21.15.76:0/1172387158 shutdown_connections 2024-03-18T14:54:14.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.315+0000 7f78f662f740 1 -- 172.21.15.76:0/1172387158 wait complete. 2024-03-18T14:54:14.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.315+0000 7f78f662f740 20 client.15816 trim_cache size 0 max 16384 2024-03-18T14:54:14.324 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T14:54:14.324 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T14:54:14.324 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-18T14:54:14.325 INFO:teuthology.orchestra.run.smithi076.stdout: "ca273113-2b4e-48a4-ad4e-eb064c6dbd50": { 2024-03-18T14:54:14.325 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-18T14:54:14.325 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "ca273113-2b4e-48a4-ad4e-eb064c6dbd50", 2024-03-18T14:54:14.325 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-18T14:54:14.325 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T14:54:14.325 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T14:54:14.325 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T14:54:14.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.316+0000 7f78f3962640 1 -- 172.21.15.76:0/576997424 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f78d80a2a90 msgr2=0x7f78d80a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.316+0000 7f78f3962640 1 --2- 172.21.15.76:0/576997424 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f78d80a2a90 0x7f78d80a4f50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009430 tx=0x7f78e402c0f0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.316+0000 7f78f3962640 1 -- 172.21.15.76:0/576997424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78ec101050 msgr2=0x7f78ec1985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:14.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.316+0000 7f78f3962640 1 --2- 172.21.15.76:0/576997424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78ec101050 0x7f78ec1985f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f78dc00ea60 tx=0x7f78dc00ef20 comp rx=0 tx=0).stop 2024-03-18T14:54:14.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.316+0000 7f78f3962640 1 -- 172.21.15.76:0/576997424 shutdown_connections 2024-03-18T14:54:14.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.316+0000 7f78f3962640 1 --2- 172.21.15.76:0/576997424 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f78d80a2a90 0x7f78d80a4f50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.316+0000 7f78f3962640 1 --2- 172.21.15.76:0/576997424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78ec1066a0 0x7f78ec19c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.316+0000 7f78f3962640 1 --2- 172.21.15.76:0/576997424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78ec101050 0x7f78ec1985f0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.316+0000 7f78f3962640 1 --2- 172.21.15.76:0/576997424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78ec100690 0x7f78ec1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:14.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.316+0000 7f78f3962640 1 -- 172.21.15.76:0/576997424 >> 172.21.15.76:0/576997424 conn(0x7f78ec076a60 msgr2=0x7f78ec06a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:14.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.316+0000 7f78f3962640 1 -- 172.21.15.76:0/576997424 shutdown_connections 2024-03-18T14:54:14.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:14.316+0000 7f78f3962640 1 -- 172.21.15.76:0/576997424 wait complete. 2024-03-18T14:54:14.327 INFO:tasks.cephfs.filesystem:scrub status for tag:ca273113-2b4e-48a4-ad4e-eb064c6dbd50 - {'path': '/', 'tag': 'ca273113-2b4e-48a4-ad4e-eb064c6dbd50', 'options': 'recursive,force'} 2024-03-18T14:54:14.467 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.169+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021f [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT auth v20 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@17},l=25120 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc4f3080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000021f:[<0x10000000215/ROSE.CPT v20>,<0x10000000009/COREL v7368>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x1000000021f:[<0x10000000215/ROSE.CPT v20>,<0x10000000009/COREL v7916>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.467 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.169+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021b [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT auth v12 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@17},l=25120 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc4f2b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000021b:[<0x10000000215/TEAPOT2.CPT v12>,<0x10000000009/COREL v7368>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x1000000021b:[<0x10000000215/TEAPOT2.CPT v12>,<0x10000000009/COREL v7916>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.467 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.171+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022b [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.PX auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03a37180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000022b:[<0x10000000229/COURSES.PX v4>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x1000000022b:[<0x10000000229/COURSES.PX v4>,<0x10000000009/PARADOX v7684>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.468 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.172+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022c [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.X04 auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03ab5600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000022c:[<0x10000000229/COURSES.X04 v6>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x1000000022c:[<0x10000000229/COURSES.X04 v6>,<0x10000000009/PARADOX v7684>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.468 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.170+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000220 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT auth v22 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@17},l=25120 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc4f3600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000220:[<0x10000000215/CORELDRW.CDT v22>,<0x10000000009/COREL v7368>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000220:[<0x10000000215/CORELDRW.CDT v22>,<0x10000000009/COREL v7916>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.468 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.170+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021a [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT auth v10 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@17},l=25120 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc4f2000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000021a:[<0x10000000215/MAILBOX2.CPT v10>,<0x10000000009/COREL v7368>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x1000000021a:[<0x10000000215/MAILBOX2.CPT v10>,<0x10000000009/COREL v7916>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.468 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.172+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000231 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG0 auth v16 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03ab5b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000231:[<0x10000000229/STUDENTS.YG0 v16>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000231:[<0x10000000229/STUDENTS.YG0 v16>,<0x10000000009/PARADOX v7684>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.469 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.172+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000237 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.PX auth v28 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03ab6100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000237:[<0x10000000229/ENROLL.PX v28>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000237:[<0x10000000229/ENROLL.PX v28>,<0x10000000009/PARADOX v7684>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.469 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.173+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022f [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.PX auth v12 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03ab6c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000022f:[<0x10000000229/STUDENTS.PX v12>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x1000000022f:[<0x10000000229/STUDENTS.PX v12>,<0x10000000009/PARADOX v7684>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.469 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.173+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000023b [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.TV auth v36 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03ae4000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000023b:[<0x10000000229/STUDENTS.TV v36>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x1000000023b:[<0x10000000229/STUDENTS.TV v36>,<0x10000000009/PARADOX v7684>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.469 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.173+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000023c [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.FSL auth v38 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03ae4580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000023c:[<0x10000000229/STUDENTS.FSL v38>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x1000000023c:[<0x10000000229/STUDENTS.FSL v38>,<0x10000000009/PARADOX v7684>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.469 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.170+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021e [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR auth v18 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@17},l=25120 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc4f3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000021e:[<0x10000000215/GRAPHIC2.CDR v18>,<0x10000000009/COREL v7368>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x1000000021e:[<0x10000000215/GRAPHIC2.CDR v18>,<0x10000000009/COREL v7916>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.469 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.171+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000218 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@17},l=25120 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc4f4100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000218:[<0x10000000215/ARTISTIC.ACL v6>,<0x10000000009/COREL v7368>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000218:[<0x10000000215/ARTISTIC.ACL v6>,<0x10000000009/COREL v7916>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.469 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.172+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021c [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT auth v14 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@17},l=25120 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc4f4680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000021c:[<0x10000000215/SIGN2.CPT v14>,<0x10000000009/COREL v7368>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x1000000021c:[<0x10000000215/SIGN2.CPT v14>,<0x10000000009/COREL v7916>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.469 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.173+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000233 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG1 auth v20 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03ae4b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000233:[<0x10000000229/STUDENTS.YG1 v20>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000233:[<0x10000000229/STUDENTS.YG1 v20>,<0x10000000009/PARADOX v7684>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.470 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.173+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000239 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.Y03 auth v32 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03ae5080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000239:[<0x10000000229/ENROLL.Y03 v32>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000239:[<0x10000000229/ENROLL.Y03 v32>,<0x10000000009/PARADOX v7684>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.470 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.172+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000217 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@17},l=25120 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc4f4c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000217:[<0x10000000215/CDRBARS.CFG v4>,<0x10000000009/COREL v7368>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000217:[<0x10000000215/CDRBARS.CFG v4>,<0x10000000009/COREL v7916>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.470 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.173+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000219 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR auth v8 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@17},l=25120 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc4f5700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000219:[<0x10000000215/GRAPHIC1.CDR v8>,<0x10000000009/COREL v7368>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000219:[<0x10000000215/GRAPHIC1.CDR v8>,<0x10000000009/COREL v7916>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.470 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.174+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000023d [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.FSL auth v40 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03ae5600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000023d:[<0x10000000229/COURSES.FSL v40>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x1000000023d:[<0x10000000229/COURSES.FSL v40>,<0x10000000009/PARADOX v7694>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.470 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.174+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000234 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.VAL auth v22 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03ae5b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000234:[<0x10000000229/STUDENTS.VAL v22>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000234:[<0x10000000229/STUDENTS.VAL v22>,<0x10000000009/PARADOX v7694>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.470 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.174+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022a [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.DB auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03ae6100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000022a:[<0x10000000229/COURSES.DB v2>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x1000000022a:[<0x10000000229/COURSES.DB v2>,<0x10000000009/PARADOX v7694>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.471 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.173+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000216 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@17},l=25120 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc275180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000216:[<0x10000000215/CDRROLS.CFG v2>,<0x10000000009/COREL v7368>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000216:[<0x10000000215/CDRROLS.CFG v2>,<0x10000000009/COREL v7916>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.471 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.173+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000226 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc274680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000226:[<0x10000000225/T1.XLS v2>,<0x10000000009/PM v7386>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000226:[<0x10000000225/T1.XLS v2>,<0x10000000009/PM v7386>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.471 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.174+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021d [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR auth v16 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@17},l=25120 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc4f5180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000021d:[<0x10000000215/TRAIN.CDR v16>,<0x10000000009/COREL v7368>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x1000000021d:[<0x10000000215/TRAIN.CDR v16>,<0x10000000009/COREL v7916>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.471 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.175+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022d [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.Y04 auth v8 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03ae6680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000022d:[<0x10000000229/COURSES.Y04 v8>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x1000000022d:[<0x10000000229/COURSES.Y04 v8>,<0x10000000009/PARADOX v7694>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.471 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.175+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000235 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.MB auth v24 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03ae7180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000235:[<0x10000000229/STUDENTS.MB v24>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000235:[<0x10000000229/STUDENTS.MB v24>,<0x10000000009/PARADOX v7694>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.472 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.174+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000227 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc4de000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000227:[<0x10000000225/EVENTS.DOC v4>,<0x10000000009/PM v7386>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000227:[<0x10000000225/EVENTS.DOC v4>,<0x10000000009/PM v7386>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.472 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.174+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000228 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc4deb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000228:[<0x10000000225/MOVED.DOC v6>,<0x10000000009/PM v7386>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000228:[<0x10000000225/MOVED.DOC v6>,<0x10000000009/PM v7386>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.472 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.175+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000238 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.X03 auth v30 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03ae6c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000238:[<0x10000000229/ENROLL.X03 v30>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000238:[<0x10000000229/ENROLL.X03 v30>,<0x10000000009/PARADOX v7694>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.472 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.175+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000023a [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.VAL auth v34 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03ae7700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000023a:[<0x10000000229/ENROLL.VAL v34>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x1000000023a:[<0x10000000229/ENROLL.VAL v34>,<0x10000000009/PARADOX v7694>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.472 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.175+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000236 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.DB auth v26 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03af2000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000236:[<0x10000000229/ENROLL.DB v26>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000236:[<0x10000000229/ENROLL.DB v26>,<0x10000000009/PARADOX v7694>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.473 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.175+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000210 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PCBENCHM.PPT auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc4dfb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000210:[<0x1000000020f/PCBENCHM.PPT v2>,<0x10000000009/PWRPNT v7418>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000210:[<0x1000000020f/PCBENCHM.PPT v2>,<0x10000000009/PWRPNT v7530>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.473 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.175+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000211 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ZD16.BMP auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc4e0100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000211:[<0x1000000020f/ZD16.BMP v4>,<0x10000000009/PWRPNT v7418>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000211:[<0x1000000020f/ZD16.BMP v4>,<0x10000000009/PWRPNT v7530>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.473 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.180+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000214 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TRIDOTS.POT auth v10 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc4e0c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000214:[<0x1000000020f/TRIDOTS.POT v10>,<0x10000000009/PWRPNT v7418>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000214:[<0x1000000020f/TRIDOTS.POT v10>,<0x10000000009/PWRPNT v7530>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.473 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.180+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000213 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TIPS.PPT auth v8 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc4e1180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000213:[<0x1000000020f/TIPS.PPT v8>,<0x10000000009/PWRPNT v7418>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000213:[<0x1000000020f/TIPS.PPT v8>,<0x10000000009/PWRPNT v7530>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.473 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T14:54:14.180+0000 7f2903089700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000212 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PPTOOLS1.PPA auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5587dc4e1700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000212:[<0x1000000020f/PPTOOLS1.PPA v6>,<0x10000000009/PWRPNT v7418>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000212:[<0x1000000020f/PPTOOLS1.PPA v6>,<0x10000000009/PWRPNT v7530>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v27>,<0x1/volumes v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.473 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.176+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000232 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG1 auth v18 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03af2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000232:[<0x10000000229/STUDENTS.XG1 v18>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000232:[<0x10000000229/STUDENTS.XG1 v18>,<0x10000000009/PARADOX v7694>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.473 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.176+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022e [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.DB auth v10 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03af3600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000022e:[<0x10000000229/STUDENTS.DB v10>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x1000000022e:[<0x10000000229/STUDENTS.DB v10>,<0x10000000009/PARADOX v7694>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.473 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T14:54:14.176+0000 7f67b50ed700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000230 [2,head] /volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG0 auth v14 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25120=0-4194304@1} caps={25120=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25120 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x562d03af3080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000230:[<0x10000000229/STUDENTS.XG0 v14>,<0x10000000009/PARADOX v7402>,<0x10000000008/~dmtmp v57>,<0x10000000007/client0 v41>,<0x10000000006/clients v31>,<0x10000000005/tmp v31>,<0x100000001fb/client.0 v32>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v46>,<0x10000000001/sv_1 v67>,<0x10000000000/qa v55>,<0x1/volumes v49>]//[]","memoryvalue":"(3)0x10000000230:[<0x10000000229/STUDENTS.XG0 v14>,<0x10000000009/PARADOX v7694>,<0x10000000008/~dmtmp v47>,<0x10000000007/client0 v44>,<0x10000000006/clients v34>,<0x10000000005/tmp v34>,<0x100000001fb/client.0 v34>,<0x100000001fa/76c47e49-4328-4dc6-9aec-d423fc6bcd3a v49>,<0x10000000001/sv_1 v69>,<0x10000000000/qa v37>,<0x1/volumes v31>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-18T14:54:14.731 INFO:tasks.workunit.client.0.smithi076.stdout: 1 39292 15.40 MB/sec warmup 15 sec latency 622.589 ms 2024-03-18T14:54:14.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:14 smithi076 ceph-mon[32205]: pgmap v32: 97 pgs: 97 active+clean; 13 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 381 op/s 2024-03-18T14:54:14.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:14 smithi174 ceph-mon[36787]: pgmap v32: 97 pgs: 97 active+clean; 13 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 381 op/s 2024-03-18T14:54:14.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:14 smithi077 ceph-mon[35865]: pgmap v32: 97 pgs: 97 active+clean; 13 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 381 op/s 2024-03-18T14:54:15.731 INFO:tasks.workunit.client.0.smithi076.stdout: 1 39292 14.44 MB/sec warmup 16 sec latency 1489.699 ms 2024-03-18T14:54:15.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: scrub queued for path: / 2024-03-18T14:54:15.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: scrub summary: idle+waiting paths [/] 2024-03-18T14:54:15.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: scrub summary: active paths [/] 2024-03-18T14:54:15.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: scrub summary: active paths [/] 2024-03-18T14:54:15.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x1000000021f (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x1000000021b (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000220 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x1000000021a (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: scrub summary: active paths [/] 2024-03-18T14:54:15.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x1000000021e (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x1000000022b (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.PX) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000218 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x1000000022c (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.X04) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000231 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG0) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x1000000021c (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000237 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.PX) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000217 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x1000000022f (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.PX) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x1000000023b (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.TV) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000219 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x1000000023c (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.FSL) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000216 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000233 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG1) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000239 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.Y03) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000226 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x1000000023d (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.FSL) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x1000000021d (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000234 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.VAL) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.913 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000227 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000228 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x1000000022a (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.DB) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x1000000022d (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.Y04) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000235 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.MB) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000210 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PCBENCHM.PPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000238 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.X03) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000211 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ZD16.BMP) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x1000000023a (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.VAL) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000236 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.DB) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000232 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG1) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x1000000022e (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.DB) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000230 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG0) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: scrub summary: idle+waiting paths [/] 2024-03-18T14:54:15.914 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000214 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TRIDOTS.POT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.915 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000213 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TIPS.PPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.915 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: Scrub error on inode 0x10000000212 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PPTOOLS1.PPA) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.915 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: scrub summary: idle+waiting paths [/] 2024-03-18T14:54:15.915 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:15 smithi076 ceph-mon[32205]: scrub summary: idle+waiting paths [/] 2024-03-18T14:54:15.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: scrub queued for path: / 2024-03-18T14:54:15.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: scrub summary: idle+waiting paths [/] 2024-03-18T14:54:15.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: scrub summary: active paths [/] 2024-03-18T14:54:15.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: scrub summary: active paths [/] 2024-03-18T14:54:15.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x1000000021f (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x1000000021b (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000220 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x1000000021a (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: scrub summary: active paths [/] 2024-03-18T14:54:15.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x1000000021e (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x1000000022b (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.PX) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000218 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x1000000022c (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.X04) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000231 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG0) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x1000000021c (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000237 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.PX) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000217 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x1000000022f (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.PX) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x1000000023b (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.TV) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000219 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x1000000023c (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.FSL) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000216 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000233 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG1) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000239 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.Y03) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000226 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x1000000023d (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.FSL) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x1000000021d (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000234 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.VAL) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000227 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000228 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x1000000022a (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.DB) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x1000000022d (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.Y04) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000235 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.MB) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000210 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PCBENCHM.PPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000238 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.X03) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000211 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ZD16.BMP) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x1000000023a (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.VAL) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000236 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.DB) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000232 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG1) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x1000000022e (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.DB) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000230 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG0) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: scrub summary: idle+waiting paths [/] 2024-03-18T14:54:15.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000214 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TRIDOTS.POT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000213 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TIPS.PPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: Scrub error on inode 0x10000000212 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PPTOOLS1.PPA) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: scrub summary: idle+waiting paths [/] 2024-03-18T14:54:15.971 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:15 smithi174 ceph-mon[36787]: scrub summary: idle+waiting paths [/] 2024-03-18T14:54:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: scrub queued for path: / 2024-03-18T14:54:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: scrub summary: idle+waiting paths [/] 2024-03-18T14:54:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: scrub summary: active paths [/] 2024-03-18T14:54:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: scrub summary: active paths [/] 2024-03-18T14:54:15.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x1000000021f (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x1000000021b (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000220 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x1000000021a (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: scrub summary: active paths [/] 2024-03-18T14:54:15.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x1000000021e (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x1000000022b (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.PX) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000218 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x1000000022c (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.X04) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000231 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG0) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x1000000021c (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000237 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.PX) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000217 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x1000000022f (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.PX) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x1000000023b (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.TV) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000219 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x1000000023c (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.FSL) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000216 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000233 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG1) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000239 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.Y03) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000226 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x1000000023d (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.FSL) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x1000000021d (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000234 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.VAL) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000227 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000228 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x1000000022a (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.DB) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x1000000022d (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.Y04) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000235 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.MB) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000210 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PCBENCHM.PPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000238 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.X03) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000211 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ZD16.BMP) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x1000000023a (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.VAL) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000236 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.DB) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000232 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG1) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x1000000022e (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.DB) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000230 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG0) see mds.i log and `damage ls` output for details 2024-03-18T14:54:15.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: scrub summary: idle+waiting paths [/] 2024-03-18T14:54:15.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000214 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TRIDOTS.POT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.981 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000213 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TIPS.PPT) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.981 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: Scrub error on inode 0x10000000212 (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PPTOOLS1.PPA) see mds.c log and `damage ls` output for details 2024-03-18T14:54:15.981 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: scrub summary: idle+waiting paths [/] 2024-03-18T14:54:15.981 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:15 smithi077 ceph-mon[35865]: scrub summary: idle+waiting paths [/] 2024-03-18T14:54:16.731 INFO:tasks.workunit.client.0.smithi076.stdout: 1 39292 13.59 MB/sec warmup 17 sec latency 2489.770 ms 2024-03-18T14:54:16.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:16 smithi076 ceph-mon[32205]: pgmap v33: 97 pgs: 97 active+clean; 11 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 351 op/s 2024-03-18T14:54:16.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:16 smithi174 ceph-mon[36787]: pgmap v33: 97 pgs: 97 active+clean; 11 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 351 op/s 2024-03-18T14:54:16.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:16 smithi077 ceph-mon[35865]: pgmap v33: 97 pgs: 97 active+clean; 11 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 351 op/s 2024-03-18T14:54:17.732 INFO:tasks.workunit.client.0.smithi076.stdout: 1 39292 12.83 MB/sec warmup 18 sec latency 3489.849 ms 2024-03-18T14:54:17.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:17 smithi076 ceph-mon[32205]: pgmap v34: 97 pgs: 97 active+clean; 11 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 351 op/s 2024-03-18T14:54:17.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:17 smithi174 ceph-mon[36787]: pgmap v34: 97 pgs: 97 active+clean; 11 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 351 op/s 2024-03-18T14:54:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:17 smithi077 ceph-mon[35865]: pgmap v34: 97 pgs: 97 active+clean; 11 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 351 op/s 2024-03-18T14:54:18.732 INFO:tasks.workunit.client.0.smithi076.stdout: 1 39379 12.16 MB/sec warmup 19 sec latency 3880.277 ms 2024-03-18T14:54:19.732 INFO:tasks.workunit.client.0.smithi076.stdout: 1 39379 11.55 MB/sec warmup 20 sec latency 1470.683 ms 2024-03-18T14:54:20.732 INFO:tasks.workunit.client.0.smithi076.stdout: 1 39379 11.00 MB/sec warmup 21 sec latency 2470.789 ms 2024-03-18T14:54:20.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:20 smithi076 ceph-mon[32205]: pgmap v35: 97 pgs: 97 active+clean; 11 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 983 KiB/s wr, 269 op/s 2024-03-18T14:54:20.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:20 smithi174 ceph-mon[36787]: pgmap v35: 97 pgs: 97 active+clean; 11 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 983 KiB/s wr, 269 op/s 2024-03-18T14:54:20.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:20 smithi077 ceph-mon[35865]: pgmap v35: 97 pgs: 97 active+clean; 11 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 983 KiB/s wr, 269 op/s 2024-03-18T14:54:21.732 INFO:tasks.workunit.client.0.smithi076.stdout: 1 39379 10.50 MB/sec warmup 22 sec latency 3470.915 ms 2024-03-18T14:54:21.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:21 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a[32201]: 2024-03-18T14:54:21.621+0000 7f4cc589e700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-18T14:54:21.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:21 smithi076 ceph-mon[32205]: pgmap v36: 97 pgs: 97 active+clean; 11 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 983 KiB/s wr, 269 op/s 2024-03-18T14:54:21.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:21 smithi076 ceph-mon[32205]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-18T14:54:21.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:21 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] up:active 2024-03-18T14:54:21.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:21 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:54:21.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:21 smithi174 ceph-mon[36787]: pgmap v36: 97 pgs: 97 active+clean; 11 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 983 KiB/s wr, 269 op/s 2024-03-18T14:54:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:21 smithi174 ceph-mon[36787]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-18T14:54:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:21 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] up:active 2024-03-18T14:54:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:21 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:54:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:21 smithi077 ceph-mon[35865]: pgmap v36: 97 pgs: 97 active+clean; 11 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 983 KiB/s wr, 269 op/s 2024-03-18T14:54:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:21 smithi077 ceph-mon[35865]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-18T14:54:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:21 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] up:active 2024-03-18T14:54:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:21 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:54:22.732 INFO:tasks.workunit.client.0.smithi076.stdout: 1 39379 10.04 MB/sec warmup 23 sec latency 4470.969 ms 2024-03-18T14:54:22.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:22 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:54:22.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:22 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:54:23.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:22 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:54:23.732 INFO:tasks.workunit.client.0.smithi076.stdout: 1 39439 9.64 MB/sec warmup 24 sec latency 4857.412 ms 2024-03-18T14:54:23.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:23 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] up:active 2024-03-18T14:54:23.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:23 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:54:23.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:23 smithi174 ceph-mon[36787]: scrub summary: idle 2024-03-18T14:54:23.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:23 smithi174 ceph-mon[36787]: scrub summary: idle 2024-03-18T14:54:23.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:23 smithi174 ceph-mon[36787]: scrub summary: idle 2024-03-18T14:54:23.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:23 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:54:23.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:23 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:54:23.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:23 smithi174 ceph-mon[36787]: pgmap v37: 97 pgs: 97 active+clean; 11 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 954 KiB/s wr, 264 op/s 2024-03-18T14:54:23.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:23 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:54:23.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:23 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:54:23.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:23 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:54:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:23 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] up:active 2024-03-18T14:54:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:23 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:54:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:23 smithi077 ceph-mon[35865]: scrub summary: idle 2024-03-18T14:54:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:23 smithi077 ceph-mon[35865]: scrub summary: idle 2024-03-18T14:54:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:23 smithi077 ceph-mon[35865]: scrub summary: idle 2024-03-18T14:54:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:23 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:54:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:23 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:54:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:23 smithi077 ceph-mon[35865]: pgmap v37: 97 pgs: 97 active+clean; 11 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 954 KiB/s wr, 264 op/s 2024-03-18T14:54:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:23 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:54:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:23 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:54:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:23 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:54:24.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:23 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] up:active 2024-03-18T14:54:24.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:23 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:54:24.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:23 smithi076 ceph-mon[32205]: scrub summary: idle 2024-03-18T14:54:24.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:23 smithi076 ceph-mon[32205]: scrub summary: idle 2024-03-18T14:54:24.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:23 smithi076 ceph-mon[32205]: scrub summary: idle 2024-03-18T14:54:24.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:23 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:54:24.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:23 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:54:24.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:23 smithi076 ceph-mon[32205]: pgmap v37: 97 pgs: 97 active+clean; 11 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 954 KiB/s wr, 264 op/s 2024-03-18T14:54:24.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:23 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:54:24.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:23 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:54:24.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:23 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:54:24.732 INFO:tasks.workunit.client.0.smithi076.stdout: 1 39439 9.25 MB/sec warmup 25 sec latency 1561.235 ms 2024-03-18T14:54:25.732 INFO:tasks.workunit.client.0.smithi076.stdout: 1 39439 8.90 MB/sec warmup 26 sec latency 2561.278 ms 2024-03-18T14:54:25.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.826+0000 7fc3284e8640 1 -- 172.21.15.76:0/998775421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc320105660 msgr2=0x7fc320105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.826+0000 7fc3284e8640 1 --2- 172.21.15.76:0/998775421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc320105660 0x7fc320105a40 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc31c009900 tx=0x7fc31c02bc30 comp rx=0 tx=0).stop 2024-03-18T14:54:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc3284e8640 1 -- 172.21.15.76:0/998775421 shutdown_connections 2024-03-18T14:54:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc3284e8640 1 --2- 172.21.15.76:0/998775421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3201012d0 0x7fc32010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc3284e8640 1 --2- 172.21.15.76:0/998775421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc320100930 0x7fc320100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc3284e8640 1 --2- 172.21.15.76:0/998775421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc320105660 0x7fc320105a40 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc3284e8640 1 -- 172.21.15.76:0/998775421 >> 172.21.15.76:0/998775421 conn(0x7fc3200774b0 msgr2=0x7fc3200778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc3284e8640 1 -- 172.21.15.76:0/998775421 shutdown_connections 2024-03-18T14:54:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc3284e8640 1 -- 172.21.15.76:0/998775421 wait complete. 2024-03-18T14:54:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc3284e8640 1 Processor -- start 2024-03-18T14:54:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc3284e8640 1 -- start start 2024-03-18T14:54:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc3284e8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc320100930 0x7fc3200fe310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc3284e8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3201012d0 0x7fc3200fe850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc3284e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc320105660 0x7fc3200fed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc3284e8640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc320110b70 con 0x7fc320100930 2024-03-18T14:54:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc3284e8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc3201109f0 con 0x7fc320105660 2024-03-18T14:54:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc3284e8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc320110cf0 con 0x7fc3201012d0 2024-03-18T14:54:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc326284640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc320100930 0x7fc3200fe310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc326284640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc320100930 0x7fc3200fe310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46284/0 (socket says 172.21.15.76:46284) 2024-03-18T14:54:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc326284640 1 -- 172.21.15.76:0/492056661 learned_addr learned my addr 172.21.15.76:0/492056661 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.827+0000 7fc326a85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc320105660 0x7fc3200fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.828+0000 7fc325a83640 1 --2- 172.21.15.76:0/492056661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3201012d0 0x7fc3200fe850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.828+0000 7fc326284640 1 -- 172.21.15.76:0/492056661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3201012d0 msgr2=0x7fc3200fe850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.828+0000 7fc326284640 1 --2- 172.21.15.76:0/492056661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3201012d0 0x7fc3200fe850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.828+0000 7fc326284640 1 -- 172.21.15.76:0/492056661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc320105660 msgr2=0x7fc3200fed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.828+0000 7fc326284640 1 --2- 172.21.15.76:0/492056661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc320105660 0x7fc3200fed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.828+0000 7fc326284640 1 -- 172.21.15.76:0/492056661 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3201a6520 con 0x7fc320100930 2024-03-18T14:54:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.828+0000 7fc326a85640 1 --2- 172.21.15.76:0/492056661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc320105660 0x7fc3200fed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.828+0000 7fc326284640 1 --2- 172.21.15.76:0/492056661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc320100930 0x7fc3200fe310 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc31c006010 tx=0x7fc31c02cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.828+0000 7fc3177fe640 1 -- 172.21.15.76:0/492056661 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c041070 con 0x7fc320100930 2024-03-18T14:54:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.828+0000 7fc3177fe640 1 -- 172.21.15.76:0/492056661 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc31c004450 con 0x7fc320100930 2024-03-18T14:54:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.828+0000 7fc3177fe640 1 -- 172.21.15.76:0/492056661 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c003e00 con 0x7fc320100930 2024-03-18T14:54:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.828+0000 7fc3284e8640 1 -- 172.21.15.76:0/492056661 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3201a67b0 con 0x7fc320100930 2024-03-18T14:54:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.828+0000 7fc3284e8640 1 -- 172.21.15.76:0/492056661 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3201a6ca0 con 0x7fc320100930 2024-03-18T14:54:25.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.830+0000 7fc3177fe640 1 -- 172.21.15.76:0/492056661 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7fc31c044070 con 0x7fc320100930 2024-03-18T14:54:25.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.831+0000 7fc32b1ae740 1 Processor -- start 2024-03-18T14:54:25.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.831+0000 7fc3177fe640 1 --2- 172.21.15.76:0/492056661 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc30c0a2b30 0x7fc30c0a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.831+0000 7fc32b1ae740 1 -- start start 2024-03-18T14:54:25.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.831+0000 7fc32b1ae740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757257f130 0x55757257f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.831+0000 7fc32b1ae740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 0x5575725882e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.831+0000 7fc326a85640 1 --2- 172.21.15.76:0/492056661 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc30c0a2b30 0x7fc30c0a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.831+0000 7fc32b1ae740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557572588820 0x55757258ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.831+0000 7fc32b1ae740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5575724e3a40 con 0x557572588820 2024-03-18T14:54:25.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.831+0000 7fc32b1ae740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55757248c920 con 0x55757257fa50 2024-03-18T14:54:25.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.831+0000 7fc32b1ae740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5575724dfae0 con 0x55757257f130 2024-03-18T14:54:25.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.831+0000 7fc3177fe640 1 -- 172.21.15.76:0/492056661 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fc31c0f0a80 con 0x7fc320100930 2024-03-18T14:54:25.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.831+0000 7fc326a85640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557572588820 0x55757258ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.831+0000 7fc325a83640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 0x5575725882e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.831+0000 7fc326284640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757257f130 0x55757257f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.831+0000 7fc326a85640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557572588820 0x55757258ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46298/0 (socket says 172.21.15.76:46298) 2024-03-18T14:54:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.831+0000 7fc326a85640 1 -- 172.21.15.76:0/1679361588 learned_addr learned my addr 172.21.15.76:0/1679361588 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.832+0000 7fc326284640 1 -- 172.21.15.76:0/1679361588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 msgr2=0x5575725882e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.832+0000 7fc326284640 1 --2- 172.21.15.76:0/1679361588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 0x5575725882e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.832+0000 7fc326284640 1 -- 172.21.15.76:0/1679361588 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557572588820 msgr2=0x55757258ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.832+0000 7fc326284640 1 --2- 172.21.15.76:0/1679361588 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557572588820 0x55757258ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.832+0000 7fc326284640 1 -- 172.21.15.76:0/1679361588 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575724a4460 con 0x55757257f130 2024-03-18T14:54:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.832+0000 7fc325a83640 1 --2- 172.21.15.76:0/1679361588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 0x5575725882e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.832+0000 7fc326a85640 1 --2- 172.21.15.76:0/492056661 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc30c0a2b30 0x7fc30c0a4ff0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc3180049d0 tx=0x7fc318013720 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.832+0000 7fc326284640 1 --2- 172.21.15.76:0/1679361588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757257f130 0x55757257f510 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc31c0ee150 tx=0x7fc31c0b92e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=896a8d593f258ad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.832+0000 7fc314bfe640 1 -- 172.21.15.76:0/1679361588 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c04b070 con 0x55757257f130 2024-03-18T14:54:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.832+0000 7fc314bfe640 1 -- 172.21.15.76:0/1679361588 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc31c02cd20 con 0x55757257f130 2024-03-18T14:54:25.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.832+0000 7fc314bfe640 1 -- 172.21.15.76:0/1679361588 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c0b96b0 con 0x55757257f130 2024-03-18T14:54:25.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.833+0000 7fc32b1ae740 1 -- 172.21.15.76:0/1679361588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757257f130 msgr2=0x55757257f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:25.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.833+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/1679361588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757257f130 0x55757257f510 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc31c0ee150 tx=0x7fc31c0b92e0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.833+0000 7fc32b1ae740 1 -- 172.21.15.76:0/1679361588 shutdown_connections 2024-03-18T14:54:25.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.833+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/1679361588 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557572588820 0x55757258ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.833+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/1679361588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 0x5575725882e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.833+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/1679361588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757257f130 0x55757257f510 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.833+0000 7fc32b1ae740 1 -- 172.21.15.76:0/1679361588 >> 172.21.15.76:0/1679361588 conn(0x557572579a50 msgr2=0x557572579e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:25.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.833+0000 7fc32b1ae740 1 -- 172.21.15.76:0/1679361588 shutdown_connections 2024-03-18T14:54:25.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.833+0000 7fc32b1ae740 1 -- 172.21.15.76:0/1679361588 wait complete. 2024-03-18T14:54:25.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.833+0000 7fc32b1ae740 1 -- start start 2024-03-18T14:54:25.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.833+0000 7fc32b1ae740 1 Processor -- start 2024-03-18T14:54:25.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.833+0000 7fc32b1ae740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757257f130 0x5575725858c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.833+0000 7fc32b1ae740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 0x557572585e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.833+0000 7fc32b1ae740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557572588820 0x557572586340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.833+0000 7fc32b1ae740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55757258b210 con 0x55757257f130 2024-03-18T14:54:25.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc32b1ae740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55757249fd70 con 0x55757257fa50 2024-03-18T14:54:25.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc32b1ae740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55757258b390 con 0x557572588820 2024-03-18T14:54:25.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc32b1ae740 10 client.0 _ll_register_callbacks cb 0x5575721f7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:25.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc326284640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757257f130 0x5575725858c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc326284640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757257f130 0x5575725858c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46310/0 (socket says 172.21.15.76:46310) 2024-03-18T14:54:25.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc326284640 1 -- 172.21.15.76:0/1059708868 learned_addr learned my addr 172.21.15.76:0/1059708868 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:25.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc325a83640 1 --2- 172.21.15.76:0/1059708868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 0x557572585e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc326a85640 1 --2- 172.21.15.76:0/1059708868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557572588820 0x557572586340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc326284640 1 -- 172.21.15.76:0/1059708868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557572588820 msgr2=0x557572586340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:25.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc326284640 1 --2- 172.21.15.76:0/1059708868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557572588820 0x557572586340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc326284640 1 -- 172.21.15.76:0/1059708868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 msgr2=0x557572585e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc326284640 1 --2- 172.21.15.76:0/1059708868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 0x557572585e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc326284640 1 -- 172.21.15.76:0/1059708868 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575721f8210 con 0x55757257f130 2024-03-18T14:54:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc326284640 1 --2- 172.21.15.76:0/1059708868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757257f130 0x5575725858c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc31c02c820 tx=0x7fc31c0b98e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ada1f520138168 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc309ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc309ffb640 1 -- 172.21.15.76:0/1059708868 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c04b040 con 0x55757257f130 2024-03-18T14:54:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc309ffb640 1 -- 172.21.15.76:0/1059708868 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fc31c0c54b0 con 0x55757257f130 2024-03-18T14:54:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc32b1ae740 1 -- 172.21.15.76:0/1059708868 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc31c02cd20 con 0x55757257f130 2024-03-18T14:54:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc309ffb640 1 -- 172.21.15.76:0/1059708868 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc31c0bc9a0 con 0x55757257f130 2024-03-18T14:54:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc32b1ae740 10 client.15828 fetch_fsmap learned FSMap version 27 2024-03-18T14:54:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc32b1ae740 1 -- 172.21.15.76:0/1059708868 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5575722c4620 con 0x55757257f130 2024-03-18T14:54:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.834+0000 7fc309ffb640 1 -- 172.21.15.76:0/1059708868 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc31c02cd20 con 0x55757257f130 2024-03-18T14:54:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.835+0000 7fc309ffb640 1 -- 172.21.15.76:0/1059708868 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc31c0bc490 con 0x55757257f130 2024-03-18T14:54:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.835+0000 7fc32b1ae740 10 client.15828 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T14:54:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.835+0000 7fc32b1ae740 10 client.15828 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.835+0000 7fc32b1ae740 20 client.15828 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.835+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/1059708868 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557572598580 0x55757259a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.835+0000 7fc32b1ae740 4 client.15828 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:54:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.835+0000 7fc32b1ae740 1 -- 172.21.15.76:0/1059708868 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575724f0f80 con 0x557572598580 2024-03-18T14:54:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.835+0000 7fc325a83640 1 --2- 172.21.15.76:0/1059708868 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557572598580 0x55757259a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.835+0000 7fc309ffb640 10 client.15828 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.835+0000 7fc325a83640 1 --2- 172.21.15.76:0/1059708868 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557572598580 0x55757259a970 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d426ac339d4f8b7d server_cookie=3b4b44ff2811707a in_seq=0 out_seq=0 2024-03-18T14:54:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.836+0000 7fc309ffb640 1 -- 172.21.15.76:0/1059708868 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5575724f0f80 con 0x557572598580 2024-03-18T14:54:25.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.836+0000 7fc309ffb640 10 client.15828 handle_command_reply: tid=0 2024-03-18T14:54:25.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.836+0000 7fc32b1ae740 1 client.15828 shutdown 2024-03-18T14:54:25.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.836+0000 7fc32b1ae740 1 -- 172.21.15.76:0/1059708868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757257f130 msgr2=0x5575725858c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:25.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.836+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/1059708868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757257f130 0x5575725858c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc31c02c820 tx=0x7fc31c0b98e0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.836+0000 7fc32b1ae740 1 -- 172.21.15.76:0/1059708868 shutdown_connections 2024-03-18T14:54:25.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.836+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/1059708868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557572588820 0x557572586340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.836+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/1059708868 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557572598580 0x55757259a970 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.836+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/1059708868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 0x557572585e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.836+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/1059708868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757257f130 0x5575725858c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.836+0000 7fc309ffb640 0 client.15828 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:25.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.836+0000 7fc32b1ae740 1 -- 172.21.15.76:0/1059708868 >> 172.21.15.76:0/1059708868 conn(0x557572579a50 msgr2=0x55757257b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:25.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.836+0000 7fc32b1ae740 1 -- 172.21.15.76:0/1059708868 shutdown_connections 2024-03-18T14:54:25.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.836+0000 7fc32b1ae740 1 -- 172.21.15.76:0/1059708868 wait complete. 2024-03-18T14:54:25.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.837+0000 7fc32b1ae740 20 client.15828 trim_cache size 0 max 16384 2024-03-18T14:54:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc32b1ae740 1 Processor -- start 2024-03-18T14:54:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc32b1ae740 1 -- start start 2024-03-18T14:54:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc32b1ae740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757257f130 0x557572593dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc32b1ae740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 0x5575725a0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc32b1ae740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557572588820 0x557572582b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc32b1ae740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc3201b1e50 con 0x55757257f130 2024-03-18T14:54:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc32b1ae740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc3201af6c0 con 0x55757257fa50 2024-03-18T14:54:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc32b1ae740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc3201ad510 con 0x557572588820 2024-03-18T14:54:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc326284640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757257f130 0x557572593dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc326284640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757257f130 0x557572593dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46326/0 (socket says 172.21.15.76:46326) 2024-03-18T14:54:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc326284640 1 -- 172.21.15.76:0/2848455379 learned_addr learned my addr 172.21.15.76:0/2848455379 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc325a83640 1 --2- 172.21.15.76:0/2848455379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 0x5575725a0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc326a85640 1 --2- 172.21.15.76:0/2848455379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557572588820 0x557572582b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc326284640 1 -- 172.21.15.76:0/2848455379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557572588820 msgr2=0x557572582b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc326284640 1 --2- 172.21.15.76:0/2848455379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557572588820 0x557572582b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc326284640 1 -- 172.21.15.76:0/2848455379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 msgr2=0x5575725a0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc326284640 1 --2- 172.21.15.76:0/2848455379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 0x5575725a0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.851+0000 7fc326284640 1 -- 172.21.15.76:0/2848455379 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55757257e410 con 0x55757257f130 2024-03-18T14:54:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.852+0000 7fc326284640 1 --2- 172.21.15.76:0/2848455379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757257f130 0x557572593dd0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc31c000c00 tx=0x7fc31c004c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a04221182a1197c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.852+0000 7fc30bfff640 1 -- 172.21.15.76:0/2848455379 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c04b070 con 0x55757257f130 2024-03-18T14:54:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.852+0000 7fc30bfff640 1 -- 172.21.15.76:0/2848455379 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc31c0bd040 con 0x55757257f130 2024-03-18T14:54:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.852+0000 7fc30bfff640 1 -- 172.21.15.76:0/2848455379 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c0bc490 con 0x55757257f130 2024-03-18T14:54:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.852+0000 7fc32b1ae740 1 -- 172.21.15.76:0/2848455379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757257f130 msgr2=0x557572593dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.852+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/2848455379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757257f130 0x557572593dd0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc31c000c00 tx=0x7fc31c004c20 comp rx=0 tx=0).stop 2024-03-18T14:54:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.852+0000 7fc32b1ae740 1 -- 172.21.15.76:0/2848455379 shutdown_connections 2024-03-18T14:54:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.852+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/2848455379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557572588820 0x557572582b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.852+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/2848455379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 0x5575725a0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.852+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/2848455379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757257f130 0x557572593dd0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.852+0000 7fc32b1ae740 1 -- 172.21.15.76:0/2848455379 >> 172.21.15.76:0/2848455379 conn(0x557572579a50 msgr2=0x55757257b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.852+0000 7fc32b1ae740 1 -- 172.21.15.76:0/2848455379 shutdown_connections 2024-03-18T14:54:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.852+0000 7fc32b1ae740 1 -- 172.21.15.76:0/2848455379 wait complete. 2024-03-18T14:54:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc32b1ae740 1 -- start start 2024-03-18T14:54:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc32b1ae740 1 Processor -- start 2024-03-18T14:54:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc32b1ae740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757257f130 0x55757258c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc32b1ae740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 0x55757258cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc32b1ae740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557572588820 0x5575725f7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc32b1ae740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc3201adb80 con 0x557572588820 2024-03-18T14:54:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc32b1ae740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc3201aeff0 con 0x55757257fa50 2024-03-18T14:54:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc32b1ae740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc3201ae250 con 0x55757257f130 2024-03-18T14:54:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc32b1ae740 10 client.0 _ll_register_callbacks cb 0x7fc318013bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc326a85640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557572588820 0x5575725f7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc326a85640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557572588820 0x5575725f7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46328/0 (socket says 172.21.15.76:46328) 2024-03-18T14:54:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc326a85640 1 -- 172.21.15.76:0/157605230 learned_addr learned my addr 172.21.15.76:0/157605230 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc325a83640 1 --2- 172.21.15.76:0/157605230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 0x55757258cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc326284640 1 --2- 172.21.15.76:0/157605230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757257f130 0x55757258c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc326a85640 1 -- 172.21.15.76:0/157605230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757257f130 msgr2=0x55757258c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc326a85640 1 --2- 172.21.15.76:0/157605230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757257f130 0x55757258c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc326a85640 1 -- 172.21.15.76:0/157605230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 msgr2=0x55757258cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc326a85640 1 --2- 172.21.15.76:0/157605230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 0x55757258cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.853+0000 7fc326a85640 1 -- 172.21.15.76:0/157605230 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575725f7bf0 con 0x557572588820 2024-03-18T14:54:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.854+0000 7fc326a85640 1 --2- 172.21.15.76:0/157605230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557572588820 0x5575725f7600 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc318009fd0 tx=0x7fc31801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709cf6e15fb67d71 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.854+0000 7fc309ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.854+0000 7fc309ffb640 1 -- 172.21.15.76:0/157605230 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc318004b20 con 0x557572588820 2024-03-18T14:54:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.854+0000 7fc309ffb640 1 -- 172.21.15.76:0/157605230 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fc31801c040 con 0x557572588820 2024-03-18T14:54:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.854+0000 7fc32b1ae740 1 -- 172.21.15.76:0/157605230 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc31c0bd040 con 0x557572588820 2024-03-18T14:54:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.854+0000 7fc309ffb640 1 -- 172.21.15.76:0/157605230 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc318004cc0 con 0x557572588820 2024-03-18T14:54:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.854+0000 7fc32b1ae740 10 client.15834 fetch_fsmap learned FSMap version 27 2024-03-18T14:54:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.854+0000 7fc32b1ae740 1 -- 172.21.15.76:0/157605230 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5575725fef80 con 0x557572588820 2024-03-18T14:54:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.854+0000 7fc309ffb640 1 -- 172.21.15.76:0/157605230 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc31c0bd040 con 0x557572588820 2024-03-18T14:54:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.854+0000 7fc309ffb640 1 -- 172.21.15.76:0/157605230 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc318020050 con 0x557572588820 2024-03-18T14:54:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.854+0000 7fc32b1ae740 10 client.15834 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T14:54:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.854+0000 7fc32b1ae740 10 client.15834 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.854+0000 7fc32b1ae740 20 client.15834 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.854+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/157605230 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5575725ff760 0x557572601b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.854+0000 7fc32b1ae740 4 client.15834 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:54:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.854+0000 7fc32b1ae740 1 -- 172.21.15.76:0/157605230 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5575724eb2d0 con 0x5575725ff760 2024-03-18T14:54:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.854+0000 7fc326284640 1 --2- 172.21.15.76:0/157605230 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5575725ff760 0x557572601b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.855+0000 7fc309ffb640 10 client.15834 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.855+0000 7fc326284640 1 --2- 172.21.15.76:0/157605230 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5575725ff760 0x557572601b80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=623322ea0da7829d server_cookie=8552dcd6f0f82f73 in_seq=0 out_seq=0 2024-03-18T14:54:25.989 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.988+0000 7fc309ffb640 1 -- 172.21.15.76:0/157605230 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5575724eb2d0 con 0x5575725ff760 2024-03-18T14:54:25.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.988+0000 7fc309ffb640 10 client.15834 handle_command_reply: tid=0 2024-03-18T14:54:25.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.988+0000 7fc32b1ae740 1 client.15834 shutdown 2024-03-18T14:54:25.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.988+0000 7fc32b1ae740 1 -- 172.21.15.76:0/157605230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557572588820 msgr2=0x5575725f7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:25.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.988+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/157605230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557572588820 0x5575725f7600 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc318009fd0 tx=0x7fc31801d000 comp rx=0 tx=0).stop 2024-03-18T14:54:25.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.988+0000 7fc32b1ae740 1 -- 172.21.15.76:0/157605230 shutdown_connections 2024-03-18T14:54:25.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.988+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/157605230 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5575725ff760 0x557572601b80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.988+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/157605230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557572588820 0x5575725f7600 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.988+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/157605230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55757257fa50 0x55757258cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.988+0000 7fc309ffb640 0 client.15834 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:25.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.988+0000 7fc32b1ae740 1 --2- 172.21.15.76:0/157605230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757257f130 0x55757258c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.988+0000 7fc32b1ae740 1 -- 172.21.15.76:0/157605230 >> 172.21.15.76:0/157605230 conn(0x557572579a50 msgr2=0x55757257aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:25.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.988+0000 7fc32b1ae740 1 -- 172.21.15.76:0/157605230 shutdown_connections 2024-03-18T14:54:25.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.988+0000 7fc32b1ae740 1 -- 172.21.15.76:0/157605230 wait complete. 2024-03-18T14:54:25.990 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.989+0000 7fc32b1ae740 20 client.15834 trim_cache size 0 max 16384 2024-03-18T14:54:25.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:54:25.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:54:25.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:54:25.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:54:25.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.989+0000 7fc3284e8640 1 -- 172.21.15.76:0/492056661 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc30c0a2b30 msgr2=0x7fc30c0a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:25.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.989+0000 7fc3284e8640 1 --2- 172.21.15.76:0/492056661 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc30c0a2b30 0x7fc30c0a4ff0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc3180049d0 tx=0x7fc318013720 comp rx=0 tx=0).stop 2024-03-18T14:54:25.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.989+0000 7fc3284e8640 1 -- 172.21.15.76:0/492056661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc320100930 msgr2=0x7fc3200fe310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:25.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.989+0000 7fc3284e8640 1 --2- 172.21.15.76:0/492056661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc320100930 0x7fc3200fe310 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc31c006010 tx=0x7fc31c02cb30 comp rx=0 tx=0).stop 2024-03-18T14:54:25.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.989+0000 7fc3284e8640 1 -- 172.21.15.76:0/492056661 shutdown_connections 2024-03-18T14:54:25.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.989+0000 7fc3284e8640 1 --2- 172.21.15.76:0/492056661 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc30c0a2b30 0x7fc30c0a4ff0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.989+0000 7fc3284e8640 1 --2- 172.21.15.76:0/492056661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3201012d0 0x7fc3200fe850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.989+0000 7fc3284e8640 1 --2- 172.21.15.76:0/492056661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc320105660 0x7fc3200fed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.989+0000 7fc3284e8640 1 --2- 172.21.15.76:0/492056661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc320100930 0x7fc3200fe310 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:25.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.989+0000 7fc3284e8640 1 -- 172.21.15.76:0/492056661 >> 172.21.15.76:0/492056661 conn(0x7fc3200774b0 msgr2=0x7fc32006a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:25.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.989+0000 7fc3284e8640 1 -- 172.21.15.76:0/492056661 shutdown_connections 2024-03-18T14:54:25.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:25.989+0000 7fc3284e8640 1 -- 172.21.15.76:0/492056661 wait complete. 2024-03-18T14:54:26.732 INFO:tasks.workunit.client.0.smithi076.stdout: 1 39439 8.57 MB/sec warmup 27 sec latency 3561.323 ms 2024-03-18T14:54:26.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:26 smithi076 ceph-mon[32205]: pgmap v38: 97 pgs: 97 active+clean; 11 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 206 KiB/s wr, 81 op/s 2024-03-18T14:54:26.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:26 smithi174 ceph-mon[36787]: pgmap v38: 97 pgs: 97 active+clean; 11 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 206 KiB/s wr, 81 op/s 2024-03-18T14:54:26.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:26 smithi077 ceph-mon[35865]: pgmap v38: 97 pgs: 97 active+clean; 11 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 206 KiB/s wr, 81 op/s 2024-03-18T14:54:27.732 INFO:tasks.workunit.client.0.smithi076.stdout: 1 39439 8.26 MB/sec warmup 28 sec latency 4561.366 ms 2024-03-18T14:54:28.625 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:28 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a[32201]: 2024-03-18T14:54:28.207+0000 7f4cc589e700 -1 log_channel(cluster) log [ERR] : Health check update: 2 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-18T14:54:28.732 INFO:tasks.workunit.client.0.smithi076.stdout: 1 40504 8.15 MB/sec warmup 29 sec latency 4952.881 ms 2024-03-18T14:54:28.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:28 smithi076 ceph-mon[32205]: pgmap v39: 97 pgs: 97 active+clean; 11 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 28 KiB/s wr, 22 op/s 2024-03-18T14:54:28.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:28 smithi076 ceph-mon[32205]: Health check update: 2 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-18T14:54:28.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:28 smithi174 ceph-mon[36787]: pgmap v39: 97 pgs: 97 active+clean; 11 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 28 KiB/s wr, 22 op/s 2024-03-18T14:54:28.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:28 smithi174 ceph-mon[36787]: Health check update: 2 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-18T14:54:28.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:28 smithi077 ceph-mon[35865]: pgmap v39: 97 pgs: 97 active+clean; 11 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 28 KiB/s wr, 22 op/s 2024-03-18T14:54:28.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:28 smithi077 ceph-mon[35865]: Health check update: 2 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-18T14:54:29.732 INFO:tasks.workunit.client.0.smithi076.stdout: 1 42969 8.35 MB/sec warmup 30 sec latency 12.362 ms 2024-03-18T14:54:30.732 INFO:tasks.workunit.client.0.smithi076.stdout: 1 45039 8.31 MB/sec warmup 31 sec latency 18.124 ms 2024-03-18T14:54:30.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:30 smithi076 ceph-mon[32205]: pgmap v40: 97 pgs: 97 active+clean; 28 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 4.4 MiB/s wr, 147 op/s 2024-03-18T14:54:30.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:30 smithi174 ceph-mon[36787]: pgmap v40: 97 pgs: 97 active+clean; 28 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 4.4 MiB/s wr, 147 op/s 2024-03-18T14:54:30.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:30 smithi077 ceph-mon[35865]: pgmap v40: 97 pgs: 97 active+clean; 28 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 4.4 MiB/s wr, 147 op/s 2024-03-18T14:54:31.732 INFO:tasks.workunit.client.0.smithi076.stdout: 1 46892 8.44 MB/sec warmup 32 sec latency 11.429 ms 2024-03-18T14:54:32.732 INFO:tasks.workunit.client.0.smithi076.stdout: 1 49821 8.71 MB/sec warmup 33 sec latency 12.204 ms 2024-03-18T14:54:32.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:32 smithi076 ceph-mon[32205]: pgmap v41: 97 pgs: 97 active+clean; 28 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 4.3 MiB/s wr, 130 op/s 2024-03-18T14:54:32.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:32 smithi174 ceph-mon[36787]: pgmap v41: 97 pgs: 97 active+clean; 28 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 4.3 MiB/s wr, 130 op/s 2024-03-18T14:54:32.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:32 smithi077 ceph-mon[35865]: pgmap v41: 97 pgs: 97 active+clean; 28 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 4.3 MiB/s wr, 130 op/s 2024-03-18T14:54:33.732 INFO:tasks.workunit.client.0.smithi076.stdout: 1 51209 8.64 MB/sec warmup 34 sec latency 15.615 ms 2024-03-18T14:54:34.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 53626 8.79 MB/sec warmup 35 sec latency 11.421 ms 2024-03-18T14:54:34.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:34 smithi076 ceph-mon[32205]: pgmap v42: 97 pgs: 97 active+clean; 33 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.0 MiB/s wr, 235 op/s 2024-03-18T14:54:34.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:34 smithi174 ceph-mon[36787]: pgmap v42: 97 pgs: 97 active+clean; 33 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.0 MiB/s wr, 235 op/s 2024-03-18T14:54:34.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:34 smithi077 ceph-mon[35865]: pgmap v42: 97 pgs: 97 active+clean; 33 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.0 MiB/s wr, 235 op/s 2024-03-18T14:54:35.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 55685 8.75 MB/sec warmup 36 sec latency 15.167 ms 2024-03-18T14:54:35.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:35 smithi174 ceph-mon[36787]: pgmap v43: 97 pgs: 97 active+clean; 33 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.0 MiB/s wr, 231 op/s 2024-03-18T14:54:35.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:35 smithi077 ceph-mon[35865]: pgmap v43: 97 pgs: 97 active+clean; 33 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.0 MiB/s wr, 231 op/s 2024-03-18T14:54:36.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:35 smithi076 ceph-mon[32205]: pgmap v43: 97 pgs: 97 active+clean; 33 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.0 MiB/s wr, 231 op/s 2024-03-18T14:54:36.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 57447 8.84 MB/sec warmup 37 sec latency 20.785 ms 2024-03-18T14:54:37.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 60412 9.07 MB/sec warmup 38 sec latency 13.906 ms 2024-03-18T14:54:38.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 61782 9.00 MB/sec warmup 39 sec latency 17.728 ms 2024-03-18T14:54:38.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:38 smithi076 ceph-mon[32205]: pgmap v44: 97 pgs: 97 active+clean; 33 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.0 MiB/s wr, 230 op/s 2024-03-18T14:54:38.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:38 smithi174 ceph-mon[36787]: pgmap v44: 97 pgs: 97 active+clean; 33 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.0 MiB/s wr, 230 op/s 2024-03-18T14:54:38.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:38 smithi077 ceph-mon[35865]: pgmap v44: 97 pgs: 97 active+clean; 33 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.0 MiB/s wr, 230 op/s 2024-03-18T14:54:39.082 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.081+0000 7f49acd87640 1 -- 172.21.15.76:0/1465576218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49a81012f0 msgr2=0x7f49a810d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:39.082 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.081+0000 7f49acd87640 1 --2- 172.21.15.76:0/1465576218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49a81012f0 0x7f49a810d510 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009ca0 tx=0x7f49a002c0f0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.082 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.081+0000 7f49acd87640 1 -- 172.21.15.76:0/1465576218 shutdown_connections 2024-03-18T14:54:39.082 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.081+0000 7f49acd87640 1 --2- 172.21.15.76:0/1465576218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49a8100930 0x7f49a8100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.082 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.081+0000 7f49acd87640 1 --2- 172.21.15.76:0/1465576218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49a81012f0 0x7f49a810d510 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.082 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.081+0000 7f49acd87640 1 --2- 172.21.15.76:0/1465576218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49a8105680 0x7f49a8105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.083 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.081+0000 7f49acd87640 1 -- 172.21.15.76:0/1465576218 >> 172.21.15.76:0/1465576218 conn(0x7f49a8077530 msgr2=0x7f49a8077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:39.083 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.081+0000 7f49acd87640 1 -- 172.21.15.76:0/1465576218 shutdown_connections 2024-03-18T14:54:39.083 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.081+0000 7f49acd87640 1 -- 172.21.15.76:0/1465576218 wait complete. 2024-03-18T14:54:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.082+0000 7f49acd87640 1 Processor -- start 2024-03-18T14:54:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.082+0000 7f49acd87640 1 -- start start 2024-03-18T14:54:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.082+0000 7f49acd87640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49a8100930 0x7f49a8102350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.082+0000 7f49acd87640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49a81012f0 0x7f49a8102890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.082+0000 7f49acd87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49a8105680 0x7f49a8102dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.082+0000 7f49acd87640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f49a8110ba0 con 0x7f49a81012f0 2024-03-18T14:54:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.082+0000 7f49acd87640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f49a8110a20 con 0x7f49a8105680 2024-03-18T14:54:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.082+0000 7f49acd87640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f49a8110d20 con 0x7f49a8100930 2024-03-18T14:54:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.082+0000 7f49a5d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49a81012f0 0x7f49a8102890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.082+0000 7f49a6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49a8105680 0x7f49a8102dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.082+0000 7f49a5d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49a81012f0 0x7f49a8102890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33192/0 (socket says 172.21.15.76:33192) 2024-03-18T14:54:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.082+0000 7f49a5d9b640 1 -- 172.21.15.76:0/1867529788 learned_addr learned my addr 172.21.15.76:0/1867529788 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.082+0000 7f49a659c640 1 --2- 172.21.15.76:0/1867529788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49a8100930 0x7f49a8102350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.083+0000 7f49a5d9b640 1 -- 172.21.15.76:0/1867529788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49a8100930 msgr2=0x7f49a8102350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.083+0000 7f49a5d9b640 1 --2- 172.21.15.76:0/1867529788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49a8100930 0x7f49a8102350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.083+0000 7f49a5d9b640 1 -- 172.21.15.76:0/1867529788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49a8105680 msgr2=0x7f49a8102dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.083+0000 7f49a5d9b640 1 --2- 172.21.15.76:0/1867529788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49a8105680 0x7f49a8102dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.083+0000 7f49a5d9b640 1 -- 172.21.15.76:0/1867529788 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a81a6590 con 0x7f49a81012f0 2024-03-18T14:54:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.083+0000 7f49a5d9b640 1 --2- 172.21.15.76:0/1867529788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49a81012f0 0x7f49a8102890 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f499c00e9f0 tx=0x7f499c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.083+0000 7f49977fe640 1 -- 172.21.15.76:0/1867529788 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499c00cd80 con 0x7f49a81012f0 2024-03-18T14:54:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.083+0000 7f49acd87640 1 -- 172.21.15.76:0/1867529788 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49a81a6880 con 0x7f49a81012f0 2024-03-18T14:54:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.083+0000 7f49977fe640 1 -- 172.21.15.76:0/1867529788 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f499c010910 con 0x7f49a81012f0 2024-03-18T14:54:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.083+0000 7f49977fe640 1 -- 172.21.15.76:0/1867529788 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499c0188f0 con 0x7f49a81012f0 2024-03-18T14:54:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.083+0000 7f49acd87640 1 -- 172.21.15.76:0/1867529788 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49a81a6e20 con 0x7f49a81012f0 2024-03-18T14:54:39.093 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.086+0000 7f49977fe640 1 -- 172.21.15.76:0/1867529788 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f499c018b70 con 0x7f49a81012f0 2024-03-18T14:54:39.093 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.086+0000 7f49af973740 1 Processor -- start 2024-03-18T14:54:39.093 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.086+0000 7f49af973740 1 -- start start 2024-03-18T14:54:39.093 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.086+0000 7f49af973740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 0x55ac807bf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.093 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.086+0000 7f49af973740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807bfa50 0x55ac807c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.093 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.086+0000 7f49af973740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807c8820 0x55ac807cac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.093 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.086+0000 7f49af973740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ac80723a40 con 0x55ac807bfa50 2024-03-18T14:54:39.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.086+0000 7f49af973740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ac806cc920 con 0x55ac807bf130 2024-03-18T14:54:39.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.086+0000 7f49af973740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ac8071fae0 con 0x55ac807c8820 2024-03-18T14:54:39.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.086+0000 7f49a659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807bfa50 0x55ac807c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.086+0000 7f49a6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 0x55ac807bf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.086+0000 7f49a659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807bfa50 0x55ac807c82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33200/0 (socket says 172.21.15.76:33200) 2024-03-18T14:54:39.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.086+0000 7f49a659c640 1 -- 172.21.15.76:0/2774907781 learned_addr learned my addr 172.21.15.76:0/2774907781 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:39.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.086+0000 7f49a5d9b640 1 --2- 172.21.15.76:0/2774907781 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807c8820 0x55ac807cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.087+0000 7f49977fe640 1 --2- 172.21.15.76:0/1867529788 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f49900a2be0 0x7f49900a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.087+0000 7f49a6d9d640 1 -- 172.21.15.76:0/2774907781 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807c8820 msgr2=0x55ac807cac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:39.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.087+0000 7f49a6d9d640 1 --2- 172.21.15.76:0/2774907781 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807c8820 0x55ac807cac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.087+0000 7f49a6d9d640 1 -- 172.21.15.76:0/2774907781 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807bfa50 msgr2=0x55ac807c82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:39.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.087+0000 7f49a6d9d640 1 --2- 172.21.15.76:0/2774907781 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807bfa50 0x55ac807c82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.087+0000 7f49a6d9d640 1 -- 172.21.15.76:0/2774907781 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac806e4460 con 0x55ac807bf130 2024-03-18T14:54:39.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.087+0000 7f49977fe640 1 -- 172.21.15.76:0/1867529788 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f499c0cf390 con 0x7f49a81012f0 2024-03-18T14:54:39.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.087+0000 7f49a6d9d640 1 --2- 172.21.15.76:0/1867529788 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f49900a2be0 0x7f49900a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.087+0000 7f49a6d9d640 1 --2- 172.21.15.76:0/2774907781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 0x55ac807bf510 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f49a002c700 tx=0x7f49a0009980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bdb94ed2ab05c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:39.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.087+0000 7f4994ff9640 1 -- 172.21.15.76:0/2774907781 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0039040 con 0x55ac807bf130 2024-03-18T14:54:39.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.087+0000 7f49a6d9d640 1 --2- 172.21.15.76:0/1867529788 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f49900a2be0 0x7f49900a50a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f49a0005cc0 tx=0x7f49a0045000 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:39.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.087+0000 7f4994ff9640 1 -- 172.21.15.76:0/2774907781 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49a0003b70 con 0x55ac807bf130 2024-03-18T14:54:39.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.087+0000 7f4994ff9640 1 -- 172.21.15.76:0/2774907781 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a004a6b0 con 0x55ac807bf130 2024-03-18T14:54:39.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.088+0000 7f49af973740 1 -- 172.21.15.76:0/2774907781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 msgr2=0x55ac807bf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:39.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.088+0000 7f49af973740 1 --2- 172.21.15.76:0/2774907781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 0x55ac807bf510 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f49a002c700 tx=0x7f49a0009980 comp rx=0 tx=0).stop 2024-03-18T14:54:39.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.088+0000 7f49af973740 1 -- 172.21.15.76:0/2774907781 shutdown_connections 2024-03-18T14:54:39.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.088+0000 7f49af973740 1 --2- 172.21.15.76:0/2774907781 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807c8820 0x55ac807cac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.088+0000 7f49af973740 1 --2- 172.21.15.76:0/2774907781 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807bfa50 0x55ac807c82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.088+0000 7f49af973740 1 --2- 172.21.15.76:0/2774907781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 0x55ac807bf510 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.088+0000 7f49af973740 1 -- 172.21.15.76:0/2774907781 >> 172.21.15.76:0/2774907781 conn(0x55ac807b9a50 msgr2=0x55ac807b9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:39.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.088+0000 7f49af973740 1 -- 172.21.15.76:0/2774907781 shutdown_connections 2024-03-18T14:54:39.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.088+0000 7f49af973740 1 -- 172.21.15.76:0/2774907781 wait complete. 2024-03-18T14:54:39.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.088+0000 7f49af973740 1 -- start start 2024-03-18T14:54:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.088+0000 7f49af973740 1 Processor -- start 2024-03-18T14:54:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49af973740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 0x55ac807c5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49af973740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807bfa50 0x55ac807c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49af973740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807c8820 0x55ac807c6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49af973740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ac807cb450 con 0x55ac807c8820 2024-03-18T14:54:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49af973740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ac806dfd70 con 0x55ac807bf130 2024-03-18T14:54:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49af973740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ac807cb5d0 con 0x55ac807bfa50 2024-03-18T14:54:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49af973740 10 client.0 _ll_register_callbacks cb 0x55ac80437a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49a6d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807c8820 0x55ac807c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49a659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 0x55ac807c5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49a6d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807c8820 0x55ac807c6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33204/0 (socket says 172.21.15.76:33204) 2024-03-18T14:54:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49a6d9d640 1 -- 172.21.15.76:0/4206063545 learned_addr learned my addr 172.21.15.76:0/4206063545 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49a5d9b640 1 --2- 172.21.15.76:0/4206063545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807bfa50 0x55ac807c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49a6d9d640 1 -- 172.21.15.76:0/4206063545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807bfa50 msgr2=0x55ac807c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49a6d9d640 1 --2- 172.21.15.76:0/4206063545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807bfa50 0x55ac807c5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49a6d9d640 1 -- 172.21.15.76:0/4206063545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 msgr2=0x55ac807c5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49a6d9d640 1 --2- 172.21.15.76:0/4206063545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 0x55ac807c5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49a6d9d640 1 -- 172.21.15.76:0/4206063545 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac80438210 con 0x55ac807c8820 2024-03-18T14:54:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49a659c640 1 --2- 172.21.15.76:0/4206063545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 0x55ac807c5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f49a6d9d640 1 --2- 172.21.15.76:0/4206063545 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807c8820 0x55ac807c6390 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f49a0037040 tx=0x7f49a00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1991f6609544f546 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f498dffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.089+0000 7f498dffb640 1 -- 172.21.15.76:0/4206063545 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0057020 con 0x55ac807c8820 2024-03-18T14:54:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.090+0000 7f498dffb640 1 -- 172.21.15.76:0/4206063545 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f49a004aaf0 con 0x55ac807c8820 2024-03-18T14:54:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.090+0000 7f49af973740 1 -- 172.21.15.76:0/4206063545 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49a0003b70 con 0x55ac807c8820 2024-03-18T14:54:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.090+0000 7f498dffb640 1 -- 172.21.15.76:0/4206063545 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f49a0062050 con 0x55ac807c8820 2024-03-18T14:54:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.090+0000 7f49af973740 10 client.15846 fetch_fsmap learned FSMap version 27 2024-03-18T14:54:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.090+0000 7f49af973740 1 -- 172.21.15.76:0/4206063545 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ac80504620 con 0x55ac807c8820 2024-03-18T14:54:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.090+0000 7f498dffb640 1 -- 172.21.15.76:0/4206063545 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49a0003b70 con 0x55ac807c8820 2024-03-18T14:54:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.090+0000 7f498dffb640 1 -- 172.21.15.76:0/4206063545 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f49a003f040 con 0x55ac807c8820 2024-03-18T14:54:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.090+0000 7f49af973740 10 client.15846 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T14:54:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.090+0000 7f49af973740 10 client.15846 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.090+0000 7f49af973740 20 client.15846 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.090+0000 7f49af973740 1 --2- 172.21.15.76:0/4206063545 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ac807d88f0 0x55ac807dace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.090+0000 7f49af973740 4 client.15846 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:54:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.090+0000 7f49af973740 1 -- 172.21.15.76:0/4206063545 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac80730f80 con 0x55ac807d88f0 2024-03-18T14:54:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.090+0000 7f49a659c640 1 --2- 172.21.15.76:0/4206063545 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ac807d88f0 0x55ac807dace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.091+0000 7f498dffb640 10 client.15846 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.091+0000 7f49a659c640 1 --2- 172.21.15.76:0/4206063545 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ac807d88f0 0x55ac807dace0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d120d854424afa server_cookie=a48ef9573aee4e22 in_seq=0 out_seq=0 2024-03-18T14:54:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.092+0000 7f498dffb640 1 -- 172.21.15.76:0/4206063545 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ac80730f80 con 0x55ac807d88f0 2024-03-18T14:54:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.092+0000 7f498dffb640 10 client.15846 handle_command_reply: tid=0 2024-03-18T14:54:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.092+0000 7f49af973740 1 client.15846 shutdown 2024-03-18T14:54:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.092+0000 7f49af973740 1 -- 172.21.15.76:0/4206063545 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807c8820 msgr2=0x55ac807c6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.092+0000 7f49af973740 1 --2- 172.21.15.76:0/4206063545 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807c8820 0x55ac807c6390 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f49a0037040 tx=0x7f49a00095c0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.092+0000 7f49af973740 1 -- 172.21.15.76:0/4206063545 shutdown_connections 2024-03-18T14:54:39.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.092+0000 7f49af973740 1 --2- 172.21.15.76:0/4206063545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807bfa50 0x55ac807c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.092+0000 7f49af973740 1 --2- 172.21.15.76:0/4206063545 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ac807d88f0 0x55ac807dace0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.092+0000 7f49af973740 1 --2- 172.21.15.76:0/4206063545 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807c8820 0x55ac807c6390 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.092+0000 7f49af973740 1 --2- 172.21.15.76:0/4206063545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 0x55ac807c5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.092+0000 7f498dffb640 0 client.15846 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:39.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.092+0000 7f49af973740 1 -- 172.21.15.76:0/4206063545 >> 172.21.15.76:0/4206063545 conn(0x55ac807b9a50 msgr2=0x55ac807bb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:39.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.092+0000 7f49af973740 1 -- 172.21.15.76:0/4206063545 shutdown_connections 2024-03-18T14:54:39.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.092+0000 7f49af973740 1 -- 172.21.15.76:0/4206063545 wait complete. 2024-03-18T14:54:39.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.092+0000 7f49af973740 20 client.15846 trim_cache size 0 max 16384 2024-03-18T14:54:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.108+0000 7f49af973740 1 Processor -- start 2024-03-18T14:54:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.108+0000 7f49af973740 1 -- start start 2024-03-18T14:54:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.108+0000 7f49af973740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 0x55ac807c8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f49af973740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807bfa50 0x55ac807d41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f49af973740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807c8820 0x55ac807c48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f49af973740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f49a81b1d70 con 0x55ac807c8820 2024-03-18T14:54:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f49af973740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f49a81af5e0 con 0x55ac807bf130 2024-03-18T14:54:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f49af973740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f49a81ad430 con 0x55ac807bfa50 2024-03-18T14:54:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f49a659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 0x55ac807c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f49a6d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807c8820 0x55ac807c48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f49a659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 0x55ac807c8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:42566/0 (socket says 172.21.15.76:42566) 2024-03-18T14:54:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f49a659c640 1 -- 172.21.15.76:0/1872410473 learned_addr learned my addr 172.21.15.76:0/1872410473 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f49a5d9b640 1 --2- 172.21.15.76:0/1872410473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807bfa50 0x55ac807d41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f49a5d9b640 1 -- 172.21.15.76:0/1872410473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 msgr2=0x55ac807c8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f49a5d9b640 1 --2- 172.21.15.76:0/1872410473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 0x55ac807c8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f49a5d9b640 1 -- 172.21.15.76:0/1872410473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807c8820 msgr2=0x55ac807c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f49a5d9b640 1 --2- 172.21.15.76:0/1872410473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807c8820 0x55ac807c48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f49a5d9b640 1 -- 172.21.15.76:0/1872410473 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac807bf850 con 0x55ac807bfa50 2024-03-18T14:54:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f49a659c640 1 --2- 172.21.15.76:0/1872410473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 0x55ac807c8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:39.112 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f49a5d9b640 1 --2- 172.21.15.76:0/1872410473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807bfa50 0x55ac807d41f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f499c0cc9b0 tx=0x7f499c097c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a2eace613f961a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:39.112 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f498ffff640 1 -- 172.21.15.76:0/1872410473 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499c018e60 con 0x55ac807bfa50 2024-03-18T14:54:39.112 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f498ffff640 1 -- 172.21.15.76:0/1872410473 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f499c09b8a0 con 0x55ac807bfa50 2024-03-18T14:54:39.112 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.109+0000 7f498ffff640 1 -- 172.21.15.76:0/1872410473 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499c097f10 con 0x55ac807bfa50 2024-03-18T14:54:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.110+0000 7f49af973740 1 -- 172.21.15.76:0/1872410473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807bfa50 msgr2=0x55ac807d41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.110+0000 7f49af973740 1 --2- 172.21.15.76:0/1872410473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807bfa50 0x55ac807d41f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f499c0cc9b0 tx=0x7f499c097c70 comp rx=0 tx=0).stop 2024-03-18T14:54:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.110+0000 7f49af973740 1 -- 172.21.15.76:0/1872410473 shutdown_connections 2024-03-18T14:54:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.110+0000 7f49af973740 1 --2- 172.21.15.76:0/1872410473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807bfa50 0x55ac807d41f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.110+0000 7f49af973740 1 --2- 172.21.15.76:0/1872410473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807c8820 0x55ac807c48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.110+0000 7f49af973740 1 --2- 172.21.15.76:0/1872410473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807bf130 0x55ac807c8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.110+0000 7f49af973740 1 -- 172.21.15.76:0/1872410473 >> 172.21.15.76:0/1872410473 conn(0x55ac807b9a50 msgr2=0x55ac807c41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.110+0000 7f49af973740 1 -- 172.21.15.76:0/1872410473 shutdown_connections 2024-03-18T14:54:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.110+0000 7f49af973740 1 -- 172.21.15.76:0/1872410473 wait complete. 2024-03-18T14:54:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.110+0000 7f49af973740 1 -- start start 2024-03-18T14:54:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.110+0000 7f49af973740 1 Processor -- start 2024-03-18T14:54:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49af973740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807bf130 0x55ac807d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49af973740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807bfa50 0x55ac807c1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49af973740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807c8820 0x55ac807c20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49af973740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f49a81adaa0 con 0x55ac807bfa50 2024-03-18T14:54:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49af973740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f49a81aef10 con 0x55ac807c8820 2024-03-18T14:54:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49af973740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f49a81ae170 con 0x55ac807bf130 2024-03-18T14:54:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49af973740 10 client.0 _ll_register_callbacks cb 0x7f4998009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49a5d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807bfa50 0x55ac807c1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49a5d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807bfa50 0x55ac807c1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33226/0 (socket says 172.21.15.76:33226) 2024-03-18T14:54:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49a659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807bf130 0x55ac807d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49a6d9d640 1 --2- 172.21.15.76:0/4100942955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807c8820 0x55ac807c20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49a5d9b640 1 -- 172.21.15.76:0/4100942955 learned_addr learned my addr 172.21.15.76:0/4100942955 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:39.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49a5d9b640 1 -- 172.21.15.76:0/4100942955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807bf130 msgr2=0x55ac807d0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:39.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49a5d9b640 1 --2- 172.21.15.76:0/4100942955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807bf130 0x55ac807d0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49a5d9b640 1 -- 172.21.15.76:0/4100942955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807c8820 msgr2=0x55ac807c20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:39.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49a5d9b640 1 --2- 172.21.15.76:0/4100942955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807c8820 0x55ac807c20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49a5d9b640 1 -- 172.21.15.76:0/4100942955 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac807c3700 con 0x55ac807bfa50 2024-03-18T14:54:39.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49a5d9b640 1 --2- 172.21.15.76:0/4100942955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807bfa50 0x55ac807c1b80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f499c098270 tx=0x7f499c099db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8667f8200e086ea9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:39.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f498dffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:39.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f498dffb640 1 -- 172.21.15.76:0/4100942955 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499c0b3020 con 0x55ac807bfa50 2024-03-18T14:54:39.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f498dffb640 1 -- 172.21.15.76:0/4100942955 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f499c0860b0 con 0x55ac807bfa50 2024-03-18T14:54:39.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.111+0000 7f49af973740 1 -- 172.21.15.76:0/4100942955 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f499c09b8a0 con 0x55ac807bfa50 2024-03-18T14:54:39.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.112+0000 7f498dffb640 1 -- 172.21.15.76:0/4100942955 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f499c018e60 con 0x55ac807bfa50 2024-03-18T14:54:39.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.112+0000 7f49af973740 10 client.15852 fetch_fsmap learned FSMap version 27 2024-03-18T14:54:39.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.112+0000 7f49af973740 1 -- 172.21.15.76:0/4100942955 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ac8083e220 con 0x55ac807bfa50 2024-03-18T14:54:39.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.112+0000 7f498dffb640 1 -- 172.21.15.76:0/4100942955 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f499c09b8a0 con 0x55ac807bfa50 2024-03-18T14:54:39.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.112+0000 7f498dffb640 1 -- 172.21.15.76:0/4100942955 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f499c0bb040 con 0x55ac807bfa50 2024-03-18T14:54:39.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.112+0000 7f49af973740 10 client.15852 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T14:54:39.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.112+0000 7f49af973740 10 client.15852 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:39.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.112+0000 7f49af973740 20 client.15852 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:39.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.112+0000 7f49af973740 1 --2- 172.21.15.76:0/4100942955 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ac8083e910 0x55ac80840d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:39.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.112+0000 7f49af973740 4 client.15852 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:54:39.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.112+0000 7f49af973740 1 -- 172.21.15.76:0/4100942955 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ac8072b2d0 con 0x55ac8083e910 2024-03-18T14:54:39.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.112+0000 7f49a659c640 1 --2- 172.21.15.76:0/4100942955 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ac8083e910 0x55ac80840d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:39.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.113+0000 7f498dffb640 10 client.15852 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:39.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.113+0000 7f49a659c640 1 --2- 172.21.15.76:0/4100942955 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ac8083e910 0x55ac80840d30 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bdb9a696d8003d1 server_cookie=291edfc6af6ff05b in_seq=0 out_seq=0 2024-03-18T14:54:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.189+0000 7f498dffb640 1 -- 172.21.15.76:0/4100942955 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ac8072b2d0 con 0x55ac8083e910 2024-03-18T14:54:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.190+0000 7f498dffb640 10 client.15852 handle_command_reply: tid=0 2024-03-18T14:54:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.190+0000 7f49af973740 1 client.15852 shutdown 2024-03-18T14:54:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.190+0000 7f49af973740 1 -- 172.21.15.76:0/4100942955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807bfa50 msgr2=0x55ac807c1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.190+0000 7f49af973740 1 --2- 172.21.15.76:0/4100942955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807bfa50 0x55ac807c1b80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f499c098270 tx=0x7f499c099db0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.190+0000 7f49af973740 1 -- 172.21.15.76:0/4100942955 shutdown_connections 2024-03-18T14:54:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.190+0000 7f49af973740 1 --2- 172.21.15.76:0/4100942955 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ac8083e910 0x55ac80840d30 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.190+0000 7f498dffb640 0 client.15852 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.190+0000 7f49af973740 1 --2- 172.21.15.76:0/4100942955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac807c8820 0x55ac807c20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.190+0000 7f49af973740 1 --2- 172.21.15.76:0/4100942955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ac807bfa50 0x55ac807c1b80 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.190+0000 7f49af973740 1 --2- 172.21.15.76:0/4100942955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ac807bf130 0x55ac807d0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.190+0000 7f49af973740 1 -- 172.21.15.76:0/4100942955 >> 172.21.15.76:0/4100942955 conn(0x55ac807b9a50 msgr2=0x55ac807c4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.190+0000 7f49af973740 1 -- 172.21.15.76:0/4100942955 shutdown_connections 2024-03-18T14:54:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.190+0000 7f49af973740 1 -- 172.21.15.76:0/4100942955 wait complete. 2024-03-18T14:54:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.190+0000 7f49af973740 20 client.15852 trim_cache size 0 max 16384 2024-03-18T14:54:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:54:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:54:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:54:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:54:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.191+0000 7f49acd87640 1 -- 172.21.15.76:0/1867529788 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f49900a2be0 msgr2=0x7f49900a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.191+0000 7f49acd87640 1 --2- 172.21.15.76:0/1867529788 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f49900a2be0 0x7f49900a50a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f49a0005cc0 tx=0x7f49a0045000 comp rx=0 tx=0).stop 2024-03-18T14:54:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.191+0000 7f49acd87640 1 -- 172.21.15.76:0/1867529788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49a81012f0 msgr2=0x7f49a8102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.191+0000 7f49acd87640 1 --2- 172.21.15.76:0/1867529788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49a81012f0 0x7f49a8102890 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f499c00e9f0 tx=0x7f499c00eeb0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.191+0000 7f49acd87640 1 -- 172.21.15.76:0/1867529788 shutdown_connections 2024-03-18T14:54:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.191+0000 7f49acd87640 1 --2- 172.21.15.76:0/1867529788 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f49900a2be0 0x7f49900a50a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.191+0000 7f49acd87640 1 --2- 172.21.15.76:0/1867529788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49a8105680 0x7f49a8102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.191+0000 7f49acd87640 1 --2- 172.21.15.76:0/1867529788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49a81012f0 0x7f49a8102890 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.191+0000 7f49acd87640 1 --2- 172.21.15.76:0/1867529788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49a8100930 0x7f49a8102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.191+0000 7f49acd87640 1 -- 172.21.15.76:0/1867529788 >> 172.21.15.76:0/1867529788 conn(0x7f49a8077530 msgr2=0x7f49a81045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.191+0000 7f49acd87640 1 -- 172.21.15.76:0/1867529788 shutdown_connections 2024-03-18T14:54:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:39.191+0000 7f49acd87640 1 -- 172.21.15.76:0/1867529788 wait complete. 2024-03-18T14:54:39.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 64222 9.13 MB/sec warmup 40 sec latency 30.874 ms 2024-03-18T14:54:39.977 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:54:39 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setuser ceph since I am not root 2024-03-18T14:54:39.977 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:54:39 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setgroup ceph since I am not root 2024-03-18T14:54:39.977 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:54:39 smithi077 ceph-mds[65404]: -- 172.21.15.77:0/3521134704 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561367141a40 con 0x561367115000 2024-03-18T14:54:39.977 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:54:39 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: starting mds.e at 2024-03-18T14:54:40.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 66260 9.08 MB/sec warmup 41 sec latency 14.742 ms 2024-03-18T14:54:40.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:40 smithi076 ceph-mon[32205]: pgmap v45: 97 pgs: 97 active+clean; 38 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 330 op/s 2024-03-18T14:54:40.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:40 smithi076 ceph-mon[32205]: daemon mds.e restarted 2024-03-18T14:54:40.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:40 smithi174 ceph-mon[36787]: pgmap v45: 97 pgs: 97 active+clean; 38 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 330 op/s 2024-03-18T14:54:40.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:40 smithi174 ceph-mon[36787]: daemon mds.e restarted 2024-03-18T14:54:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:40 smithi077 ceph-mon[35865]: pgmap v45: 97 pgs: 97 active+clean; 38 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 330 op/s 2024-03-18T14:54:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:40 smithi077 ceph-mon[35865]: daemon mds.e restarted 2024-03-18T14:54:41.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 68027 9.15 MB/sec warmup 42 sec latency 15.914 ms 2024-03-18T14:54:41.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:41 smithi076 ceph-mon[32205]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T14:54:41.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:41 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.77:6834/4143695203,v1:172.21.15.77:6835/4143695203] up:boot 2024-03-18T14:54:41.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:41 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T14:54:41.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:41 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:54:41.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:41 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:54:41.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:41 smithi174 ceph-mon[36787]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T14:54:41.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:41 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.77:6834/4143695203,v1:172.21.15.77:6835/4143695203] up:boot 2024-03-18T14:54:41.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:41 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T14:54:41.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:41 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:54:41.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:41 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:54:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:41 smithi077 ceph-mon[35865]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T14:54:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:41 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.77:6834/4143695203,v1:172.21.15.77:6835/4143695203] up:boot 2024-03-18T14:54:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:41 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T14:54:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:41 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:54:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:41 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:54:42.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 70968 9.34 MB/sec warmup 43 sec latency 13.860 ms 2024-03-18T14:54:42.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:42 smithi076 ceph-mon[32205]: pgmap v47: 97 pgs: 97 active+clean; 38 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 246 op/s 2024-03-18T14:54:42.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:42 smithi174 ceph-mon[36787]: pgmap v47: 97 pgs: 97 active+clean; 38 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 246 op/s 2024-03-18T14:54:42.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:42 smithi077 ceph-mon[35865]: pgmap v47: 97 pgs: 97 active+clean; 38 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 246 op/s 2024-03-18T14:54:43.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 72348 9.28 MB/sec warmup 44 sec latency 19.496 ms 2024-03-18T14:54:44.329 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T14:54:44.329 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T14:54:44.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.440+0000 7f70ce37e640 1 -- 172.21.15.76:0/3801503326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70c8069e90 msgr2=0x7f70c806a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.440+0000 7f70ce37e640 1 --2- 172.21.15.76:0/3801503326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70c8069e90 0x7f70c806a310 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009930 tx=0x7f70bc02bc30 comp rx=0 tx=0).stop 2024-03-18T14:54:44.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.440+0000 7f70ce37e640 1 -- 172.21.15.76:0/3801503326 shutdown_connections 2024-03-18T14:54:44.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.440+0000 7f70ce37e640 1 --2- 172.21.15.76:0/3801503326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c806a850 0x7f70c810e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.440+0000 7f70ce37e640 1 --2- 172.21.15.76:0/3801503326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70c8069e90 0x7f70c806a310 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.440+0000 7f70ce37e640 1 --2- 172.21.15.76:0/3801503326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70c8107850 0x7f70c8107c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.440+0000 7f70ce37e640 1 -- 172.21.15.76:0/3801503326 >> 172.21.15.76:0/3801503326 conn(0x7f70c80777b0 msgr2=0x7f70c8077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:44.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.440+0000 7f70ce37e640 1 -- 172.21.15.76:0/3801503326 shutdown_connections 2024-03-18T14:54:44.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.440+0000 7f70ce37e640 1 -- 172.21.15.76:0/3801503326 wait complete. 2024-03-18T14:54:44.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.441+0000 7f70ce37e640 1 Processor -- start 2024-03-18T14:54:44.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.441+0000 7f70ce37e640 1 -- start start 2024-03-18T14:54:44.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70ce37e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c8069e90 0x7f70c819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70ce37e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70c806a850 0x7f70c819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70ce37e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70c8107850 0x7f70c819eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70ce37e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f70c8111ba0 con 0x7f70c8107850 2024-03-18T14:54:44.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70ce37e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f70c8111a20 con 0x7f70c8069e90 2024-03-18T14:54:44.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70ce37e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f70c8111d20 con 0x7f70c806a850 2024-03-18T14:54:44.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70cc91b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70c8107850 0x7f70c819eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70c7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c8069e90 0x7f70c819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70c7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c8069e90 0x7f70c819a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:42594/0 (socket says 172.21.15.76:42594) 2024-03-18T14:54:44.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70cc91b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70c8107850 0x7f70c819eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33250/0 (socket says 172.21.15.76:33250) 2024-03-18T14:54:44.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70c7fff640 1 -- 172.21.15.76:0/431439071 learned_addr learned my addr 172.21.15.76:0/431439071 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:44.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70c77fe640 1 --2- 172.21.15.76:0/431439071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70c806a850 0x7f70c819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70c7fff640 1 -- 172.21.15.76:0/431439071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70c806a850 msgr2=0x7f70c819a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70c7fff640 1 --2- 172.21.15.76:0/431439071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70c806a850 0x7f70c819a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70c7fff640 1 -- 172.21.15.76:0/431439071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70c8107850 msgr2=0x7f70c819eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70c7fff640 1 --2- 172.21.15.76:0/431439071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70c8107850 0x7f70c819eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70c7fff640 1 -- 172.21.15.76:0/431439071 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c819f230 con 0x7f70c8069e90 2024-03-18T14:54:44.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70cc91b640 1 --2- 172.21.15.76:0/431439071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70c8107850 0x7f70c819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:44.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70c7fff640 1 --2- 172.21.15.76:0/431439071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c8069e90 0x7f70c819a240 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f70b800e9f0 tx=0x7f70b800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:44.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.442+0000 7f70c57fa640 1 -- 172.21.15.76:0/431439071 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b800cd80 con 0x7f70c8069e90 2024-03-18T14:54:44.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.443+0000 7f70ce37e640 1 -- 172.21.15.76:0/431439071 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70c819f520 con 0x7f70c8069e90 2024-03-18T14:54:44.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.443+0000 7f70c57fa640 1 -- 172.21.15.76:0/431439071 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70b8005500 con 0x7f70c8069e90 2024-03-18T14:54:44.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.443+0000 7f70c57fa640 1 -- 172.21.15.76:0/431439071 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b8010690 con 0x7f70c8069e90 2024-03-18T14:54:44.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.443+0000 7f70ce37e640 1 -- 172.21.15.76:0/431439071 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70c81a6db0 con 0x7f70c8069e90 2024-03-18T14:54:44.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.445+0000 7f70c57fa640 1 -- 172.21.15.76:0/431439071 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f70b80046c0 con 0x7f70c8069e90 2024-03-18T14:54:44.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70d0f6d740 1 Processor -- start 2024-03-18T14:54:44.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70d0f6d740 1 -- start start 2024-03-18T14:54:44.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70d0f6d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220150 0x560c42220530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70d0f6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42220a70 0x560c42229110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70d0f6d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42229650 0x560c4222ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70d0f6d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x560c42184500 con 0x560c42220150 2024-03-18T14:54:44.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70d0f6d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560c4212cbe0 con 0x560c42229650 2024-03-18T14:54:44.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70d0f6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560c421805a0 con 0x560c42220a70 2024-03-18T14:54:44.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70c57fa640 1 --2- 172.21.15.76:0/431439071 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f70b00a2a90 0x7f70b00a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70cc91b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220150 0x560c42220530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70c7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42220a70 0x560c42229110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70c77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42229650 0x560c4222ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70cc91b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220150 0x560c42220530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33254/0 (socket says 172.21.15.76:33254) 2024-03-18T14:54:44.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70cc91b640 1 -- 172.21.15.76:0/4081376285 learned_addr learned my addr 172.21.15.76:0/4081376285 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:44.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70cc91b640 1 --2- 172.21.15.76:0/431439071 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f70b00a2a90 0x7f70b00a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70c57fa640 1 -- 172.21.15.76:0/431439071 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f70b80cf4c0 con 0x7f70c8069e90 2024-03-18T14:54:44.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70c77fe640 1 -- 172.21.15.76:0/4081376285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42220a70 msgr2=0x560c42229110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:44.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70c77fe640 1 --2- 172.21.15.76:0/4081376285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42220a70 0x560c42229110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70c77fe640 1 -- 172.21.15.76:0/4081376285 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220150 msgr2=0x560c42220530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:44.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70c77fe640 1 --2- 172.21.15.76:0/4081376285 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220150 0x560c42220530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.446+0000 7f70c77fe640 1 -- 172.21.15.76:0/4081376285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c421404b0 con 0x560c42229650 2024-03-18T14:54:44.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.447+0000 7f70cc91b640 1 --2- 172.21.15.76:0/431439071 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f70b00a2a90 0x7f70b00a4f50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f70c00099e0 tx=0x7f70c0009500 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:44.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.447+0000 7f70c77fe640 1 --2- 172.21.15.76:0/4081376285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42229650 0x560c4222ba40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009230 tx=0x7f70bc009670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8037ccd59e1b3c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:44.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.447+0000 7f70ae7fc640 1 -- 172.21.15.76:0/4081376285 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc042070 con 0x560c42229650 2024-03-18T14:54:44.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.447+0000 7f70ae7fc640 1 -- 172.21.15.76:0/4081376285 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70bc02c8b0 con 0x560c42229650 2024-03-18T14:54:44.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.447+0000 7f70ae7fc640 1 -- 172.21.15.76:0/4081376285 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc004c00 con 0x560c42229650 2024-03-18T14:54:44.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.447+0000 7f70d0f6d740 1 -- 172.21.15.76:0/4081376285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42229650 msgr2=0x560c4222ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.447+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/4081376285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42229650 0x560c4222ba40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009230 tx=0x7f70bc009670 comp rx=0 tx=0).stop 2024-03-18T14:54:44.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.447+0000 7f70d0f6d740 1 -- 172.21.15.76:0/4081376285 shutdown_connections 2024-03-18T14:54:44.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.447+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/4081376285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42220a70 0x560c42229110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.447+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/4081376285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42229650 0x560c4222ba40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.447+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/4081376285 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220150 0x560c42220530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.447+0000 7f70d0f6d740 1 -- 172.21.15.76:0/4081376285 >> 172.21.15.76:0/4081376285 conn(0x560c4221a880 msgr2=0x560c4221ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:44.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.447+0000 7f70d0f6d740 1 -- 172.21.15.76:0/4081376285 shutdown_connections 2024-03-18T14:54:44.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.447+0000 7f70d0f6d740 1 -- 172.21.15.76:0/4081376285 wait complete. 2024-03-18T14:54:44.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.448+0000 7f70d0f6d740 1 -- start start 2024-03-18T14:54:44.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.448+0000 7f70d0f6d740 1 Processor -- start 2024-03-18T14:54:44.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.448+0000 7f70d0f6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42220150 0x560c422266c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.448+0000 7f70d0f6d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220a70 0x560c42226c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.448+0000 7f70d0f6d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42229650 0x560c42227140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.448+0000 7f70d0f6d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x560c4222c3e0 con 0x560c42220a70 2024-03-18T14:54:44.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.448+0000 7f70d0f6d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560c42140160 con 0x560c42229650 2024-03-18T14:54:44.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.448+0000 7f70d0f6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560c4222c560 con 0x560c42220150 2024-03-18T14:54:44.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.448+0000 7f70d0f6d740 10 client.0 _ll_register_callbacks cb 0x560c42124150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:44.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.448+0000 7f70c77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220a70 0x560c42226c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.448+0000 7f70c77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220a70 0x560c42226c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33266/0 (socket says 172.21.15.76:33266) 2024-03-18T14:54:44.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.448+0000 7f70c77fe640 1 -- 172.21.15.76:0/3778130029 learned_addr learned my addr 172.21.15.76:0/3778130029 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:44.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.448+0000 7f70c7fff640 1 --2- 172.21.15.76:0/3778130029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42220150 0x560c422266c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.448+0000 7f70cc91b640 1 --2- 172.21.15.76:0/3778130029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42229650 0x560c42227140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.449+0000 7f70c77fe640 1 -- 172.21.15.76:0/3778130029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42220150 msgr2=0x560c422266c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:44.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.449+0000 7f70c77fe640 1 --2- 172.21.15.76:0/3778130029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42220150 0x560c422266c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.449+0000 7f70c77fe640 1 -- 172.21.15.76:0/3778130029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42229650 msgr2=0x560c42227140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:44.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.449+0000 7f70c77fe640 1 --2- 172.21.15.76:0/3778130029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42229650 0x560c42227140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.449+0000 7f70c77fe640 1 -- 172.21.15.76:0/3778130029 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c41f627d0 con 0x560c42220a70 2024-03-18T14:54:44.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.449+0000 7f70c77fe640 1 --2- 172.21.15.76:0/3778130029 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220a70 0x560c42226c00 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f70bc02c170 tx=0x7f70bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b33fe8fd8667fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:44.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.449+0000 7f70a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:44.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.449+0000 7f70a77fe640 1 -- 172.21.15.76:0/3778130029 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc04d030 con 0x560c42220a70 2024-03-18T14:54:44.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.449+0000 7f70a77fe640 1 -- 172.21.15.76:0/3778130029 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f70bc004b90 con 0x560c42220a70 2024-03-18T14:54:44.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.449+0000 7f70d0f6d740 1 -- 172.21.15.76:0/3778130029 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70bc02c8b0 con 0x560c42220a70 2024-03-18T14:54:44.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.449+0000 7f70a77fe640 1 -- 172.21.15.76:0/3778130029 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f70bc056020 con 0x560c42220a70 2024-03-18T14:54:44.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.449+0000 7f70d0f6d740 10 client.15870 fetch_fsmap learned FSMap version 29 2024-03-18T14:54:44.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.449+0000 7f70d0f6d740 1 -- 172.21.15.76:0/3778130029 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560c41e983f0 con 0x560c42220a70 2024-03-18T14:54:44.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.449+0000 7f70a77fe640 1 -- 172.21.15.76:0/3778130029 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70bc02c8b0 con 0x560c42220a70 2024-03-18T14:54:44.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.450+0000 7f70a77fe640 1 -- 172.21.15.76:0/3778130029 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f70bc0478a0 con 0x560c42220a70 2024-03-18T14:54:44.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.450+0000 7f70d0f6d740 10 client.15870 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:54:44.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.450+0000 7f70d0f6d740 10 client.15870 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:44.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.450+0000 7f70d0f6d740 20 client.15870 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:44.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.450+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/3778130029 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x560c422394c0 0x560c4223b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.450+0000 7f70d0f6d740 4 client.15870 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:54:44.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.450+0000 7f70d0f6d740 1 -- 172.21.15.76:0/3778130029 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c42191a40 con 0x560c422394c0 2024-03-18T14:54:44.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.450+0000 7f70c7fff640 1 --2- 172.21.15.76:0/3778130029 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x560c422394c0 0x560c4223b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.450+0000 7f70c7fff640 1 --2- 172.21.15.76:0/3778130029 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x560c422394c0 0x560c4223b8b0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cec8999cec9b22fb server_cookie=34639c5e1de0f4 in_seq=0 out_seq=0 2024-03-18T14:54:44.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.450+0000 7f70a77fe640 10 client.15870 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:44.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.451+0000 7f70a77fe640 1 -- 172.21.15.76:0/3778130029 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560c42191a40 con 0x560c422394c0 2024-03-18T14:54:44.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.451+0000 7f70a77fe640 10 client.15870 handle_command_reply: tid=0 2024-03-18T14:54:44.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.451+0000 7f70d0f6d740 1 client.15870 shutdown 2024-03-18T14:54:44.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.451+0000 7f70d0f6d740 1 -- 172.21.15.76:0/3778130029 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220a70 msgr2=0x560c42226c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.451+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/3778130029 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220a70 0x560c42226c00 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f70bc02c170 tx=0x7f70bc0046c0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.451+0000 7f70d0f6d740 1 -- 172.21.15.76:0/3778130029 shutdown_connections 2024-03-18T14:54:44.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.451+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/3778130029 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x560c422394c0 0x560c4223b8b0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.451+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/3778130029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42229650 0x560c42227140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.451+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/3778130029 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220a70 0x560c42226c00 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.451+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/3778130029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42220150 0x560c422266c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.451+0000 7f70d0f6d740 1 -- 172.21.15.76:0/3778130029 >> 172.21.15.76:0/3778130029 conn(0x560c4221a880 msgr2=0x560c4221c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:44.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.451+0000 7f70a77fe640 0 client.15870 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:44.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.452+0000 7f70d0f6d740 1 -- 172.21.15.76:0/3778130029 shutdown_connections 2024-03-18T14:54:44.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.452+0000 7f70d0f6d740 1 -- 172.21.15.76:0/3778130029 wait complete. 2024-03-18T14:54:44.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.452+0000 7f70d0f6d740 20 client.15870 trim_cache size 0 max 16384 2024-03-18T14:54:44.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.470+0000 7f70d0f6d740 1 Processor -- start 2024-03-18T14:54:44.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.470+0000 7f70d0f6d740 1 -- start start 2024-03-18T14:54:44.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.470+0000 7f70d0f6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42220150 0x560c42228df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.470+0000 7f70d0f6d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42220a70 0x560c42225600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.470+0000 7f70d0f6d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42229650 0x560c42235180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.470+0000 7f70d0f6d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f70c81b1d40 con 0x560c42229650 2024-03-18T14:54:44.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.470+0000 7f70d0f6d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f70c81af5b0 con 0x560c42220a70 2024-03-18T14:54:44.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.470+0000 7f70d0f6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f70c81ad400 con 0x560c42220150 2024-03-18T14:54:44.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.470+0000 7f70cc91b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42229650 0x560c42235180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.470+0000 7f70c7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42220150 0x560c42228df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.470+0000 7f70cc91b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42229650 0x560c42235180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33268/0 (socket says 172.21.15.76:33268) 2024-03-18T14:54:44.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.470+0000 7f70cc91b640 1 -- 172.21.15.76:0/6914544 learned_addr learned my addr 172.21.15.76:0/6914544 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:44.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.470+0000 7f70c77fe640 1 --2- 172.21.15.76:0/6914544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42220a70 0x560c42225600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.471+0000 7f70cc91b640 1 -- 172.21.15.76:0/6914544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42220150 msgr2=0x560c42228df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:44.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.471+0000 7f70cc91b640 1 --2- 172.21.15.76:0/6914544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42220150 0x560c42228df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.471+0000 7f70cc91b640 1 -- 172.21.15.76:0/6914544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42220a70 msgr2=0x560c42225600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:44.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.471+0000 7f70cc91b640 1 --2- 172.21.15.76:0/6914544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42220a70 0x560c42225600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.471+0000 7f70cc91b640 1 -- 172.21.15.76:0/6914544 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c41e580f0 con 0x560c42229650 2024-03-18T14:54:44.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.471+0000 7f70c7fff640 1 --2- 172.21.15.76:0/6914544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42220150 0x560c42228df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:44.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.471+0000 7f70c77fe640 1 --2- 172.21.15.76:0/6914544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42220a70 0x560c42225600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:44.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.471+0000 7f70cc91b640 1 --2- 172.21.15.76:0/6914544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42229650 0x560c42235180 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f70c0011700 tx=0x7f70c0008000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0a80b6b8e76cb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:44.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.471+0000 7f70adffb640 1 -- 172.21.15.76:0/6914544 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c002f070 con 0x560c42229650 2024-03-18T14:54:44.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.471+0000 7f70adffb640 1 -- 172.21.15.76:0/6914544 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70c00045b0 con 0x560c42229650 2024-03-18T14:54:44.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.471+0000 7f70adffb640 1 -- 172.21.15.76:0/6914544 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c00206b0 con 0x560c42229650 2024-03-18T14:54:44.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.472+0000 7f70d0f6d740 1 -- 172.21.15.76:0/6914544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42229650 msgr2=0x560c42235180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.472+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/6914544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42229650 0x560c42235180 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f70c0011700 tx=0x7f70c0008000 comp rx=0 tx=0).stop 2024-03-18T14:54:44.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.472+0000 7f70d0f6d740 1 -- 172.21.15.76:0/6914544 shutdown_connections 2024-03-18T14:54:44.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.472+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/6914544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42229650 0x560c42235180 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.472+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/6914544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42220a70 0x560c42225600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.472+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/6914544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42220150 0x560c42228df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.472+0000 7f70d0f6d740 1 -- 172.21.15.76:0/6914544 >> 172.21.15.76:0/6914544 conn(0x560c4221a880 msgr2=0x560c42219900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:44.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.472+0000 7f70d0f6d740 1 -- 172.21.15.76:0/6914544 shutdown_connections 2024-03-18T14:54:44.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.472+0000 7f70d0f6d740 1 -- 172.21.15.76:0/6914544 wait complete. 2024-03-18T14:54:44.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.472+0000 7f70d0f6d740 1 -- start start 2024-03-18T14:54:44.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.472+0000 7f70d0f6d740 1 Processor -- start 2024-03-18T14:54:44.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.472+0000 7f70d0f6d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42220150 0x560c4222d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.472+0000 7f70d0f6d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220a70 0x560c4222d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.472+0000 7f70d0f6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42229650 0x560c4222dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70d0f6d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f70c81ada70 con 0x560c42220a70 2024-03-18T14:54:44.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70d0f6d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f70c81aeee0 con 0x560c42220150 2024-03-18T14:54:44.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70d0f6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f70c81ae140 con 0x560c42229650 2024-03-18T14:54:44.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70d0f6d740 10 client.0 _ll_register_callbacks cb 0x7f70c0011810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:44.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70c77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220a70 0x560c4222d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70c7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42220150 0x560c4222d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70c77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220a70 0x560c4222d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33284/0 (socket says 172.21.15.76:33284) 2024-03-18T14:54:44.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70cc91b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42229650 0x560c4222dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70c77fe640 1 -- 172.21.15.76:0/1531842621 learned_addr learned my addr 172.21.15.76:0/1531842621 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:44.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70c7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42220150 0x560c4222d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:42640/0 (socket says 172.21.15.76:42640) 2024-03-18T14:54:44.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70c77fe640 1 -- 172.21.15.76:0/1531842621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42229650 msgr2=0x560c4222dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:44.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70c77fe640 1 --2- 172.21.15.76:0/1531842621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42229650 0x560c4222dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70c77fe640 1 -- 172.21.15.76:0/1531842621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42220150 msgr2=0x560c4222d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:44.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70c77fe640 1 --2- 172.21.15.76:0/1531842621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42220150 0x560c4222d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70c77fe640 1 -- 172.21.15.76:0/1531842621 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c4214b9d0 con 0x560c42220a70 2024-03-18T14:54:44.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70c7fff640 1 --2- 172.21.15.76:0/1531842621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42220150 0x560c4222d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:44.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70c77fe640 1 --2- 172.21.15.76:0/1531842621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220a70 0x560c4222d5a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x560c42227c90 tx=0x7f70bc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=204fd1082f0a28c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:44.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:44.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70a77fe640 1 -- 172.21.15.76:0/1531842621 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc04d070 con 0x560c42220a70 2024-03-18T14:54:44.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70d0f6d740 1 -- 172.21.15.76:0/1531842621 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70c00045b0 con 0x560c42220a70 2024-03-18T14:54:44.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70a77fe640 1 -- 172.21.15.76:0/1531842621 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f70bc005110 con 0x560c42220a70 2024-03-18T14:54:44.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70a77fe640 1 -- 172.21.15.76:0/1531842621 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f70bc042040 con 0x560c42220a70 2024-03-18T14:54:44.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70d0f6d740 10 client.15876 fetch_fsmap learned FSMap version 29 2024-03-18T14:54:44.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.473+0000 7f70d0f6d740 1 -- 172.21.15.76:0/1531842621 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560c4229f650 con 0x560c42220a70 2024-03-18T14:54:44.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.474+0000 7f70a77fe640 1 -- 172.21.15.76:0/1531842621 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70c00045b0 con 0x560c42220a70 2024-03-18T14:54:44.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.474+0000 7f70a77fe640 1 -- 172.21.15.76:0/1531842621 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f70bc034920 con 0x560c42220a70 2024-03-18T14:54:44.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.474+0000 7f70d0f6d740 10 client.15876 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:54:44.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.474+0000 7f70d0f6d740 10 client.15876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:44.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.474+0000 7f70d0f6d740 20 client.15876 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:44.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.474+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/1531842621 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x560c4229fe00 0x560c422a2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.474+0000 7f70d0f6d740 4 client.15876 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T14:54:44.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.474+0000 7f70d0f6d740 1 -- 172.21.15.76:0/1531842621 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560c4218bd90 con 0x560c4229fe00 2024-03-18T14:54:44.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.474+0000 7f70c7fff640 1 --2- 172.21.15.76:0/1531842621 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x560c4229fe00 0x560c422a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.474+0000 7f70c7fff640 1 --2- 172.21.15.76:0/1531842621 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x560c4229fe00 0x560c422a2220 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bff297c79fce09e5 server_cookie=845cab86ae91bbb1 in_seq=0 out_seq=0 2024-03-18T14:54:44.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.474+0000 7f70a77fe640 10 client.15876 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:44.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70a77fe640 1 -- 172.21.15.76:0/1531842621 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560c4218bd90 con 0x560c4229fe00 2024-03-18T14:54:44.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70a77fe640 10 client.15876 handle_command_reply: tid=0 2024-03-18T14:54:44.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70d0f6d740 1 client.15876 shutdown 2024-03-18T14:54:44.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70d0f6d740 1 -- 172.21.15.76:0/1531842621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220a70 msgr2=0x560c4222d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/1531842621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220a70 0x560c4222d5a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x560c42227c90 tx=0x7f70bc000bc0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70d0f6d740 1 -- 172.21.15.76:0/1531842621 shutdown_connections 2024-03-18T14:54:44.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/1531842621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c42229650 0x560c4222dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/1531842621 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x560c4229fe00 0x560c422a2220 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/1531842621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560c42220a70 0x560c4222d5a0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70d0f6d740 1 --2- 172.21.15.76:0/1531842621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560c42220150 0x560c4222d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70d0f6d740 1 -- 172.21.15.76:0/1531842621 >> 172.21.15.76:0/1531842621 conn(0x560c4221a880 msgr2=0x560c42226660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:44.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70a77fe640 0 client.15876 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:44.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70d0f6d740 1 -- 172.21.15.76:0/1531842621 shutdown_connections 2024-03-18T14:54:44.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70d0f6d740 1 -- 172.21.15.76:0/1531842621 wait complete. 2024-03-18T14:54:44.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70d0f6d740 20 client.15876 trim_cache size 0 max 16384 2024-03-18T14:54:44.483 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T14:54:44.484 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-18T14:54:44.484 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-18T14:54:44.484 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T14:54:44.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70ce37e640 1 -- 172.21.15.76:0/431439071 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f70b00a2a90 msgr2=0x7f70b00a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70ce37e640 1 --2- 172.21.15.76:0/431439071 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f70b00a2a90 0x7f70b00a4f50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f70c00099e0 tx=0x7f70c0009500 comp rx=0 tx=0).stop 2024-03-18T14:54:44.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70ce37e640 1 -- 172.21.15.76:0/431439071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c8069e90 msgr2=0x7f70c819a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.475+0000 7f70ce37e640 1 --2- 172.21.15.76:0/431439071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c8069e90 0x7f70c819a240 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f70b800e9f0 tx=0x7f70b800edf0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.476+0000 7f70ce37e640 1 -- 172.21.15.76:0/431439071 shutdown_connections 2024-03-18T14:54:44.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.476+0000 7f70ce37e640 1 --2- 172.21.15.76:0/431439071 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f70b00a2a90 0x7f70b00a4f50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.476+0000 7f70ce37e640 1 --2- 172.21.15.76:0/431439071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70c806a850 0x7f70c819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.476+0000 7f70ce37e640 1 --2- 172.21.15.76:0/431439071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70c8107850 0x7f70c819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.476+0000 7f70ce37e640 1 --2- 172.21.15.76:0/431439071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c8069e90 0x7f70c819a240 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.476+0000 7f70ce37e640 1 -- 172.21.15.76:0/431439071 >> 172.21.15.76:0/431439071 conn(0x7f70c80777b0 msgr2=0x7f70c80fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:44.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.476+0000 7f70ce37e640 1 -- 172.21.15.76:0/431439071 shutdown_connections 2024-03-18T14:54:44.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.476+0000 7f70ce37e640 1 -- 172.21.15.76:0/431439071 wait complete. 2024-03-18T14:54:44.487 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T14:54:44.487 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:54:44.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.595+0000 7f9a84a0b640 1 -- 172.21.15.76:0/32378370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a80101770 msgr2=0x7f9a80101bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.595+0000 7f9a84a0b640 1 --2- 172.21.15.76:0/32378370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a80101770 0x7f9a80101bf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9a74009930 tx=0x7f9a7402bc30 comp rx=0 tx=0).stop 2024-03-18T14:54:44.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.595+0000 7f9a84a0b640 1 -- 172.21.15.76:0/32378370 shutdown_connections 2024-03-18T14:54:44.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.595+0000 7f9a84a0b640 1 --2- 172.21.15.76:0/32378370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a80101770 0x7f9a80101bf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.595+0000 7f9a84a0b640 1 --2- 172.21.15.76:0/32378370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a80102130 0x7f9a8010e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.595+0000 7f9a84a0b640 1 --2- 172.21.15.76:0/32378370 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a80107780 0x7f9a80107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.595+0000 7f9a84a0b640 1 -- 172.21.15.76:0/32378370 >> 172.21.15.76:0/32378370 conn(0x7f9a800fd280 msgr2=0x7f9a800ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:44.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.595+0000 7f9a84a0b640 1 -- 172.21.15.76:0/32378370 shutdown_connections 2024-03-18T14:54:44.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.595+0000 7f9a84a0b640 1 -- 172.21.15.76:0/32378370 wait complete. 2024-03-18T14:54:44.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a84a0b640 1 Processor -- start 2024-03-18T14:54:44.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a84a0b640 1 -- start start 2024-03-18T14:54:44.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a84a0b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a80101770 0x7f9a801980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a84a0b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a80102130 0x7f9a80198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a84a0b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a80107780 0x7f9a8019c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a84a0b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9a80111b50 con 0x7f9a80102130 2024-03-18T14:54:44.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a84a0b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9a801119d0 con 0x7f9a80107780 2024-03-18T14:54:44.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a84a0b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9a80111cd0 con 0x7f9a80101770 2024-03-18T14:54:44.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a7dd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a80102130 0x7f9a80198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a7dd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a80102130 0x7f9a80198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33300/0 (socket says 172.21.15.76:33300) 2024-03-18T14:54:44.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a7dd9b640 1 -- 172.21.15.76:0/438176479 learned_addr learned my addr 172.21.15.76:0/438176479 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:44.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a7e59c640 1 --2- 172.21.15.76:0/438176479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a80101770 0x7f9a801980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a7ed9d640 1 --2- 172.21.15.76:0/438176479 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a80107780 0x7f9a8019c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a7dd9b640 1 -- 172.21.15.76:0/438176479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a80101770 msgr2=0x7f9a801980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a7dd9b640 1 --2- 172.21.15.76:0/438176479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a80101770 0x7f9a801980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a7dd9b640 1 -- 172.21.15.76:0/438176479 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a80107780 msgr2=0x7f9a8019c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a7dd9b640 1 --2- 172.21.15.76:0/438176479 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a80107780 0x7f9a8019c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.596+0000 7f9a7dd9b640 1 -- 172.21.15.76:0/438176479 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8019d0c0 con 0x7f9a80102130 2024-03-18T14:54:44.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.597+0000 7f9a7dd9b640 1 --2- 172.21.15.76:0/438176479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a80102130 0x7f9a80198610 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9a74009230 tx=0x7f9a7402cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:44.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.597+0000 7f9a6f7fe640 1 -- 172.21.15.76:0/438176479 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74041070 con 0x7f9a80102130 2024-03-18T14:54:44.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.597+0000 7f9a6f7fe640 1 -- 172.21.15.76:0/438176479 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a74004450 con 0x7f9a80102130 2024-03-18T14:54:44.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.597+0000 7f9a84a0b640 1 -- 172.21.15.76:0/438176479 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a8019d350 con 0x7f9a80102130 2024-03-18T14:54:44.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.597+0000 7f9a6f7fe640 1 -- 172.21.15.76:0/438176479 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74003de0 con 0x7f9a80102130 2024-03-18T14:54:44.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.597+0000 7f9a84a0b640 1 -- 172.21.15.76:0/438176479 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a801a4c40 con 0x7f9a80102130 2024-03-18T14:54:44.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.598+0000 7f9a84a0b640 1 -- 172.21.15.76:0/438176479 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a80068a20 con 0x7f9a80102130 2024-03-18T14:54:44.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.599+0000 7f9a6f7fe640 1 -- 172.21.15.76:0/438176479 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f9a74044070 con 0x7f9a80102130 2024-03-18T14:54:44.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.599+0000 7f9a6f7fe640 1 --2- 172.21.15.76:0/438176479 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f9a680a2a90 0x7f9a680a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.599+0000 7f9a7e59c640 1 --2- 172.21.15.76:0/438176479 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f9a680a2a90 0x7f9a680a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.599+0000 7f9a6f7fe640 1 -- 172.21.15.76:0/438176479 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f9a740f0900 con 0x7f9a80102130 2024-03-18T14:54:44.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.600+0000 7f9a7e59c640 1 --2- 172.21.15.76:0/438176479 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f9a680a2a90 0x7f9a680a4f50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9a700095f0 tx=0x7f9a70005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:44.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.604+0000 7f9a6f7fe640 1 -- 172.21.15.76:0/438176479 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9a740bd570 con 0x7f9a80102130 2024-03-18T14:54:44.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 74848 9.38 MB/sec warmup 45 sec latency 15.288 ms 2024-03-18T14:54:44.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.836+0000 7f9a84a0b640 1 -- 172.21.15.76:0/438176479 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a80199420 con 0x7f9a80102130 2024-03-18T14:54:44.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.836+0000 7f9a6f7fe640 1 -- 172.21.15.76:0/438176479 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11868 (secure 0 0 0) 0x7f9a740bce70 con 0x7f9a80102130 2024-03-18T14:54:44.837 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T14:54:44.837 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/2801069369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":2801069369},{"type":"v1","addr":"172.21.15.76:6839","nonce":2801069369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2645229232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2645229232},{"type":"v1","addr":"172.21.15.76:6841","nonce":2645229232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T14:51:01.293009+0000","modified":"2024-03-18T14:54:40.637881+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24490,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":56,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":55,"addr":"172.21.15.174:6839/399810652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":399810652},{"type":"v1","addr":"172.21.15.174:6839","nonce":399810652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1685041985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1685041985},{"type":"v1","addr":"172.21.15.77:6839","nonce":1685041985}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25715":{"gid":25715,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/4143695203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4143695203},{"type":"v1","addr":"172.21.15.77:6835","nonce":4143695203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24463,24490,24524]},"id":1}]} 2024-03-18T14:54:44.837 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 29 2024-03-18T14:54:44.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.838+0000 7f9a84a0b640 1 -- 172.21.15.76:0/438176479 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f9a680a2a90 msgr2=0x7f9a680a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.838+0000 7f9a84a0b640 1 --2- 172.21.15.76:0/438176479 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f9a680a2a90 0x7f9a680a4f50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9a700095f0 tx=0x7f9a70005f90 comp rx=0 tx=0).stop 2024-03-18T14:54:44.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.838+0000 7f9a84a0b640 1 -- 172.21.15.76:0/438176479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a80102130 msgr2=0x7f9a80198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.838+0000 7f9a84a0b640 1 --2- 172.21.15.76:0/438176479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a80102130 0x7f9a80198610 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9a74009230 tx=0x7f9a7402cb30 comp rx=0 tx=0).stop 2024-03-18T14:54:44.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.838+0000 7f9a84a0b640 1 -- 172.21.15.76:0/438176479 shutdown_connections 2024-03-18T14:54:44.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.838+0000 7f9a84a0b640 1 --2- 172.21.15.76:0/438176479 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f9a680a2a90 0x7f9a680a4f50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.838+0000 7f9a84a0b640 1 --2- 172.21.15.76:0/438176479 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a80107780 0x7f9a8019c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.838+0000 7f9a84a0b640 1 --2- 172.21.15.76:0/438176479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a80102130 0x7f9a80198610 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.838+0000 7f9a84a0b640 1 --2- 172.21.15.76:0/438176479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a80101770 0x7f9a801980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.838+0000 7f9a84a0b640 1 -- 172.21.15.76:0/438176479 >> 172.21.15.76:0/438176479 conn(0x7f9a800fd280 msgr2=0x7f9a800fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:44.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.838+0000 7f9a84a0b640 1 -- 172.21.15.76:0/438176479 shutdown_connections 2024-03-18T14:54:44.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.839+0000 7f9a84a0b640 1 -- 172.21.15.76:0/438176479 wait complete. 2024-03-18T14:54:44.850 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T14:54:44.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:44 smithi076 ceph-mon[32205]: pgmap v48: 97 pgs: 97 active+clean; 36 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 299 op/s 2024-03-18T14:54:44.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.961+0000 7fa88e95b640 1 -- 172.21.15.76:0/4210310042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa888101420 msgr2=0x7fa888101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.961+0000 7fa88e95b640 1 --2- 172.21.15.76:0/4210310042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa888101420 0x7fa888101800 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fa878009900 tx=0x7fa87802bc30 comp rx=0 tx=0).stop 2024-03-18T14:54:44.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.961+0000 7fa88e95b640 1 -- 172.21.15.76:0/4210310042 shutdown_connections 2024-03-18T14:54:44.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.961+0000 7fa88e95b640 1 --2- 172.21.15.76:0/4210310042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa888105aa0 0x7fa88810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.961+0000 7fa88e95b640 1 --2- 172.21.15.76:0/4210310042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa888101dd0 0x7fa888105370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.961+0000 7fa88e95b640 1 --2- 172.21.15.76:0/4210310042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa888101420 0x7fa888101800 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.961+0000 7fa88e95b640 1 -- 172.21.15.76:0/4210310042 >> 172.21.15.76:0/4210310042 conn(0x7fa888075a20 msgr2=0x7fa888075e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:44.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.961+0000 7fa88e95b640 1 -- 172.21.15.76:0/4210310042 shutdown_connections 2024-03-18T14:54:44.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.961+0000 7fa88e95b640 1 -- 172.21.15.76:0/4210310042 wait complete. 2024-03-18T14:54:44.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.961+0000 7fa88e95b640 1 Processor -- start 2024-03-18T14:54:44.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.961+0000 7fa88e95b640 1 -- start start 2024-03-18T14:54:44.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa88e95b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa888101420 0x7fa8881980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa88e95b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa888105aa0 0x7fa888198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa88e95b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa88819c940 0x7fa88819cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa88e95b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa888111c10 con 0x7fa88819c940 2024-03-18T14:54:44.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa88e95b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa888111a90 con 0x7fa888101420 2024-03-18T14:54:44.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa88e95b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa888111d90 con 0x7fa888105aa0 2024-03-18T14:54:44.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa88cef8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa88819c940 0x7fa88819cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa887fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa888101420 0x7fa8881980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa88cef8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa88819c940 0x7fa88819cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33322/0 (socket says 172.21.15.76:33322) 2024-03-18T14:54:44.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa88cef8640 1 -- 172.21.15.76:0/1105385720 learned_addr learned my addr 172.21.15.76:0/1105385720 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:44.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa8877fe640 1 --2- 172.21.15.76:0/1105385720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa888105aa0 0x7fa888198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa88cef8640 1 -- 172.21.15.76:0/1105385720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa888105aa0 msgr2=0x7fa888198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa88cef8640 1 --2- 172.21.15.76:0/1105385720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa888105aa0 0x7fa888198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa88cef8640 1 -- 172.21.15.76:0/1105385720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa888101420 msgr2=0x7fa8881980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa88cef8640 1 --2- 172.21.15.76:0/1105385720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa888101420 0x7fa8881980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa88cef8640 1 -- 172.21.15.76:0/1105385720 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa88819d430 con 0x7fa88819c940 2024-03-18T14:54:44.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa887fff640 1 --2- 172.21.15.76:0/1105385720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa888101420 0x7fa8881980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:44.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa88cef8640 1 --2- 172.21.15.76:0/1105385720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa88819c940 0x7fa88819cdf0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fa88000eb10 tx=0x7fa88000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:44.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa8857fa640 1 -- 172.21.15.76:0/1105385720 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880004b20 con 0x7fa88819c940 2024-03-18T14:54:44.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa8857fa640 1 -- 172.21.15.76:0/1105385720 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa880004cc0 con 0x7fa88819c940 2024-03-18T14:54:44.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa8857fa640 1 -- 172.21.15.76:0/1105385720 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880010760 con 0x7fa88819c940 2024-03-18T14:54:44.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.962+0000 7fa88e95b640 1 -- 172.21.15.76:0/1105385720 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa88819d720 con 0x7fa88819c940 2024-03-18T14:54:44.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.963+0000 7fa88e95b640 1 -- 172.21.15.76:0/1105385720 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8881a4f10 con 0x7fa88819c940 2024-03-18T14:54:44.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:44 smithi174 ceph-mon[36787]: pgmap v48: 97 pgs: 97 active+clean; 36 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 299 op/s 2024-03-18T14:54:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.965+0000 7fa8857fa640 1 -- 172.21.15.76:0/1105385720 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7fa880005020 con 0x7fa88819c940 2024-03-18T14:54:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.965+0000 7fa891624740 1 Processor -- start 2024-03-18T14:54:44.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.965+0000 7fa891624740 1 -- start start 2024-03-18T14:54:44.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.965+0000 7fa891624740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06eb830 0x55bfb06ebc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.965+0000 7fa891624740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06ec150 0x55bfb06f47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.965+0000 7fa8857fa640 1 --2- 172.21.15.76:0/1105385720 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa8700a2a90 0x7fa8700a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.965+0000 7fa891624740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06f4d30 0x55bfb06f7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.965+0000 7fa891624740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55bfb064f490 con 0x55bfb06ec150 2024-03-18T14:54:44.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.965+0000 7fa891624740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bfb05f7c90 con 0x55bfb06f4d30 2024-03-18T14:54:44.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.965+0000 7fa891624740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bfb064b530 con 0x55bfb06eb830 2024-03-18T14:54:44.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.965+0000 7fa887fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06ec150 0x55bfb06f47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.965+0000 7fa887fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06ec150 0x55bfb06f47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33332/0 (socket says 172.21.15.76:33332) 2024-03-18T14:54:44.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.965+0000 7fa8877fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06eb830 0x55bfb06ebc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.966+0000 7fa887fff640 1 -- 172.21.15.76:0/3767872042 learned_addr learned my addr 172.21.15.76:0/3767872042 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:44.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.966+0000 7fa88cef8640 1 --2- 172.21.15.76:0/3767872042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06f4d30 0x55bfb06f7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.966+0000 7fa8877fe640 1 --2- 172.21.15.76:0/1105385720 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa8700a2a90 0x7fa8700a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.966+0000 7fa8857fa640 1 -- 172.21.15.76:0/1105385720 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fa8800cf7a0 con 0x7fa88819c940 2024-03-18T14:54:44.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.966+0000 7fa887fff640 1 -- 172.21.15.76:0/3767872042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06eb830 msgr2=0x55bfb06ebc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:44.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.966+0000 7fa887fff640 1 --2- 172.21.15.76:0/3767872042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06eb830 0x55bfb06ebc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.966+0000 7fa887fff640 1 -- 172.21.15.76:0/3767872042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06f4d30 msgr2=0x55bfb06f7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:44.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.966+0000 7fa887fff640 1 --2- 172.21.15.76:0/3767872042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06f4d30 0x55bfb06f7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.966+0000 7fa887fff640 1 -- 172.21.15.76:0/3767872042 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfb06b2500 con 0x55bfb06ec150 2024-03-18T14:54:44.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.966+0000 7fa88cef8640 1 --2- 172.21.15.76:0/3767872042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06f4d30 0x55bfb06f7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:44.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.966+0000 7fa8877fe640 1 --2- 172.21.15.76:0/1105385720 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa8700a2a90 0x7fa8700a4f50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa87c009730 tx=0x7fa87c0097f0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:44.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.966+0000 7fa887fff640 1 --2- 172.21.15.76:0/3767872042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06ec150 0x55bfb06f47f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa878009230 tx=0x7fa878009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7247a8f3b42d2e5a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:44.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.966+0000 7fa86e7fc640 1 -- 172.21.15.76:0/3767872042 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa878045070 con 0x55bfb06ec150 2024-03-18T14:54:44.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.966+0000 7fa86e7fc640 1 -- 172.21.15.76:0/3767872042 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa87802c650 con 0x55bfb06ec150 2024-03-18T14:54:44.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.966+0000 7fa86e7fc640 1 -- 172.21.15.76:0/3767872042 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa878004f20 con 0x55bfb06ec150 2024-03-18T14:54:44.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.967+0000 7fa891624740 1 -- 172.21.15.76:0/3767872042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06ec150 msgr2=0x55bfb06f47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.967+0000 7fa891624740 1 --2- 172.21.15.76:0/3767872042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06ec150 0x55bfb06f47f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa878009230 tx=0x7fa878009560 comp rx=0 tx=0).stop 2024-03-18T14:54:44.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.967+0000 7fa891624740 1 -- 172.21.15.76:0/3767872042 shutdown_connections 2024-03-18T14:54:44.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.967+0000 7fa891624740 1 --2- 172.21.15.76:0/3767872042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06f4d30 0x55bfb06f7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.967+0000 7fa891624740 1 --2- 172.21.15.76:0/3767872042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06ec150 0x55bfb06f47f0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.967+0000 7fa891624740 1 --2- 172.21.15.76:0/3767872042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06eb830 0x55bfb06ebc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.967+0000 7fa891624740 1 -- 172.21.15.76:0/3767872042 >> 172.21.15.76:0/3767872042 conn(0x55bfb06e5f60 msgr2=0x55bfb06e6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:44.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.967+0000 7fa891624740 1 -- 172.21.15.76:0/3767872042 shutdown_connections 2024-03-18T14:54:44.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.967+0000 7fa891624740 1 -- 172.21.15.76:0/3767872042 wait complete. 2024-03-18T14:54:44.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.967+0000 7fa891624740 1 -- start start 2024-03-18T14:54:44.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.967+0000 7fa891624740 1 Processor -- start 2024-03-18T14:54:44.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa891624740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06eb830 0x55bfb06f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa891624740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06ec150 0x55bfb06f22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa891624740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06f4d30 0x55bfb06f2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa891624740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55bfb06f7ac0 con 0x55bfb06eb830 2024-03-18T14:54:44.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa891624740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bfb060b230 con 0x55bfb06f4d30 2024-03-18T14:54:44.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa891624740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bfb06f7c40 con 0x55bfb06ec150 2024-03-18T14:54:44.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa891624740 10 client.0 _ll_register_callbacks cb 0x55bfb0647010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:44.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa887fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06eb830 0x55bfb06f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa887fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06eb830 0x55bfb06f1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33344/0 (socket says 172.21.15.76:33344) 2024-03-18T14:54:44.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa887fff640 1 -- 172.21.15.76:0/4237864045 learned_addr learned my addr 172.21.15.76:0/4237864045 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:44.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa88cef8640 1 --2- 172.21.15.76:0/4237864045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06f4d30 0x55bfb06f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa8877fe640 1 --2- 172.21.15.76:0/4237864045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06ec150 0x55bfb06f22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa887fff640 1 -- 172.21.15.76:0/4237864045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06ec150 msgr2=0x55bfb06f22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:44.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa887fff640 1 --2- 172.21.15.76:0/4237864045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06ec150 0x55bfb06f22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa887fff640 1 -- 172.21.15.76:0/4237864045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06f4d30 msgr2=0x55bfb06f2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:44.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa887fff640 1 --2- 172.21.15.76:0/4237864045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06f4d30 0x55bfb06f2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa887fff640 1 -- 172.21.15.76:0/4237864045 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfb061a100 con 0x55bfb06eb830 2024-03-18T14:54:44.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa88cef8640 1 --2- 172.21.15.76:0/4237864045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06f4d30 0x55bfb06f2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:44.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa887fff640 1 --2- 172.21.15.76:0/4237864045 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06eb830 0x55bfb06f1d90 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa87802c240 tx=0x7fa878004d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dacecb8511f9965 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:44.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa8677fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:44.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa8677fe640 1 -- 172.21.15.76:0/4237864045 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa878045040 con 0x55bfb06eb830 2024-03-18T14:54:44.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa8677fe640 1 -- 172.21.15.76:0/4237864045 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fa878005330 con 0x55bfb06eb830 2024-03-18T14:54:44.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.968+0000 7fa891624740 1 -- 172.21.15.76:0/4237864045 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa87802c650 con 0x55bfb06eb830 2024-03-18T14:54:44.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.969+0000 7fa8677fe640 1 -- 172.21.15.76:0/4237864045 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa87805a050 con 0x55bfb06eb830 2024-03-18T14:54:44.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.969+0000 7fa891624740 10 client.15894 fetch_fsmap learned FSMap version 29 2024-03-18T14:54:44.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.969+0000 7fa891624740 1 -- 172.21.15.76:0/4237864045 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bfb03633f0 con 0x55bfb06eb830 2024-03-18T14:54:44.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.969+0000 7fa8677fe640 1 -- 172.21.15.76:0/4237864045 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa87802c650 con 0x55bfb06eb830 2024-03-18T14:54:44.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.969+0000 7fa8677fe640 1 -- 172.21.15.76:0/4237864045 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fa87803f440 con 0x55bfb06eb830 2024-03-18T14:54:44.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.969+0000 7fa891624740 10 client.15894 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:54:44.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.969+0000 7fa891624740 10 client.15894 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T14:54:44.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.969+0000 7fa891624740 20 client.15894 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:44.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.969+0000 7fa891624740 1 --2- 172.21.15.76:0/4237864045 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55bfb0704f10 0x55bfb0707300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.969+0000 7fa891624740 4 client.15894 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:54:44.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.969+0000 7fa891624740 1 -- 172.21.15.76:0/4237864045 --> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfb065c9d0 con 0x55bfb0704f10 2024-03-18T14:54:44.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.969+0000 7fa8877fe640 1 --2- 172.21.15.76:0/4237864045 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55bfb0704f10 0x55bfb0707300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.970+0000 7fa8677fe640 10 client.15894 ms_handle_connect on v2:172.21.15.174:6836/239386414 2024-03-18T14:54:44.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.970+0000 7fa8877fe640 1 --2- 172.21.15.76:0/4237864045 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55bfb0704f10 0x55bfb0707300 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3ce240976074f40b server_cookie=1c8978dd9896e919 in_seq=0 out_seq=0 2024-03-18T14:54:44.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.970+0000 7fa8677fe640 1 -- 172.21.15.76:0/4237864045 <== mds.2 v2:172.21.15.174:6836/239386414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bfb065c9d0 con 0x55bfb0704f10 2024-03-18T14:54:44.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.970+0000 7fa8677fe640 10 client.15894 handle_command_reply: tid=0 2024-03-18T14:54:44.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.970+0000 7fa891624740 1 client.15894 shutdown 2024-03-18T14:54:44.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.970+0000 7fa891624740 1 -- 172.21.15.76:0/4237864045 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06eb830 msgr2=0x55bfb06f1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.970+0000 7fa891624740 1 --2- 172.21.15.76:0/4237864045 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06eb830 0x55bfb06f1d90 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa87802c240 tx=0x7fa878004d20 comp rx=0 tx=0).stop 2024-03-18T14:54:44.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.970+0000 7fa891624740 1 -- 172.21.15.76:0/4237864045 shutdown_connections 2024-03-18T14:54:44.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.970+0000 7fa891624740 1 --2- 172.21.15.76:0/4237864045 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55bfb0704f10 0x55bfb0707300 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.970+0000 7fa891624740 1 --2- 172.21.15.76:0/4237864045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06ec150 0x55bfb06f22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.970+0000 7fa891624740 1 --2- 172.21.15.76:0/4237864045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06f4d30 0x55bfb06f2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.970+0000 7fa8677fe640 0 client.15894 ms_handle_reset on v2:172.21.15.174:6836/239386414 2024-03-18T14:54:44.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.970+0000 7fa891624740 1 --2- 172.21.15.76:0/4237864045 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06eb830 0x55bfb06f1d90 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.970+0000 7fa891624740 1 -- 172.21.15.76:0/4237864045 >> 172.21.15.76:0/4237864045 conn(0x55bfb06e5f60 msgr2=0x55bfb06e7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:44.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.971+0000 7fa891624740 1 -- 172.21.15.76:0/4237864045 shutdown_connections 2024-03-18T14:54:44.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.971+0000 7fa891624740 1 -- 172.21.15.76:0/4237864045 wait complete. 2024-03-18T14:54:44.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.971+0000 7fa891624740 20 client.15894 trim_cache size 0 max 16384 2024-03-18T14:54:44.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:44 smithi077 ceph-mon[35865]: pgmap v48: 97 pgs: 97 active+clean; 36 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 299 op/s 2024-03-18T14:54:44.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa891624740 1 Processor -- start 2024-03-18T14:54:44.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa891624740 1 -- start start 2024-03-18T14:54:44.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa891624740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06eb830 0x55bfb06f44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa891624740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06ec150 0x55bfb0700860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa891624740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06f4d30 0x55bfb06f0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa891624740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa8881afd10 con 0x55bfb06ec150 2024-03-18T14:54:44.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa891624740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa8881ad580 con 0x55bfb06f4d30 2024-03-18T14:54:44.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa891624740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa8881ab3d0 con 0x55bfb06eb830 2024-03-18T14:54:44.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa8877fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06ec150 0x55bfb0700860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa8877fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06ec150 0x55bfb0700860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33346/0 (socket says 172.21.15.76:33346) 2024-03-18T14:54:44.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa8877fe640 1 -- 172.21.15.76:0/2353919572 learned_addr learned my addr 172.21.15.76:0/2353919572 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:44.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa887fff640 1 --2- 172.21.15.76:0/2353919572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06eb830 0x55bfb06f44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa88cef8640 1 --2- 172.21.15.76:0/2353919572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06f4d30 0x55bfb06f0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa8877fe640 1 -- 172.21.15.76:0/2353919572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06eb830 msgr2=0x55bfb06f44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:44.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa8877fe640 1 --2- 172.21.15.76:0/2353919572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06eb830 0x55bfb06f44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa8877fe640 1 -- 172.21.15.76:0/2353919572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06f4d30 msgr2=0x55bfb06f0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:44.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa8877fe640 1 --2- 172.21.15.76:0/2353919572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06f4d30 0x55bfb06f0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa8877fe640 1 -- 172.21.15.76:0/2353919572 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfb042d7d0 con 0x55bfb06ec150 2024-03-18T14:54:44.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.985+0000 7fa8877fe640 1 --2- 172.21.15.76:0/2353919572 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06ec150 0x55bfb0700860 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fa87c011610 tx=0x7fa87c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faac14e984755b42 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:44.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.986+0000 7fa86dffb640 1 -- 172.21.15.76:0/2353919572 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c02dba0 con 0x55bfb06ec150 2024-03-18T14:54:44.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.986+0000 7fa86dffb640 1 -- 172.21.15.76:0/2353919572 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa87c02dd40 con 0x55bfb06ec150 2024-03-18T14:54:44.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.986+0000 7fa86dffb640 1 -- 172.21.15.76:0/2353919572 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c004730 con 0x55bfb06ec150 2024-03-18T14:54:44.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.986+0000 7fa891624740 1 -- 172.21.15.76:0/2353919572 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06ec150 msgr2=0x55bfb0700860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.986+0000 7fa891624740 1 --2- 172.21.15.76:0/2353919572 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06ec150 0x55bfb0700860 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fa87c011610 tx=0x7fa87c01d000 comp rx=0 tx=0).stop 2024-03-18T14:54:44.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.986+0000 7fa891624740 1 -- 172.21.15.76:0/2353919572 shutdown_connections 2024-03-18T14:54:44.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.986+0000 7fa891624740 1 --2- 172.21.15.76:0/2353919572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06f4d30 0x55bfb06f0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.986+0000 7fa891624740 1 --2- 172.21.15.76:0/2353919572 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06ec150 0x55bfb0700860 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.986+0000 7fa891624740 1 --2- 172.21.15.76:0/2353919572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06eb830 0x55bfb06f44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.986+0000 7fa891624740 1 -- 172.21.15.76:0/2353919572 >> 172.21.15.76:0/2353919572 conn(0x55bfb06e5f60 msgr2=0x55bfb06f0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:44.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.986+0000 7fa891624740 1 -- 172.21.15.76:0/2353919572 shutdown_connections 2024-03-18T14:54:44.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.986+0000 7fa891624740 1 -- 172.21.15.76:0/2353919572 wait complete. 2024-03-18T14:54:44.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.987+0000 7fa891624740 1 -- start start 2024-03-18T14:54:44.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.987+0000 7fa891624740 1 Processor -- start 2024-03-18T14:54:44.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.987+0000 7fa891624740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06eb830 0x55bfb06f80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.987+0000 7fa891624740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06ec150 0x55bfb06f85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.987+0000 7fa891624740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06f4d30 0x55bfb0762fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.987+0000 7fa891624740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa8881aba40 con 0x55bfb06f4d30 2024-03-18T14:54:44.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.987+0000 7fa891624740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa8881aceb0 con 0x55bfb06ec150 2024-03-18T14:54:44.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.987+0000 7fa891624740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa8881ac110 con 0x55bfb06eb830 2024-03-18T14:54:44.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.987+0000 7fa891624740 10 client.0 _ll_register_callbacks cb 0x7fa87c009b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:44.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.987+0000 7fa8877fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06ec150 0x55bfb06f85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.987+0000 7fa887fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06eb830 0x55bfb06f80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.987+0000 7fa88cef8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06f4d30 0x55bfb0762fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.987+0000 7fa8877fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06ec150 0x55bfb06f85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:42718/0 (socket says 172.21.15.76:42718) 2024-03-18T14:54:44.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.987+0000 7fa8877fe640 1 -- 172.21.15.76:0/2814483847 learned_addr learned my addr 172.21.15.76:0/2814483847 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:44.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.988+0000 7fa8877fe640 1 -- 172.21.15.76:0/2814483847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06eb830 msgr2=0x55bfb06f80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:44.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.988+0000 7fa8877fe640 1 --2- 172.21.15.76:0/2814483847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06eb830 0x55bfb06f80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.988+0000 7fa8877fe640 1 -- 172.21.15.76:0/2814483847 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06f4d30 msgr2=0x55bfb0762fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:44.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.988+0000 7fa8877fe640 1 --2- 172.21.15.76:0/2814483847 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06f4d30 0x55bfb0762fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.988+0000 7fa8877fe640 1 -- 172.21.15.76:0/2814483847 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfb0763610 con 0x55bfb06ec150 2024-03-18T14:54:44.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.988+0000 7fa8877fe640 1 --2- 172.21.15.76:0/2814483847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06ec150 0x55bfb06f85e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa87c00f800 tx=0x7fa87c02df10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78c0f1cea38496f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:44.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.988+0000 7fa8677fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:54:44.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.989+0000 7fa8677fe640 1 -- 172.21.15.76:0/2814483847 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c03c030 con 0x55bfb06ec150 2024-03-18T14:54:44.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.989+0000 7fa8677fe640 1 -- 172.21.15.76:0/2814483847 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fa87c032040 con 0x55bfb06ec150 2024-03-18T14:54:44.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.989+0000 7fa891624740 1 -- 172.21.15.76:0/2814483847 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa87c02dd40 con 0x55bfb06ec150 2024-03-18T14:54:44.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.989+0000 7fa8677fe640 1 -- 172.21.15.76:0/2814483847 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa87c046050 con 0x55bfb06ec150 2024-03-18T14:54:44.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.989+0000 7fa891624740 10 client.25757 fetch_fsmap learned FSMap version 29 2024-03-18T14:54:44.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.989+0000 7fa891624740 1 -- 172.21.15.76:0/2814483847 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bfb076a9c0 con 0x55bfb06ec150 2024-03-18T14:54:44.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.989+0000 7fa8677fe640 1 -- 172.21.15.76:0/2814483847 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa87c02dd40 con 0x55bfb06ec150 2024-03-18T14:54:44.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.989+0000 7fa8677fe640 1 -- 172.21.15.76:0/2814483847 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fa87c0046c0 con 0x55bfb06ec150 2024-03-18T14:54:44.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.989+0000 7fa891624740 10 client.25757 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:54:44.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.989+0000 7fa891624740 10 client.25757 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T14:54:44.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.989+0000 7fa891624740 20 client.25757 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:44.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.989+0000 7fa891624740 1 --2- 172.21.15.76:0/2814483847 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55bfb0704f10 0x55bfb076d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:44.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.989+0000 7fa891624740 4 client.25757 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T14:54:44.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.989+0000 7fa891624740 1 -- 172.21.15.76:0/2814483847 --> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bfb0656d20 con 0x55bfb0704f10 2024-03-18T14:54:44.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.990+0000 7fa887fff640 1 --2- 172.21.15.76:0/2814483847 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55bfb0704f10 0x55bfb076d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:44.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.990+0000 7fa8677fe640 10 client.25757 ms_handle_connect on v2:172.21.15.174:6836/239386414 2024-03-18T14:54:44.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.990+0000 7fa887fff640 1 --2- 172.21.15.76:0/2814483847 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55bfb0704f10 0x55bfb076d140 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=12b1df27ec9c4e64 server_cookie=f010c3c5f90d008 in_seq=0 out_seq=0 2024-03-18T14:54:44.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.991+0000 7fa8677fe640 1 -- 172.21.15.76:0/2814483847 <== mds.2 v2:172.21.15.174:6836/239386414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+4367 (crc 0 0 0) 0x55bfb0656d20 con 0x55bfb0704f10 2024-03-18T14:54:44.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.991+0000 7fa8677fe640 10 client.25757 handle_command_reply: tid=0 2024-03-18T14:54:44.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.991+0000 7fa891624740 1 client.25757 shutdown 2024-03-18T14:54:44.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.991+0000 7fa891624740 1 -- 172.21.15.76:0/2814483847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06ec150 msgr2=0x55bfb06f85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:44.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.991+0000 7fa891624740 1 --2- 172.21.15.76:0/2814483847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06ec150 0x55bfb06f85e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa87c00f800 tx=0x7fa87c02df10 comp rx=0 tx=0).stop 2024-03-18T14:54:44.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.991+0000 7fa891624740 1 -- 172.21.15.76:0/2814483847 shutdown_connections 2024-03-18T14:54:44.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.991+0000 7fa891624740 1 --2- 172.21.15.76:0/2814483847 >> [v2:172.21.15.174:6836/239386414,v1:172.21.15.174:6837/239386414] conn(0x55bfb0704f10 0x55bfb076d140 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.991+0000 7fa891624740 1 --2- 172.21.15.76:0/2814483847 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bfb06f4d30 0x55bfb0762fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.991+0000 7fa891624740 1 --2- 172.21.15.76:0/2814483847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bfb06ec150 0x55bfb06f85e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.991+0000 7fa891624740 1 --2- 172.21.15.76:0/2814483847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bfb06eb830 0x55bfb06f80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:44.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.991+0000 7fa8677fe640 0 client.25757 ms_handle_reset on v2:172.21.15.174:6836/239386414 2024-03-18T14:54:44.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.991+0000 7fa891624740 1 -- 172.21.15.76:0/2814483847 >> 172.21.15.76:0/2814483847 conn(0x55bfb06e5f60 msgr2=0x55bfb06f0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:44.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.991+0000 7fa891624740 1 -- 172.21.15.76:0/2814483847 shutdown_connections 2024-03-18T14:54:44.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.991+0000 7fa891624740 1 -- 172.21.15.76:0/2814483847 wait complete. 2024-03-18T14:54:44.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.991+0000 7fa891624740 20 client.25757 trim_cache size 0 max 16384 2024-03-18T14:54:44.997 INFO:teuthology.orchestra.run.smithi076.stdout:[ 2024-03-18T14:54:44.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:44.997 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:44.997 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 94420810, 2024-03-18T14:54:44.997 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628308, 2024-03-18T14:54:44.997 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TRIDOTS.POT" 2024-03-18T14:54:44.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:44.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:44.997 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:44.997 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 314228977, 2024-03-18T14:54:44.997 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628318, 2024-03-18T14:54:44.997 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR" 2024-03-18T14:54:44.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:44.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:44.998 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:44.998 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 398535490, 2024-03-18T14:54:44.998 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628326, 2024-03-18T14:54:44.998 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS" 2024-03-18T14:54:44.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:44.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:44.998 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:44.998 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 459283709, 2024-03-18T14:54:44.998 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628316, 2024-03-18T14:54:44.998 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT" 2024-03-18T14:54:44.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:44.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:44.999 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:44.999 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 557683196, 2024-03-18T14:54:44.999 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628304, 2024-03-18T14:54:44.999 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PCBENCHM.PPT" 2024-03-18T14:54:44.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:44.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:44.999 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:44.999 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 905097938, 2024-03-18T14:54:44.999 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628307, 2024-03-18T14:54:44.999 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TIPS.PPT" 2024-03-18T14:54:44.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:44.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:44.999 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:44.999 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 1123835392, 2024-03-18T14:54:44.999 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628327, 2024-03-18T14:54:45.000 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC" 2024-03-18T14:54:45.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.000 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.000 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 1581852873, 2024-03-18T14:54:45.000 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628312, 2024-03-18T14:54:45.000 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL" 2024-03-18T14:54:45.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.000 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.000 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 2106152765, 2024-03-18T14:54:45.000 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628319, 2024-03-18T14:54:45.000 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT" 2024-03-18T14:54:45.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.001 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.001 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 2110312286, 2024-03-18T14:54:45.001 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628317, 2024-03-18T14:54:45.001 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR" 2024-03-18T14:54:45.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.001 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.001 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 2516886946, 2024-03-18T14:54:45.001 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628314, 2024-03-18T14:54:45.001 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT" 2024-03-18T14:54:45.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.001 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.002 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 2615612840, 2024-03-18T14:54:45.002 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628305, 2024-03-18T14:54:45.002 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ZD16.BMP" 2024-03-18T14:54:45.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.002 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.002 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 2858841007, 2024-03-18T14:54:45.002 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628315, 2024-03-18T14:54:45.002 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT" 2024-03-18T14:54:45.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.002 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.002 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 2885331974, 2024-03-18T14:54:45.002 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628311, 2024-03-18T14:54:45.003 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG" 2024-03-18T14:54:45.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.003 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.003 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 3082383376, 2024-03-18T14:54:45.003 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628313, 2024-03-18T14:54:45.003 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR" 2024-03-18T14:54:45.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.003 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.003 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 3934675315, 2024-03-18T14:54:45.003 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628328, 2024-03-18T14:54:45.003 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC" 2024-03-18T14:54:45.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.004 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.004 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 4185442196, 2024-03-18T14:54:45.004 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628306, 2024-03-18T14:54:45.004 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PPTOOLS1.PPA" 2024-03-18T14:54:45.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.004 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.004 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 4205518695, 2024-03-18T14:54:45.004 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628310, 2024-03-18T14:54:45.004 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG" 2024-03-18T14:54:45.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.004 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.005 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 4245226452, 2024-03-18T14:54:45.005 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628320, 2024-03-18T14:54:45.005 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT" 2024-03-18T14:54:45.005 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T14:54:45.005 INFO:teuthology.orchestra.run.smithi076.stdout:] 2024-03-18T14:54:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.992+0000 7fa88e95b640 1 -- 172.21.15.76:0/1105385720 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa8700a2a90 msgr2=0x7fa8700a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.992+0000 7fa88e95b640 1 --2- 172.21.15.76:0/1105385720 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa8700a2a90 0x7fa8700a4f50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa87c009730 tx=0x7fa87c0097f0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.992+0000 7fa88e95b640 1 -- 172.21.15.76:0/1105385720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa88819c940 msgr2=0x7fa88819cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.992+0000 7fa88e95b640 1 --2- 172.21.15.76:0/1105385720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa88819c940 0x7fa88819cdf0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fa88000eb10 tx=0x7fa88000c2d0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.992+0000 7fa88e95b640 1 -- 172.21.15.76:0/1105385720 shutdown_connections 2024-03-18T14:54:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.992+0000 7fa88e95b640 1 --2- 172.21.15.76:0/1105385720 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa8700a2a90 0x7fa8700a4f50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.992+0000 7fa88e95b640 1 --2- 172.21.15.76:0/1105385720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa888105aa0 0x7fa888198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.992+0000 7fa88e95b640 1 --2- 172.21.15.76:0/1105385720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa88819c940 0x7fa88819cdf0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.992+0000 7fa88e95b640 1 --2- 172.21.15.76:0/1105385720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa888101420 0x7fa8881980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.992+0000 7fa88e95b640 1 -- 172.21.15.76:0/1105385720 >> 172.21.15.76:0/1105385720 conn(0x7fa888075a20 msgr2=0x7fa8880fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:45.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.992+0000 7fa88e95b640 1 -- 172.21.15.76:0/1105385720 shutdown_connections 2024-03-18T14:54:45.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:44.992+0000 7fa88e95b640 1 -- 172.21.15.76:0/1105385720 wait complete. 2024-03-18T14:54:45.007 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T14:54:45.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.115+0000 7f2100952640 1 -- 172.21.15.76:0/449454811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f20fc101dd0 msgr2=0x7f20fc105370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.115+0000 7f2100952640 1 --2- 172.21.15.76:0/449454811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f20fc101dd0 0x7f20fc105370 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f20f0009900 tx=0x7f20f002bc30 comp rx=0 tx=0).stop 2024-03-18T14:54:45.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.115+0000 7f2100952640 1 -- 172.21.15.76:0/449454811 shutdown_connections 2024-03-18T14:54:45.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.115+0000 7f2100952640 1 --2- 172.21.15.76:0/449454811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20fc105aa0 0x7f20fc10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.115+0000 7f2100952640 1 --2- 172.21.15.76:0/449454811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f20fc101dd0 0x7f20fc105370 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.115+0000 7f2100952640 1 --2- 172.21.15.76:0/449454811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20fc101420 0x7f20fc101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.115+0000 7f2100952640 1 -- 172.21.15.76:0/449454811 >> 172.21.15.76:0/449454811 conn(0x7f20fc075a20 msgr2=0x7f20fc075e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:45.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.115+0000 7f2100952640 1 -- 172.21.15.76:0/449454811 shutdown_connections 2024-03-18T14:54:45.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.115+0000 7f2100952640 1 -- 172.21.15.76:0/449454811 wait complete. 2024-03-18T14:54:45.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f2100952640 1 Processor -- start 2024-03-18T14:54:45.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f2100952640 1 -- start start 2024-03-18T14:54:45.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f2100952640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f20fc101420 0x7f20fc1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f2100952640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20fc105aa0 0x7f20fc198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f2100952640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20fc19c950 0x7f20fc19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f2100952640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc111be0 con 0x7f20fc101420 2024-03-18T14:54:45.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f2100952640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc111a60 con 0x7f20fc19c950 2024-03-18T14:54:45.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f2100952640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc111d60 con 0x7f20fc105aa0 2024-03-18T14:54:45.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f20fa59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f20fc101420 0x7f20fc1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f20fa59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f20fc101420 0x7f20fc1980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33366/0 (socket says 172.21.15.76:33366) 2024-03-18T14:54:45.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f20fad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20fc19c950 0x7f20fc19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f20fa59c640 1 -- 172.21.15.76:0/3880463659 learned_addr learned my addr 172.21.15.76:0/3880463659 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:45.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f20f9d9b640 1 --2- 172.21.15.76:0/3880463659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20fc105aa0 0x7f20fc198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f20fa59c640 1 -- 172.21.15.76:0/3880463659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20fc105aa0 msgr2=0x7f20fc198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f20fa59c640 1 --2- 172.21.15.76:0/3880463659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20fc105aa0 0x7f20fc198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f20fa59c640 1 -- 172.21.15.76:0/3880463659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20fc19c950 msgr2=0x7f20fc19ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f20fa59c640 1 --2- 172.21.15.76:0/3880463659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20fc19c950 0x7f20fc19ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f20fa59c640 1 -- 172.21.15.76:0/3880463659 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20fc19d440 con 0x7f20fc101420 2024-03-18T14:54:45.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.116+0000 7f20fad9d640 1 --2- 172.21.15.76:0/3880463659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20fc19c950 0x7f20fc19ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:45.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.117+0000 7f20fa59c640 1 --2- 172.21.15.76:0/3880463659 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f20fc101420 0x7f20fc1980e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e9f0 tx=0x7f20ec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:45.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.117+0000 7f20eb7fe640 1 -- 172.21.15.76:0/3880463659 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec00cd80 con 0x7f20fc101420 2024-03-18T14:54:45.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.117+0000 7f20eb7fe640 1 -- 172.21.15.76:0/3880463659 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20ec005500 con 0x7f20fc101420 2024-03-18T14:54:45.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.117+0000 7f2100952640 1 -- 172.21.15.76:0/3880463659 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20fc19d730 con 0x7f20fc101420 2024-03-18T14:54:45.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.117+0000 7f20eb7fe640 1 -- 172.21.15.76:0/3880463659 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec010690 con 0x7f20fc101420 2024-03-18T14:54:45.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.117+0000 7f2100952640 1 -- 172.21.15.76:0/3880463659 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20fc1a4ea0 con 0x7f20fc101420 2024-03-18T14:54:45.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.119+0000 7f20eb7fe640 1 -- 172.21.15.76:0/3880463659 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f20ec0046c0 con 0x7f20fc101420 2024-03-18T14:54:45.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.119+0000 7f2103610740 1 Processor -- start 2024-03-18T14:54:45.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.119+0000 7f2103610740 1 -- start start 2024-03-18T14:54:45.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f2103610740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb4830 0x5627c5fb4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f2103610740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb5150 0x5627c5fbd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f2103610740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 0x5627c5fc0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f2103610740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5627c5f18490 con 0x5627c5fb5150 2024-03-18T14:54:45.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f2103610740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5627c5ec0c90 con 0x5627c5fb4830 2024-03-18T14:54:45.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f2103610740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5627c5f14530 con 0x5627c5fbdd30 2024-03-18T14:54:45.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f20eb7fe640 1 --2- 172.21.15.76:0/3880463659 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f20e40a2a90 0x7f20e40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f20fa59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb5150 0x5627c5fbd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f20fa59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb5150 0x5627c5fbd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33380/0 (socket says 172.21.15.76:33380) 2024-03-18T14:54:45.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f20fa59c640 1 -- 172.21.15.76:0/572618063 learned_addr learned my addr 172.21.15.76:0/572618063 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:45.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f20f9d9b640 1 --2- 172.21.15.76:0/572618063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 0x5627c5fc0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f20fad9d640 1 --2- 172.21.15.76:0/572618063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb4830 0x5627c5fb4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f20fad9d640 1 --2- 172.21.15.76:0/3880463659 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f20e40a2a90 0x7f20e40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f20eb7fe640 1 -- 172.21.15.76:0/3880463659 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f20ec0cf4c0 con 0x7f20fc101420 2024-03-18T14:54:45.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f20fa59c640 1 -- 172.21.15.76:0/572618063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 msgr2=0x5627c5fc0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:45.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f20fa59c640 1 --2- 172.21.15.76:0/572618063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 0x5627c5fc0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f20fa59c640 1 -- 172.21.15.76:0/572618063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb4830 msgr2=0x5627c5fb4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:45.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f20fa59c640 1 --2- 172.21.15.76:0/572618063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb4830 0x5627c5fb4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.120+0000 7f20fa59c640 1 -- 172.21.15.76:0/572618063 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627c5f7b500 con 0x5627c5fb5150 2024-03-18T14:54:45.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.121+0000 7f20fa59c640 1 --2- 172.21.15.76:0/572618063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb5150 0x5627c5fbd7f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0982f0 tx=0x7f20ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7548fc624069a10 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:45.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.121+0000 7f20fad9d640 1 --2- 172.21.15.76:0/3880463659 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f20e40a2a90 0x7f20e40a4f50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f20f40099e0 tx=0x7f20f4009500 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:45.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.121+0000 7f20e8ff9640 1 -- 172.21.15.76:0/572618063 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec084e60 con 0x5627c5fb5150 2024-03-18T14:54:45.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.121+0000 7f20e8ff9640 1 -- 172.21.15.76:0/572618063 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20ec09c020 con 0x5627c5fb5150 2024-03-18T14:54:45.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.121+0000 7f20e8ff9640 1 -- 172.21.15.76:0/572618063 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec0860e0 con 0x5627c5fb5150 2024-03-18T14:54:45.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.122+0000 7f2103610740 1 -- 172.21.15.76:0/572618063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb5150 msgr2=0x5627c5fbd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.122+0000 7f2103610740 1 --2- 172.21.15.76:0/572618063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb5150 0x5627c5fbd7f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0982f0 tx=0x7f20ec004a40 comp rx=0 tx=0).stop 2024-03-18T14:54:45.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.122+0000 7f2103610740 1 -- 172.21.15.76:0/572618063 shutdown_connections 2024-03-18T14:54:45.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.122+0000 7f2103610740 1 --2- 172.21.15.76:0/572618063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 0x5627c5fc0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.122+0000 7f2103610740 1 --2- 172.21.15.76:0/572618063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb5150 0x5627c5fbd7f0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.122+0000 7f2103610740 1 --2- 172.21.15.76:0/572618063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb4830 0x5627c5fb4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.122+0000 7f2103610740 1 -- 172.21.15.76:0/572618063 >> 172.21.15.76:0/572618063 conn(0x5627c5faef60 msgr2=0x5627c5faf340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:45.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.123+0000 7f2103610740 1 -- 172.21.15.76:0/572618063 shutdown_connections 2024-03-18T14:54:45.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.123+0000 7f2103610740 1 -- 172.21.15.76:0/572618063 wait complete. 2024-03-18T14:54:45.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.123+0000 7f2103610740 1 -- start start 2024-03-18T14:54:45.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.123+0000 7f2103610740 1 Processor -- start 2024-03-18T14:54:45.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.124+0000 7f2103610740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb4830 0x5627c5fbadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.124+0000 7f2103610740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb5150 0x5627c5fbb310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.124+0000 7f2103610740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 0x5627c5fbb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.124+0000 7f2103610740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5627c5fc0b20 con 0x5627c5fb5150 2024-03-18T14:54:45.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.124+0000 7f2103610740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5627c5ed4230 con 0x5627c5fb4830 2024-03-18T14:54:45.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.124+0000 7f2103610740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5627c5fc0ca0 con 0x5627c5fbdd30 2024-03-18T14:54:45.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.124+0000 7f2103610740 10 client.0 _ll_register_callbacks cb 0x5627c5f10010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:45.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.124+0000 7f20fa59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb4830 0x5627c5fbadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.124+0000 7f20fa59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb4830 0x5627c5fbadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:42766/0 (socket says 172.21.15.76:42766) 2024-03-18T14:54:45.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.124+0000 7f20fa59c640 1 -- 172.21.15.76:0/354209498 learned_addr learned my addr 172.21.15.76:0/354209498 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:45.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.124+0000 7f20f9d9b640 1 --2- 172.21.15.76:0/354209498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb5150 0x5627c5fbb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.125+0000 7f20fad9d640 1 --2- 172.21.15.76:0/354209498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 0x5627c5fbb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.125+0000 7f20fa59c640 1 -- 172.21.15.76:0/354209498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 msgr2=0x5627c5fbb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:45.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.125+0000 7f20fa59c640 1 --2- 172.21.15.76:0/354209498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 0x5627c5fbb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.125+0000 7f20fa59c640 1 -- 172.21.15.76:0/354209498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb5150 msgr2=0x5627c5fbb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:45.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.125+0000 7f20fa59c640 1 --2- 172.21.15.76:0/354209498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb5150 0x5627c5fbb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.125+0000 7f20fa59c640 1 -- 172.21.15.76:0/354209498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627c5ee3100 con 0x5627c5fb4830 2024-03-18T14:54:45.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.125+0000 7f20fa59c640 1 --2- 172.21.15.76:0/354209498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb4830 0x5627c5fbadd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0982f0 tx=0x7f20ec0a4130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7e1cf35f451f58a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:45.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.125+0000 7f20e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:54:45.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.125+0000 7f20e1ffb640 1 -- 172.21.15.76:0/354209498 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec0b9060 con 0x5627c5fb4830 2024-03-18T14:54:45.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.126+0000 7f20e1ffb640 1 -- 172.21.15.76:0/354209498 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f20ec0ccdc0 con 0x5627c5fb4830 2024-03-18T14:54:45.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.126+0000 7f2103610740 1 -- 172.21.15.76:0/354209498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20ec09c020 con 0x5627c5fb4830 2024-03-18T14:54:45.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.126+0000 7f20e1ffb640 1 -- 172.21.15.76:0/354209498 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f20ec084e60 con 0x5627c5fb4830 2024-03-18T14:54:45.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.126+0000 7f2103610740 10 client.25769 fetch_fsmap learned FSMap version 29 2024-03-18T14:54:45.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.126+0000 7f2103610740 1 -- 172.21.15.76:0/354209498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5627c5c2c3f0 con 0x5627c5fb4830 2024-03-18T14:54:45.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.127+0000 7f20e1ffb640 1 -- 172.21.15.76:0/354209498 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20ec09c020 con 0x5627c5fb4830 2024-03-18T14:54:45.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.127+0000 7f20e1ffb640 1 -- 172.21.15.76:0/354209498 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f20ec0cf040 con 0x5627c5fb4830 2024-03-18T14:54:45.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.127+0000 7f2103610740 10 client.25769 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:54:45.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.127+0000 7f2103610740 10 client.25769 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T14:54:45.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.127+0000 7f2103610740 20 client.25769 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:45.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.127+0000 7f2103610740 1 --2- 172.21.15.76:0/354209498 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x5627c5fcdfa0 0x5627c5fd0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.127+0000 7f2103610740 4 client.25769 mds_command: new command op to 24490 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:54:45.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.127+0000 7f2103610740 1 -- 172.21.15.76:0/354209498 --> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627c5f259d0 con 0x5627c5fcdfa0 2024-03-18T14:54:45.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.127+0000 7f20f9d9b640 1 --2- 172.21.15.76:0/354209498 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x5627c5fcdfa0 0x5627c5fd0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.129+0000 7f20f9d9b640 1 --2- 172.21.15.76:0/354209498 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x5627c5fcdfa0 0x5627c5fd0390 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=703dd590166251 server_cookie=286d3e521678a678 in_seq=0 out_seq=0 2024-03-18T14:54:45.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.129+0000 7f20e1ffb640 10 client.25769 ms_handle_connect on v2:172.21.15.174:6838/399810652 2024-03-18T14:54:45.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.130+0000 7f20e1ffb640 1 -- 172.21.15.76:0/354209498 <== mds.1 v2:172.21.15.174:6838/399810652 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5627c5f259d0 con 0x5627c5fcdfa0 2024-03-18T14:54:45.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.130+0000 7f20e1ffb640 10 client.25769 handle_command_reply: tid=0 2024-03-18T14:54:45.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.130+0000 7f2103610740 1 client.25769 shutdown 2024-03-18T14:54:45.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.130+0000 7f2103610740 1 -- 172.21.15.76:0/354209498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb4830 msgr2=0x5627c5fbadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.130+0000 7f2103610740 1 --2- 172.21.15.76:0/354209498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb4830 0x5627c5fbadd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0982f0 tx=0x7f20ec0a4130 comp rx=0 tx=0).stop 2024-03-18T14:54:45.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.131+0000 7f2103610740 1 -- 172.21.15.76:0/354209498 shutdown_connections 2024-03-18T14:54:45.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.131+0000 7f2103610740 1 --2- 172.21.15.76:0/354209498 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x5627c5fcdfa0 0x5627c5fd0390 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.131+0000 7f2103610740 1 --2- 172.21.15.76:0/354209498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 0x5627c5fbb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.131+0000 7f20e1ffb640 0 client.25769 ms_handle_reset on v2:172.21.15.174:6838/399810652 2024-03-18T14:54:45.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.131+0000 7f2103610740 1 --2- 172.21.15.76:0/354209498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb5150 0x5627c5fbb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.131+0000 7f2103610740 1 --2- 172.21.15.76:0/354209498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb4830 0x5627c5fbadd0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.131+0000 7f2103610740 1 -- 172.21.15.76:0/354209498 >> 172.21.15.76:0/354209498 conn(0x5627c5faef60 msgr2=0x5627c5fb0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:45.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.131+0000 7f2103610740 1 -- 172.21.15.76:0/354209498 shutdown_connections 2024-03-18T14:54:45.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.131+0000 7f2103610740 1 -- 172.21.15.76:0/354209498 wait complete. 2024-03-18T14:54:45.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.131+0000 7f2103610740 20 client.25769 trim_cache size 0 max 16384 2024-03-18T14:54:45.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.146+0000 7f2103610740 1 Processor -- start 2024-03-18T14:54:45.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.146+0000 7f2103610740 1 -- start start 2024-03-18T14:54:45.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.146+0000 7f2103610740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb4830 0x5627c5fbd500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.146+0000 7f2103610740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb5150 0x5627c5fade60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.146+0000 7f2103610740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 0x5627c5fd33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.146+0000 7f2103610740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc1adb20 con 0x5627c5fb4830 2024-03-18T14:54:45.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.146+0000 7f2103610740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc1ab390 con 0x5627c5fb5150 2024-03-18T14:54:45.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.146+0000 7f2103610740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc1a9220 con 0x5627c5fbdd30 2024-03-18T14:54:45.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.146+0000 7f20fa59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb4830 0x5627c5fbd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.146+0000 7f20f9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb5150 0x5627c5fade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.146+0000 7f20fad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 0x5627c5fd33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.146+0000 7f20f9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb5150 0x5627c5fade60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:42768/0 (socket says 172.21.15.76:42768) 2024-03-18T14:54:45.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.146+0000 7f20f9d9b640 1 -- 172.21.15.76:0/2801988907 learned_addr learned my addr 172.21.15.76:0/2801988907 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:45.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.147+0000 7f20f9d9b640 1 -- 172.21.15.76:0/2801988907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 msgr2=0x5627c5fd33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:45.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.147+0000 7f20f9d9b640 1 --2- 172.21.15.76:0/2801988907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 0x5627c5fd33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.147+0000 7f20f9d9b640 1 -- 172.21.15.76:0/2801988907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb4830 msgr2=0x5627c5fbd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:45.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.147+0000 7f20f9d9b640 1 --2- 172.21.15.76:0/2801988907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb4830 0x5627c5fbd500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.147+0000 7f20f9d9b640 1 -- 172.21.15.76:0/2801988907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627c5fb4f80 con 0x5627c5fb5150 2024-03-18T14:54:45.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.147+0000 7f20fa59c640 1 --2- 172.21.15.76:0/2801988907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb4830 0x5627c5fbd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:45.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.147+0000 7f20f9d9b640 1 --2- 172.21.15.76:0/2801988907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb5150 0x5627c5fade60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f20fc076ff0 tx=0x7f20f002e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87751eb51693d5cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:45.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.147+0000 7f20e3fff640 1 -- 172.21.15.76:0/2801988907 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f0049d10 con 0x5627c5fb5150 2024-03-18T14:54:45.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.147+0000 7f20e3fff640 1 -- 172.21.15.76:0/2801988907 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20f0004040 con 0x5627c5fb5150 2024-03-18T14:54:45.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.147+0000 7f20e3fff640 1 -- 172.21.15.76:0/2801988907 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f002cca0 con 0x5627c5fb5150 2024-03-18T14:54:45.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.148+0000 7f2103610740 1 -- 172.21.15.76:0/2801988907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb5150 msgr2=0x5627c5fade60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.148+0000 7f2103610740 1 --2- 172.21.15.76:0/2801988907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb5150 0x5627c5fade60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f20fc076ff0 tx=0x7f20f002e040 comp rx=0 tx=0).stop 2024-03-18T14:54:45.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.148+0000 7f2103610740 1 -- 172.21.15.76:0/2801988907 shutdown_connections 2024-03-18T14:54:45.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.148+0000 7f2103610740 1 --2- 172.21.15.76:0/2801988907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 0x5627c5fd33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.148+0000 7f2103610740 1 --2- 172.21.15.76:0/2801988907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb5150 0x5627c5fade60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.148+0000 7f2103610740 1 --2- 172.21.15.76:0/2801988907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb4830 0x5627c5fbd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.148+0000 7f2103610740 1 -- 172.21.15.76:0/2801988907 >> 172.21.15.76:0/2801988907 conn(0x5627c5faef60 msgr2=0x5627c5fad750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:45.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.148+0000 7f2103610740 1 -- 172.21.15.76:0/2801988907 shutdown_connections 2024-03-18T14:54:45.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.148+0000 7f2103610740 1 -- 172.21.15.76:0/2801988907 wait complete. 2024-03-18T14:54:45.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.148+0000 7f2103610740 1 -- start start 2024-03-18T14:54:45.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.148+0000 7f2103610740 1 Processor -- start 2024-03-18T14:54:45.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f2103610740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb4830 0x5627c5fd3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f2103610740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb5150 0x5627c5fd0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f2103610740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 0x5627c5fd1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f2103610740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc1a9870 con 0x5627c5fb4830 2024-03-18T14:54:45.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f2103610740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc1aacc0 con 0x5627c5fb5150 2024-03-18T14:54:45.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f2103610740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc1a9f20 con 0x5627c5fbdd30 2024-03-18T14:54:45.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f2103610740 10 client.0 _ll_register_callbacks cb 0x7f20f4011760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:45.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f20fa59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb4830 0x5627c5fd3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f20fa59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb4830 0x5627c5fd3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33408/0 (socket says 172.21.15.76:33408) 2024-03-18T14:54:45.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f20fa59c640 1 -- 172.21.15.76:0/605954351 learned_addr learned my addr 172.21.15.76:0/605954351 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:45.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f20f9d9b640 1 --2- 172.21.15.76:0/605954351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb5150 0x5627c5fd0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f20fad9d640 1 --2- 172.21.15.76:0/605954351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 0x5627c5fd1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f20fa59c640 1 -- 172.21.15.76:0/605954351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 msgr2=0x5627c5fd1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:45.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f20fa59c640 1 --2- 172.21.15.76:0/605954351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 0x5627c5fd1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f20fa59c640 1 -- 172.21.15.76:0/605954351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb5150 msgr2=0x5627c5fd0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:45.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f20fa59c640 1 --2- 172.21.15.76:0/605954351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb5150 0x5627c5fd0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f20fa59c640 1 -- 172.21.15.76:0/605954351 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627c5cf67d0 con 0x5627c5fb4830 2024-03-18T14:54:45.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f20f9d9b640 1 --2- 172.21.15.76:0/605954351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb5150 0x5627c5fd0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:45.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f20fa59c640 1 --2- 172.21.15.76:0/605954351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb4830 0x5627c5fd3830 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0cd260 tx=0x7f20ec0a40c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cecbef0ca249e3b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:45.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f20e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:45.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f20e1ffb640 1 -- 172.21.15.76:0/605954351 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec0b9070 con 0x5627c5fb4830 2024-03-18T14:54:45.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f20e1ffb640 1 -- 172.21.15.76:0/605954351 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f20ec09af80 con 0x5627c5fb4830 2024-03-18T14:54:45.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f2103610740 1 -- 172.21.15.76:0/605954351 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20f0004040 con 0x5627c5fb4830 2024-03-18T14:54:45.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f20e1ffb640 1 -- 172.21.15.76:0/605954351 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f20ec07c020 con 0x5627c5fb4830 2024-03-18T14:54:45.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.149+0000 7f2103610740 10 client.15918 fetch_fsmap learned FSMap version 29 2024-03-18T14:54:45.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.150+0000 7f2103610740 1 -- 172.21.15.76:0/605954351 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5627c6033f30 con 0x5627c5fb4830 2024-03-18T14:54:45.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.150+0000 7f20e1ffb640 1 -- 172.21.15.76:0/605954351 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20f0004040 con 0x5627c5fb4830 2024-03-18T14:54:45.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.150+0000 7f20e1ffb640 1 -- 172.21.15.76:0/605954351 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f20ec07ac10 con 0x5627c5fb4830 2024-03-18T14:54:45.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.150+0000 7f2103610740 10 client.15918 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:54:45.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.150+0000 7f2103610740 10 client.15918 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T14:54:45.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.150+0000 7f2103610740 20 client.15918 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:45.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.150+0000 7f2103610740 1 --2- 172.21.15.76:0/605954351 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x5627c6034410 0x5627c6036800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.150+0000 7f2103610740 4 client.15918 mds_command: new command op to 24490 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T14:54:45.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.150+0000 7f2103610740 1 -- 172.21.15.76:0/605954351 --> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5627c5f1fd20 con 0x5627c6034410 2024-03-18T14:54:45.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.150+0000 7f20f9d9b640 1 --2- 172.21.15.76:0/605954351 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x5627c6034410 0x5627c6036800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.150+0000 7f20e1ffb640 10 client.15918 ms_handle_connect on v2:172.21.15.174:6838/399810652 2024-03-18T14:54:45.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.150+0000 7f20f9d9b640 1 --2- 172.21.15.76:0/605954351 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x5627c6034410 0x5627c6036800 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d3a4fcda9b18503e server_cookie=b65a89da8ab970e6 in_seq=0 out_seq=0 2024-03-18T14:54:45.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.151+0000 7f20e1ffb640 1 -- 172.21.15.76:0/605954351 <== mds.1 v2:172.21.15.174:6838/399810652 1 ==== command_reply(tid 0: 0 ) ==== 8+0+4655 (crc 0 0 0) 0x5627c5f1fd20 con 0x5627c6034410 2024-03-18T14:54:45.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.151+0000 7f20e1ffb640 10 client.15918 handle_command_reply: tid=0 2024-03-18T14:54:45.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.151+0000 7f2103610740 1 client.15918 shutdown 2024-03-18T14:54:45.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.151+0000 7f2103610740 1 -- 172.21.15.76:0/605954351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb4830 msgr2=0x5627c5fd3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.151+0000 7f2103610740 1 --2- 172.21.15.76:0/605954351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb4830 0x5627c5fd3830 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0cd260 tx=0x7f20ec0a40c0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.151+0000 7f2103610740 1 -- 172.21.15.76:0/605954351 shutdown_connections 2024-03-18T14:54:45.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.151+0000 7f2103610740 1 --2- 172.21.15.76:0/605954351 >> [v2:172.21.15.174:6838/399810652,v1:172.21.15.174:6839/399810652] conn(0x5627c6034410 0x5627c6036800 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.151+0000 7f2103610740 1 --2- 172.21.15.76:0/605954351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5627c5fbdd30 0x5627c5fd1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.151+0000 7f20e1ffb640 0 client.15918 ms_handle_reset on v2:172.21.15.174:6838/399810652 2024-03-18T14:54:45.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.151+0000 7f2103610740 1 --2- 172.21.15.76:0/605954351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627c5fb5150 0x5627c5fd0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.151+0000 7f2103610740 1 --2- 172.21.15.76:0/605954351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627c5fb4830 0x5627c5fd3830 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.151+0000 7f2103610740 1 -- 172.21.15.76:0/605954351 >> 172.21.15.76:0/605954351 conn(0x5627c5faef60 msgr2=0x5627c5fbb600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:45.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.151+0000 7f2103610740 1 -- 172.21.15.76:0/605954351 shutdown_connections 2024-03-18T14:54:45.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.151+0000 7f2103610740 1 -- 172.21.15.76:0/605954351 wait complete. 2024-03-18T14:54:45.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.151+0000 7f2103610740 20 client.15918 trim_cache size 0 max 16384 2024-03-18T14:54:45.160 INFO:teuthology.orchestra.run.smithi076.stdout:[ 2024-03-18T14:54:45.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.160 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.160 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 200727419, 2024-03-18T14:54:45.160 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628344, 2024-03-18T14:54:45.160 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.X03" 2024-03-18T14:54:45.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.161 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.161 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 369657943, 2024-03-18T14:54:45.161 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628348, 2024-03-18T14:54:45.161 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.FSL" 2024-03-18T14:54:45.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.161 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.161 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 390129868, 2024-03-18T14:54:45.161 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628336, 2024-03-18T14:54:45.161 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG0" 2024-03-18T14:54:45.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.162 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.162 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 694303819, 2024-03-18T14:54:45.162 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628339, 2024-03-18T14:54:45.162 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG1" 2024-03-18T14:54:45.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.162 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.162 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 869040883, 2024-03-18T14:54:45.162 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628333, 2024-03-18T14:54:45.162 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.Y04" 2024-03-18T14:54:45.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.162 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.162 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 1055885286, 2024-03-18T14:54:45.163 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628349, 2024-03-18T14:54:45.163 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.FSL" 2024-03-18T14:54:45.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.163 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.163 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 1181399849, 2024-03-18T14:54:45.163 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628334, 2024-03-18T14:54:45.163 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.DB" 2024-03-18T14:54:45.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.164 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.164 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 1196012775, 2024-03-18T14:54:45.164 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628346, 2024-03-18T14:54:45.164 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.VAL" 2024-03-18T14:54:45.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.164 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.164 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 1511329952, 2024-03-18T14:54:45.164 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628347, 2024-03-18T14:54:45.164 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.TV" 2024-03-18T14:54:45.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.164 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.164 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 2152505386, 2024-03-18T14:54:45.165 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628335, 2024-03-18T14:54:45.165 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.PX" 2024-03-18T14:54:45.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.165 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.165 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 2461033744, 2024-03-18T14:54:45.165 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628342, 2024-03-18T14:54:45.165 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.DB" 2024-03-18T14:54:45.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.165 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.165 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 2530979369, 2024-03-18T14:54:45.165 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628340, 2024-03-18T14:54:45.165 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.VAL" 2024-03-18T14:54:45.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.166 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.166 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 2685593990, 2024-03-18T14:54:45.166 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628332, 2024-03-18T14:54:45.166 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.X04" 2024-03-18T14:54:45.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.166 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.166 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 2729367069, 2024-03-18T14:54:45.166 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628338, 2024-03-18T14:54:45.166 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG1" 2024-03-18T14:54:45.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.167 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.167 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 2785978108, 2024-03-18T14:54:45.167 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628330, 2024-03-18T14:54:45.167 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.DB" 2024-03-18T14:54:45.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.167 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.167 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 2885482703, 2024-03-18T14:54:45.167 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628331, 2024-03-18T14:54:45.167 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.PX" 2024-03-18T14:54:45.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.167 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.167 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 3346329482, 2024-03-18T14:54:45.167 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628337, 2024-03-18T14:54:45.168 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG0" 2024-03-18T14:54:45.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.168 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.168 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 3517481119, 2024-03-18T14:54:45.168 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628345, 2024-03-18T14:54:45.168 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.Y03" 2024-03-18T14:54:45.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.168 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.168 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 3623539156, 2024-03-18T14:54:45.168 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628341, 2024-03-18T14:54:45.168 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.MB" 2024-03-18T14:54:45.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T14:54:45.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T14:54:45.169 INFO:teuthology.orchestra.run.smithi076.stdout: "damage_type": "backtrace", 2024-03-18T14:54:45.169 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 3701451795, 2024-03-18T14:54:45.169 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 1099511628343, 2024-03-18T14:54:45.169 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.PX" 2024-03-18T14:54:45.169 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T14:54:45.169 INFO:teuthology.orchestra.run.smithi076.stdout:] 2024-03-18T14:54:45.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.152+0000 7f2100952640 1 -- 172.21.15.76:0/3880463659 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f20e40a2a90 msgr2=0x7f20e40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.152+0000 7f2100952640 1 --2- 172.21.15.76:0/3880463659 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f20e40a2a90 0x7f20e40a4f50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f20f40099e0 tx=0x7f20f4009500 comp rx=0 tx=0).stop 2024-03-18T14:54:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.152+0000 7f2100952640 1 -- 172.21.15.76:0/3880463659 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f20fc101420 msgr2=0x7f20fc1980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.152+0000 7f2100952640 1 --2- 172.21.15.76:0/3880463659 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f20fc101420 0x7f20fc1980e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e9f0 tx=0x7f20ec00edf0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.152+0000 7f2100952640 1 -- 172.21.15.76:0/3880463659 shutdown_connections 2024-03-18T14:54:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.152+0000 7f2100952640 1 --2- 172.21.15.76:0/3880463659 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f20e40a2a90 0x7f20e40a4f50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.152+0000 7f2100952640 1 --2- 172.21.15.76:0/3880463659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20fc105aa0 0x7f20fc198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.152+0000 7f2100952640 1 --2- 172.21.15.76:0/3880463659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20fc19c950 0x7f20fc19ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.152+0000 7f2100952640 1 --2- 172.21.15.76:0/3880463659 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f20fc101420 0x7f20fc1980e0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.152+0000 7f2100952640 1 -- 172.21.15.76:0/3880463659 >> 172.21.15.76:0/3880463659 conn(0x7f20fc075a20 msgr2=0x7f20fc0fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.152+0000 7f2100952640 1 -- 172.21.15.76:0/3880463659 shutdown_connections 2024-03-18T14:54:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.152+0000 7f2100952640 1 -- 172.21.15.76:0/3880463659 wait complete. 2024-03-18T14:54:45.172 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T14:54:45.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.286+0000 7f71c6352640 1 -- 172.21.15.76:0/491376845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71c0100650 msgr2=0x7f71c0100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.286+0000 7f71c6352640 1 --2- 172.21.15.76:0/491376845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71c0100650 0x7f71c0100ad0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f71b4009930 tx=0x7f71b402bc30 comp rx=0 tx=0).stop 2024-03-18T14:54:45.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.286+0000 7f71c6352640 1 -- 172.21.15.76:0/491376845 shutdown_connections 2024-03-18T14:54:45.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.286+0000 7f71c6352640 1 --2- 172.21.15.76:0/491376845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71c0101010 0x7f71c010d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.286+0000 7f71c6352640 1 --2- 172.21.15.76:0/491376845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71c0100650 0x7f71c0100ad0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.286+0000 7f71c6352640 1 --2- 172.21.15.76:0/491376845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71c0106660 0x7f71c0106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.286+0000 7f71c6352640 1 -- 172.21.15.76:0/491376845 >> 172.21.15.76:0/491376845 conn(0x7f71c00777e0 msgr2=0x7f71c0077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:45.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.286+0000 7f71c6352640 1 -- 172.21.15.76:0/491376845 shutdown_connections 2024-03-18T14:54:45.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.286+0000 7f71c6352640 1 -- 172.21.15.76:0/491376845 wait complete. 2024-03-18T14:54:45.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.286+0000 7f71c6352640 1 Processor -- start 2024-03-18T14:54:45.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.287+0000 7f71c6352640 1 -- start start 2024-03-18T14:54:45.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.287+0000 7f71c6352640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71c0100650 0x7f71c01980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.287+0000 7f71c6352640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71c0101010 0x7f71c0198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.287+0000 7f71c6352640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71c0106660 0x7f71c019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.287+0000 7f71c6352640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f71c0110a70 con 0x7f71c0100650 2024-03-18T14:54:45.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.287+0000 7f71c6352640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01108f0 con 0x7f71c0106660 2024-03-18T14:54:45.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.287+0000 7f71c6352640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f71c0110bf0 con 0x7f71c0101010 2024-03-18T14:54:45.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.287+0000 7f71bffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71c0100650 0x7f71c01980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.287+0000 7f71bf7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71c0101010 0x7f71c0198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.287+0000 7f71c48ef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71c0106660 0x7f71c019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.287+0000 7f71bffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71c0100650 0x7f71c01980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33420/0 (socket says 172.21.15.76:33420) 2024-03-18T14:54:45.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.287+0000 7f71bffff640 1 -- 172.21.15.76:0/2139251585 learned_addr learned my addr 172.21.15.76:0/2139251585 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:45.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.287+0000 7f71c48ef640 1 -- 172.21.15.76:0/2139251585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71c0101010 msgr2=0x7f71c0198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.287+0000 7f71c48ef640 1 --2- 172.21.15.76:0/2139251585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71c0101010 0x7f71c0198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.287+0000 7f71c48ef640 1 -- 172.21.15.76:0/2139251585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71c0100650 msgr2=0x7f71c01980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.287+0000 7f71c48ef640 1 --2- 172.21.15.76:0/2139251585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71c0100650 0x7f71c01980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.288+0000 7f71c48ef640 1 -- 172.21.15.76:0/2139251585 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c019d080 con 0x7f71c0106660 2024-03-18T14:54:45.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.288+0000 7f71bffff640 1 --2- 172.21.15.76:0/2139251585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71c0100650 0x7f71c01980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:45.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.288+0000 7f71c48ef640 1 --2- 172.21.15.76:0/2139251585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71c0106660 0x7f71c019c950 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f71b8009fd0 tx=0x7f71b800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:45.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.288+0000 7f71bd7fa640 1 -- 172.21.15.76:0/2139251585 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b800ce20 con 0x7f71c0106660 2024-03-18T14:54:45.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.288+0000 7f71c6352640 1 -- 172.21.15.76:0/2139251585 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71c019d370 con 0x7f71c0106660 2024-03-18T14:54:45.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.288+0000 7f71bd7fa640 1 -- 172.21.15.76:0/2139251585 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71b8005020 con 0x7f71c0106660 2024-03-18T14:54:45.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.288+0000 7f71c6352640 1 -- 172.21.15.76:0/2139251585 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71c01a4c00 con 0x7f71c0106660 2024-03-18T14:54:45.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.288+0000 7f71bd7fa640 1 -- 172.21.15.76:0/2139251585 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b8010700 con 0x7f71c0106660 2024-03-18T14:54:45.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.290+0000 7f71bd7fa640 1 -- 172.21.15.76:0/2139251585 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f71b8005540 con 0x7f71c0106660 2024-03-18T14:54:45.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71c9010740 1 Processor -- start 2024-03-18T14:54:45.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71c9010740 1 -- start start 2024-03-18T14:54:45.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71bd7fa640 1 --2- 172.21.15.76:0/2139251585 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f71a80a2a90 0x7f71a80a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71c9010740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa49d830 0x5628aa49dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71c9010740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49e150 0x5628aa4a67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71bf7fe640 1 --2- 172.21.15.76:0/2139251585 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f71a80a2a90 0x7f71a80a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71c9010740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa4a6d30 0x5628aa4a9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71c9010740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5628aa401490 con 0x5628aa49e150 2024-03-18T14:54:45.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71c9010740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5628aa3a9c90 con 0x5628aa49d830 2024-03-18T14:54:45.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71c9010740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5628aa3fd530 con 0x5628aa4a6d30 2024-03-18T14:54:45.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71bd7fa640 1 -- 172.21.15.76:0/2139251585 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f71b80cf750 con 0x7f71c0106660 2024-03-18T14:54:45.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71bffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa49d830 0x5628aa49dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71bffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa49d830 0x5628aa49dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:42800/0 (socket says 172.21.15.76:42800) 2024-03-18T14:54:45.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71bffff640 1 -- 172.21.15.76:0/637420125 learned_addr learned my addr 172.21.15.76:0/637420125 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:45.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71c48ef640 1 --2- 172.21.15.76:0/637420125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa4a6d30 0x5628aa4a9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71bf7fe640 1 --2- 172.21.15.76:0/637420125 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49e150 0x5628aa4a67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71bffff640 1 -- 172.21.15.76:0/637420125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa4a6d30 msgr2=0x5628aa4a9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:45.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71bffff640 1 --2- 172.21.15.76:0/637420125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa4a6d30 0x5628aa4a9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71bffff640 1 -- 172.21.15.76:0/637420125 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49e150 msgr2=0x5628aa4a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:45.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71bffff640 1 --2- 172.21.15.76:0/637420125 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49e150 0x5628aa4a67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.291+0000 7f71bffff640 1 -- 172.21.15.76:0/637420125 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628aa464500 con 0x5628aa49d830 2024-03-18T14:54:45.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.292+0000 7f71bf7fe640 1 --2- 172.21.15.76:0/2139251585 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f71a80a2a90 0x7f71a80a4f50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f71b4005ab0 tx=0x7f71b402c980 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:45.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.292+0000 7f71bffff640 1 --2- 172.21.15.76:0/637420125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa49d830 0x5628aa49dc10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f71b00115f0 tx=0x7f71b0011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afba843a3d96a835 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:45.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.292+0000 7f71a67fc640 1 -- 172.21.15.76:0/637420125 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b001e070 con 0x5628aa49d830 2024-03-18T14:54:45.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.292+0000 7f71a67fc640 1 -- 172.21.15.76:0/637420125 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71b000fd50 con 0x5628aa49d830 2024-03-18T14:54:45.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.292+0000 7f71a67fc640 1 -- 172.21.15.76:0/637420125 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b000e3b0 con 0x5628aa49d830 2024-03-18T14:54:45.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.292+0000 7f71c9010740 1 -- 172.21.15.76:0/637420125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa49d830 msgr2=0x5628aa49dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.292+0000 7f71c9010740 1 --2- 172.21.15.76:0/637420125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa49d830 0x5628aa49dc10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f71b00115f0 tx=0x7f71b0011ab0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.293+0000 7f71c9010740 1 -- 172.21.15.76:0/637420125 shutdown_connections 2024-03-18T14:54:45.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.293+0000 7f71c9010740 1 --2- 172.21.15.76:0/637420125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa4a6d30 0x5628aa4a9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.293+0000 7f71c9010740 1 --2- 172.21.15.76:0/637420125 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49e150 0x5628aa4a67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.293+0000 7f71c9010740 1 --2- 172.21.15.76:0/637420125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa49d830 0x5628aa49dc10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.293+0000 7f71c9010740 1 -- 172.21.15.76:0/637420125 >> 172.21.15.76:0/637420125 conn(0x5628aa497f60 msgr2=0x5628aa498340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:45.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.293+0000 7f71c9010740 1 -- 172.21.15.76:0/637420125 shutdown_connections 2024-03-18T14:54:45.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.293+0000 7f71c9010740 1 -- 172.21.15.76:0/637420125 wait complete. 2024-03-18T14:54:45.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.293+0000 7f71c9010740 1 -- start start 2024-03-18T14:54:45.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.293+0000 7f71c9010740 1 Processor -- start 2024-03-18T14:54:45.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.293+0000 7f71c9010740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49d830 0x5628aa4a3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.293+0000 7f71c9010740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49e150 0x5628aa4a4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.293+0000 7f71c9010740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa4a6d30 0x5628aa4a4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.293+0000 7f71c9010740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5628aa4a9b20 con 0x5628aa49d830 2024-03-18T14:54:45.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.293+0000 7f71c9010740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5628aa3bd230 con 0x5628aa4a6d30 2024-03-18T14:54:45.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.293+0000 7f71c9010740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5628aa4a9ca0 con 0x5628aa49e150 2024-03-18T14:54:45.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.293+0000 7f71c9010740 10 client.0 _ll_register_callbacks cb 0x5628aa3f9010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:45.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f71bffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49d830 0x5628aa4a3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f71bf7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49e150 0x5628aa4a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f71c48ef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa4a6d30 0x5628aa4a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f71bffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49d830 0x5628aa4a3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33440/0 (socket says 172.21.15.76:33440) 2024-03-18T14:54:45.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f71bf7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49e150 0x5628aa4a4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:58318/0 (socket says 172.21.15.76:58318) 2024-03-18T14:54:45.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f71bffff640 1 -- 172.21.15.76:0/3735541821 learned_addr learned my addr 172.21.15.76:0/3735541821 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:45.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f71c48ef640 1 -- 172.21.15.76:0/3735541821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49e150 msgr2=0x5628aa4a4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:45.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f71c48ef640 1 --2- 172.21.15.76:0/3735541821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49e150 0x5628aa4a4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f71c48ef640 1 -- 172.21.15.76:0/3735541821 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49d830 msgr2=0x5628aa4a3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:45.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f71c48ef640 1 --2- 172.21.15.76:0/3735541821 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49d830 0x5628aa4a3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f71c48ef640 1 -- 172.21.15.76:0/3735541821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628aa3cc100 con 0x5628aa4a6d30 2024-03-18T14:54:45.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f71bf7fe640 1 --2- 172.21.15.76:0/3735541821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49e150 0x5628aa4a4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:45.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f71bffff640 1 --2- 172.21.15.76:0/3735541821 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49d830 0x5628aa4a3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:45.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f71c48ef640 1 --2- 172.21.15.76:0/3735541821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa4a6d30 0x5628aa4a4770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f71b80ccd70 tx=0x7f71b809bc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83bff0714d87a659 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:45.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f719f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:54:45.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f719f7fe640 1 -- 172.21.15.76:0/3735541821 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b80a40c0 con 0x5628aa4a6d30 2024-03-18T14:54:45.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f719f7fe640 1 -- 172.21.15.76:0/3735541821 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f71b80cd050 con 0x5628aa4a6d30 2024-03-18T14:54:45.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f71c9010740 1 -- 172.21.15.76:0/3735541821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71b000fd50 con 0x5628aa4a6d30 2024-03-18T14:54:45.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f719f7fe640 1 -- 172.21.15.76:0/3735541821 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f71b80cd250 con 0x5628aa4a6d30 2024-03-18T14:54:45.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f71c9010740 10 client.25787 fetch_fsmap learned FSMap version 29 2024-03-18T14:54:45.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.294+0000 7f71c9010740 1 -- 172.21.15.76:0/3735541821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5628aa1153f0 con 0x5628aa4a6d30 2024-03-18T14:54:45.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.295+0000 7f719f7fe640 1 -- 172.21.15.76:0/3735541821 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71b000fd50 con 0x5628aa4a6d30 2024-03-18T14:54:45.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.295+0000 7f719f7fe640 1 -- 172.21.15.76:0/3735541821 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f71b809ac70 con 0x5628aa4a6d30 2024-03-18T14:54:45.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.295+0000 7f71c9010740 10 client.25787 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:54:45.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.295+0000 7f71c9010740 10 client.25787 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:45.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.295+0000 7f71c9010740 20 client.25787 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:45.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.295+0000 7f71c9010740 1 --2- 172.21.15.76:0/3735541821 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5628aa4b7130 0x5628aa4b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.295+0000 7f71c9010740 4 client.25787 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:54:45.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.295+0000 7f71c9010740 1 -- 172.21.15.76:0/3735541821 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628aa40e9d0 con 0x5628aa4b7130 2024-03-18T14:54:45.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.295+0000 7f71bffff640 1 --2- 172.21.15.76:0/3735541821 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5628aa4b7130 0x5628aa4b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.295+0000 7f719f7fe640 10 client.25787 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:45.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.295+0000 7f71bffff640 1 --2- 172.21.15.76:0/3735541821 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5628aa4b7130 0x5628aa4b95d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7725c69ac5c102 server_cookie=f9778fd4cb2d6499 in_seq=0 out_seq=0 2024-03-18T14:54:45.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.296+0000 7f719f7fe640 1 -- 172.21.15.76:0/3735541821 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5628aa40e9d0 con 0x5628aa4b7130 2024-03-18T14:54:45.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.296+0000 7f719f7fe640 10 client.25787 handle_command_reply: tid=0 2024-03-18T14:54:45.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.296+0000 7f71c9010740 1 client.25787 shutdown 2024-03-18T14:54:45.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.296+0000 7f71c9010740 1 -- 172.21.15.76:0/3735541821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa4a6d30 msgr2=0x5628aa4a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.296+0000 7f71c9010740 1 --2- 172.21.15.76:0/3735541821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa4a6d30 0x5628aa4a4770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f71b80ccd70 tx=0x7f71b809bc70 comp rx=0 tx=0).stop 2024-03-18T14:54:45.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.296+0000 7f71c9010740 1 -- 172.21.15.76:0/3735541821 shutdown_connections 2024-03-18T14:54:45.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.296+0000 7f71c9010740 1 --2- 172.21.15.76:0/3735541821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49e150 0x5628aa4a4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.296+0000 7f71c9010740 1 --2- 172.21.15.76:0/3735541821 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5628aa4b7130 0x5628aa4b95d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.296+0000 7f719f7fe640 0 client.25787 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:45.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.296+0000 7f71c9010740 1 --2- 172.21.15.76:0/3735541821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa4a6d30 0x5628aa4a4770 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.296+0000 7f71c9010740 1 --2- 172.21.15.76:0/3735541821 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49d830 0x5628aa4a3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.296+0000 7f71c9010740 1 -- 172.21.15.76:0/3735541821 >> 172.21.15.76:0/3735541821 conn(0x5628aa497f60 msgr2=0x5628aa499cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:45.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.297+0000 7f71c9010740 1 -- 172.21.15.76:0/3735541821 shutdown_connections 2024-03-18T14:54:45.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.297+0000 7f71c9010740 1 -- 172.21.15.76:0/3735541821 wait complete. 2024-03-18T14:54:45.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.297+0000 7f71c9010740 20 client.25787 trim_cache size 0 max 16384 2024-03-18T14:54:45.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.311+0000 7f71c9010740 1 Processor -- start 2024-03-18T14:54:45.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.311+0000 7f71c9010740 1 -- start start 2024-03-18T14:54:45.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.311+0000 7f71c9010740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49d830 0x5628aa4a6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.311+0000 7f71c9010740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa49e150 0x5628aa4b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.311+0000 7f71c9010740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa4a6d30 0x5628aa4beea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.311+0000 7f71c9010740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01afcc0 con 0x5628aa4a6d30 2024-03-18T14:54:45.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.311+0000 7f71c9010740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01ad530 con 0x5628aa49e150 2024-03-18T14:54:45.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.311+0000 7f71c9010740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01ab380 con 0x5628aa49d830 2024-03-18T14:54:45.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.312+0000 7f71c48ef640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa4a6d30 0x5628aa4beea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.312+0000 7f71bf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa49e150 0x5628aa4b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.312+0000 7f71bffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49d830 0x5628aa4a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.312+0000 7f71bf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa49e150 0x5628aa4b28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:42808/0 (socket says 172.21.15.76:42808) 2024-03-18T14:54:45.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.312+0000 7f71c48ef640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa4a6d30 0x5628aa4beea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33450/0 (socket says 172.21.15.76:33450) 2024-03-18T14:54:45.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.312+0000 7f71bf7fe640 1 -- 172.21.15.76:0/4070869446 learned_addr learned my addr 172.21.15.76:0/4070869446 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:45.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.312+0000 7f71c48ef640 1 -- 172.21.15.76:0/4070869446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49d830 msgr2=0x5628aa4a6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:45.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.312+0000 7f71c48ef640 1 --2- 172.21.15.76:0/4070869446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49d830 0x5628aa4a6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.312+0000 7f71c48ef640 1 -- 172.21.15.76:0/4070869446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa49e150 msgr2=0x5628aa4b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:45.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.312+0000 7f71c48ef640 1 --2- 172.21.15.76:0/4070869446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa49e150 0x5628aa4b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.312+0000 7f71c48ef640 1 -- 172.21.15.76:0/4070869446 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628aa1df7d0 con 0x5628aa4a6d30 2024-03-18T14:54:45.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.312+0000 7f71bf7fe640 1 --2- 172.21.15.76:0/4070869446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa49e150 0x5628aa4b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:45.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.312+0000 7f71c48ef640 1 --2- 172.21.15.76:0/4070869446 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa4a6d30 0x5628aa4beea0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f71b8098580 tx=0x7f71b8005940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e4b1ac8b06d0704 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:45.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.312+0000 7f71a5ffb640 1 -- 172.21.15.76:0/4070869446 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b809c2b0 con 0x5628aa4a6d30 2024-03-18T14:54:45.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.312+0000 7f71a5ffb640 1 -- 172.21.15.76:0/4070869446 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71b809fd00 con 0x5628aa4a6d30 2024-03-18T14:54:45.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.312+0000 7f71a5ffb640 1 -- 172.21.15.76:0/4070869446 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b80a4120 con 0x5628aa4a6d30 2024-03-18T14:54:45.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.313+0000 7f71c9010740 1 -- 172.21.15.76:0/4070869446 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa4a6d30 msgr2=0x5628aa4beea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.313+0000 7f71c9010740 1 --2- 172.21.15.76:0/4070869446 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa4a6d30 0x5628aa4beea0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f71b8098580 tx=0x7f71b8005940 comp rx=0 tx=0).stop 2024-03-18T14:54:45.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.313+0000 7f71c9010740 1 -- 172.21.15.76:0/4070869446 shutdown_connections 2024-03-18T14:54:45.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.313+0000 7f71c9010740 1 --2- 172.21.15.76:0/4070869446 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa4a6d30 0x5628aa4beea0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.313+0000 7f71c9010740 1 --2- 172.21.15.76:0/4070869446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa49e150 0x5628aa4b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.313+0000 7f71c9010740 1 --2- 172.21.15.76:0/4070869446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49d830 0x5628aa4a6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.313+0000 7f71c9010740 1 -- 172.21.15.76:0/4070869446 >> 172.21.15.76:0/4070869446 conn(0x5628aa497f60 msgr2=0x5628aa4a3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:45.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.313+0000 7f71c9010740 1 -- 172.21.15.76:0/4070869446 shutdown_connections 2024-03-18T14:54:45.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.313+0000 7f71c9010740 1 -- 172.21.15.76:0/4070869446 wait complete. 2024-03-18T14:54:45.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.314+0000 7f71c9010740 1 -- start start 2024-03-18T14:54:45.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.314+0000 7f71c9010740 1 Processor -- start 2024-03-18T14:54:45.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.314+0000 7f71c9010740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49d830 0x5628aa515840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.314+0000 7f71c9010740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49e150 0x5628aa515d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.314+0000 7f71c9010740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa4a6d30 0x5628aa4aa000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.314+0000 7f71c9010740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01ab9f0 con 0x5628aa49d830 2024-03-18T14:54:45.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.314+0000 7f71c9010740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01ace60 con 0x5628aa4a6d30 2024-03-18T14:54:45.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.314+0000 7f71c9010740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01ac0c0 con 0x5628aa49e150 2024-03-18T14:54:45.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.314+0000 7f71c9010740 10 client.0 _ll_register_callbacks cb 0x7f71b809b710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:45.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.314+0000 7f71bffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49d830 0x5628aa515840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.314+0000 7f71bffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49d830 0x5628aa515840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33452/0 (socket says 172.21.15.76:33452) 2024-03-18T14:54:45.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.314+0000 7f71bffff640 1 -- 172.21.15.76:0/722252716 learned_addr learned my addr 172.21.15.76:0/722252716 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:45.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.314+0000 7f71bf7fe640 1 --2- 172.21.15.76:0/722252716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49e150 0x5628aa515d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.314+0000 7f71c48ef640 1 --2- 172.21.15.76:0/722252716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa4a6d30 0x5628aa4aa000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.315+0000 7f71bffff640 1 -- 172.21.15.76:0/722252716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49e150 msgr2=0x5628aa515d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:45.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.315+0000 7f71bffff640 1 --2- 172.21.15.76:0/722252716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49e150 0x5628aa515d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.315+0000 7f71bffff640 1 -- 172.21.15.76:0/722252716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa4a6d30 msgr2=0x5628aa4aa000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:45.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.315+0000 7f71bffff640 1 --2- 172.21.15.76:0/722252716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa4a6d30 0x5628aa4aa000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.315+0000 7f71bffff640 1 -- 172.21.15.76:0/722252716 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628aa4aa6f0 con 0x5628aa49d830 2024-03-18T14:54:45.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.315+0000 7f71c48ef640 1 --2- 172.21.15.76:0/722252716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa4a6d30 0x5628aa4aa000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:45.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.315+0000 7f71bf7fe640 1 --2- 172.21.15.76:0/722252716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49e150 0x5628aa515d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:45.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.315+0000 7f71bffff640 1 --2- 172.21.15.76:0/722252716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49d830 0x5628aa515840 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f71b0019c70 tx=0x7f71b0019b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c17ea8b93ef6c771 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:45.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.315+0000 7f719f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:45.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.315+0000 7f719f7fe640 1 -- 172.21.15.76:0/722252716 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b001e040 con 0x5628aa49d830 2024-03-18T14:54:45.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.315+0000 7f71c9010740 1 -- 172.21.15.76:0/722252716 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71b809fd00 con 0x5628aa49d830 2024-03-18T14:54:45.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.315+0000 7f719f7fe640 1 -- 172.21.15.76:0/722252716 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f71b0005040 con 0x5628aa49d830 2024-03-18T14:54:45.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.315+0000 7f719f7fe640 1 -- 172.21.15.76:0/722252716 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f71b0033020 con 0x5628aa49d830 2024-03-18T14:54:45.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.315+0000 7f71c9010740 10 client.15936 fetch_fsmap learned FSMap version 29 2024-03-18T14:54:45.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.315+0000 7f71c9010740 1 -- 172.21.15.76:0/722252716 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5628aa51cf90 con 0x5628aa49d830 2024-03-18T14:54:45.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.316+0000 7f719f7fe640 1 -- 172.21.15.76:0/722252716 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71b809fd00 con 0x5628aa49d830 2024-03-18T14:54:45.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.316+0000 7f719f7fe640 1 -- 172.21.15.76:0/722252716 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f71b0004c10 con 0x5628aa49d830 2024-03-18T14:54:45.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.316+0000 7f71c9010740 10 client.15936 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:54:45.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.316+0000 7f71c9010740 10 client.15936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:45.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.316+0000 7f71c9010740 20 client.15936 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:45.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.316+0000 7f71c9010740 1 --2- 172.21.15.76:0/722252716 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5628aa4b7130 0x5628aa51f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:45.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.316+0000 7f71c9010740 4 client.15936 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T14:54:45.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.316+0000 7f71c9010740 1 -- 172.21.15.76:0/722252716 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5628aa408d20 con 0x5628aa4b7130 2024-03-18T14:54:45.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.316+0000 7f71bf7fe640 1 --2- 172.21.15.76:0/722252716 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5628aa4b7130 0x5628aa51f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:45.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.316+0000 7f71bf7fe640 1 --2- 172.21.15.76:0/722252716 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5628aa4b7130 0x5628aa51f570 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f6926f663568da server_cookie=fe472845ea314d32 in_seq=0 out_seq=0 2024-03-18T14:54:45.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.316+0000 7f719f7fe640 10 client.15936 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:45.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.317+0000 7f719f7fe640 1 -- 172.21.15.76:0/722252716 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5628aa408d20 con 0x5628aa4b7130 2024-03-18T14:54:45.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.317+0000 7f719f7fe640 10 client.15936 handle_command_reply: tid=0 2024-03-18T14:54:45.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.317+0000 7f71c9010740 1 client.15936 shutdown 2024-03-18T14:54:45.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.317+0000 7f71c9010740 1 -- 172.21.15.76:0/722252716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49d830 msgr2=0x5628aa515840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.317+0000 7f71c9010740 1 --2- 172.21.15.76:0/722252716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49d830 0x5628aa515840 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f71b0019c70 tx=0x7f71b0019b90 comp rx=0 tx=0).stop 2024-03-18T14:54:45.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.317+0000 7f71c9010740 1 -- 172.21.15.76:0/722252716 shutdown_connections 2024-03-18T14:54:45.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.317+0000 7f71c9010740 1 --2- 172.21.15.76:0/722252716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628aa49e150 0x5628aa515d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.317+0000 7f71c9010740 1 --2- 172.21.15.76:0/722252716 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5628aa4b7130 0x5628aa51f570 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.317+0000 7f71c9010740 1 --2- 172.21.15.76:0/722252716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628aa4a6d30 0x5628aa4aa000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.317+0000 7f71c9010740 1 --2- 172.21.15.76:0/722252716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628aa49d830 0x5628aa515840 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.317+0000 7f719f7fe640 0 client.15936 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:45.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.317+0000 7f71c9010740 1 -- 172.21.15.76:0/722252716 >> 172.21.15.76:0/722252716 conn(0x5628aa497f60 msgr2=0x5628aa499b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:45.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.317+0000 7f71c9010740 1 -- 172.21.15.76:0/722252716 shutdown_connections 2024-03-18T14:54:45.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.317+0000 7f71c9010740 1 -- 172.21.15.76:0/722252716 wait complete. 2024-03-18T14:54:45.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.317+0000 7f71c9010740 20 client.15936 trim_cache size 0 max 16384 2024-03-18T14:54:45.325 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-18T14:54:45.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.318+0000 7f71c6352640 1 -- 172.21.15.76:0/2139251585 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f71a80a2a90 msgr2=0x7f71a80a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.318+0000 7f71c6352640 1 --2- 172.21.15.76:0/2139251585 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f71a80a2a90 0x7f71a80a4f50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f71b4005ab0 tx=0x7f71b402c980 comp rx=0 tx=0).stop 2024-03-18T14:54:45.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.318+0000 7f71c6352640 1 -- 172.21.15.76:0/2139251585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71c0106660 msgr2=0x7f71c019c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:45.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.318+0000 7f71c6352640 1 --2- 172.21.15.76:0/2139251585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71c0106660 0x7f71c019c950 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f71b8009fd0 tx=0x7f71b800ef10 comp rx=0 tx=0).stop 2024-03-18T14:54:45.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.318+0000 7f71c6352640 1 -- 172.21.15.76:0/2139251585 shutdown_connections 2024-03-18T14:54:45.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.318+0000 7f71c6352640 1 --2- 172.21.15.76:0/2139251585 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f71a80a2a90 0x7f71a80a4f50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.318+0000 7f71c6352640 1 --2- 172.21.15.76:0/2139251585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71c0101010 0x7f71c0198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.318+0000 7f71c6352640 1 --2- 172.21.15.76:0/2139251585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71c0106660 0x7f71c019c950 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.318+0000 7f71c6352640 1 --2- 172.21.15.76:0/2139251585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71c0100650 0x7f71c01980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:45.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.318+0000 7f71c6352640 1 -- 172.21.15.76:0/2139251585 >> 172.21.15.76:0/2139251585 conn(0x7f71c00777e0 msgr2=0x7f71c0102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:45.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.318+0000 7f71c6352640 1 -- 172.21.15.76:0/2139251585 shutdown_connections 2024-03-18T14:54:45.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T14:54:45.318+0000 7f71c6352640 1 -- 172.21.15.76:0/2139251585 wait complete. 2024-03-18T14:54:45.330 ERROR:tasks.fwd_scrub:rank 2 damaged: [{'damage_type': 'backtrace', 'id': 94420810, 'ino': 1099511628308, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TRIDOTS.POT'}, {'damage_type': 'backtrace', 'id': 314228977, 'ino': 1099511628318, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR'}, {'damage_type': 'backtrace', 'id': 398535490, 'ino': 1099511628326, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS'}, {'damage_type': 'backtrace', 'id': 459283709, 'ino': 1099511628316, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT'}, {'damage_type': 'backtrace', 'id': 557683196, 'ino': 1099511628304, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PCBENCHM.PPT'}, {'damage_type': 'backtrace', 'id': 905097938, 'ino': 1099511628307, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TIPS.PPT'}, {'damage_type': 'backtrace', 'id': 1123835392, 'ino': 1099511628327, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC'}, {'damage_type': 'backtrace', 'id': 1581852873, 'ino': 1099511628312, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL'}, {'damage_type': 'backtrace', 'id': 2106152765, 'ino': 1099511628319, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT'}, {'damage_type': 'backtrace', 'id': 2110312286, 'ino': 1099511628317, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR'}, {'damage_type': 'backtrace', 'id': 2516886946, 'ino': 1099511628314, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT'}, {'damage_type': 'backtrace', 'id': 2615612840, 'ino': 1099511628305, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ZD16.BMP'}, {'damage_type': 'backtrace', 'id': 2858841007, 'ino': 1099511628315, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT'}, {'damage_type': 'backtrace', 'id': 2885331974, 'ino': 1099511628311, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG'}, {'damage_type': 'backtrace', 'id': 3082383376, 'ino': 1099511628313, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR'}, {'damage_type': 'backtrace', 'id': 3934675315, 'ino': 1099511628328, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC'}, {'damage_type': 'backtrace', 'id': 4185442196, 'ino': 1099511628306, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PPTOOLS1.PPA'}, {'damage_type': 'backtrace', 'id': 4205518695, 'ino': 1099511628310, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG'}, {'damage_type': 'backtrace', 'id': 4245226452, 'ino': 1099511628320, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT'}] 2024-03-18T14:54:45.330 ERROR:tasks.fwd_scrub:rank 1 damaged: [{'damage_type': 'backtrace', 'id': 200727419, 'ino': 1099511628344, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.X03'}, {'damage_type': 'backtrace', 'id': 369657943, 'ino': 1099511628348, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.FSL'}, {'damage_type': 'backtrace', 'id': 390129868, 'ino': 1099511628336, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG0'}, {'damage_type': 'backtrace', 'id': 694303819, 'ino': 1099511628339, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG1'}, {'damage_type': 'backtrace', 'id': 869040883, 'ino': 1099511628333, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.Y04'}, {'damage_type': 'backtrace', 'id': 1055885286, 'ino': 1099511628349, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.FSL'}, {'damage_type': 'backtrace', 'id': 1181399849, 'ino': 1099511628334, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.DB'}, {'damage_type': 'backtrace', 'id': 1196012775, 'ino': 1099511628346, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.VAL'}, {'damage_type': 'backtrace', 'id': 1511329952, 'ino': 1099511628347, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.TV'}, {'damage_type': 'backtrace', 'id': 2152505386, 'ino': 1099511628335, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.PX'}, {'damage_type': 'backtrace', 'id': 2461033744, 'ino': 1099511628342, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.DB'}, {'damage_type': 'backtrace', 'id': 2530979369, 'ino': 1099511628340, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.VAL'}, {'damage_type': 'backtrace', 'id': 2685593990, 'ino': 1099511628332, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.X04'}, {'damage_type': 'backtrace', 'id': 2729367069, 'ino': 1099511628338, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG1'}, {'damage_type': 'backtrace', 'id': 2785978108, 'ino': 1099511628330, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.DB'}, {'damage_type': 'backtrace', 'id': 2885482703, 'ino': 1099511628331, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.PX'}, {'damage_type': 'backtrace', 'id': 3346329482, 'ino': 1099511628337, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG0'}, {'damage_type': 'backtrace', 'id': 3517481119, 'ino': 1099511628345, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.Y03'}, {'damage_type': 'backtrace', 'id': 3623539156, 'ino': 1099511628341, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.MB'}, {'damage_type': 'backtrace', 'id': 3701451795, 'ino': 1099511628343, 'path': '/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.PX'}] 2024-03-18T14:54:45.330 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/fwd_scrub.py", line 79, in _scrub self._check_damage() File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/fwd_scrub.py", line 90, in _check_damage raise RuntimeError(f"rank damage found: {types}") RuntimeError: rank damage found: {'backtrace'} 2024-03-18T14:54:45.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 76727 9.31 MB/sec warmup 46 sec latency 17.556 ms 2024-03-18T14:54:45.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:45 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/438176479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:54:45.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:45 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/438176479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:54:45.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:45 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/438176479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T14:54:46.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 78641 9.40 MB/sec warmup 47 sec latency 23.892 ms 2024-03-18T14:54:46.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:46 smithi076 ceph-mon[32205]: pgmap v49: 97 pgs: 97 active+clean; 36 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 299 op/s 2024-03-18T14:54:46.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:46 smithi174 ceph-mon[36787]: pgmap v49: 97 pgs: 97 active+clean; 36 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 299 op/s 2024-03-18T14:54:46.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:46 smithi077 ceph-mon[35865]: pgmap v49: 97 pgs: 97 active+clean; 36 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 299 op/s 2024-03-18T14:54:47.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 81259 9.48 MB/sec warmup 48 sec latency 17.150 ms 2024-03-18T14:54:47.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:47 smithi076 ceph-mon[32205]: pgmap v50: 97 pgs: 97 active+clean; 36 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 299 op/s 2024-03-18T14:54:47.969 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:47 smithi174 ceph-mon[36787]: pgmap v50: 97 pgs: 97 active+clean; 36 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 299 op/s 2024-03-18T14:54:47.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:47 smithi077 ceph-mon[35865]: pgmap v50: 97 pgs: 97 active+clean; 36 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 299 op/s 2024-03-18T14:54:48.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 82673 9.44 MB/sec warmup 49 sec latency 15.409 ms 2024-03-18T14:54:49.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 85465 9.59 MB/sec warmup 50 sec latency 11.862 ms 2024-03-18T14:54:49.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:49 smithi076 ceph-mon[32205]: mgrmap e25: x(active, since 91s), standbys: z, y 2024-03-18T14:54:49.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:49 smithi076 ceph-mon[32205]: pgmap v51: 97 pgs: 97 active+clean; 41 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 297 op/s 2024-03-18T14:54:49.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:49 smithi174 ceph-mon[36787]: mgrmap e25: x(active, since 91s), standbys: z, y 2024-03-18T14:54:49.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:49 smithi174 ceph-mon[36787]: pgmap v51: 97 pgs: 97 active+clean; 41 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 297 op/s 2024-03-18T14:54:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:49 smithi077 ceph-mon[35865]: mgrmap e25: x(active, since 91s), standbys: z, y 2024-03-18T14:54:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:49 smithi077 ceph-mon[35865]: pgmap v51: 97 pgs: 97 active+clean; 41 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 297 op/s 2024-03-18T14:54:50.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 87243 9.52 MB/sec warmup 51 sec latency 18.490 ms 2024-03-18T14:54:51.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 89259 9.60 MB/sec warmup 52 sec latency 10.489 ms 2024-03-18T14:54:52.284 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.281+0000 7ff6cd00b640 1 -- 172.21.15.76:0/3553322930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6c8100930 msgr2=0x7ff6c8100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:52.284 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.281+0000 7ff6cd00b640 1 --2- 172.21.15.76:0/3553322930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6c8100930 0x7ff6c8100d90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc009900 tx=0x7ff6bc02bc30 comp rx=0 tx=0).stop 2024-03-18T14:54:52.284 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.281+0000 7ff6cd00b640 1 -- 172.21.15.76:0/3553322930 shutdown_connections 2024-03-18T14:54:52.284 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.281+0000 7ff6cd00b640 1 --2- 172.21.15.76:0/3553322930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6c81012d0 0x7ff6c810d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.284 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.281+0000 7ff6cd00b640 1 --2- 172.21.15.76:0/3553322930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6c8100930 0x7ff6c8100d90 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.284 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.281+0000 7ff6cd00b640 1 --2- 172.21.15.76:0/3553322930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6c8106760 0x7ff6c8106b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.284 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.281+0000 7ff6cd00b640 1 -- 172.21.15.76:0/3553322930 >> 172.21.15.76:0/3553322930 conn(0x7ff6c8077540 msgr2=0x7ff6c8077950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:52.284 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.281+0000 7ff6cd00b640 1 -- 172.21.15.76:0/3553322930 shutdown_connections 2024-03-18T14:54:52.285 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.281+0000 7ff6cd00b640 1 -- 172.21.15.76:0/3553322930 wait complete. 2024-03-18T14:54:52.285 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.282+0000 7ff6cd00b640 1 Processor -- start 2024-03-18T14:54:52.285 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.282+0000 7ff6cd00b640 1 -- start start 2024-03-18T14:54:52.285 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.282+0000 7ff6cd00b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6c8100930 0x7ff6c8198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.285 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.282+0000 7ff6cd00b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6c81012d0 0x7ff6c8198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.285 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.282+0000 7ff6cd00b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6c8106760 0x7ff6c819c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.285 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.282+0000 7ff6cd00b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c8110ae0 con 0x7ff6c81012d0 2024-03-18T14:54:52.285 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.282+0000 7ff6cd00b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c8110960 con 0x7ff6c8106760 2024-03-18T14:54:52.285 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.282+0000 7ff6cd00b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c8110c60 con 0x7ff6c8100930 2024-03-18T14:54:52.285 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.282+0000 7ff6c659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6c81012d0 0x7ff6c8198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.285 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.282+0000 7ff6c759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6c8106760 0x7ff6c819c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.285 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.282+0000 7ff6c6d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6c8100930 0x7ff6c8198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.285 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.282+0000 7ff6c659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6c81012d0 0x7ff6c8198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46478/0 (socket says 172.21.15.76:46478) 2024-03-18T14:54:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.282+0000 7ff6c759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6c8106760 0x7ff6c819c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:35070/0 (socket says 172.21.15.76:35070) 2024-03-18T14:54:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.282+0000 7ff6c659c640 1 -- 172.21.15.76:0/110745026 learned_addr learned my addr 172.21.15.76:0/110745026 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.282+0000 7ff6c6d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6c8100930 0x7ff6c8198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:35472/0 (socket says 172.21.15.76:35472) 2024-03-18T14:54:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.283+0000 7ff6c659c640 1 -- 172.21.15.76:0/110745026 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6c8100930 msgr2=0x7ff6c8198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.283+0000 7ff6c659c640 1 --2- 172.21.15.76:0/110745026 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6c8100930 0x7ff6c8198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.283+0000 7ff6c659c640 1 -- 172.21.15.76:0/110745026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6c8106760 msgr2=0x7ff6c819c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.283+0000 7ff6c659c640 1 --2- 172.21.15.76:0/110745026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6c8106760 0x7ff6c819c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.283+0000 7ff6c659c640 1 -- 172.21.15.76:0/110745026 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c819cfd0 con 0x7ff6c81012d0 2024-03-18T14:54:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.283+0000 7ff6c759e640 1 --2- 172.21.15.76:0/110745026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6c8106760 0x7ff6c819c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:54:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.283+0000 7ff6c659c640 1 --2- 172.21.15.76:0/110745026 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6c81012d0 0x7ff6c8198570 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc006010 tx=0x7ff6bc02cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.283+0000 7ff6affff640 1 -- 172.21.15.76:0/110745026 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc041070 con 0x7ff6c81012d0 2024-03-18T14:54:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.283+0000 7ff6cd00b640 1 -- 172.21.15.76:0/110745026 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6c819d260 con 0x7ff6c81012d0 2024-03-18T14:54:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.283+0000 7ff6affff640 1 -- 172.21.15.76:0/110745026 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6bc004450 con 0x7ff6c81012d0 2024-03-18T14:54:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.283+0000 7ff6cd00b640 1 -- 172.21.15.76:0/110745026 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6c81a4b50 con 0x7ff6c81012d0 2024-03-18T14:54:52.287 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.283+0000 7ff6affff640 1 -- 172.21.15.76:0/110745026 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc04b060 con 0x7ff6c81012d0 2024-03-18T14:54:52.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.285+0000 7ff6affff640 1 -- 172.21.15.76:0/110745026 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7ff6bc044070 con 0x7ff6c81012d0 2024-03-18T14:54:52.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.285+0000 7ff6cfbf7740 1 Processor -- start 2024-03-18T14:54:52.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.285+0000 7ff6cfbf7740 1 -- start start 2024-03-18T14:54:52.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6cfbf7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e50e130 0x55da8e50e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6cfbf7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e50ea50 0x55da8e5172e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6cfbf7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e517820 0x55da8e519c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6cfbf7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55da8e472a40 con 0x55da8e517820 2024-03-18T14:54:52.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6cfbf7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55da8e41b920 con 0x55da8e50e130 2024-03-18T14:54:52.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6cfbf7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55da8e46eae0 con 0x55da8e50ea50 2024-03-18T14:54:52.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6c759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e50e130 0x55da8e50e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6c659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e517820 0x55da8e519c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6c759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e50e130 0x55da8e50e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:35078/0 (socket says 172.21.15.76:35078) 2024-03-18T14:54:52.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6c759e640 1 -- 172.21.15.76:0/94070487 learned_addr learned my addr 172.21.15.76:0/94070487 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:52.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6c6d9d640 1 --2- 172.21.15.76:0/94070487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e50ea50 0x55da8e5172e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6affff640 1 --2- 172.21.15.76:0/110745026 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6b00a2a10 0x7ff6b00a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6c759e640 1 -- 172.21.15.76:0/94070487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e50ea50 msgr2=0x55da8e5172e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6c759e640 1 --2- 172.21.15.76:0/94070487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e50ea50 0x55da8e5172e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6c759e640 1 -- 172.21.15.76:0/94070487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e517820 msgr2=0x55da8e519c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6c759e640 1 --2- 172.21.15.76:0/94070487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e517820 0x55da8e519c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6c759e640 1 -- 172.21.15.76:0/94070487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da8e433460 con 0x55da8e50e130 2024-03-18T14:54:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6affff640 1 -- 172.21.15.76:0/110745026 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7ff6bc0f0760 con 0x7ff6c81012d0 2024-03-18T14:54:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6c759e640 1 --2- 172.21.15.76:0/110745026 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6b00a2a10 0x7ff6b00a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.286+0000 7ff6c759e640 1 --2- 172.21.15.76:0/94070487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e50e130 0x55da8e50e510 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00117f0 tx=0x7ff6c0011cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abe5057de205cdef server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.287+0000 7ff6ad7fa640 1 -- 172.21.15.76:0/94070487 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0020070 con 0x55da8e50e130 2024-03-18T14:54:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.287+0000 7ff6ad7fa640 1 -- 172.21.15.76:0/94070487 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6c000fc20 con 0x55da8e50e130 2024-03-18T14:54:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.287+0000 7ff6ad7fa640 1 -- 172.21.15.76:0/94070487 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c001b8a0 con 0x55da8e50e130 2024-03-18T14:54:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.287+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/94070487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e50e130 msgr2=0x55da8e50e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.287+0000 7ff6c759e640 1 --2- 172.21.15.76:0/110745026 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6b00a2a10 0x7ff6b00a4ed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff6c001d000 tx=0x7ff6c000e3b0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.287+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/94070487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e50e130 0x55da8e50e510 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00117f0 tx=0x7ff6c0011cb0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.287+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/94070487 shutdown_connections 2024-03-18T14:54:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.287+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/94070487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e50ea50 0x55da8e5172e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.287+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/94070487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e517820 0x55da8e519c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.287+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/94070487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e50e130 0x55da8e50e510 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.287+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/94070487 >> 172.21.15.76:0/94070487 conn(0x55da8e508a50 msgr2=0x55da8e508e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.287+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/94070487 shutdown_connections 2024-03-18T14:54:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.287+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/94070487 wait complete. 2024-03-18T14:54:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.288+0000 7ff6cfbf7740 1 -- start start 2024-03-18T14:54:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.288+0000 7ff6cfbf7740 1 Processor -- start 2024-03-18T14:54:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.288+0000 7ff6cfbf7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 0x55da8e514810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.288+0000 7ff6cfbf7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e50ea50 0x55da8e514d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.288+0000 7ff6cfbf7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e517820 0x55da8e515290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.288+0000 7ff6cfbf7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55da8e51a3f0 con 0x55da8e50e130 2024-03-18T14:54:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.288+0000 7ff6cfbf7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55da8e42ed70 con 0x55da8e517820 2024-03-18T14:54:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.288+0000 7ff6cfbf7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55da8e51a570 con 0x55da8e50ea50 2024-03-18T14:54:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.288+0000 7ff6cfbf7740 10 client.0 _ll_register_callbacks cb 0x55da8e186a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.288+0000 7ff6c6d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 0x55da8e514810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.288+0000 7ff6c659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e50ea50 0x55da8e514d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.288+0000 7ff6c759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e517820 0x55da8e515290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.288+0000 7ff6c759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e517820 0x55da8e515290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:35094/0 (socket says 172.21.15.76:35094) 2024-03-18T14:54:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.288+0000 7ff6c6d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 0x55da8e514810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46504/0 (socket says 172.21.15.76:46504) 2024-03-18T14:54:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.288+0000 7ff6c659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e50ea50 0x55da8e514d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:35494/0 (socket says 172.21.15.76:35494) 2024-03-18T14:54:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.288+0000 7ff6c759e640 1 -- 172.21.15.76:0/2324920480 learned_addr learned my addr 172.21.15.76:0/2324920480 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.289+0000 7ff6c759e640 1 -- 172.21.15.76:0/2324920480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e50ea50 msgr2=0x55da8e514d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.289+0000 7ff6c759e640 1 --2- 172.21.15.76:0/2324920480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e50ea50 0x55da8e514d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.289+0000 7ff6c759e640 1 -- 172.21.15.76:0/2324920480 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 msgr2=0x55da8e514810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.289+0000 7ff6c759e640 1 --2- 172.21.15.76:0/2324920480 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 0x55da8e514810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.289+0000 7ff6c759e640 1 -- 172.21.15.76:0/2324920480 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da8e187210 con 0x55da8e517820 2024-03-18T14:54:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.289+0000 7ff6c759e640 1 --2- 172.21.15.76:0/2324920480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e517820 0x55da8e515290 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff6c001b500 tx=0x7ff6c0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d04c4f48c2a4ac41 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.289+0000 7ff6aaff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:54:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.289+0000 7ff6aaff5640 1 -- 172.21.15.76:0/2324920480 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0020050 con 0x55da8e517820 2024-03-18T14:54:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.289+0000 7ff6aaff5640 1 -- 172.21.15.76:0/2324920480 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7ff6c000eb10 con 0x55da8e517820 2024-03-18T14:54:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.289+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/2324920480 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6c000fc20 con 0x55da8e517820 2024-03-18T14:54:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.289+0000 7ff6aaff5640 1 -- 172.21.15.76:0/2324920480 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff6c000ed10 con 0x55da8e517820 2024-03-18T14:54:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.289+0000 7ff6cfbf7740 10 client.25805 fetch_fsmap learned FSMap version 29 2024-03-18T14:54:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.289+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/2324920480 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55da8e253620 con 0x55da8e517820 2024-03-18T14:54:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.290+0000 7ff6aaff5640 1 -- 172.21.15.76:0/2324920480 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6c000fc20 con 0x55da8e517820 2024-03-18T14:54:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.290+0000 7ff6aaff5640 1 -- 172.21.15.76:0/2324920480 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7ff6c0004ce0 con 0x55da8e517820 2024-03-18T14:54:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.290+0000 7ff6cfbf7740 10 client.25805 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:54:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.290+0000 7ff6cfbf7740 10 client.25805 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.290+0000 7ff6cfbf7740 20 client.25805 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.290+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/2324920480 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55da8e5278f0 0x55da8e529ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.290+0000 7ff6cfbf7740 4 client.25805 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:54:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.290+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/2324920480 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da8e47ff80 con 0x55da8e5278f0 2024-03-18T14:54:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.290+0000 7ff6c6d9d640 1 --2- 172.21.15.76:0/2324920480 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55da8e5278f0 0x55da8e529ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.290+0000 7ff6aaff5640 10 client.25805 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.290+0000 7ff6c6d9d640 1 --2- 172.21.15.76:0/2324920480 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55da8e5278f0 0x55da8e529ce0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbe82cf42020ea1d server_cookie=377ea15257289592 in_seq=0 out_seq=0 2024-03-18T14:54:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.291+0000 7ff6aaff5640 1 -- 172.21.15.76:0/2324920480 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55da8e47ff80 con 0x55da8e5278f0 2024-03-18T14:54:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.291+0000 7ff6aaff5640 10 client.25805 handle_command_reply: tid=0 2024-03-18T14:54:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.291+0000 7ff6cfbf7740 1 client.25805 shutdown 2024-03-18T14:54:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.291+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/2324920480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e517820 msgr2=0x55da8e515290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:52.300 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.291+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/2324920480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e517820 0x55da8e515290 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff6c001b500 tx=0x7ff6c0016000 comp rx=0 tx=0).stop 2024-03-18T14:54:52.300 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.291+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/2324920480 shutdown_connections 2024-03-18T14:54:52.300 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.291+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/2324920480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e50ea50 0x55da8e514d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.300 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.291+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/2324920480 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55da8e5278f0 0x55da8e529ce0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.300 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.291+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/2324920480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e517820 0x55da8e515290 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.300 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.291+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/2324920480 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 0x55da8e514810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.300 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.291+0000 7ff6aaff5640 0 client.25805 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:52.300 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.291+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/2324920480 >> 172.21.15.76:0/2324920480 conn(0x55da8e508a50 msgr2=0x55da8e50a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:52.300 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.291+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/2324920480 shutdown_connections 2024-03-18T14:54:52.300 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.291+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/2324920480 wait complete. 2024-03-18T14:54:52.300 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.291+0000 7ff6cfbf7740 20 client.25805 trim_cache size 0 max 16384 2024-03-18T14:54:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.306+0000 7ff6cfbf7740 1 Processor -- start 2024-03-18T14:54:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.306+0000 7ff6cfbf7740 1 -- start start 2024-03-18T14:54:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.306+0000 7ff6cfbf7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 0x55da8e523190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.306+0000 7ff6cfbf7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e50ea50 0x55da8e52f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.306+0000 7ff6cfbf7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e517820 0x55da8e51f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.307+0000 7ff6cfbf7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c81afcd0 con 0x55da8e50e130 2024-03-18T14:54:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.307+0000 7ff6cfbf7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c81ad540 con 0x55da8e50ea50 2024-03-18T14:54:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.307+0000 7ff6cfbf7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c81ab390 con 0x55da8e517820 2024-03-18T14:54:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.307+0000 7ff6c6d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 0x55da8e523190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.307+0000 7ff6c6d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 0x55da8e523190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46508/0 (socket says 172.21.15.76:46508) 2024-03-18T14:54:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.307+0000 7ff6c6d9d640 1 -- 172.21.15.76:0/217787792 learned_addr learned my addr 172.21.15.76:0/217787792 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.307+0000 7ff6c659c640 1 --2- 172.21.15.76:0/217787792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e50ea50 0x55da8e52f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.307+0000 7ff6c759e640 1 --2- 172.21.15.76:0/217787792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e517820 0x55da8e51f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.307+0000 7ff6c6d9d640 1 -- 172.21.15.76:0/217787792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e517820 msgr2=0x55da8e51f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.307+0000 7ff6c6d9d640 1 --2- 172.21.15.76:0/217787792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e517820 0x55da8e51f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.307+0000 7ff6c6d9d640 1 -- 172.21.15.76:0/217787792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e50ea50 msgr2=0x55da8e52f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.307+0000 7ff6c6d9d640 1 --2- 172.21.15.76:0/217787792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e50ea50 0x55da8e52f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.307+0000 7ff6c6d9d640 1 -- 172.21.15.76:0/217787792 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da8e52a270 con 0x55da8e50e130 2024-03-18T14:54:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.307+0000 7ff6c6d9d640 1 --2- 172.21.15.76:0/217787792 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 0x55da8e523190 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8009c90 tx=0x7ff6b8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ecc155035e7708c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.307+0000 7ff6acff9640 1 -- 172.21.15.76:0/217787792 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8024030 con 0x55da8e50e130 2024-03-18T14:54:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.307+0000 7ff6acff9640 1 -- 172.21.15.76:0/217787792 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6b8005500 con 0x55da8e50e130 2024-03-18T14:54:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.307+0000 7ff6acff9640 1 -- 172.21.15.76:0/217787792 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b80203e0 con 0x55da8e50e130 2024-03-18T14:54:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.308+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/217787792 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 msgr2=0x55da8e523190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.308+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/217787792 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 0x55da8e523190 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8009c90 tx=0x7ff6b8005f90 comp rx=0 tx=0).stop 2024-03-18T14:54:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.308+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/217787792 shutdown_connections 2024-03-18T14:54:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.308+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/217787792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e517820 0x55da8e51f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.308+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/217787792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e50ea50 0x55da8e52f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.308+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/217787792 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 0x55da8e523190 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.308+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/217787792 >> 172.21.15.76:0/217787792 conn(0x55da8e508a50 msgr2=0x55da8e507320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.308+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/217787792 shutdown_connections 2024-03-18T14:54:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.308+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/217787792 wait complete. 2024-03-18T14:54:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.308+0000 7ff6cfbf7740 1 -- start start 2024-03-18T14:54:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.308+0000 7ff6cfbf7740 1 Processor -- start 2024-03-18T14:54:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.308+0000 7ff6cfbf7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 0x55da8e511b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.308+0000 7ff6cfbf7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e50ea50 0x55da8e5120c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.308+0000 7ff6cfbf7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e517820 0x55da8e58cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.308+0000 7ff6cfbf7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c81aba00 con 0x55da8e50e130 2024-03-18T14:54:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.308+0000 7ff6cfbf7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c81ace70 con 0x55da8e517820 2024-03-18T14:54:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6cfbf7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c81ac0d0 con 0x55da8e50ea50 2024-03-18T14:54:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6cfbf7740 10 client.0 _ll_register_callbacks cb 0x7ff6c001e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:54:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6c6d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 0x55da8e511b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6c6d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 0x55da8e511b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46524/0 (socket says 172.21.15.76:46524) 2024-03-18T14:54:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6c6d9d640 1 -- 172.21.15.76:0/1978768085 learned_addr learned my addr 172.21.15.76:0/1978768085 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:54:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6c659c640 1 --2- 172.21.15.76:0/1978768085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e50ea50 0x55da8e5120c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6c759e640 1 --2- 172.21.15.76:0/1978768085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e517820 0x55da8e58cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6c6d9d640 1 -- 172.21.15.76:0/1978768085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e50ea50 msgr2=0x55da8e5120c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6c6d9d640 1 --2- 172.21.15.76:0/1978768085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e50ea50 0x55da8e5120c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6c6d9d640 1 -- 172.21.15.76:0/1978768085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e517820 msgr2=0x55da8e58cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:54:52.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6c6d9d640 1 --2- 172.21.15.76:0/1978768085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e517820 0x55da8e58cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6c6d9d640 1 -- 172.21.15.76:0/1978768085 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da8e5236d0 con 0x55da8e50e130 2024-03-18T14:54:52.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6c6d9d640 1 --2- 172.21.15.76:0/1978768085 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 0x55da8e511b80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8000c00 tx=0x7ff6b8005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c558f289938858 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:54:52.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6aaff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:54:52.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6aaff5640 1 -- 172.21.15.76:0/1978768085 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b802d020 con 0x55da8e50e130 2024-03-18T14:54:52.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6aaff5640 1 -- 172.21.15.76:0/1978768085 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7ff6b8004700 con 0x55da8e50e130 2024-03-18T14:54:52.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/1978768085 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6b8005500 con 0x55da8e50e130 2024-03-18T14:54:52.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6aaff5640 1 -- 172.21.15.76:0/1978768085 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff6b8038050 con 0x55da8e50e130 2024-03-18T14:54:52.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6cfbf7740 10 client.15954 fetch_fsmap learned FSMap version 29 2024-03-18T14:54:52.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.309+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/1978768085 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55da8e58e240 con 0x55da8e50e130 2024-03-18T14:54:52.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.310+0000 7ff6aaff5640 1 -- 172.21.15.76:0/1978768085 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6b8005500 con 0x55da8e50e130 2024-03-18T14:54:52.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.310+0000 7ff6aaff5640 1 -- 172.21.15.76:0/1978768085 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7ff6b8029470 con 0x55da8e50e130 2024-03-18T14:54:52.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.310+0000 7ff6cfbf7740 10 client.15954 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:54:52.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.310+0000 7ff6cfbf7740 10 client.15954 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:54:52.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.310+0000 7ff6cfbf7740 20 client.15954 populate_metadata read hostname 'smithi076' 2024-03-18T14:54:52.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.310+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/1978768085 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55da8e58ea40 0x55da8e590e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:54:52.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.310+0000 7ff6cfbf7740 4 client.15954 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:54:52.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.310+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/1978768085 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55da8e47a2d0 con 0x55da8e58ea40 2024-03-18T14:54:52.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.310+0000 7ff6c659c640 1 --2- 172.21.15.76:0/1978768085 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55da8e58ea40 0x55da8e590e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:54:52.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.310+0000 7ff6aaff5640 10 client.15954 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:52.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.310+0000 7ff6c659c640 1 --2- 172.21.15.76:0/1978768085 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55da8e58ea40 0x55da8e590e60 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e92dcd88576dc2a1 server_cookie=2b59f4c0103faf44 in_seq=0 out_seq=0 2024-03-18T14:54:52.386 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.385+0000 7ff6aaff5640 1 -- 172.21.15.76:0/1978768085 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55da8e47a2d0 con 0x55da8e58ea40 2024-03-18T14:54:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.385+0000 7ff6aaff5640 10 client.15954 handle_command_reply: tid=0 2024-03-18T14:54:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.385+0000 7ff6cfbf7740 1 client.15954 shutdown 2024-03-18T14:54:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.385+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/1978768085 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 msgr2=0x55da8e511b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.385+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/1978768085 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 0x55da8e511b80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8000c00 tx=0x7ff6b8005740 comp rx=0 tx=0).stop 2024-03-18T14:54:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.385+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/1978768085 shutdown_connections 2024-03-18T14:54:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.385+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/1978768085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da8e50ea50 0x55da8e5120c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.385+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/1978768085 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55da8e58ea40 0x55da8e590e60 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.385+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/1978768085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8e517820 0x55da8e58cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.385+0000 7ff6aaff5640 0 client.15954 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:54:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.385+0000 7ff6cfbf7740 1 --2- 172.21.15.76:0/1978768085 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55da8e50e130 0x55da8e511b80 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.385+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/1978768085 >> 172.21.15.76:0/1978768085 conn(0x55da8e508a50 msgr2=0x55da8e507bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.385+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/1978768085 shutdown_connections 2024-03-18T14:54:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.385+0000 7ff6cfbf7740 1 -- 172.21.15.76:0/1978768085 wait complete. 2024-03-18T14:54:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.386+0000 7ff6cfbf7740 20 client.15954 trim_cache size 0 max 16384 2024-03-18T14:54:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:54:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:54:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:54:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:54:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.386+0000 7ff6cd00b640 1 -- 172.21.15.76:0/110745026 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6b00a2a10 msgr2=0x7ff6b00a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.386+0000 7ff6cd00b640 1 --2- 172.21.15.76:0/110745026 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6b00a2a10 0x7ff6b00a4ed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff6c001d000 tx=0x7ff6c000e3b0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.386+0000 7ff6cd00b640 1 -- 172.21.15.76:0/110745026 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6c81012d0 msgr2=0x7ff6c8198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:54:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.386+0000 7ff6cd00b640 1 --2- 172.21.15.76:0/110745026 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6c81012d0 0x7ff6c8198570 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc006010 tx=0x7ff6bc02cb30 comp rx=0 tx=0).stop 2024-03-18T14:54:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.386+0000 7ff6cd00b640 1 -- 172.21.15.76:0/110745026 shutdown_connections 2024-03-18T14:54:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.386+0000 7ff6cd00b640 1 --2- 172.21.15.76:0/110745026 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6b00a2a10 0x7ff6b00a4ed0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.386+0000 7ff6cd00b640 1 --2- 172.21.15.76:0/110745026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6c8106760 0x7ff6c819c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.386+0000 7ff6cd00b640 1 --2- 172.21.15.76:0/110745026 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6c81012d0 0x7ff6c8198570 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.386+0000 7ff6cd00b640 1 --2- 172.21.15.76:0/110745026 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6c8100930 0x7ff6c8198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:54:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.386+0000 7ff6cd00b640 1 -- 172.21.15.76:0/110745026 >> 172.21.15.76:0/110745026 conn(0x7ff6c8077540 msgr2=0x7ff6c80ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:54:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.387+0000 7ff6cd00b640 1 -- 172.21.15.76:0/110745026 shutdown_connections 2024-03-18T14:54:52.390 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:54:52.387+0000 7ff6cd00b640 1 -- 172.21.15.76:0/110745026 wait complete. 2024-03-18T14:54:52.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 91716 9.65 MB/sec warmup 53 sec latency 44.292 ms 2024-03-18T14:54:52.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:52 smithi076 ceph-mon[32205]: pgmap v52: 97 pgs: 97 active+clean; 41 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 272 op/s 2024-03-18T14:54:52.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:52 smithi174 ceph-mon[36787]: pgmap v52: 97 pgs: 97 active+clean; 41 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 272 op/s 2024-03-18T14:54:52.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:52 smithi077 ceph-mon[35865]: pgmap v52: 97 pgs: 97 active+clean; 41 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 272 op/s 2024-03-18T14:54:53.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 93172 9.61 MB/sec warmup 54 sec latency 16.598 ms 2024-03-18T14:54:54.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 96042 9.76 MB/sec warmup 55 sec latency 11.287 ms 2024-03-18T14:54:54.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:54 smithi076 ceph-mon[32205]: pgmap v53: 97 pgs: 97 active+clean; 42 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 404 op/s 2024-03-18T14:54:54.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:54 smithi174 ceph-mon[36787]: pgmap v53: 97 pgs: 97 active+clean; 42 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 404 op/s 2024-03-18T14:54:54.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:54 smithi077 ceph-mon[35865]: pgmap v53: 97 pgs: 97 active+clean; 42 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 404 op/s 2024-03-18T14:54:55.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 97375 9.69 MB/sec warmup 56 sec latency 16.812 ms 2024-03-18T14:54:56.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 99768 9.76 MB/sec warmup 57 sec latency 10.593 ms 2024-03-18T14:54:56.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:56 smithi076 ceph-mon[32205]: pgmap v54: 97 pgs: 97 active+clean; 42 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.2 MiB/s wr, 255 op/s 2024-03-18T14:54:56.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:56 smithi174 ceph-mon[36787]: pgmap v54: 97 pgs: 97 active+clean; 42 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.2 MiB/s wr, 255 op/s 2024-03-18T14:54:56.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:56 smithi077 ceph-mon[35865]: pgmap v54: 97 pgs: 97 active+clean; 42 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.2 MiB/s wr, 255 op/s 2024-03-18T14:54:57.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 101839 9.73 MB/sec warmup 58 sec latency 15.046 ms 2024-03-18T14:54:58.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 103511 9.76 MB/sec warmup 59 sec latency 20.352 ms 2024-03-18T14:54:58.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:58 smithi076 ceph-mon[32205]: pgmap v55: 97 pgs: 97 active+clean; 42 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.2 MiB/s wr, 255 op/s 2024-03-18T14:54:58.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:58 smithi174 ceph-mon[36787]: pgmap v55: 97 pgs: 97 active+clean; 42 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.2 MiB/s wr, 255 op/s 2024-03-18T14:54:58.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:58 smithi077 ceph-mon[35865]: pgmap v55: 97 pgs: 97 active+clean; 42 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.2 MiB/s wr, 255 op/s 2024-03-18T14:54:59.733 INFO:tasks.workunit.client.0.smithi076.stdout: 1 106406 9.88 MB/sec warmup 60 sec latency 13.153 ms 2024-03-18T14:54:59.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:54:59 smithi174 ceph-mon[36787]: pgmap v56: 97 pgs: 97 active+clean; 48 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 355 op/s 2024-03-18T14:54:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:54:59 smithi077 ceph-mon[35865]: pgmap v56: 97 pgs: 97 active+clean; 48 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 355 op/s 2024-03-18T14:55:00.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:54:59 smithi076 ceph-mon[32205]: pgmap v56: 97 pgs: 97 active+clean; 48 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 355 op/s 2024-03-18T14:55:00.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 107459 9.77 MB/sec warmup 61 sec latency 20.715 ms 2024-03-18T14:55:01.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 109482 9.80 MB/sec warmup 62 sec latency 62.349 ms 2024-03-18T14:55:02.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 110872 9.75 MB/sec warmup 63 sec latency 20.559 ms 2024-03-18T14:55:02.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:02 smithi076 ceph-mon[32205]: pgmap v57: 97 pgs: 97 active+clean; 48 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.2 MiB/s wr, 256 op/s 2024-03-18T14:55:02.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:02 smithi174 ceph-mon[36787]: pgmap v57: 97 pgs: 97 active+clean; 48 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.2 MiB/s wr, 256 op/s 2024-03-18T14:55:02.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:02 smithi077 ceph-mon[35865]: pgmap v57: 97 pgs: 97 active+clean; 48 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.2 MiB/s wr, 256 op/s 2024-03-18T14:55:03.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 113506 9.86 MB/sec warmup 64 sec latency 15.521 ms 2024-03-18T14:55:04.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 114728 9.77 MB/sec warmup 65 sec latency 15.422 ms 2024-03-18T14:55:04.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:04 smithi076 ceph-mon[32205]: pgmap v58: 97 pgs: 97 active+clean; 54 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.4 MiB/s wr, 400 op/s 2024-03-18T14:55:04.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:04 smithi174 ceph-mon[36787]: pgmap v58: 97 pgs: 97 active+clean; 54 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.4 MiB/s wr, 400 op/s 2024-03-18T14:55:04.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:04 smithi077 ceph-mon[35865]: pgmap v58: 97 pgs: 97 active+clean; 54 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.4 MiB/s wr, 400 op/s 2024-03-18T14:55:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.475+0000 7f612b6ea640 1 -- 172.21.15.76:0/4103167973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6124068e60 msgr2=0x7f61240692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.475+0000 7f612b6ea640 1 --2- 172.21.15.76:0/4103167973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6124068e60 0x7f61240692c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f6114009930 tx=0x7f611402bc30 comp rx=0 tx=0).stop 2024-03-18T14:55:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.475+0000 7f612b6ea640 1 -- 172.21.15.76:0/4103167973 shutdown_connections 2024-03-18T14:55:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.475+0000 7f612b6ea640 1 --2- 172.21.15.76:0/4103167973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6124069800 0x7f612410e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.475+0000 7f612b6ea640 1 --2- 172.21.15.76:0/4103167973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6124068e60 0x7f61240692c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.475+0000 7f612b6ea640 1 --2- 172.21.15.76:0/4103167973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61241078b0 0x7f6124107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.476+0000 7f612b6ea640 1 -- 172.21.15.76:0/4103167973 >> 172.21.15.76:0/4103167973 conn(0x7f6124077530 msgr2=0x7f6124077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.476+0000 7f612b6ea640 1 -- 172.21.15.76:0/4103167973 shutdown_connections 2024-03-18T14:55:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.476+0000 7f612b6ea640 1 -- 172.21.15.76:0/4103167973 wait complete. 2024-03-18T14:55:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.476+0000 7f612b6ea640 1 Processor -- start 2024-03-18T14:55:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.476+0000 7f612b6ea640 1 -- start start 2024-03-18T14:55:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.476+0000 7f612b6ea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6124068e60 0x7f6124197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.476+0000 7f612b6ea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6124069800 0x7f61241984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.476+0000 7f612b6ea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61241078b0 0x7f612419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.476+0000 7f612b6ea640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6124111ba0 con 0x7f6124069800 2024-03-18T14:55:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.476+0000 7f612b6ea640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6124111a20 con 0x7f6124068e60 2024-03-18T14:55:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.476+0000 7f612b6ea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6124111d20 con 0x7f61241078b0 2024-03-18T14:55:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.477+0000 7f6128c85640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6124069800 0x7f61241984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.477+0000 7f6128c85640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6124069800 0x7f61241984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45708/0 (socket says 172.21.15.76:45708) 2024-03-18T14:55:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.477+0000 7f6128c85640 1 -- 172.21.15.76:0/4133748334 learned_addr learned my addr 172.21.15.76:0/4133748334 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.477+0000 7f6129c87640 1 --2- 172.21.15.76:0/4133748334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61241078b0 0x7f612419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.477+0000 7f6129486640 1 --2- 172.21.15.76:0/4133748334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6124068e60 0x7f6124197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.477+0000 7f6128c85640 1 -- 172.21.15.76:0/4133748334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61241078b0 msgr2=0x7f612419c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.477+0000 7f6128c85640 1 --2- 172.21.15.76:0/4133748334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61241078b0 0x7f612419c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.477+0000 7f6128c85640 1 -- 172.21.15.76:0/4133748334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6124068e60 msgr2=0x7f6124197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.477+0000 7f6128c85640 1 --2- 172.21.15.76:0/4133748334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6124068e60 0x7f6124197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.477+0000 7f6128c85640 1 -- 172.21.15.76:0/4133748334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612419cf70 con 0x7f6124069800 2024-03-18T14:55:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.477+0000 7f6129486640 1 --2- 172.21.15.76:0/4133748334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6124068e60 0x7f6124197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.477+0000 7f6128c85640 1 --2- 172.21.15.76:0/4133748334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6124069800 0x7f61241984c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6114009230 tx=0x7f6114004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.477+0000 7f611a7fc640 1 -- 172.21.15.76:0/4133748334 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6114041070 con 0x7f6124069800 2024-03-18T14:55:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.477+0000 7f611a7fc640 1 -- 172.21.15.76:0/4133748334 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61140054e0 con 0x7f6124069800 2024-03-18T14:55:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.477+0000 7f612b6ea640 1 -- 172.21.15.76:0/4133748334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f612419d200 con 0x7f6124069800 2024-03-18T14:55:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.477+0000 7f611a7fc640 1 -- 172.21.15.76:0/4133748334 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6114003c90 con 0x7f6124069800 2024-03-18T14:55:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.477+0000 7f612b6ea640 1 -- 172.21.15.76:0/4133748334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61241a4af0 con 0x7f6124069800 2024-03-18T14:55:05.488 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.480+0000 7f611a7fc640 1 -- 172.21.15.76:0/4133748334 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f6114044070 con 0x7f6124069800 2024-03-18T14:55:05.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.480+0000 7f612e3b5740 1 Processor -- start 2024-03-18T14:55:05.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.480+0000 7f612e3b5740 1 -- start start 2024-03-18T14:55:05.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.480+0000 7f612e3b5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe545e130 0x55bbe545e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.480+0000 7f612e3b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545ea50 0x55bbe54672e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.480+0000 7f612e3b5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe5467820 0x55bbe5469c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.480+0000 7f612e3b5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55bbe53c2a40 con 0x55bbe545e130 2024-03-18T14:55:05.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.480+0000 7f612e3b5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bbe536b920 con 0x55bbe5467820 2024-03-18T14:55:05.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.480+0000 7f612e3b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bbe53beae0 con 0x55bbe545ea50 2024-03-18T14:55:05.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.480+0000 7f611a7fc640 1 --2- 172.21.15.76:0/4133748334 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f61100a2a10 0x7f61100a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f6128c85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe5467820 0x55bbe5469c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f6129486640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545ea50 0x55bbe54672e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f6129c87640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe545e130 0x55bbe545e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f6129486640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545ea50 0x55bbe54672e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:57554/0 (socket says 172.21.15.76:57554) 2024-03-18T14:55:05.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f6129486640 1 -- 172.21.15.76:0/1452324961 learned_addr learned my addr 172.21.15.76:0/1452324961 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f6129c87640 1 --2- 172.21.15.76:0/4133748334 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f61100a2a10 0x7f61100a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f611a7fc640 1 -- 172.21.15.76:0/4133748334 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f61140f08d0 con 0x7f6124069800 2024-03-18T14:55:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f6129486640 1 -- 172.21.15.76:0/1452324961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe5467820 msgr2=0x55bbe5469c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f6129486640 1 --2- 172.21.15.76:0/1452324961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe5467820 0x55bbe5469c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f6129486640 1 -- 172.21.15.76:0/1452324961 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe545e130 msgr2=0x55bbe545e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f6129486640 1 --2- 172.21.15.76:0/1452324961 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe545e130 0x55bbe545e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f6129486640 1 -- 172.21.15.76:0/1452324961 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbe5383460 con 0x55bbe545ea50 2024-03-18T14:55:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f6128c85640 1 --2- 172.21.15.76:0/1452324961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe5467820 0x55bbe5469c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f6129c87640 1 --2- 172.21.15.76:0/4133748334 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f61100a2a10 0x7f61100a4ed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f611c009660 tx=0x7f611c005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f6129486640 1 --2- 172.21.15.76:0/1452324961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545ea50 0x55bbe54672e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6120011860 tx=0x7f6120011d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1a3db6ff2ab25e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f610f7fe640 1 -- 172.21.15.76:0/1452324961 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61200112f0 con 0x55bbe545ea50 2024-03-18T14:55:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f610f7fe640 1 -- 172.21.15.76:0/1452324961 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f612000f9a0 con 0x55bbe545ea50 2024-03-18T14:55:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.481+0000 7f610f7fe640 1 -- 172.21.15.76:0/1452324961 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6120005700 con 0x55bbe545ea50 2024-03-18T14:55:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.482+0000 7f612e3b5740 1 -- 172.21.15.76:0/1452324961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545ea50 msgr2=0x55bbe54672e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.482+0000 7f612e3b5740 1 --2- 172.21.15.76:0/1452324961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545ea50 0x55bbe54672e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6120011860 tx=0x7f6120011d20 comp rx=0 tx=0).stop 2024-03-18T14:55:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.482+0000 7f612e3b5740 1 -- 172.21.15.76:0/1452324961 shutdown_connections 2024-03-18T14:55:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.482+0000 7f612e3b5740 1 --2- 172.21.15.76:0/1452324961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545ea50 0x55bbe54672e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.482+0000 7f612e3b5740 1 --2- 172.21.15.76:0/1452324961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe5467820 0x55bbe5469c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.482+0000 7f612e3b5740 1 --2- 172.21.15.76:0/1452324961 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe545e130 0x55bbe545e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.482+0000 7f612e3b5740 1 -- 172.21.15.76:0/1452324961 >> 172.21.15.76:0/1452324961 conn(0x55bbe5458a50 msgr2=0x55bbe5458e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.482+0000 7f612e3b5740 1 -- 172.21.15.76:0/1452324961 shutdown_connections 2024-03-18T14:55:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.482+0000 7f612e3b5740 1 -- 172.21.15.76:0/1452324961 wait complete. 2024-03-18T14:55:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.482+0000 7f612e3b5740 1 -- start start 2024-03-18T14:55:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f612e3b5740 1 Processor -- start 2024-03-18T14:55:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f612e3b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545e130 0x55bbe54647d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f612e3b5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe545ea50 0x55bbe5464d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f612e3b5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe5467820 0x55bbe5465250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f612e3b5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55bbe546a450 con 0x55bbe545ea50 2024-03-18T14:55:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f612e3b5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bbe537ed70 con 0x55bbe5467820 2024-03-18T14:55:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f612e3b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bbe546a5d0 con 0x55bbe545e130 2024-03-18T14:55:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f612e3b5740 10 client.0 _ll_register_callbacks cb 0x55bbe50d6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:55:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f6128c85640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe545ea50 0x55bbe5464d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f6128c85640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe545ea50 0x55bbe5464d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45726/0 (socket says 172.21.15.76:45726) 2024-03-18T14:55:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f6128c85640 1 -- 172.21.15.76:0/2861927291 learned_addr learned my addr 172.21.15.76:0/2861927291 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f6129486640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545e130 0x55bbe54647d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f6129c87640 1 --2- 172.21.15.76:0/2861927291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe5467820 0x55bbe5465250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f6128c85640 1 -- 172.21.15.76:0/2861927291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545e130 msgr2=0x55bbe54647d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f6128c85640 1 --2- 172.21.15.76:0/2861927291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545e130 0x55bbe54647d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f6128c85640 1 -- 172.21.15.76:0/2861927291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe5467820 msgr2=0x55bbe5465250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f6128c85640 1 --2- 172.21.15.76:0/2861927291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe5467820 0x55bbe5465250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f6128c85640 1 -- 172.21.15.76:0/2861927291 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbe50d7210 con 0x55bbe545ea50 2024-03-18T14:55:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.483+0000 7f6128c85640 1 --2- 172.21.15.76:0/2861927291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe545ea50 0x55bbe5464d10 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6114009430 tx=0x7f6114003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a6f0d73d7fe9913 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.484+0000 7f610cff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:55:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.484+0000 7f610cff9640 1 -- 172.21.15.76:0/2861927291 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611404b070 con 0x55bbe545ea50 2024-03-18T14:55:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.484+0000 7f610cff9640 1 -- 172.21.15.76:0/2861927291 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f61140b97b0 con 0x55bbe545ea50 2024-03-18T14:55:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.484+0000 7f612e3b5740 1 -- 172.21.15.76:0/2861927291 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f612000f9a0 con 0x55bbe545ea50 2024-03-18T14:55:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.484+0000 7f610cff9640 1 -- 172.21.15.76:0/2861927291 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f61140bcc80 con 0x55bbe545ea50 2024-03-18T14:55:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.484+0000 7f612e3b5740 10 client.15966 fetch_fsmap learned FSMap version 29 2024-03-18T14:55:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.484+0000 7f612e3b5740 1 -- 172.21.15.76:0/2861927291 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bbe51a3620 con 0x55bbe545ea50 2024-03-18T14:55:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.484+0000 7f610cff9640 1 -- 172.21.15.76:0/2861927291 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f612000f9a0 con 0x55bbe545ea50 2024-03-18T14:55:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.484+0000 7f610cff9640 1 -- 172.21.15.76:0/2861927291 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f61140eb410 con 0x55bbe545ea50 2024-03-18T14:55:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.484+0000 7f612e3b5740 10 client.15966 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:55:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.484+0000 7f612e3b5740 10 client.15966 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:55:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.484+0000 7f612e3b5740 20 client.15966 populate_metadata read hostname 'smithi076' 2024-03-18T14:55:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.484+0000 7f612e3b5740 1 --2- 172.21.15.76:0/2861927291 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55bbe5477970 0x55bbe5479db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.484+0000 7f612e3b5740 4 client.15966 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:55:05.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.484+0000 7f612e3b5740 1 -- 172.21.15.76:0/2861927291 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bbe53cff80 con 0x55bbe5477970 2024-03-18T14:55:05.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.484+0000 7f6129486640 1 --2- 172.21.15.76:0/2861927291 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55bbe5477970 0x55bbe5479db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.485+0000 7f610cff9640 10 client.15966 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:05.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.485+0000 7f6129486640 1 --2- 172.21.15.76:0/2861927291 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55bbe5477970 0x55bbe5479db0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af8a2765701b1517 server_cookie=a66ad04b43794e92 in_seq=0 out_seq=0 2024-03-18T14:55:05.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.485+0000 7f610cff9640 1 -- 172.21.15.76:0/2861927291 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bbe53cff80 con 0x55bbe5477970 2024-03-18T14:55:05.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.485+0000 7f610cff9640 10 client.15966 handle_command_reply: tid=0 2024-03-18T14:55:05.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.485+0000 7f612e3b5740 1 client.15966 shutdown 2024-03-18T14:55:05.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.486+0000 7f612e3b5740 1 -- 172.21.15.76:0/2861927291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe545ea50 msgr2=0x55bbe5464d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:05.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.486+0000 7f612e3b5740 1 --2- 172.21.15.76:0/2861927291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe545ea50 0x55bbe5464d10 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6114009430 tx=0x7f6114003760 comp rx=0 tx=0).stop 2024-03-18T14:55:05.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.486+0000 7f612e3b5740 1 -- 172.21.15.76:0/2861927291 shutdown_connections 2024-03-18T14:55:05.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.486+0000 7f612e3b5740 1 --2- 172.21.15.76:0/2861927291 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55bbe5477970 0x55bbe5479db0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.486+0000 7f612e3b5740 1 --2- 172.21.15.76:0/2861927291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe5467820 0x55bbe5465250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.486+0000 7f610cff9640 0 client.15966 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:05.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.486+0000 7f612e3b5740 1 --2- 172.21.15.76:0/2861927291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe545ea50 0x55bbe5464d10 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.486+0000 7f612e3b5740 1 --2- 172.21.15.76:0/2861927291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545e130 0x55bbe54647d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.486+0000 7f612e3b5740 1 -- 172.21.15.76:0/2861927291 >> 172.21.15.76:0/2861927291 conn(0x55bbe5458a50 msgr2=0x55bbe545a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:05.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.486+0000 7f612e3b5740 1 -- 172.21.15.76:0/2861927291 shutdown_connections 2024-03-18T14:55:05.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.486+0000 7f612e3b5740 1 -- 172.21.15.76:0/2861927291 wait complete. 2024-03-18T14:55:05.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.486+0000 7f612e3b5740 20 client.15966 trim_cache size 0 max 16384 2024-03-18T14:55:05.503 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f612e3b5740 1 Processor -- start 2024-03-18T14:55:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f612e3b5740 1 -- start start 2024-03-18T14:55:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f612e3b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545e130 0x55bbe54731f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f612e3b5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe545ea50 0x55bbe547f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f612e3b5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe5467820 0x55bbe5457320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f612e3b5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f61241afb80 con 0x55bbe5467820 2024-03-18T14:55:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f612e3b5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f61241ad3f0 con 0x55bbe545ea50 2024-03-18T14:55:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f612e3b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f61241ab240 con 0x55bbe545e130 2024-03-18T14:55:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f6129c87640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe5467820 0x55bbe5457320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f6128c85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe545ea50 0x55bbe547f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f6129486640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545e130 0x55bbe54731f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f6128c85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe545ea50 0x55bbe547f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:33536/0 (socket says 172.21.15.76:33536) 2024-03-18T14:55:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f6128c85640 1 -- 172.21.15.76:0/3276150297 learned_addr learned my addr 172.21.15.76:0/3276150297 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f6129c87640 1 -- 172.21.15.76:0/3276150297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545e130 msgr2=0x55bbe54731f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f6129c87640 1 --2- 172.21.15.76:0/3276150297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545e130 0x55bbe54731f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f6129c87640 1 -- 172.21.15.76:0/3276150297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe545ea50 msgr2=0x55bbe547f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f6129c87640 1 --2- 172.21.15.76:0/3276150297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe545ea50 0x55bbe547f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f6129c87640 1 -- 172.21.15.76:0/3276150297 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbe5454310 con 0x55bbe5467820 2024-03-18T14:55:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f6129c87640 1 --2- 172.21.15.76:0/3276150297 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe5467820 0x55bbe5457320 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f611c0058c0 tx=0x7f611c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77708dc29f15447b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f610effd640 1 -- 172.21.15.76:0/3276150297 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c004400 con 0x55bbe5467820 2024-03-18T14:55:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f610effd640 1 -- 172.21.15.76:0/3276150297 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f611c005500 con 0x55bbe5467820 2024-03-18T14:55:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.502+0000 7f610effd640 1 -- 172.21.15.76:0/3276150297 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c004dd0 con 0x55bbe5467820 2024-03-18T14:55:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.503+0000 7f612e3b5740 1 -- 172.21.15.76:0/3276150297 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe5467820 msgr2=0x55bbe5457320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.503+0000 7f612e3b5740 1 --2- 172.21.15.76:0/3276150297 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe5467820 0x55bbe5457320 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f611c0058c0 tx=0x7f611c005850 comp rx=0 tx=0).stop 2024-03-18T14:55:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.503+0000 7f612e3b5740 1 -- 172.21.15.76:0/3276150297 shutdown_connections 2024-03-18T14:55:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.503+0000 7f612e3b5740 1 --2- 172.21.15.76:0/3276150297 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe5467820 0x55bbe5457320 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.503+0000 7f612e3b5740 1 --2- 172.21.15.76:0/3276150297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe545ea50 0x55bbe547f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.503+0000 7f612e3b5740 1 --2- 172.21.15.76:0/3276150297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545e130 0x55bbe54731f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.503+0000 7f612e3b5740 1 -- 172.21.15.76:0/3276150297 >> 172.21.15.76:0/3276150297 conn(0x55bbe5458a50 msgr2=0x55bbe545efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.503+0000 7f612e3b5740 1 -- 172.21.15.76:0/3276150297 shutdown_connections 2024-03-18T14:55:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.503+0000 7f612e3b5740 1 -- 172.21.15.76:0/3276150297 wait complete. 2024-03-18T14:55:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.503+0000 7f612e3b5740 1 -- start start 2024-03-18T14:55:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.503+0000 7f612e3b5740 1 Processor -- start 2024-03-18T14:55:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f612e3b5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe545e130 0x55bbe546b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f612e3b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545ea50 0x55bbe546b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f612e3b5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe5467820 0x55bbe546bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f612e3b5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f61241ab8b0 con 0x55bbe5467820 2024-03-18T14:55:05.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f612e3b5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f61241acd20 con 0x55bbe545e130 2024-03-18T14:55:05.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f612e3b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f61241abf80 con 0x55bbe545ea50 2024-03-18T14:55:05.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f612e3b5740 10 client.0 _ll_register_callbacks cb 0x7f61140057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:55:05.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f6129c87640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe5467820 0x55bbe546bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f6129486640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe545e130 0x55bbe546b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f6128c85640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545ea50 0x55bbe546b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f6129486640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe545e130 0x55bbe546b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:33548/0 (socket says 172.21.15.76:33548) 2024-03-18T14:55:05.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f6129486640 1 -- 172.21.15.76:0/2899912628 learned_addr learned my addr 172.21.15.76:0/2899912628 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:05.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f6129c87640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe5467820 0x55bbe546bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45744/0 (socket says 172.21.15.76:45744) 2024-03-18T14:55:05.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f6128c85640 1 -- 172.21.15.76:0/2899912628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe545e130 msgr2=0x55bbe546b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:05.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f6128c85640 1 --2- 172.21.15.76:0/2899912628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe545e130 0x55bbe546b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f6128c85640 1 -- 172.21.15.76:0/2899912628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe5467820 msgr2=0x55bbe546bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f6128c85640 1 --2- 172.21.15.76:0/2899912628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe5467820 0x55bbe546bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f6128c85640 1 -- 172.21.15.76:0/2899912628 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbe54dcfe0 con 0x55bbe545ea50 2024-03-18T14:55:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f6129486640 1 --2- 172.21.15.76:0/2899912628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe545e130 0x55bbe546b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f6128c85640 1 --2- 172.21.15.76:0/2899912628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545ea50 0x55bbe546b940 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f61140b95a0 tx=0x7f61140bd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a0195c0f418c1c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f610cff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-18T14:55:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.504+0000 7f610cff9640 1 -- 172.21.15.76:0/2899912628 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611404b070 con 0x55bbe545ea50 2024-03-18T14:55:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.505+0000 7f610cff9640 1 -- 172.21.15.76:0/2899912628 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f61140c53a0 con 0x55bbe545ea50 2024-03-18T14:55:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.505+0000 7f612e3b5740 1 -- 172.21.15.76:0/2899912628 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f611c005500 con 0x55bbe545ea50 2024-03-18T14:55:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.505+0000 7f610cff9640 1 -- 172.21.15.76:0/2899912628 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f61140bcaa0 con 0x55bbe545ea50 2024-03-18T14:55:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.505+0000 7f612e3b5740 10 client.25822 fetch_fsmap learned FSMap version 29 2024-03-18T14:55:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.505+0000 7f612e3b5740 1 -- 172.21.15.76:0/2899912628 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bbe54de2f0 con 0x55bbe545ea50 2024-03-18T14:55:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.505+0000 7f610cff9640 1 -- 172.21.15.76:0/2899912628 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f611c005500 con 0x55bbe545ea50 2024-03-18T14:55:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.505+0000 7f610cff9640 1 -- 172.21.15.76:0/2899912628 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f61140dbb90 con 0x55bbe545ea50 2024-03-18T14:55:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.505+0000 7f612e3b5740 10 client.25822 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:55:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.505+0000 7f612e3b5740 10 client.25822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:55:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.505+0000 7f612e3b5740 20 client.25822 populate_metadata read hostname 'smithi076' 2024-03-18T14:55:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.505+0000 7f612e3b5740 1 --2- 172.21.15.76:0/2899912628 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55bbe54de8e0 0x55bbe54e0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.505+0000 7f612e3b5740 4 client.25822 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:55:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.505+0000 7f612e3b5740 1 -- 172.21.15.76:0/2899912628 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bbe53ca2d0 con 0x55bbe54de8e0 2024-03-18T14:55:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.505+0000 7f6129486640 1 --2- 172.21.15.76:0/2899912628 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55bbe54de8e0 0x55bbe54e0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.506+0000 7f6129486640 1 --2- 172.21.15.76:0/2899912628 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55bbe54de8e0 0x55bbe54e0d00 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfc782a8c78bb5e2 server_cookie=1ab790a1a81311e3 in_seq=0 out_seq=0 2024-03-18T14:55:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.506+0000 7f610cff9640 10 client.25822 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.570+0000 7f610cff9640 1 -- 172.21.15.76:0/2899912628 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bbe53ca2d0 con 0x55bbe54de8e0 2024-03-18T14:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.570+0000 7f610cff9640 10 client.25822 handle_command_reply: tid=0 2024-03-18T14:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.570+0000 7f612e3b5740 1 client.25822 shutdown 2024-03-18T14:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.570+0000 7f612e3b5740 1 -- 172.21.15.76:0/2899912628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545ea50 msgr2=0x55bbe546b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.570+0000 7f612e3b5740 1 --2- 172.21.15.76:0/2899912628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545ea50 0x55bbe546b940 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f61140b95a0 tx=0x7f61140bd040 comp rx=0 tx=0).stop 2024-03-18T14:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.570+0000 7f612e3b5740 1 -- 172.21.15.76:0/2899912628 shutdown_connections 2024-03-18T14:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.570+0000 7f612e3b5740 1 --2- 172.21.15.76:0/2899912628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbe545ea50 0x55bbe546b940 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.570+0000 7f612e3b5740 1 --2- 172.21.15.76:0/2899912628 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55bbe54de8e0 0x55bbe54e0d00 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.570+0000 7f612e3b5740 1 --2- 172.21.15.76:0/2899912628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bbe5467820 0x55bbe546bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.570+0000 7f610cff9640 0 client.25822 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.570+0000 7f612e3b5740 1 --2- 172.21.15.76:0/2899912628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbe545e130 0x55bbe546b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.570+0000 7f612e3b5740 1 -- 172.21.15.76:0/2899912628 >> 172.21.15.76:0/2899912628 conn(0x55bbe5458a50 msgr2=0x55bbe5457320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.570+0000 7f612e3b5740 1 -- 172.21.15.76:0/2899912628 shutdown_connections 2024-03-18T14:55:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.570+0000 7f612e3b5740 1 -- 172.21.15.76:0/2899912628 wait complete. 2024-03-18T14:55:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.571+0000 7f612e3b5740 20 client.25822 trim_cache size 0 max 16384 2024-03-18T14:55:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:55:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:55:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:55:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:55:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.571+0000 7f612b6ea640 1 -- 172.21.15.76:0/4133748334 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f61100a2a10 msgr2=0x7f61100a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.571+0000 7f612b6ea640 1 --2- 172.21.15.76:0/4133748334 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f61100a2a10 0x7f61100a4ed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f611c009660 tx=0x7f611c005f90 comp rx=0 tx=0).stop 2024-03-18T14:55:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.571+0000 7f612b6ea640 1 -- 172.21.15.76:0/4133748334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6124069800 msgr2=0x7f61241984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.571+0000 7f612b6ea640 1 --2- 172.21.15.76:0/4133748334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6124069800 0x7f61241984c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6114009230 tx=0x7f6114004270 comp rx=0 tx=0).stop 2024-03-18T14:55:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.571+0000 7f612b6ea640 1 -- 172.21.15.76:0/4133748334 shutdown_connections 2024-03-18T14:55:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.571+0000 7f612b6ea640 1 --2- 172.21.15.76:0/4133748334 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f61100a2a10 0x7f61100a4ed0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.571+0000 7f612b6ea640 1 --2- 172.21.15.76:0/4133748334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61241078b0 0x7f612419c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.571+0000 7f612b6ea640 1 --2- 172.21.15.76:0/4133748334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6124069800 0x7f61241984c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.571+0000 7f612b6ea640 1 --2- 172.21.15.76:0/4133748334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6124068e60 0x7f6124197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.571+0000 7f612b6ea640 1 -- 172.21.15.76:0/4133748334 >> 172.21.15.76:0/4133748334 conn(0x7f6124077530 msgr2=0x7f61240fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.571+0000 7f612b6ea640 1 -- 172.21.15.76:0/4133748334 shutdown_connections 2024-03-18T14:55:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:05.571+0000 7f612b6ea640 1 -- 172.21.15.76:0/4133748334 wait complete. 2024-03-18T14:55:05.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 117332 9.87 MB/sec warmup 66 sec latency 39.724 ms 2024-03-18T14:55:06.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 118739 9.81 MB/sec warmup 67 sec latency 47.956 ms 2024-03-18T14:55:06.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:06 smithi076 ceph-mon[32205]: pgmap v59: 97 pgs: 97 active+clean; 54 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 243 op/s 2024-03-18T14:55:06.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:06 smithi174 ceph-mon[36787]: pgmap v59: 97 pgs: 97 active+clean; 54 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 243 op/s 2024-03-18T14:55:06.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:06 smithi077 ceph-mon[35865]: pgmap v59: 97 pgs: 97 active+clean; 54 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 243 op/s 2024-03-18T14:55:07.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 121060 9.87 MB/sec warmup 68 sec latency 16.927 ms 2024-03-18T14:55:07.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:07 smithi174 ceph-mon[36787]: pgmap v60: 97 pgs: 97 active+clean; 54 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 243 op/s 2024-03-18T14:55:07.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:07 smithi077 ceph-mon[35865]: pgmap v60: 97 pgs: 97 active+clean; 54 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 243 op/s 2024-03-18T14:55:08.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:07 smithi076 ceph-mon[32205]: pgmap v60: 97 pgs: 97 active+clean; 54 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 243 op/s 2024-03-18T14:55:08.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 123391 9.89 MB/sec warmup 69 sec latency 14.332 ms 2024-03-18T14:55:09.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 125057 9.87 MB/sec warmup 70 sec latency 13.317 ms 2024-03-18T14:55:10.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 127947 9.99 MB/sec warmup 71 sec latency 17.118 ms 2024-03-18T14:55:10.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:10 smithi076 ceph-mon[32205]: pgmap v61: 97 pgs: 97 active+clean; 50 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.2 MiB/s wr, 419 op/s 2024-03-18T14:55:10.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:10 smithi174 ceph-mon[36787]: pgmap v61: 97 pgs: 97 active+clean; 50 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.2 MiB/s wr, 419 op/s 2024-03-18T14:55:10.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:10 smithi077 ceph-mon[35865]: pgmap v61: 97 pgs: 97 active+clean; 50 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.2 MiB/s wr, 419 op/s 2024-03-18T14:55:11.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 129215 9.92 MB/sec warmup 72 sec latency 41.632 ms 2024-03-18T14:55:12.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 131588 9.98 MB/sec warmup 73 sec latency 20.101 ms 2024-03-18T14:55:12.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:12 smithi076 ceph-mon[32205]: pgmap v62: 97 pgs: 97 active+clean; 50 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 318 op/s 2024-03-18T14:55:12.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:12 smithi174 ceph-mon[36787]: pgmap v62: 97 pgs: 97 active+clean; 50 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 318 op/s 2024-03-18T14:55:12.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:12 smithi077 ceph-mon[35865]: pgmap v62: 97 pgs: 97 active+clean; 50 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 318 op/s 2024-03-18T14:55:13.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 133612 9.94 MB/sec warmup 74 sec latency 16.910 ms 2024-03-18T14:55:14.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:14 smithi077 ceph-mon[35865]: pgmap v63: 97 pgs: 97 active+clean; 58 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 445 op/s 2024-03-18T14:55:14.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:14 smithi076 ceph-mon[32205]: pgmap v63: 97 pgs: 97 active+clean; 58 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 445 op/s 2024-03-18T14:55:14.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:14 smithi174 ceph-mon[36787]: pgmap v63: 97 pgs: 97 active+clean; 58 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 445 op/s 2024-03-18T14:55:14.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 135402 9.97 MB/sec warmup 75 sec latency 16.936 ms 2024-03-18T14:55:15.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 138197 10.03 MB/sec warmup 76 sec latency 20.932 ms 2024-03-18T14:55:16.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 139427 9.98 MB/sec warmup 77 sec latency 39.092 ms 2024-03-18T14:55:16.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:16 smithi076 ceph-mon[32205]: pgmap v64: 97 pgs: 97 active+clean; 58 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 302 op/s 2024-03-18T14:55:16.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:16 smithi174 ceph-mon[36787]: pgmap v64: 97 pgs: 97 active+clean; 58 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 302 op/s 2024-03-18T14:55:16.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:16 smithi077 ceph-mon[35865]: pgmap v64: 97 pgs: 97 active+clean; 58 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 302 op/s 2024-03-18T14:55:17.734 INFO:tasks.workunit.client.0.smithi076.stdout: 1 142125 10.07 MB/sec warmup 78 sec latency 47.604 ms 2024-03-18T14:55:18.666 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.664+0000 7fe28c836640 1 -- 172.21.15.76:0/3926933978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2881078b0 msgr2=0x7fe288107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.664+0000 7fe28c836640 1 --2- 172.21.15.76:0/3926933978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2881078b0 0x7fe288107c90 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe278009930 tx=0x7fe27802bc30 comp rx=0 tx=0).stop 2024-03-18T14:55:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.664+0000 7fe28c836640 1 -- 172.21.15.76:0/3926933978 shutdown_connections 2024-03-18T14:55:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.664+0000 7fe28c836640 1 --2- 172.21.15.76:0/3926933978 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe288069800 0x7fe28810e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.664+0000 7fe28c836640 1 --2- 172.21.15.76:0/3926933978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe288068e60 0x7fe2880692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.664+0000 7fe28c836640 1 --2- 172.21.15.76:0/3926933978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2881078b0 0x7fe288107c90 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.664+0000 7fe28c836640 1 -- 172.21.15.76:0/3926933978 >> 172.21.15.76:0/3926933978 conn(0x7fe288077530 msgr2=0x7fe288077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.664+0000 7fe28c836640 1 -- 172.21.15.76:0/3926933978 shutdown_connections 2024-03-18T14:55:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.664+0000 7fe28c836640 1 -- 172.21.15.76:0/3926933978 wait complete. 2024-03-18T14:55:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe28c836640 1 Processor -- start 2024-03-18T14:55:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe28c836640 1 -- start start 2024-03-18T14:55:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe28c836640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe288068e60 0x7fe288198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.668 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe28c836640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe288069800 0x7fe288198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.668 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe28c836640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2881078b0 0x7fe28819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.668 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe28c836640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe288111ba0 con 0x7fe288069800 2024-03-18T14:55:18.668 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe28c836640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe288111a20 con 0x7fe2881078b0 2024-03-18T14:55:18.668 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe28c836640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe288111d20 con 0x7fe288068e60 2024-03-18T14:55:18.668 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe285d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe288069800 0x7fe288198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.668 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe28659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe288068e60 0x7fe288198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.668 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe286d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2881078b0 0x7fe28819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.668 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe28659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe288068e60 0x7fe288198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:47056/0 (socket says 172.21.15.76:47056) 2024-03-18T14:55:18.668 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe286d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2881078b0 0x7fe28819c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:49842/0 (socket says 172.21.15.76:49842) 2024-03-18T14:55:18.668 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe28659c640 1 -- 172.21.15.76:0/2998050957 learned_addr learned my addr 172.21.15.76:0/2998050957 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:18.668 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe285d9b640 1 -- 172.21.15.76:0/2998050957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe288068e60 msgr2=0x7fe288198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:18.668 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe285d9b640 1 --2- 172.21.15.76:0/2998050957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe288068e60 0x7fe288198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.668 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe285d9b640 1 -- 172.21.15.76:0/2998050957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2881078b0 msgr2=0x7fe28819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:18.669 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe285d9b640 1 --2- 172.21.15.76:0/2998050957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2881078b0 0x7fe28819c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.669 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe285d9b640 1 -- 172.21.15.76:0/2998050957 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28819cff0 con 0x7fe288069800 2024-03-18T14:55:18.669 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.665+0000 7fe286d9d640 1 --2- 172.21.15.76:0/2998050957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2881078b0 0x7fe28819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:18.669 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.666+0000 7fe285d9b640 1 --2- 172.21.15.76:0/2998050957 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe288069800 0x7fe288198540 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00e9f0 tx=0x7fe27c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:18.669 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.666+0000 7fe2777fe640 1 -- 172.21.15.76:0/2998050957 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c00ccc0 con 0x7fe288069800 2024-03-18T14:55:18.669 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.666+0000 7fe2777fe640 1 -- 172.21.15.76:0/2998050957 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe27c010910 con 0x7fe288069800 2024-03-18T14:55:18.669 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.666+0000 7fe2777fe640 1 -- 172.21.15.76:0/2998050957 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c018960 con 0x7fe288069800 2024-03-18T14:55:18.669 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.666+0000 7fe28c836640 1 -- 172.21.15.76:0/2998050957 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe28819d2e0 con 0x7fe288069800 2024-03-18T14:55:18.669 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.666+0000 7fe28c836640 1 -- 172.21.15.76:0/2998050957 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2881a4b70 con 0x7fe288069800 2024-03-18T14:55:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.668+0000 7fe2777fe640 1 -- 172.21.15.76:0/2998050957 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fe27c010ac0 con 0x7fe288069800 2024-03-18T14:55:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.669+0000 7fe2777fe640 1 --2- 172.21.15.76:0/2998050957 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe2700a2a10 0x7fe2700a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.676 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.669+0000 7fe28f423740 1 Processor -- start 2024-03-18T14:55:18.676 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.669+0000 7fe28659c640 1 --2- 172.21.15.76:0/2998050957 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe2700a2a10 0x7fe2700a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.676 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.669+0000 7fe2777fe640 1 -- 172.21.15.76:0/2998050957 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fe27c0cf1d0 con 0x7fe288069800 2024-03-18T14:55:18.676 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.669+0000 7fe28f423740 1 -- start start 2024-03-18T14:55:18.676 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.669+0000 7fe28f423740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fad3130 0x560d0fad3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.676 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.669+0000 7fe28f423740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fad3a50 0x560d0fadc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.676 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.669+0000 7fe28f423740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fadc820 0x560d0fadec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.676 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.669+0000 7fe28f423740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x560d0fa37a40 con 0x560d0fad3130 2024-03-18T14:55:18.676 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.669+0000 7fe28f423740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560d0f9e0920 con 0x560d0fadc820 2024-03-18T14:55:18.676 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.669+0000 7fe28f423740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560d0fa33ae0 con 0x560d0fad3a50 2024-03-18T14:55:18.676 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.670+0000 7fe28659c640 1 --2- 172.21.15.76:0/2998050957 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe2700a2a10 0x7fe2700a4ed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe278000c00 tx=0x7fe2780095e0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:18.676 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.670+0000 7fe28659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fad3130 0x560d0fad3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.676 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.670+0000 7fe28659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fad3130 0x560d0fad3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48174/0 (socket says 172.21.15.76:48174) 2024-03-18T14:55:18.676 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.670+0000 7fe28659c640 1 -- 172.21.15.76:0/3694798062 learned_addr learned my addr 172.21.15.76:0/3694798062 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:18.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.670+0000 7fe285d9b640 1 --2- 172.21.15.76:0/3694798062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fad3a50 0x560d0fadc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.670+0000 7fe286d9d640 1 --2- 172.21.15.76:0/3694798062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fadc820 0x560d0fadec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.670+0000 7fe28659c640 1 -- 172.21.15.76:0/3694798062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fad3a50 msgr2=0x560d0fadc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:18.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.670+0000 7fe28659c640 1 --2- 172.21.15.76:0/3694798062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fad3a50 0x560d0fadc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.670+0000 7fe28659c640 1 -- 172.21.15.76:0/3694798062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fadc820 msgr2=0x560d0fadec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:18.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.670+0000 7fe28659c640 1 --2- 172.21.15.76:0/3694798062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fadc820 0x560d0fadec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.670+0000 7fe28659c640 1 -- 172.21.15.76:0/3694798062 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d0f9f8460 con 0x560d0fad3130 2024-03-18T14:55:18.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.670+0000 7fe285d9b640 1 --2- 172.21.15.76:0/3694798062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fad3a50 0x560d0fadc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:18.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.670+0000 7fe286d9d640 1 --2- 172.21.15.76:0/3694798062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fadc820 0x560d0fadec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:18.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.670+0000 7fe28659c640 1 --2- 172.21.15.76:0/3694798062 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fad3130 0x560d0fad3510 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe278036040 tx=0x7fe27803f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e2dfb7040f7add1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:18.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.670+0000 7fe274ff9640 1 -- 172.21.15.76:0/3694798062 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27802cb70 con 0x560d0fad3130 2024-03-18T14:55:18.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.670+0000 7fe274ff9640 1 -- 172.21.15.76:0/3694798062 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe278050070 con 0x560d0fad3130 2024-03-18T14:55:18.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.670+0000 7fe274ff9640 1 -- 172.21.15.76:0/3694798062 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2780055e0 con 0x560d0fad3130 2024-03-18T14:55:18.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.671+0000 7fe28f423740 1 -- 172.21.15.76:0/3694798062 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fad3130 msgr2=0x560d0fad3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:18.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.671+0000 7fe28f423740 1 --2- 172.21.15.76:0/3694798062 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fad3130 0x560d0fad3510 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe278036040 tx=0x7fe27803f000 comp rx=0 tx=0).stop 2024-03-18T14:55:18.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.671+0000 7fe28f423740 1 -- 172.21.15.76:0/3694798062 shutdown_connections 2024-03-18T14:55:18.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.671+0000 7fe28f423740 1 --2- 172.21.15.76:0/3694798062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fad3a50 0x560d0fadc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.671+0000 7fe28f423740 1 --2- 172.21.15.76:0/3694798062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fadc820 0x560d0fadec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.671+0000 7fe28f423740 1 --2- 172.21.15.76:0/3694798062 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fad3130 0x560d0fad3510 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.671+0000 7fe28f423740 1 -- 172.21.15.76:0/3694798062 >> 172.21.15.76:0/3694798062 conn(0x560d0facda50 msgr2=0x560d0facde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:18.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.671+0000 7fe28f423740 1 -- 172.21.15.76:0/3694798062 shutdown_connections 2024-03-18T14:55:18.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.671+0000 7fe28f423740 1 -- 172.21.15.76:0/3694798062 wait complete. 2024-03-18T14:55:18.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.671+0000 7fe28f423740 1 -- start start 2024-03-18T14:55:18.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe28f423740 1 Processor -- start 2024-03-18T14:55:18.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe28f423740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fad3130 0x560d0fad9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe28f423740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fad3a50 0x560d0fad9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe28f423740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fadc820 0x560d0fada2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe28f423740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x560d0fadf3f0 con 0x560d0fadc820 2024-03-18T14:55:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe28f423740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560d0f9f3d70 con 0x560d0fad3a50 2024-03-18T14:55:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe28f423740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560d0fadf570 con 0x560d0fad3130 2024-03-18T14:55:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe28f423740 10 client.0 _ll_register_callbacks cb 0x560d0f74ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:55:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe286d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fadc820 0x560d0fada2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe285d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fad3a50 0x560d0fad9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe28659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fad3130 0x560d0fad9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe286d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fadc820 0x560d0fada2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48190/0 (socket says 172.21.15.76:48190) 2024-03-18T14:55:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe286d9d640 1 -- 172.21.15.76:0/1131812222 learned_addr learned my addr 172.21.15.76:0/1131812222 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe286d9d640 1 -- 172.21.15.76:0/1131812222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fad3130 msgr2=0x560d0fad9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe286d9d640 1 --2- 172.21.15.76:0/1131812222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fad3130 0x560d0fad9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe286d9d640 1 -- 172.21.15.76:0/1131812222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fad3a50 msgr2=0x560d0fad9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe286d9d640 1 --2- 172.21.15.76:0/1131812222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fad3a50 0x560d0fad9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe286d9d640 1 -- 172.21.15.76:0/1131812222 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d0f74c210 con 0x560d0fadc820 2024-03-18T14:55:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe285d9b640 1 --2- 172.21.15.76:0/1131812222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fad3a50 0x560d0fad9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe286d9d640 1 --2- 172.21.15.76:0/1131812222 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fadc820 0x560d0fada2f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe28000fe50 tx=0x7fe280012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72047f8ac292bc61 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe26dffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:55:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.672+0000 7fe26dffb640 1 -- 172.21.15.76:0/1131812222 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280003970 con 0x560d0fadc820 2024-03-18T14:55:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.673+0000 7fe26dffb640 1 -- 172.21.15.76:0/1131812222 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fe280003b10 con 0x560d0fadc820 2024-03-18T14:55:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.673+0000 7fe28f423740 1 -- 172.21.15.76:0/1131812222 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe278050070 con 0x560d0fadc820 2024-03-18T14:55:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.673+0000 7fe26dffb640 1 -- 172.21.15.76:0/1131812222 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe280003d10 con 0x560d0fadc820 2024-03-18T14:55:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.673+0000 7fe28f423740 10 client.15984 fetch_fsmap learned FSMap version 29 2024-03-18T14:55:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.673+0000 7fe28f423740 1 -- 172.21.15.76:0/1131812222 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560d0f818620 con 0x560d0fadc820 2024-03-18T14:55:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.673+0000 7fe26dffb640 1 -- 172.21.15.76:0/1131812222 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe278050070 con 0x560d0fadc820 2024-03-18T14:55:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.673+0000 7fe26dffb640 1 -- 172.21.15.76:0/1131812222 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fe2800257a0 con 0x560d0fadc820 2024-03-18T14:55:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.673+0000 7fe28f423740 10 client.15984 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:55:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.673+0000 7fe28f423740 10 client.15984 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:55:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.673+0000 7fe28f423740 20 client.15984 populate_metadata read hostname 'smithi076' 2024-03-18T14:55:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.673+0000 7fe28f423740 1 --2- 172.21.15.76:0/1131812222 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x560d0faeca10 0x560d0faeee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.673+0000 7fe28f423740 4 client.15984 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:55:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.673+0000 7fe28f423740 1 -- 172.21.15.76:0/1131812222 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d0fa44f80 con 0x560d0faeca10 2024-03-18T14:55:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.673+0000 7fe28659c640 1 --2- 172.21.15.76:0/1131812222 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x560d0faeca10 0x560d0faeee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.674+0000 7fe28659c640 1 --2- 172.21.15.76:0/1131812222 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x560d0faeca10 0x560d0faeee00 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a19c80a2cd13096f server_cookie=63204aba2d9475b0 in_seq=0 out_seq=0 2024-03-18T14:55:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.674+0000 7fe26dffb640 10 client.15984 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.674+0000 7fe26dffb640 1 -- 172.21.15.76:0/1131812222 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560d0fa44f80 con 0x560d0faeca10 2024-03-18T14:55:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.674+0000 7fe26dffb640 10 client.15984 handle_command_reply: tid=0 2024-03-18T14:55:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.674+0000 7fe28f423740 1 client.15984 shutdown 2024-03-18T14:55:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.674+0000 7fe28f423740 1 -- 172.21.15.76:0/1131812222 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fadc820 msgr2=0x560d0fada2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.674+0000 7fe28f423740 1 --2- 172.21.15.76:0/1131812222 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fadc820 0x560d0fada2f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe28000fe50 tx=0x7fe280012820 comp rx=0 tx=0).stop 2024-03-18T14:55:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.674+0000 7fe28f423740 1 -- 172.21.15.76:0/1131812222 shutdown_connections 2024-03-18T14:55:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.674+0000 7fe28f423740 1 --2- 172.21.15.76:0/1131812222 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x560d0faeca10 0x560d0faeee00 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.674+0000 7fe28f423740 1 --2- 172.21.15.76:0/1131812222 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fadc820 0x560d0fada2f0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.674+0000 7fe28f423740 1 --2- 172.21.15.76:0/1131812222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fad3a50 0x560d0fad9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.674+0000 7fe28f423740 1 --2- 172.21.15.76:0/1131812222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fad3130 0x560d0fad9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.674+0000 7fe26dffb640 0 client.15984 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.674+0000 7fe28f423740 1 -- 172.21.15.76:0/1131812222 >> 172.21.15.76:0/1131812222 conn(0x560d0facda50 msgr2=0x560d0facf7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.675+0000 7fe28f423740 1 -- 172.21.15.76:0/1131812222 shutdown_connections 2024-03-18T14:55:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.675+0000 7fe28f423740 1 -- 172.21.15.76:0/1131812222 wait complete. 2024-03-18T14:55:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.675+0000 7fe28f423740 20 client.15984 trim_cache size 0 max 16384 2024-03-18T14:55:18.694 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.689+0000 7fe28f423740 1 Processor -- start 2024-03-18T14:55:18.694 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.689+0000 7fe28f423740 1 -- start start 2024-03-18T14:55:18.694 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.689+0000 7fe28f423740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fad3130 0x560d0fadbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.694 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.689+0000 7fe28f423740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fad3a50 0x560d0fae8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.694 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.689+0000 7fe28f423740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fadc820 0x560d0fada110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.694 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.689+0000 7fe28f423740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe2881afc20 con 0x560d0fad3130 2024-03-18T14:55:18.694 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.689+0000 7fe28f423740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe2881ad490 con 0x560d0fad3a50 2024-03-18T14:55:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.689+0000 7fe28f423740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe2881ab2e0 con 0x560d0fadc820 2024-03-18T14:55:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.689+0000 7fe28659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fad3130 0x560d0fadbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.689+0000 7fe28659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fad3130 0x560d0fadbfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48194/0 (socket says 172.21.15.76:48194) 2024-03-18T14:55:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.689+0000 7fe28659c640 1 -- 172.21.15.76:0/2389146977 learned_addr learned my addr 172.21.15.76:0/2389146977 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.689+0000 7fe285d9b640 1 --2- 172.21.15.76:0/2389146977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fad3a50 0x560d0fae8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.690+0000 7fe286d9d640 1 --2- 172.21.15.76:0/2389146977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fadc820 0x560d0fada110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.690+0000 7fe28659c640 1 -- 172.21.15.76:0/2389146977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fadc820 msgr2=0x560d0fada110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.690+0000 7fe28659c640 1 --2- 172.21.15.76:0/2389146977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fadc820 0x560d0fada110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.690+0000 7fe28659c640 1 -- 172.21.15.76:0/2389146977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fad3a50 msgr2=0x560d0fae8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.690+0000 7fe28659c640 1 --2- 172.21.15.76:0/2389146977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fad3a50 0x560d0fae8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.690+0000 7fe28659c640 1 -- 172.21.15.76:0/2389146977 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d0fac7fc0 con 0x560d0fad3130 2024-03-18T14:55:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.690+0000 7fe28659c640 1 --2- 172.21.15.76:0/2389146977 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fad3130 0x560d0fadbfa0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe27802e010 tx=0x7fe27802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a51c2ec4a616c8da server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.690+0000 7fe26ffff640 1 -- 172.21.15.76:0/2389146977 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe278004f80 con 0x560d0fad3130 2024-03-18T14:55:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.690+0000 7fe26ffff640 1 -- 172.21.15.76:0/2389146977 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe278050030 con 0x560d0fad3130 2024-03-18T14:55:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.690+0000 7fe26ffff640 1 -- 172.21.15.76:0/2389146977 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27805e2c0 con 0x560d0fad3130 2024-03-18T14:55:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 1 -- 172.21.15.76:0/2389146977 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fad3130 msgr2=0x560d0fadbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 1 --2- 172.21.15.76:0/2389146977 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fad3130 0x560d0fadbfa0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe27802e010 tx=0x7fe27802e040 comp rx=0 tx=0).stop 2024-03-18T14:55:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 1 -- 172.21.15.76:0/2389146977 shutdown_connections 2024-03-18T14:55:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 1 --2- 172.21.15.76:0/2389146977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fadc820 0x560d0fada110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 1 --2- 172.21.15.76:0/2389146977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fad3a50 0x560d0fae8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 1 --2- 172.21.15.76:0/2389146977 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fad3130 0x560d0fadbfa0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 1 -- 172.21.15.76:0/2389146977 >> 172.21.15.76:0/2389146977 conn(0x560d0facda50 msgr2=0x560d0fac8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 1 -- 172.21.15.76:0/2389146977 shutdown_connections 2024-03-18T14:55:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 1 -- 172.21.15.76:0/2389146977 wait complete. 2024-03-18T14:55:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 1 -- start start 2024-03-18T14:55:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 1 Processor -- start 2024-03-18T14:55:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fad3130 0x560d0fae4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fad3a50 0x560d0fad60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fadc820 0x560d0fad6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe2881ab950 con 0x560d0fadc820 2024-03-18T14:55:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe2881acdc0 con 0x560d0fad3a50 2024-03-18T14:55:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe2881ac020 con 0x560d0fad3130 2024-03-18T14:55:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28f423740 10 client.0 _ll_register_callbacks cb 0x7fe28000f4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:55:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe286d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fadc820 0x560d0fad6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe285d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fad3a50 0x560d0fad60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe28659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fad3130 0x560d0fae4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe286d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fadc820 0x560d0fad6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48202/0 (socket says 172.21.15.76:48202) 2024-03-18T14:55:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.691+0000 7fe286d9d640 1 -- 172.21.15.76:0/33750071 learned_addr learned my addr 172.21.15.76:0/33750071 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.692+0000 7fe286d9d640 1 -- 172.21.15.76:0/33750071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fad3130 msgr2=0x560d0fae4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.692+0000 7fe286d9d640 1 --2- 172.21.15.76:0/33750071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fad3130 0x560d0fae4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.692+0000 7fe286d9d640 1 -- 172.21.15.76:0/33750071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fad3a50 msgr2=0x560d0fad60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.692+0000 7fe286d9d640 1 --2- 172.21.15.76:0/33750071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fad3a50 0x560d0fad60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.692+0000 7fe286d9d640 1 -- 172.21.15.76:0/33750071 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d0fad6e50 con 0x560d0fadc820 2024-03-18T14:55:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.692+0000 7fe285d9b640 1 --2- 172.21.15.76:0/33750071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fad3a50 0x560d0fad60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.692+0000 7fe286d9d640 1 --2- 172.21.15.76:0/33750071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fadc820 0x560d0fad6620 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe280025b40 tx=0x7fe280003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15ca998a7922998a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.692+0000 7fe26dffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:55:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.692+0000 7fe26dffb640 1 -- 172.21.15.76:0/33750071 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280034060 con 0x560d0fadc820 2024-03-18T14:55:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.692+0000 7fe26dffb640 1 -- 172.21.15.76:0/33750071 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fe280018040 con 0x560d0fadc820 2024-03-18T14:55:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.692+0000 7fe28f423740 1 -- 172.21.15.76:0/33750071 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe278050030 con 0x560d0fadc820 2024-03-18T14:55:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.692+0000 7fe26dffb640 1 -- 172.21.15.76:0/33750071 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe28002b040 con 0x560d0fadc820 2024-03-18T14:55:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.692+0000 7fe28f423740 10 client.15990 fetch_fsmap learned FSMap version 29 2024-03-18T14:55:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.692+0000 7fe28f423740 1 -- 172.21.15.76:0/33750071 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560d0fb52ce0 con 0x560d0fadc820 2024-03-18T14:55:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.692+0000 7fe26dffb640 1 -- 172.21.15.76:0/33750071 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe278050030 con 0x560d0fadc820 2024-03-18T14:55:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.693+0000 7fe26dffb640 1 -- 172.21.15.76:0/33750071 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fe280005620 con 0x560d0fadc820 2024-03-18T14:55:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.693+0000 7fe28f423740 10 client.15990 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:55:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.693+0000 7fe28f423740 10 client.15990 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:55:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.693+0000 7fe28f423740 20 client.15990 populate_metadata read hostname 'smithi076' 2024-03-18T14:55:18.699 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.693+0000 7fe28f423740 1 --2- 172.21.15.76:0/33750071 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x560d0fb534c0 0x560d0fb558e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:18.699 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.693+0000 7fe28f423740 4 client.15990 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:55:18.699 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.693+0000 7fe28f423740 1 -- 172.21.15.76:0/33750071 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560d0fa3f2d0 con 0x560d0fb534c0 2024-03-18T14:55:18.699 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.693+0000 7fe28659c640 1 --2- 172.21.15.76:0/33750071 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x560d0fb534c0 0x560d0fb558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:18.699 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.693+0000 7fe26dffb640 10 client.15990 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:18.699 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.693+0000 7fe28659c640 1 --2- 172.21.15.76:0/33750071 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x560d0fb534c0 0x560d0fb558e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28b54b6709fe2dd4 server_cookie=e03bf5d58025085c in_seq=0 out_seq=0 2024-03-18T14:55:18.735 INFO:tasks.workunit.client.0.smithi076.stdout: 1 143337 10.01 MB/sec warmup 79 sec latency 39.253 ms 2024-03-18T14:55:18.759 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.758+0000 7fe26dffb640 1 -- 172.21.15.76:0/33750071 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560d0fa3f2d0 con 0x560d0fb534c0 2024-03-18T14:55:18.759 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.758+0000 7fe26dffb640 10 client.15990 handle_command_reply: tid=0 2024-03-18T14:55:18.759 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.758+0000 7fe28f423740 1 client.15990 shutdown 2024-03-18T14:55:18.760 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.758+0000 7fe28f423740 1 -- 172.21.15.76:0/33750071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fadc820 msgr2=0x560d0fad6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:18.760 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.758+0000 7fe28f423740 1 --2- 172.21.15.76:0/33750071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fadc820 0x560d0fad6620 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe280025b40 tx=0x7fe280003c80 comp rx=0 tx=0).stop 2024-03-18T14:55:18.760 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.758+0000 7fe28f423740 1 -- 172.21.15.76:0/33750071 shutdown_connections 2024-03-18T14:55:18.760 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.758+0000 7fe28f423740 1 --2- 172.21.15.76:0/33750071 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x560d0fb534c0 0x560d0fb558e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.760 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.758+0000 7fe28f423740 1 --2- 172.21.15.76:0/33750071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560d0fadc820 0x560d0fad6620 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.760 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.758+0000 7fe28f423740 1 --2- 172.21.15.76:0/33750071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d0fad3a50 0x560d0fad60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.760 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.758+0000 7fe26dffb640 0 client.15990 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:18.760 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.758+0000 7fe28f423740 1 --2- 172.21.15.76:0/33750071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d0fad3130 0x560d0fae4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.760 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.758+0000 7fe28f423740 1 -- 172.21.15.76:0/33750071 >> 172.21.15.76:0/33750071 conn(0x560d0facda50 msgr2=0x560d0facc320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:18.760 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.758+0000 7fe28f423740 1 -- 172.21.15.76:0/33750071 shutdown_connections 2024-03-18T14:55:18.760 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.758+0000 7fe28f423740 1 -- 172.21.15.76:0/33750071 wait complete. 2024-03-18T14:55:18.760 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.758+0000 7fe28f423740 20 client.15990 trim_cache size 0 max 16384 2024-03-18T14:55:18.761 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:55:18.761 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:55:18.761 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:55:18.761 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:55:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.759+0000 7fe28c836640 1 -- 172.21.15.76:0/2998050957 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe2700a2a10 msgr2=0x7fe2700a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.759+0000 7fe28c836640 1 --2- 172.21.15.76:0/2998050957 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe2700a2a10 0x7fe2700a4ed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe278000c00 tx=0x7fe2780095e0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.759+0000 7fe28c836640 1 -- 172.21.15.76:0/2998050957 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe288069800 msgr2=0x7fe288198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.759+0000 7fe28c836640 1 --2- 172.21.15.76:0/2998050957 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe288069800 0x7fe288198540 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00e9f0 tx=0x7fe27c00eeb0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.759+0000 7fe28c836640 1 -- 172.21.15.76:0/2998050957 shutdown_connections 2024-03-18T14:55:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.759+0000 7fe28c836640 1 --2- 172.21.15.76:0/2998050957 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe2700a2a10 0x7fe2700a4ed0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.759+0000 7fe28c836640 1 --2- 172.21.15.76:0/2998050957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2881078b0 0x7fe28819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.759+0000 7fe28c836640 1 --2- 172.21.15.76:0/2998050957 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe288069800 0x7fe288198540 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.759+0000 7fe28c836640 1 --2- 172.21.15.76:0/2998050957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe288068e60 0x7fe288198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.759+0000 7fe28c836640 1 -- 172.21.15.76:0/2998050957 >> 172.21.15.76:0/2998050957 conn(0x7fe288077530 msgr2=0x7fe2880fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.759+0000 7fe28c836640 1 -- 172.21.15.76:0/2998050957 shutdown_connections 2024-03-18T14:55:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:18.759+0000 7fe28c836640 1 -- 172.21.15.76:0/2998050957 wait complete. 2024-03-18T14:55:18.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:18 smithi076 ceph-mon[32205]: pgmap v65: 97 pgs: 97 active+clean; 58 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 302 op/s 2024-03-18T14:55:18.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:18 smithi174 ceph-mon[36787]: pgmap v65: 97 pgs: 97 active+clean; 58 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 302 op/s 2024-03-18T14:55:18.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:18 smithi077 ceph-mon[35865]: pgmap v65: 97 pgs: 97 active+clean; 58 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 302 op/s 2024-03-18T14:55:19.735 INFO:tasks.workunit.client.0.smithi076.stdout: 1 145758 10.06 MB/sec warmup 80 sec latency 42.638 ms 2024-03-18T14:55:19.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:19 smithi174 ceph-mon[36787]: pgmap v66: 97 pgs: 97 active+clean; 63 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 426 op/s 2024-03-18T14:55:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:19 smithi077 ceph-mon[35865]: pgmap v66: 97 pgs: 97 active+clean; 63 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 426 op/s 2024-03-18T14:55:20.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:19 smithi076 ceph-mon[32205]: pgmap v66: 97 pgs: 97 active+clean; 63 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 426 op/s 2024-03-18T14:55:20.735 INFO:tasks.workunit.client.0.smithi076.stdout: 1 147439 10.01 MB/sec warmup 81 sec latency 23.021 ms 2024-03-18T14:55:21.735 INFO:tasks.workunit.client.0.smithi076.stdout: 1 149506 10.05 MB/sec warmup 82 sec latency 10.324 ms 2024-03-18T14:55:22.735 INFO:tasks.workunit.client.0.smithi076.stdout: 1 151606 10.03 MB/sec warmup 83 sec latency 22.342 ms 2024-03-18T14:55:22.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:22 smithi076 ceph-mon[32205]: pgmap v67: 97 pgs: 97 active+clean; 63 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 251 op/s 2024-03-18T14:55:22.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:22 smithi174 ceph-mon[36787]: pgmap v67: 97 pgs: 97 active+clean; 63 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 251 op/s 2024-03-18T14:55:22.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:22 smithi077 ceph-mon[35865]: pgmap v67: 97 pgs: 97 active+clean; 63 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 251 op/s 2024-03-18T14:55:23.735 INFO:tasks.workunit.client.0.smithi076.stdout: 1 153319 10.04 MB/sec warmup 84 sec latency 20.455 ms 2024-03-18T14:55:24.735 INFO:tasks.workunit.client.0.smithi076.stdout: 1 156036 10.13 MB/sec warmup 85 sec latency 12.351 ms 2024-03-18T14:55:24.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:24 smithi076 ceph-mon[32205]: pgmap v68: 97 pgs: 97 active+clean; 59 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 418 op/s 2024-03-18T14:55:24.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:24 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:55:24.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:24 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:55:24.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:24 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:55:24.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:24 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:55:24.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:24 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:55:24.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:24 smithi174 ceph-mon[36787]: pgmap v68: 97 pgs: 97 active+clean; 59 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 418 op/s 2024-03-18T14:55:24.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:24 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:55:24.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:24 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:55:24.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:24 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:55:24.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:24 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:55:24.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:24 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:55:24.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:24 smithi077 ceph-mon[35865]: pgmap v68: 97 pgs: 97 active+clean; 59 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 418 op/s 2024-03-18T14:55:24.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:24 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:55:24.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:24 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:55:24.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:24 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:55:24.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:24 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:55:24.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:24 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:55:25.735 INFO:tasks.workunit.client.0.smithi076.stdout: 1 157217 10.05 MB/sec warmup 86 sec latency 17.435 ms 2024-03-18T14:55:25.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:25 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:55:25.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:25 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:55:25.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:25 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:55:25.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:25 smithi076 ceph-mon[32205]: pgmap v69: 97 pgs: 97 active+clean; 59 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 290 op/s 2024-03-18T14:55:25.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:25 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:55:25.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:25 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:55:25.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:25 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:55:25.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:25 smithi174 ceph-mon[36787]: pgmap v69: 97 pgs: 97 active+clean; 59 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 290 op/s 2024-03-18T14:55:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:25 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:55:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:25 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:55:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:25 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:55:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:25 smithi077 ceph-mon[35865]: pgmap v69: 97 pgs: 97 active+clean; 59 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 290 op/s 2024-03-18T14:55:26.735 INFO:tasks.workunit.client.0.smithi076.stdout: 1 159905 10.13 MB/sec warmup 87 sec latency 14.035 ms 2024-03-18T14:55:27.735 INFO:tasks.workunit.client.0.smithi076.stdout: 1 161288 10.08 MB/sec warmup 88 sec latency 32.746 ms 2024-03-18T14:55:28.735 INFO:tasks.workunit.client.0.smithi076.stdout: 1 163609 10.12 MB/sec warmup 89 sec latency 11.076 ms 2024-03-18T14:55:28.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:28 smithi076 ceph-mon[32205]: pgmap v70: 97 pgs: 97 active+clean; 59 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 290 op/s 2024-03-18T14:55:28.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:28 smithi174 ceph-mon[36787]: pgmap v70: 97 pgs: 97 active+clean; 59 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 290 op/s 2024-03-18T14:55:28.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:28 smithi077 ceph-mon[35865]: pgmap v70: 97 pgs: 97 active+clean; 59 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 290 op/s 2024-03-18T14:55:29.735 INFO:tasks.workunit.client.0.smithi076.stdout: 1 165661 10.09 MB/sec warmup 90 sec latency 17.421 ms 2024-03-18T14:55:29.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:29 smithi174 ceph-mon[36787]: pgmap v71: 97 pgs: 97 active+clean; 64 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 414 op/s 2024-03-18T14:55:29.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:29 smithi077 ceph-mon[35865]: pgmap v71: 97 pgs: 97 active+clean; 64 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 414 op/s 2024-03-18T14:55:30.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:29 smithi076 ceph-mon[32205]: pgmap v71: 97 pgs: 97 active+clean; 64 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 414 op/s 2024-03-18T14:55:30.735 INFO:tasks.workunit.client.0.smithi076.stdout: 1 167400 10.11 MB/sec warmup 91 sec latency 15.721 ms 2024-03-18T14:55:31.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 170239 10.19 MB/sec warmup 92 sec latency 17.074 ms 2024-03-18T14:55:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.849+0000 7f680364b640 1 -- 172.21.15.76:0/1963900843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67fc1077a0 msgr2=0x7f67fc107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.849+0000 7f680364b640 1 --2- 172.21.15.76:0/1963900843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67fc1077a0 0x7f67fc107b80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f67f8009930 tx=0x7f67f802bc30 comp rx=0 tx=0).stop 2024-03-18T14:55:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.849+0000 7f680364b640 1 -- 172.21.15.76:0/1963900843 shutdown_connections 2024-03-18T14:55:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.849+0000 7f680364b640 1 --2- 172.21.15.76:0/1963900843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67fc102150 0x7f67fc10e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.849+0000 7f680364b640 1 --2- 172.21.15.76:0/1963900843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67fc101790 0x7f67fc101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.849+0000 7f680364b640 1 --2- 172.21.15.76:0/1963900843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67fc1077a0 0x7f67fc107b80 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.849+0000 7f680364b640 1 -- 172.21.15.76:0/1963900843 >> 172.21.15.76:0/1963900843 conn(0x7f67fc0fd2a0 msgr2=0x7f67fc0ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.849+0000 7f680364b640 1 -- 172.21.15.76:0/1963900843 shutdown_connections 2024-03-18T14:55:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.849+0000 7f680364b640 1 -- 172.21.15.76:0/1963900843 wait complete. 2024-03-18T14:55:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f680364b640 1 Processor -- start 2024-03-18T14:55:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f680364b640 1 -- start start 2024-03-18T14:55:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f680364b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67fc101790 0x7f67fc197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f680364b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67fc102150 0x7f67fc198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f680364b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67fc1077a0 0x7f67fc19c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f680364b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc111b70 con 0x7f67fc102150 2024-03-18T14:55:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f680364b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc1119f0 con 0x7f67fc1077a0 2024-03-18T14:55:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f680364b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc111cf0 con 0x7f67fc101790 2024-03-18T14:55:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f6800be6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67fc102150 0x7f67fc198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f6800be6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67fc102150 0x7f67fc198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51356/0 (socket says 172.21.15.76:51356) 2024-03-18T14:55:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f6800be6640 1 -- 172.21.15.76:0/4080288795 learned_addr learned my addr 172.21.15.76:0/4080288795 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f68013e7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67fc101790 0x7f67fc197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f6801be8640 1 --2- 172.21.15.76:0/4080288795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67fc1077a0 0x7f67fc19c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f6800be6640 1 -- 172.21.15.76:0/4080288795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67fc101790 msgr2=0x7f67fc197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f6800be6640 1 --2- 172.21.15.76:0/4080288795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67fc101790 0x7f67fc197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f6800be6640 1 -- 172.21.15.76:0/4080288795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67fc1077a0 msgr2=0x7f67fc19c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f6800be6640 1 --2- 172.21.15.76:0/4080288795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67fc1077a0 0x7f67fc19c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.850+0000 7f6800be6640 1 -- 172.21.15.76:0/4080288795 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67fc19cfe0 con 0x7f67fc102150 2024-03-18T14:55:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.851+0000 7f6801be8640 1 --2- 172.21.15.76:0/4080288795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67fc1077a0 0x7f67fc19c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.851+0000 7f6800be6640 1 --2- 172.21.15.76:0/4080288795 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67fc102150 0x7f67fc198530 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00e9f0 tx=0x7f67ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.851+0000 7f67f27fc640 1 -- 172.21.15.76:0/4080288795 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec00cd80 con 0x7f67fc102150 2024-03-18T14:55:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.851+0000 7f67f27fc640 1 -- 172.21.15.76:0/4080288795 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67ec010910 con 0x7f67fc102150 2024-03-18T14:55:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.851+0000 7f680364b640 1 -- 172.21.15.76:0/4080288795 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67fc19d2d0 con 0x7f67fc102150 2024-03-18T14:55:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.851+0000 7f67f27fc640 1 -- 172.21.15.76:0/4080288795 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec018990 con 0x7f67fc102150 2024-03-18T14:55:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.851+0000 7f680364b640 1 -- 172.21.15.76:0/4080288795 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67fc1a4b60 con 0x7f67fc102150 2024-03-18T14:55:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.853+0000 7f67f27fc640 1 -- 172.21.15.76:0/4080288795 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f67ec010ac0 con 0x7f67fc102150 2024-03-18T14:55:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.854+0000 7f6806239740 1 Processor -- start 2024-03-18T14:55:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.854+0000 7f67f27fc640 1 --2- 172.21.15.76:0/4080288795 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f67e80a2a10 0x7f67e80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.854+0000 7f6806239740 1 -- start start 2024-03-18T14:55:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.854+0000 7f6801be8640 1 --2- 172.21.15.76:0/4080288795 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f67e80a2a10 0x7f67e80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.854+0000 7f67f27fc640 1 -- 172.21.15.76:0/4080288795 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f67ec0cf0b0 con 0x7f67fc102150 2024-03-18T14:55:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.854+0000 7f6806239740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11130 0x561f51c11510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.854+0000 7f6806239740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c11a50 0x561f51c1a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.854+0000 7f6806239740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c1a820 0x561f51c1cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.854+0000 7f6806239740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561f51b75a40 con 0x561f51c11130 2024-03-18T14:55:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.854+0000 7f6806239740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561f51b1e920 con 0x561f51c11a50 2024-03-18T14:55:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.854+0000 7f6806239740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561f51b71ae0 con 0x561f51c1a820 2024-03-18T14:55:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.854+0000 7f68013e7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11130 0x561f51c11510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.854+0000 7f68013e7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11130 0x561f51c11510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51370/0 (socket says 172.21.15.76:51370) 2024-03-18T14:55:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.854+0000 7f6800be6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c11a50 0x561f51c1a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.854+0000 7f68013e7640 1 -- 172.21.15.76:0/1259413669 learned_addr learned my addr 172.21.15.76:0/1259413669 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.855+0000 7f6801be8640 1 --2- 172.21.15.76:0/1259413669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c1a820 0x561f51c1cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.855+0000 7f6801be8640 1 --2- 172.21.15.76:0/4080288795 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f67e80a2a10 0x7f67e80a4ed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f67f4005fd0 tx=0x7f67f4011b50 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.855+0000 7f6800be6640 1 -- 172.21.15.76:0/1259413669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c1a820 msgr2=0x561f51c1cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.855+0000 7f6800be6640 1 --2- 172.21.15.76:0/1259413669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c1a820 0x561f51c1cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.855+0000 7f6800be6640 1 -- 172.21.15.76:0/1259413669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11130 msgr2=0x561f51c11510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.855+0000 7f6800be6640 1 --2- 172.21.15.76:0/1259413669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11130 0x561f51c11510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.855+0000 7f6800be6640 1 -- 172.21.15.76:0/1259413669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f51b36460 con 0x561f51c11a50 2024-03-18T14:55:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.855+0000 7f68013e7640 1 --2- 172.21.15.76:0/1259413669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11130 0x561f51c11510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.855+0000 7f6800be6640 1 --2- 172.21.15.76:0/1259413669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c11a50 0x561f51c1a2e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f67ec0cc8f0 tx=0x7f67ec09b970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=366f3877a60c342d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.855+0000 7f67e77fe640 1 -- 172.21.15.76:0/1259413669 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec099bb0 con 0x561f51c11a50 2024-03-18T14:55:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.855+0000 7f67e77fe640 1 -- 172.21.15.76:0/1259413669 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67ec0ccb70 con 0x561f51c11a50 2024-03-18T14:55:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.855+0000 7f67e77fe640 1 -- 172.21.15.76:0/1259413669 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec0afde0 con 0x561f51c11a50 2024-03-18T14:55:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.856+0000 7f6806239740 1 -- 172.21.15.76:0/1259413669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c11a50 msgr2=0x561f51c1a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.856+0000 7f6806239740 1 --2- 172.21.15.76:0/1259413669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c11a50 0x561f51c1a2e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f67ec0cc8f0 tx=0x7f67ec09b970 comp rx=0 tx=0).stop 2024-03-18T14:55:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.856+0000 7f6806239740 1 -- 172.21.15.76:0/1259413669 shutdown_connections 2024-03-18T14:55:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.856+0000 7f6806239740 1 --2- 172.21.15.76:0/1259413669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c1a820 0x561f51c1cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.856+0000 7f6806239740 1 --2- 172.21.15.76:0/1259413669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c11a50 0x561f51c1a2e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.856+0000 7f6806239740 1 --2- 172.21.15.76:0/1259413669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11130 0x561f51c11510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.856+0000 7f6806239740 1 -- 172.21.15.76:0/1259413669 >> 172.21.15.76:0/1259413669 conn(0x561f51c0ba50 msgr2=0x561f51c0be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.856+0000 7f6806239740 1 -- 172.21.15.76:0/1259413669 shutdown_connections 2024-03-18T14:55:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.856+0000 7f6806239740 1 -- 172.21.15.76:0/1259413669 wait complete. 2024-03-18T14:55:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.856+0000 7f6806239740 1 -- start start 2024-03-18T14:55:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.856+0000 7f6806239740 1 Processor -- start 2024-03-18T14:55:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6806239740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c11130 0x561f51c17830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6806239740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c17d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.876 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6806239740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c1a820 0x561f51c182b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.876 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6806239740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561f51c1d450 con 0x561f51c11a50 2024-03-18T14:55:31.876 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6806239740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561f51b31d70 con 0x561f51c1a820 2024-03-18T14:55:31.876 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6806239740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561f51c1d5d0 con 0x561f51c11130 2024-03-18T14:55:31.876 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6806239740 10 client.0 _ll_register_callbacks cb 0x561f51889a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:55:31.876 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6800be6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c17d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.877 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6800be6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c17d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51384/0 (socket says 172.21.15.76:51384) 2024-03-18T14:55:31.877 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6800be6640 1 -- 172.21.15.76:0/2928195576 learned_addr learned my addr 172.21.15.76:0/2928195576 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:31.877 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f68013e7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c11130 0x561f51c17830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.877 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6801be8640 1 --2- 172.21.15.76:0/2928195576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c1a820 0x561f51c182b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.877 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6801be8640 1 -- 172.21.15.76:0/2928195576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c11130 msgr2=0x561f51c17830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:31.877 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6801be8640 1 --2- 172.21.15.76:0/2928195576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c11130 0x561f51c17830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.877 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6801be8640 1 -- 172.21.15.76:0/2928195576 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 msgr2=0x561f51c17d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:31.877 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6801be8640 1 --2- 172.21.15.76:0/2928195576 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c17d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.877 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6801be8640 1 -- 172.21.15.76:0/2928195576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f5188a210 con 0x561f51c1a820 2024-03-18T14:55:31.877 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6800be6640 1 --2- 172.21.15.76:0/2928195576 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c17d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:31.877 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f6801be8640 1 --2- 172.21.15.76:0/2928195576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c1a820 0x561f51c182b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f67f401b000 tx=0x7f67f4004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea5fd1f94828df85 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:31.877 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f67e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:55:31.877 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f67e4ff9640 1 -- 172.21.15.76:0/2928195576 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f4017040 con 0x561f51c1a820 2024-03-18T14:55:31.877 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.857+0000 7f67e4ff9640 1 -- 172.21.15.76:0/2928195576 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f67f4028440 con 0x561f51c1a820 2024-03-18T14:55:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.858+0000 7f6806239740 1 -- 172.21.15.76:0/2928195576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67ec0ccb70 con 0x561f51c1a820 2024-03-18T14:55:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.858+0000 7f67e4ff9640 1 -- 172.21.15.76:0/2928195576 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f67f4028640 con 0x561f51c1a820 2024-03-18T14:55:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.858+0000 7f6806239740 10 client.25859 fetch_fsmap learned FSMap version 29 2024-03-18T14:55:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.858+0000 7f6806239740 1 -- 172.21.15.76:0/2928195576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561f51956620 con 0x561f51c1a820 2024-03-18T14:55:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.858+0000 7f67e4ff9640 1 -- 172.21.15.76:0/2928195576 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67ec0ccb70 con 0x561f51c1a820 2024-03-18T14:55:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.858+0000 7f67e4ff9640 1 -- 172.21.15.76:0/2928195576 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f67f4026020 con 0x561f51c1a820 2024-03-18T14:55:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.858+0000 7f6806239740 10 client.25859 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:55:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.858+0000 7f6806239740 10 client.25859 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:55:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.858+0000 7f6806239740 20 client.25859 populate_metadata read hostname 'smithi076' 2024-03-18T14:55:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.858+0000 7f6806239740 1 --2- 172.21.15.76:0/2928195576 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561f51c2aa20 0x561f51c2ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.858+0000 7f6806239740 4 client.25859 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:55:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.858+0000 7f6806239740 1 -- 172.21.15.76:0/2928195576 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f51b82f80 con 0x561f51c2aa20 2024-03-18T14:55:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.858+0000 7f68013e7640 1 --2- 172.21.15.76:0/2928195576 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561f51c2aa20 0x561f51c2ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.859+0000 7f67e4ff9640 10 client.25859 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.859+0000 7f68013e7640 1 --2- 172.21.15.76:0/2928195576 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561f51c2aa20 0x561f51c2ce10 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6423fe625d955d4a server_cookie=24fe478dd01ec0a5 in_seq=0 out_seq=0 2024-03-18T14:55:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.859+0000 7f67e4ff9640 1 -- 172.21.15.76:0/2928195576 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561f51b82f80 con 0x561f51c2aa20 2024-03-18T14:55:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.859+0000 7f67e4ff9640 10 client.25859 handle_command_reply: tid=0 2024-03-18T14:55:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.859+0000 7f6806239740 1 client.25859 shutdown 2024-03-18T14:55:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.859+0000 7f6806239740 1 -- 172.21.15.76:0/2928195576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c1a820 msgr2=0x561f51c182b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.859+0000 7f6806239740 1 --2- 172.21.15.76:0/2928195576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c1a820 0x561f51c182b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f67f401b000 tx=0x7f67f4004ab0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.859+0000 7f6806239740 1 -- 172.21.15.76:0/2928195576 shutdown_connections 2024-03-18T14:55:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.859+0000 7f6806239740 1 --2- 172.21.15.76:0/2928195576 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561f51c2aa20 0x561f51c2ce10 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.859+0000 7f6806239740 1 --2- 172.21.15.76:0/2928195576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c1a820 0x561f51c182b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.859+0000 7f67e4ff9640 0 client.25859 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.859+0000 7f6806239740 1 --2- 172.21.15.76:0/2928195576 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c17d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.859+0000 7f6806239740 1 --2- 172.21.15.76:0/2928195576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c11130 0x561f51c17830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.859+0000 7f6806239740 1 -- 172.21.15.76:0/2928195576 >> 172.21.15.76:0/2928195576 conn(0x561f51c0ba50 msgr2=0x561f51c0d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.860+0000 7f6806239740 1 -- 172.21.15.76:0/2928195576 shutdown_connections 2024-03-18T14:55:31.880 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.860+0000 7f6806239740 1 -- 172.21.15.76:0/2928195576 wait complete. 2024-03-18T14:55:31.880 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.860+0000 7f6806239740 20 client.25859 trim_cache size 0 max 16384 2024-03-18T14:55:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.874+0000 7f6806239740 1 Processor -- start 2024-03-18T14:55:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.874+0000 7f6806239740 1 -- start start 2024-03-18T14:55:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.874+0000 7f6806239740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c11130 0x561f51c19f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.874+0000 7f6806239740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c06a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.874+0000 7f6806239740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c1a820 0x561f51c261f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.874+0000 7f6806239740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc1afc90 con 0x561f51c11a50 2024-03-18T14:55:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.874+0000 7f6806239740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc1ad500 con 0x561f51c1a820 2024-03-18T14:55:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.874+0000 7f6806239740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc1ab350 con 0x561f51c11130 2024-03-18T14:55:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.875+0000 7f6800be6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c06a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.875+0000 7f6800be6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c06a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51390/0 (socket says 172.21.15.76:51390) 2024-03-18T14:55:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.875+0000 7f6800be6640 1 -- 172.21.15.76:0/3074020150 learned_addr learned my addr 172.21.15.76:0/3074020150 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.875+0000 7f6801be8640 1 --2- 172.21.15.76:0/3074020150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c1a820 0x561f51c261f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.886 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.875+0000 7f68013e7640 1 --2- 172.21.15.76:0/3074020150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c11130 0x561f51c19f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.886 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.875+0000 7f6800be6640 1 -- 172.21.15.76:0/3074020150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c11130 msgr2=0x561f51c19f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:31.886 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.875+0000 7f6800be6640 1 --2- 172.21.15.76:0/3074020150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c11130 0x561f51c19f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.886 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.875+0000 7f6800be6640 1 -- 172.21.15.76:0/3074020150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c1a820 msgr2=0x561f51c261f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:31.886 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.875+0000 7f6800be6640 1 --2- 172.21.15.76:0/3074020150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c1a820 0x561f51c261f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.886 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.875+0000 7f6800be6640 1 -- 172.21.15.76:0/3074020150 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f51c167e0 con 0x561f51c11a50 2024-03-18T14:55:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.875+0000 7f6801be8640 1 --2- 172.21.15.76:0/3074020150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c1a820 0x561f51c261f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.875+0000 7f6800be6640 1 --2- 172.21.15.76:0/3074020150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c06a40 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00c6a0 tx=0x7f67ec097c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eccd8a552137409b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.875+0000 7f67e6ffd640 1 -- 172.21.15.76:0/3074020150 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec0b4070 con 0x561f51c11a50 2024-03-18T14:55:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.875+0000 7f67e6ffd640 1 -- 172.21.15.76:0/3074020150 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67ec086070 con 0x561f51c11a50 2024-03-18T14:55:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.875+0000 7f67e6ffd640 1 -- 172.21.15.76:0/3074020150 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec09cd90 con 0x561f51c11a50 2024-03-18T14:55:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.876+0000 7f6806239740 1 -- 172.21.15.76:0/3074020150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 msgr2=0x561f51c06a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.876+0000 7f6806239740 1 --2- 172.21.15.76:0/3074020150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c06a40 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00c6a0 tx=0x7f67ec097c80 comp rx=0 tx=0).stop 2024-03-18T14:55:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.876+0000 7f6806239740 1 -- 172.21.15.76:0/3074020150 shutdown_connections 2024-03-18T14:55:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.876+0000 7f6806239740 1 --2- 172.21.15.76:0/3074020150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c1a820 0x561f51c261f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.876+0000 7f6806239740 1 --2- 172.21.15.76:0/3074020150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c06a40 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.876+0000 7f6806239740 1 --2- 172.21.15.76:0/3074020150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c11130 0x561f51c19f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.876+0000 7f6806239740 1 -- 172.21.15.76:0/3074020150 >> 172.21.15.76:0/3074020150 conn(0x561f51c0ba50 msgr2=0x561f51c0abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.876+0000 7f6806239740 1 -- 172.21.15.76:0/3074020150 shutdown_connections 2024-03-18T14:55:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.876+0000 7f6806239740 1 -- 172.21.15.76:0/3074020150 wait complete. 2024-03-18T14:55:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.876+0000 7f6806239740 1 -- start start 2024-03-18T14:55:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.876+0000 7f6806239740 1 Processor -- start 2024-03-18T14:55:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6806239740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c11130 0x561f51c1e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6806239740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c1e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6806239740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c1a820 0x561f51c1ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6806239740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc1ab9c0 con 0x561f51c11a50 2024-03-18T14:55:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6806239740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc1ace30 con 0x561f51c11130 2024-03-18T14:55:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6806239740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc1ac090 con 0x561f51c1a820 2024-03-18T14:55:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6806239740 10 client.0 _ll_register_callbacks cb 0x7f67f4003a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:55:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6800be6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c1e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f68013e7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c11130 0x561f51c1e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6800be6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c1e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51396/0 (socket says 172.21.15.76:51396) 2024-03-18T14:55:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6800be6640 1 -- 172.21.15.76:0/3356765689 learned_addr learned my addr 172.21.15.76:0/3356765689 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f68013e7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c11130 0x561f51c1e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:36628/0 (socket says 172.21.15.76:36628) 2024-03-18T14:55:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6801be8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c1a820 0x561f51c1ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6800be6640 1 -- 172.21.15.76:0/3356765689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c1a820 msgr2=0x561f51c1ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6800be6640 1 --2- 172.21.15.76:0/3356765689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c1a820 0x561f51c1ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6800be6640 1 -- 172.21.15.76:0/3356765689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c11130 msgr2=0x561f51c1e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6800be6640 1 --2- 172.21.15.76:0/3356765689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c11130 0x561f51c1e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6800be6640 1 -- 172.21.15.76:0/3356765689 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f51c8fde0 con 0x561f51c11a50 2024-03-18T14:55:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f68013e7640 1 --2- 172.21.15.76:0/3356765689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c11130 0x561f51c1e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6800be6640 1 --2- 172.21.15.76:0/3356765689 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c1e700 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f67ec09b5b0 tx=0x7f67ec0afc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e05dcc53daaf82ea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f67e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:55:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f67e4ff9640 1 -- 172.21.15.76:0/3356765689 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec0b9060 con 0x561f51c11a50 2024-03-18T14:55:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f67e4ff9640 1 -- 172.21.15.76:0/3356765689 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f67ec018b30 con 0x561f51c11a50 2024-03-18T14:55:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.877+0000 7f6806239740 1 -- 172.21.15.76:0/3356765689 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67ec086070 con 0x561f51c11a50 2024-03-18T14:55:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.878+0000 7f67e4ff9640 1 -- 172.21.15.76:0/3356765689 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f67ec0b4040 con 0x561f51c11a50 2024-03-18T14:55:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.878+0000 7f6806239740 10 client.16008 fetch_fsmap learned FSMap version 29 2024-03-18T14:55:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.878+0000 7f6806239740 1 -- 172.21.15.76:0/3356765689 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561f51c910d0 con 0x561f51c11a50 2024-03-18T14:55:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.878+0000 7f67e4ff9640 1 -- 172.21.15.76:0/3356765689 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67ec086070 con 0x561f51c11a50 2024-03-18T14:55:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.878+0000 7f67e4ff9640 1 -- 172.21.15.76:0/3356765689 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f67ec097600 con 0x561f51c11a50 2024-03-18T14:55:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.878+0000 7f6806239740 10 client.16008 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:55:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.878+0000 7f6806239740 10 client.16008 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:55:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.878+0000 7f6806239740 20 client.16008 populate_metadata read hostname 'smithi076' 2024-03-18T14:55:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.878+0000 7f6806239740 1 --2- 172.21.15.76:0/3356765689 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561f51c912f0 0x561f51c93710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.878+0000 7f6806239740 4 client.16008 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:55:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.878+0000 7f6806239740 1 -- 172.21.15.76:0/3356765689 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f51b7d2d0 con 0x561f51c912f0 2024-03-18T14:55:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.878+0000 7f68013e7640 1 --2- 172.21.15.76:0/3356765689 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561f51c912f0 0x561f51c93710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.878+0000 7f67e4ff9640 10 client.16008 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.878+0000 7f68013e7640 1 --2- 172.21.15.76:0/3356765689 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561f51c912f0 0x561f51c93710 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=264b4f4a1b5873f8 server_cookie=e66fb0ed43d62a03 in_seq=0 out_seq=0 2024-03-18T14:55:31.946 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.944+0000 7f67e4ff9640 1 -- 172.21.15.76:0/3356765689 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561f51b7d2d0 con 0x561f51c912f0 2024-03-18T14:55:31.946 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.944+0000 7f67e4ff9640 10 client.16008 handle_command_reply: tid=0 2024-03-18T14:55:31.946 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.944+0000 7f6806239740 1 client.16008 shutdown 2024-03-18T14:55:31.946 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.944+0000 7f6806239740 1 -- 172.21.15.76:0/3356765689 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 msgr2=0x561f51c1e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:31.946 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.944+0000 7f6806239740 1 --2- 172.21.15.76:0/3356765689 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c1e700 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f67ec09b5b0 tx=0x7f67ec0afc50 comp rx=0 tx=0).stop 2024-03-18T14:55:31.946 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.944+0000 7f6806239740 1 -- 172.21.15.76:0/3356765689 shutdown_connections 2024-03-18T14:55:31.946 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.944+0000 7f6806239740 1 --2- 172.21.15.76:0/3356765689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f51c1a820 0x561f51c1ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.946 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.944+0000 7f6806239740 1 --2- 172.21.15.76:0/3356765689 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561f51c912f0 0x561f51c93710 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.946 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.945+0000 7f6806239740 1 --2- 172.21.15.76:0/3356765689 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f51c11a50 0x561f51c1e700 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.946 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.945+0000 7f6806239740 1 --2- 172.21.15.76:0/3356765689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f51c11130 0x561f51c1e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.946 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.945+0000 7f67e4ff9640 0 client.16008 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:31.946 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.945+0000 7f6806239740 1 -- 172.21.15.76:0/3356765689 >> 172.21.15.76:0/3356765689 conn(0x561f51c0ba50 msgr2=0x561f51c177f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:31.947 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.945+0000 7f6806239740 1 -- 172.21.15.76:0/3356765689 shutdown_connections 2024-03-18T14:55:31.947 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.945+0000 7f6806239740 1 -- 172.21.15.76:0/3356765689 wait complete. 2024-03-18T14:55:31.947 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.945+0000 7f6806239740 20 client.16008 trim_cache size 0 max 16384 2024-03-18T14:55:31.947 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:55:31.947 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:55:31.947 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:55:31.947 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:55:31.948 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.945+0000 7f680364b640 1 -- 172.21.15.76:0/4080288795 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f67e80a2a10 msgr2=0x7f67e80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:31.948 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.945+0000 7f680364b640 1 --2- 172.21.15.76:0/4080288795 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f67e80a2a10 0x7f67e80a4ed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f67f4005fd0 tx=0x7f67f4011b50 comp rx=0 tx=0).stop 2024-03-18T14:55:31.948 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.945+0000 7f680364b640 1 -- 172.21.15.76:0/4080288795 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67fc102150 msgr2=0x7f67fc198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:31.948 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.945+0000 7f680364b640 1 --2- 172.21.15.76:0/4080288795 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67fc102150 0x7f67fc198530 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00e9f0 tx=0x7f67ec00eeb0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.948 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.945+0000 7f680364b640 1 -- 172.21.15.76:0/4080288795 shutdown_connections 2024-03-18T14:55:31.948 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.945+0000 7f680364b640 1 --2- 172.21.15.76:0/4080288795 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f67e80a2a10 0x7f67e80a4ed0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.949 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.945+0000 7f680364b640 1 --2- 172.21.15.76:0/4080288795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67fc1077a0 0x7f67fc19c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.949 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.945+0000 7f680364b640 1 --2- 172.21.15.76:0/4080288795 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67fc102150 0x7f67fc198530 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.949 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.945+0000 7f680364b640 1 --2- 172.21.15.76:0/4080288795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67fc101790 0x7f67fc197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:31.949 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.945+0000 7f680364b640 1 -- 172.21.15.76:0/4080288795 >> 172.21.15.76:0/4080288795 conn(0x7f67fc0fd2a0 msgr2=0x7f67fc0fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:31.949 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.946+0000 7f680364b640 1 -- 172.21.15.76:0/4080288795 shutdown_connections 2024-03-18T14:55:31.949 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:31.946+0000 7f680364b640 1 -- 172.21.15.76:0/4080288795 wait complete. 2024-03-18T14:55:32.735 INFO:tasks.workunit.client.0.smithi076.stdout: 1 171331 10.12 MB/sec warmup 93 sec latency 65.152 ms 2024-03-18T14:55:32.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:32 smithi076 ceph-mon[32205]: pgmap v72: 97 pgs: 97 active+clean; 64 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 290 op/s 2024-03-18T14:55:32.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:32 smithi174 ceph-mon[36787]: pgmap v72: 97 pgs: 97 active+clean; 64 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 290 op/s 2024-03-18T14:55:32.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:32 smithi077 ceph-mon[35865]: pgmap v72: 97 pgs: 97 active+clean; 64 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 290 op/s 2024-03-18T14:55:33.735 INFO:tasks.workunit.client.0.smithi076.stdout: 1 174087 10.19 MB/sec warmup 94 sec latency 14.055 ms 2024-03-18T14:55:33.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:33 smithi076 ceph-mon[32205]: pgmap v73: 97 pgs: 97 active+clean; 61 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 456 op/s 2024-03-18T14:55:33.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:33 smithi174 ceph-mon[36787]: pgmap v73: 97 pgs: 97 active+clean; 61 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 456 op/s 2024-03-18T14:55:33.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:33 smithi077 ceph-mon[35865]: pgmap v73: 97 pgs: 97 active+clean; 61 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 456 op/s 2024-03-18T14:55:34.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 175633 10.14 MB/sec warmup 95 sec latency 13.496 ms 2024-03-18T14:55:35.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 177813 10.18 MB/sec warmup 96 sec latency 19.952 ms 2024-03-18T14:55:36.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 180122 10.19 MB/sec warmup 97 sec latency 15.449 ms 2024-03-18T14:55:36.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:36 smithi076 ceph-mon[32205]: pgmap v74: 97 pgs: 97 active+clean; 61 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 289 op/s 2024-03-18T14:55:36.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:36 smithi174 ceph-mon[36787]: pgmap v74: 97 pgs: 97 active+clean; 61 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 289 op/s 2024-03-18T14:55:36.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:36 smithi077 ceph-mon[35865]: pgmap v74: 97 pgs: 97 active+clean; 61 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 289 op/s 2024-03-18T14:55:37.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 181480 10.17 MB/sec warmup 98 sec latency 41.186 ms 2024-03-18T14:55:37.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:37 smithi076 ceph-mon[32205]: pgmap v75: 97 pgs: 97 active+clean; 61 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 289 op/s 2024-03-18T14:55:37.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:37 smithi174 ceph-mon[36787]: pgmap v75: 97 pgs: 97 active+clean; 61 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 289 op/s 2024-03-18T14:55:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:37 smithi077 ceph-mon[35865]: pgmap v75: 97 pgs: 97 active+clean; 61 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 289 op/s 2024-03-18T14:55:38.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 184282 10.21 MB/sec warmup 99 sec latency 15.321 ms 2024-03-18T14:55:39.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 185559 10.17 MB/sec warmup 100 sec latency 14.691 ms 2024-03-18T14:55:39.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:39 smithi174 ceph-mon[36787]: pgmap v76: 97 pgs: 97 active+clean; 66 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 409 op/s 2024-03-18T14:55:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:39 smithi077 ceph-mon[35865]: pgmap v76: 97 pgs: 97 active+clean; 66 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 409 op/s 2024-03-18T14:55:40.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:39 smithi076 ceph-mon[32205]: pgmap v76: 97 pgs: 97 active+clean; 66 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 409 op/s 2024-03-18T14:55:40.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 188274 10.24 MB/sec warmup 101 sec latency 15.335 ms 2024-03-18T14:55:41.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 189782 10.19 MB/sec warmup 102 sec latency 16.192 ms 2024-03-18T14:55:41.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:41 smithi174 ceph-mon[36787]: pgmap v77: 97 pgs: 97 active+clean; 66 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 286 op/s 2024-03-18T14:55:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:41 smithi077 ceph-mon[35865]: pgmap v77: 97 pgs: 97 active+clean; 66 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 286 op/s 2024-03-18T14:55:42.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:41 smithi076 ceph-mon[32205]: pgmap v77: 97 pgs: 97 active+clean; 66 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 286 op/s 2024-03-18T14:55:42.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 191966 10.23 MB/sec warmup 103 sec latency 81.799 ms 2024-03-18T14:55:43.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 194143 10.21 MB/sec warmup 104 sec latency 22.311 ms 2024-03-18T14:55:43.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:43 smithi174 ceph-mon[36787]: pgmap v78: 97 pgs: 97 active+clean; 70 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 406 op/s 2024-03-18T14:55:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:43 smithi077 ceph-mon[35865]: pgmap v78: 97 pgs: 97 active+clean; 70 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 406 op/s 2024-03-18T14:55:44.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:43 smithi076 ceph-mon[32205]: pgmap v78: 97 pgs: 97 active+clean; 70 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 406 op/s 2024-03-18T14:55:44.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 195818 10.22 MB/sec warmup 105 sec latency 11.778 ms 2024-03-18T14:55:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.035+0000 7fb89191b640 1 -- 172.21.15.76:0/3589380720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb88c105660 msgr2=0x7fb88c105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.035+0000 7fb89191b640 1 --2- 172.21.15.76:0/3589380720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb88c105660 0x7fb88c105a40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009900 tx=0x7fb87c02bc30 comp rx=0 tx=0).stop 2024-03-18T14:55:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.035+0000 7fb89191b640 1 -- 172.21.15.76:0/3589380720 shutdown_connections 2024-03-18T14:55:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.035+0000 7fb89191b640 1 --2- 172.21.15.76:0/3589380720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88c100930 0x7fb88c100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.035+0000 7fb89191b640 1 --2- 172.21.15.76:0/3589380720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb88c1012d0 0x7fb88c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.035+0000 7fb89191b640 1 --2- 172.21.15.76:0/3589380720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb88c105660 0x7fb88c105a40 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.035+0000 7fb89191b640 1 -- 172.21.15.76:0/3589380720 >> 172.21.15.76:0/3589380720 conn(0x7fb88c0774b0 msgr2=0x7fb88c0778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.035+0000 7fb89191b640 1 -- 172.21.15.76:0/3589380720 shutdown_connections 2024-03-18T14:55:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.035+0000 7fb89191b640 1 -- 172.21.15.76:0/3589380720 wait complete. 2024-03-18T14:55:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.035+0000 7fb89191b640 1 Processor -- start 2024-03-18T14:55:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb89191b640 1 -- start start 2024-03-18T14:55:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb89191b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb88c100930 0x7fb88c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb89191b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb88c1012d0 0x7fb88c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb89191b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88c105660 0x7fb88c19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb89191b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c110b70 con 0x7fb88c100930 2024-03-18T14:55:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb89191b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c1109f0 con 0x7fb88c1012d0 2024-03-18T14:55:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb89191b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c110cf0 con 0x7fb88c105660 2024-03-18T14:55:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb88affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb88c100930 0x7fb88c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb88affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb88c100930 0x7fb88c198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41756/0 (socket says 172.21.15.76:41756) 2024-03-18T14:55:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb88affd640 1 -- 172.21.15.76:0/701271933 learned_addr learned my addr 172.21.15.76:0/701271933 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb883fff640 1 --2- 172.21.15.76:0/701271933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb88c1012d0 0x7fb88c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb88b7fe640 1 --2- 172.21.15.76:0/701271933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88c105660 0x7fb88c19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb88affd640 1 -- 172.21.15.76:0/701271933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88c105660 msgr2=0x7fb88c19c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb88affd640 1 --2- 172.21.15.76:0/701271933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88c105660 0x7fb88c19c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb88affd640 1 -- 172.21.15.76:0/701271933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb88c1012d0 msgr2=0x7fb88c198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb88affd640 1 --2- 172.21.15.76:0/701271933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb88c1012d0 0x7fb88c198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb88affd640 1 -- 172.21.15.76:0/701271933 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb88c19d020 con 0x7fb88c100930 2024-03-18T14:55:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb883fff640 1 --2- 172.21.15.76:0/701271933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb88c1012d0 0x7fb88c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.036+0000 7fb88affd640 1 --2- 172.21.15.76:0/701271933 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb88c100930 0x7fb88c198030 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb87c006010 tx=0x7fb87c02cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.037+0000 7fb890919640 1 -- 172.21.15.76:0/701271933 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c041070 con 0x7fb88c100930 2024-03-18T14:55:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.037+0000 7fb890919640 1 -- 172.21.15.76:0/701271933 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb87c004450 con 0x7fb88c100930 2024-03-18T14:55:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.037+0000 7fb89191b640 1 -- 172.21.15.76:0/701271933 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb88c19d2b0 con 0x7fb88c100930 2024-03-18T14:55:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.037+0000 7fb890919640 1 -- 172.21.15.76:0/701271933 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c003e00 con 0x7fb88c100930 2024-03-18T14:55:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.037+0000 7fb89191b640 1 -- 172.21.15.76:0/701271933 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb88c1a4ba0 con 0x7fb88c100930 2024-03-18T14:55:45.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.039+0000 7fb890919640 1 -- 172.21.15.76:0/701271933 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fb87c044070 con 0x7fb88c100930 2024-03-18T14:55:45.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.039+0000 7fb8945e2740 1 Processor -- start 2024-03-18T14:55:45.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.039+0000 7fb8945e2740 1 -- start start 2024-03-18T14:55:45.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb8945e2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3130 0x55c1041f3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb8945e2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041f3a50 0x55c1041fc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb8945e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041fc820 0x55c1041fec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb8945e2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c104157a40 con 0x55c1041f3130 2024-03-18T14:55:45.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb8945e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c104100920 con 0x55c1041fc820 2024-03-18T14:55:45.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb8945e2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c104153ae0 con 0x55c1041f3a50 2024-03-18T14:55:45.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb890919640 1 --2- 172.21.15.76:0/701271933 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb8780a2a10 0x7fb8780a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb88b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3130 0x55c1041f3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb883fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041fc820 0x55c1041fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb88affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041f3a50 0x55c1041fc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb88b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3130 0x55c1041f3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41768/0 (socket says 172.21.15.76:41768) 2024-03-18T14:55:45.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb88b7fe640 1 -- 172.21.15.76:0/3286215132 learned_addr learned my addr 172.21.15.76:0/3286215132 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:45.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb88b7fe640 1 --2- 172.21.15.76:0/701271933 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb8780a2a10 0x7fb8780a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb890919640 1 -- 172.21.15.76:0/701271933 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fb87c0f07c0 con 0x7fb88c100930 2024-03-18T14:55:45.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb883fff640 1 -- 172.21.15.76:0/3286215132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041f3a50 msgr2=0x55c1041fc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:45.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb883fff640 1 --2- 172.21.15.76:0/3286215132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041f3a50 0x55c1041fc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb883fff640 1 -- 172.21.15.76:0/3286215132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3130 msgr2=0x55c1041f3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:45.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb883fff640 1 --2- 172.21.15.76:0/3286215132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3130 0x55c1041f3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb883fff640 1 -- 172.21.15.76:0/3286215132 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c104118460 con 0x55c1041fc820 2024-03-18T14:55:45.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb88b7fe640 1 --2- 172.21.15.76:0/701271933 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb8780a2a10 0x7fb8780a4ed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb8840099e0 tx=0x7fb884009500 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:45.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.040+0000 7fb883fff640 1 --2- 172.21.15.76:0/3286215132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041fc820 0x55c1041fec10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb874009fa0 tx=0x7fb8740118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5e0533f9d74faba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:45.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.041+0000 7fb881ffb640 1 -- 172.21.15.76:0/3286215132 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb874004310 con 0x55c1041fc820 2024-03-18T14:55:45.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.041+0000 7fb881ffb640 1 -- 172.21.15.76:0/3286215132 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8740044b0 con 0x55c1041fc820 2024-03-18T14:55:45.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.041+0000 7fb881ffb640 1 -- 172.21.15.76:0/3286215132 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87400fae0 con 0x55c1041fc820 2024-03-18T14:55:45.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.041+0000 7fb8945e2740 1 -- 172.21.15.76:0/3286215132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041fc820 msgr2=0x55c1041fec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:45.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.041+0000 7fb8945e2740 1 --2- 172.21.15.76:0/3286215132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041fc820 0x55c1041fec10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb874009fa0 tx=0x7fb8740118d0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.041+0000 7fb8945e2740 1 -- 172.21.15.76:0/3286215132 shutdown_connections 2024-03-18T14:55:45.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.041+0000 7fb8945e2740 1 --2- 172.21.15.76:0/3286215132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041f3a50 0x55c1041fc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.041+0000 7fb8945e2740 1 --2- 172.21.15.76:0/3286215132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041fc820 0x55c1041fec10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.041+0000 7fb8945e2740 1 --2- 172.21.15.76:0/3286215132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3130 0x55c1041f3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.041+0000 7fb8945e2740 1 -- 172.21.15.76:0/3286215132 >> 172.21.15.76:0/3286215132 conn(0x55c1041eda50 msgr2=0x55c1041ede30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:45.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.041+0000 7fb8945e2740 1 -- 172.21.15.76:0/3286215132 shutdown_connections 2024-03-18T14:55:45.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.041+0000 7fb8945e2740 1 -- 172.21.15.76:0/3286215132 wait complete. 2024-03-18T14:55:45.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.042+0000 7fb8945e2740 1 -- start start 2024-03-18T14:55:45.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.042+0000 7fb8945e2740 1 Processor -- start 2024-03-18T14:55:45.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.042+0000 7fb8945e2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041f3130 0x55c1041f98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.042+0000 7fb8945e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041f3a50 0x55c1041f9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.042+0000 7fb8945e2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041fc820 0x55c1041fa320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.042+0000 7fb8945e2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c1041ff450 con 0x55c1041fc820 2024-03-18T14:55:45.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.042+0000 7fb8945e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c104113d70 con 0x55c1041f3a50 2024-03-18T14:55:45.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.042+0000 7fb8945e2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c1041ff5d0 con 0x55c1041f3130 2024-03-18T14:55:45.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.042+0000 7fb8945e2740 10 client.0 _ll_register_callbacks cb 0x55c103e6ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:55:45.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.042+0000 7fb88b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041fc820 0x55c1041fa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.050 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.042+0000 7fb883fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041f3a50 0x55c1041f9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.050 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.042+0000 7fb88b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041fc820 0x55c1041fa320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41774/0 (socket says 172.21.15.76:41774) 2024-03-18T14:55:45.050 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.042+0000 7fb88b7fe640 1 -- 172.21.15.76:0/334082138 learned_addr learned my addr 172.21.15.76:0/334082138 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:45.050 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.042+0000 7fb88affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041f3130 0x55c1041f98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.050 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.042+0000 7fb88b7fe640 1 -- 172.21.15.76:0/334082138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041f3130 msgr2=0x55c1041f98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:45.050 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.042+0000 7fb88b7fe640 1 --2- 172.21.15.76:0/334082138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041f3130 0x55c1041f98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.050 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.043+0000 7fb88b7fe640 1 -- 172.21.15.76:0/334082138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041f3a50 msgr2=0x55c1041f9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:45.050 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.043+0000 7fb88b7fe640 1 --2- 172.21.15.76:0/334082138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041f3a50 0x55c1041f9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.050 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.043+0000 7fb88b7fe640 1 -- 172.21.15.76:0/334082138 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c103e6c210 con 0x55c1041fc820 2024-03-18T14:55:45.050 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.043+0000 7fb88b7fe640 1 --2- 172.21.15.76:0/334082138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041fc820 0x55c1041fa320 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb884011fd0 tx=0x7fb884004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d81fb936b58be0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:45.050 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.043+0000 7fb86affd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:55:45.050 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.043+0000 7fb86affd640 1 -- 172.21.15.76:0/334082138 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb884004b20 con 0x55c1041fc820 2024-03-18T14:55:45.050 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.043+0000 7fb86affd640 1 -- 172.21.15.76:0/334082138 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fb884004cc0 con 0x55c1041fc820 2024-03-18T14:55:45.050 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.043+0000 7fb8945e2740 1 -- 172.21.15.76:0/334082138 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8740044b0 con 0x55c1041fc820 2024-03-18T14:55:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.043+0000 7fb86affd640 1 -- 172.21.15.76:0/334082138 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb884020420 con 0x55c1041fc820 2024-03-18T14:55:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.043+0000 7fb8945e2740 10 client.16020 fetch_fsmap learned FSMap version 29 2024-03-18T14:55:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.043+0000 7fb8945e2740 1 -- 172.21.15.76:0/334082138 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c103f38620 con 0x55c1041fc820 2024-03-18T14:55:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.043+0000 7fb86affd640 1 -- 172.21.15.76:0/334082138 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8740044b0 con 0x55c1041fc820 2024-03-18T14:55:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.043+0000 7fb86affd640 1 -- 172.21.15.76:0/334082138 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fb884028070 con 0x55c1041fc820 2024-03-18T14:55:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.044+0000 7fb8945e2740 10 client.16020 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:55:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.044+0000 7fb8945e2740 10 client.16020 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:55:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.044+0000 7fb8945e2740 20 client.16020 populate_metadata read hostname 'smithi076' 2024-03-18T14:55:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.044+0000 7fb8945e2740 1 --2- 172.21.15.76:0/334082138 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55c10420cb10 0x55c10420ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.044+0000 7fb8945e2740 4 client.16020 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:55:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.044+0000 7fb8945e2740 1 -- 172.21.15.76:0/334082138 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c104164f80 con 0x55c10420cb10 2024-03-18T14:55:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.044+0000 7fb88affd640 1 --2- 172.21.15.76:0/334082138 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55c10420cb10 0x55c10420ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.044+0000 7fb88affd640 1 --2- 172.21.15.76:0/334082138 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55c10420cb10 0x55c10420ef00 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6095a24a7d7633 server_cookie=fb07db1995677726 in_seq=0 out_seq=0 2024-03-18T14:55:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.044+0000 7fb86affd640 10 client.16020 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.045+0000 7fb86affd640 1 -- 172.21.15.76:0/334082138 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c104164f80 con 0x55c10420cb10 2024-03-18T14:55:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.045+0000 7fb86affd640 10 client.16020 handle_command_reply: tid=0 2024-03-18T14:55:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.045+0000 7fb8945e2740 1 client.16020 shutdown 2024-03-18T14:55:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.045+0000 7fb8945e2740 1 -- 172.21.15.76:0/334082138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041fc820 msgr2=0x55c1041fa320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.045+0000 7fb8945e2740 1 --2- 172.21.15.76:0/334082138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041fc820 0x55c1041fa320 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb884011fd0 tx=0x7fb884004a40 comp rx=0 tx=0).stop 2024-03-18T14:55:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.045+0000 7fb8945e2740 1 -- 172.21.15.76:0/334082138 shutdown_connections 2024-03-18T14:55:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.045+0000 7fb8945e2740 1 --2- 172.21.15.76:0/334082138 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55c10420cb10 0x55c10420ef00 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.045+0000 7fb8945e2740 1 --2- 172.21.15.76:0/334082138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041fc820 0x55c1041fa320 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.045+0000 7fb8945e2740 1 --2- 172.21.15.76:0/334082138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041f3a50 0x55c1041f9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.045+0000 7fb8945e2740 1 --2- 172.21.15.76:0/334082138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041f3130 0x55c1041f98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.045+0000 7fb86affd640 0 client.16020 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.045+0000 7fb8945e2740 1 -- 172.21.15.76:0/334082138 >> 172.21.15.76:0/334082138 conn(0x55c1041eda50 msgr2=0x55c1041ef7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.045+0000 7fb8945e2740 1 -- 172.21.15.76:0/334082138 shutdown_connections 2024-03-18T14:55:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.045+0000 7fb8945e2740 1 -- 172.21.15.76:0/334082138 wait complete. 2024-03-18T14:55:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.045+0000 7fb8945e2740 20 client.16020 trim_cache size 0 max 16384 2024-03-18T14:55:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb8945e2740 1 Processor -- start 2024-03-18T14:55:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb8945e2740 1 -- start start 2024-03-18T14:55:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb8945e2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041f3130 0x55c1042081f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb8945e2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3a50 0x55c1041fa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb8945e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041fc820 0x55c1042147f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb8945e2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c1afb80 con 0x55c1041f3a50 2024-03-18T14:55:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb8945e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c1ad3f0 con 0x55c1041fc820 2024-03-18T14:55:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb8945e2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c1ab240 con 0x55c1041f3130 2024-03-18T14:55:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb883fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3a50 0x55c1041fa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb883fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3a50 0x55c1041fa140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41784/0 (socket says 172.21.15.76:41784) 2024-03-18T14:55:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb883fff640 1 -- 172.21.15.76:0/3699357669 learned_addr learned my addr 172.21.15.76:0/3699357669 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb88b7fe640 1 --2- 172.21.15.76:0/3699357669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041fc820 0x55c1042147f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb88affd640 1 --2- 172.21.15.76:0/3699357669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041f3130 0x55c1042081f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb883fff640 1 -- 172.21.15.76:0/3699357669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041f3130 msgr2=0x55c1042081f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb883fff640 1 --2- 172.21.15.76:0/3699357669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041f3130 0x55c1042081f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb883fff640 1 -- 172.21.15.76:0/3699357669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041fc820 msgr2=0x55c1042147f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb883fff640 1 --2- 172.21.15.76:0/3699357669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041fc820 0x55c1042147f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb883fff640 1 -- 172.21.15.76:0/3699357669 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1041f8850 con 0x55c1041f3a50 2024-03-18T14:55:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.061+0000 7fb88b7fe640 1 --2- 172.21.15.76:0/3699357669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041fc820 0x55c1042147f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.062+0000 7fb883fff640 1 --2- 172.21.15.76:0/3699357669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3a50 0x55c1041fa140 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb874009380 tx=0x7fb87400f680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c74ded307ce9be72 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.062+0000 7fb8817fa640 1 -- 172.21.15.76:0/3699357669 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87402a070 con 0x55c1041f3a50 2024-03-18T14:55:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.062+0000 7fb8817fa640 1 -- 172.21.15.76:0/3699357669 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb87400f7b0 con 0x55c1041f3a50 2024-03-18T14:55:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.062+0000 7fb8817fa640 1 -- 172.21.15.76:0/3699357669 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87401d040 con 0x55c1041f3a50 2024-03-18T14:55:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.062+0000 7fb8945e2740 1 -- 172.21.15.76:0/3699357669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3a50 msgr2=0x55c1041fa140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.062+0000 7fb8945e2740 1 --2- 172.21.15.76:0/3699357669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3a50 0x55c1041fa140 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb874009380 tx=0x7fb87400f680 comp rx=0 tx=0).stop 2024-03-18T14:55:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.062+0000 7fb8945e2740 1 -- 172.21.15.76:0/3699357669 shutdown_connections 2024-03-18T14:55:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.062+0000 7fb8945e2740 1 --2- 172.21.15.76:0/3699357669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041fc820 0x55c1042147f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.062+0000 7fb8945e2740 1 --2- 172.21.15.76:0/3699357669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3a50 0x55c1041fa140 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.062+0000 7fb8945e2740 1 --2- 172.21.15.76:0/3699357669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041f3130 0x55c1042081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.062+0000 7fb8945e2740 1 -- 172.21.15.76:0/3699357669 >> 172.21.15.76:0/3699357669 conn(0x55c1041eda50 msgr2=0x55c1041ef730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.063+0000 7fb8945e2740 1 -- 172.21.15.76:0/3699357669 shutdown_connections 2024-03-18T14:55:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.063+0000 7fb8945e2740 1 -- 172.21.15.76:0/3699357669 wait complete. 2024-03-18T14:55:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.063+0000 7fb8945e2740 1 -- start start 2024-03-18T14:55:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.063+0000 7fb8945e2740 1 Processor -- start 2024-03-18T14:55:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.063+0000 7fb8945e2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3130 0x55c1041f6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.063+0000 7fb8945e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041f3a50 0x55c1041f6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.063+0000 7fb8945e2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041fc820 0x55c1041f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.063+0000 7fb8945e2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c1ab8b0 con 0x55c1041f3130 2024-03-18T14:55:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.063+0000 7fb8945e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c1acd20 con 0x55c1041f3a50 2024-03-18T14:55:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.063+0000 7fb8945e2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c1abf80 con 0x55c1041fc820 2024-03-18T14:55:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.063+0000 7fb8945e2740 10 client.0 _ll_register_callbacks cb 0x7fb884011e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:55:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.063+0000 7fb88affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3130 0x55c1041f6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.063+0000 7fb88affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3130 0x55c1041f6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41800/0 (socket says 172.21.15.76:41800) 2024-03-18T14:55:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.063+0000 7fb883fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041f3a50 0x55c1041f6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.063+0000 7fb88affd640 1 -- 172.21.15.76:0/2432231645 learned_addr learned my addr 172.21.15.76:0/2432231645 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.063+0000 7fb88b7fe640 1 --2- 172.21.15.76:0/2432231645 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041fc820 0x55c1041f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb88affd640 1 -- 172.21.15.76:0/2432231645 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041fc820 msgr2=0x55c1041f70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb88affd640 1 --2- 172.21.15.76:0/2432231645 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041fc820 0x55c1041f70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb88affd640 1 -- 172.21.15.76:0/2432231645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041f3a50 msgr2=0x55c1041f6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb88affd640 1 --2- 172.21.15.76:0/2432231645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041f3a50 0x55c1041f6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb88affd640 1 -- 172.21.15.76:0/2432231645 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c104271ff0 con 0x55c1041f3130 2024-03-18T14:55:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb88b7fe640 1 --2- 172.21.15.76:0/2432231645 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041fc820 0x55c1041f70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:55:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb88affd640 1 --2- 172.21.15.76:0/2432231645 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3130 0x55c1041f6620 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb87c000c00 tx=0x7fb87c08caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a3c81468bcb45c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb86affd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:55:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb86affd640 1 -- 172.21.15.76:0/2432231645 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c04b040 con 0x55c1041f3130 2024-03-18T14:55:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb86affd640 1 -- 172.21.15.76:0/2432231645 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fb87c0bb2d0 con 0x55c1041f3130 2024-03-18T14:55:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb8945e2740 1 -- 172.21.15.76:0/2432231645 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb87400f7b0 con 0x55c1041f3130 2024-03-18T14:55:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb86affd640 1 -- 172.21.15.76:0/2432231645 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb87c0c1070 con 0x55c1041f3130 2024-03-18T14:55:45.072 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb8945e2740 10 client.16026 fetch_fsmap learned FSMap version 29 2024-03-18T14:55:45.072 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb8945e2740 1 -- 172.21.15.76:0/2432231645 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c1042732c0 con 0x55c1041f3130 2024-03-18T14:55:45.072 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb86affd640 1 -- 172.21.15.76:0/2432231645 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb87400f7b0 con 0x55c1041f3130 2024-03-18T14:55:45.072 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb86affd640 1 -- 172.21.15.76:0/2432231645 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fb87c0ee180 con 0x55c1041f3130 2024-03-18T14:55:45.072 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb8945e2740 10 client.16026 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T14:55:45.072 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb8945e2740 10 client.16026 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:55:45.072 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb8945e2740 20 client.16026 populate_metadata read hostname 'smithi076' 2024-03-18T14:55:45.072 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb8945e2740 1 --2- 172.21.15.76:0/2432231645 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55c104273940 0x55c104275d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:45.072 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb8945e2740 4 client.16026 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:55:45.072 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.064+0000 7fb8945e2740 1 -- 172.21.15.76:0/2432231645 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c10415f2d0 con 0x55c104273940 2024-03-18T14:55:45.072 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.065+0000 7fb883fff640 1 --2- 172.21.15.76:0/2432231645 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55c104273940 0x55c104275d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:45.072 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.065+0000 7fb86affd640 10 client.16026 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:45.072 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.065+0000 7fb883fff640 1 --2- 172.21.15.76:0/2432231645 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55c104273940 0x55c104275d70 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5c598f80868dd83 server_cookie=8214843f999d09d4 in_seq=0 out_seq=0 2024-03-18T14:55:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.127+0000 7fb86affd640 1 -- 172.21.15.76:0/2432231645 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c10415f2d0 con 0x55c104273940 2024-03-18T14:55:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.127+0000 7fb86affd640 10 client.16026 handle_command_reply: tid=0 2024-03-18T14:55:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.127+0000 7fb8945e2740 1 client.16026 shutdown 2024-03-18T14:55:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.127+0000 7fb8945e2740 1 -- 172.21.15.76:0/2432231645 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3130 msgr2=0x55c1041f6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.127+0000 7fb8945e2740 1 --2- 172.21.15.76:0/2432231645 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3130 0x55c1041f6620 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb87c000c00 tx=0x7fb87c08caa0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.127+0000 7fb8945e2740 1 -- 172.21.15.76:0/2432231645 shutdown_connections 2024-03-18T14:55:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.127+0000 7fb8945e2740 1 --2- 172.21.15.76:0/2432231645 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1041fc820 0x55c1041f70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.127+0000 7fb8945e2740 1 --2- 172.21.15.76:0/2432231645 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55c104273940 0x55c104275d70 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.127+0000 7fb8945e2740 1 --2- 172.21.15.76:0/2432231645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1041f3a50 0x55c1041f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.127+0000 7fb8945e2740 1 --2- 172.21.15.76:0/2432231645 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c1041f3130 0x55c1041f6620 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.127+0000 7fb86affd640 0 client.16026 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.127+0000 7fb8945e2740 1 -- 172.21.15.76:0/2432231645 >> 172.21.15.76:0/2432231645 conn(0x55c1041eda50 msgr2=0x55c1041eee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.127+0000 7fb8945e2740 1 -- 172.21.15.76:0/2432231645 shutdown_connections 2024-03-18T14:55:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.127+0000 7fb8945e2740 1 -- 172.21.15.76:0/2432231645 wait complete. 2024-03-18T14:55:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.127+0000 7fb8945e2740 20 client.16026 trim_cache size 0 max 16384 2024-03-18T14:55:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:55:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:55:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:55:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:55:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.128+0000 7fb89191b640 1 -- 172.21.15.76:0/701271933 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb8780a2a10 msgr2=0x7fb8780a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.128+0000 7fb89191b640 1 --2- 172.21.15.76:0/701271933 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb8780a2a10 0x7fb8780a4ed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb8840099e0 tx=0x7fb884009500 comp rx=0 tx=0).stop 2024-03-18T14:55:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.128+0000 7fb89191b640 1 -- 172.21.15.76:0/701271933 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb88c100930 msgr2=0x7fb88c198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.128+0000 7fb89191b640 1 --2- 172.21.15.76:0/701271933 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb88c100930 0x7fb88c198030 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb87c006010 tx=0x7fb87c02cb30 comp rx=0 tx=0).stop 2024-03-18T14:55:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.128+0000 7fb89191b640 1 -- 172.21.15.76:0/701271933 shutdown_connections 2024-03-18T14:55:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.128+0000 7fb89191b640 1 --2- 172.21.15.76:0/701271933 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb8780a2a10 0x7fb8780a4ed0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.128+0000 7fb89191b640 1 --2- 172.21.15.76:0/701271933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88c105660 0x7fb88c19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.128+0000 7fb89191b640 1 --2- 172.21.15.76:0/701271933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb88c1012d0 0x7fb88c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.128+0000 7fb89191b640 1 --2- 172.21.15.76:0/701271933 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb88c100930 0x7fb88c198030 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.128+0000 7fb89191b640 1 -- 172.21.15.76:0/701271933 >> 172.21.15.76:0/701271933 conn(0x7fb88c0774b0 msgr2=0x7fb88c1044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.128+0000 7fb89191b640 1 -- 172.21.15.76:0/701271933 shutdown_connections 2024-03-18T14:55:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:45.128+0000 7fb89191b640 1 -- 172.21.15.76:0/701271933 wait complete. 2024-03-18T14:55:45.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:45 smithi076 ceph-mon[32205]: daemon mds.e restarted 2024-03-18T14:55:45.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:45 smithi174 ceph-mon[36787]: daemon mds.e restarted 2024-03-18T14:55:45.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:45 smithi077 ceph-mon[35865]: daemon mds.e restarted 2024-03-18T14:55:45.727 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:55:45 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setuser ceph since I am not root 2024-03-18T14:55:45.727 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:55:45 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setgroup ceph since I am not root 2024-03-18T14:55:45.727 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:55:45 smithi077 ceph-mds[65404]: -- 172.21.15.77:0/1585029592 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5627544cba40 con 0x56275449f000 2024-03-18T14:55:45.727 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:55:45 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: starting mds.e at 2024-03-18T14:55:45.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 198603 10.28 MB/sec warmup 106 sec latency 23.245 ms 2024-03-18T14:55:46.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:46 smithi076 ceph-mon[32205]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T14:55:46.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:46 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.77:6834/515747233,v1:172.21.15.77:6835/515747233] up:boot 2024-03-18T14:55:46.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:46 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T14:55:46.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:46 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:55:46.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:46 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:55:46.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:46 smithi076 ceph-mon[32205]: pgmap v80: 97 pgs: 97 active+clean; 70 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 289 op/s 2024-03-18T14:55:46.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:46 smithi174 ceph-mon[36787]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T14:55:46.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:46 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.77:6834/515747233,v1:172.21.15.77:6835/515747233] up:boot 2024-03-18T14:55:46.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:46 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T14:55:46.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:46 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:55:46.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:46 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:55:46.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:46 smithi174 ceph-mon[36787]: pgmap v80: 97 pgs: 97 active+clean; 70 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 289 op/s 2024-03-18T14:55:46.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:46 smithi077 ceph-mon[35865]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T14:55:46.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:46 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.77:6834/515747233,v1:172.21.15.77:6835/515747233] up:boot 2024-03-18T14:55:46.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:46 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T14:55:46.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:46 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:55:46.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:46 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:55:46.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:46 smithi077 ceph-mon[35865]: pgmap v80: 97 pgs: 97 active+clean; 70 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 289 op/s 2024-03-18T14:55:46.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 199923 10.22 MB/sec warmup 107 sec latency 17.977 ms 2024-03-18T14:55:47.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 202487 10.28 MB/sec warmup 108 sec latency 26.066 ms 2024-03-18T14:55:47.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:47 smithi174 ceph-mon[36787]: pgmap v81: 97 pgs: 97 active+clean; 70 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 289 op/s 2024-03-18T14:55:47.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:47 smithi077 ceph-mon[35865]: pgmap v81: 97 pgs: 97 active+clean; 70 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 289 op/s 2024-03-18T14:55:48.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:47 smithi076 ceph-mon[32205]: pgmap v81: 97 pgs: 97 active+clean; 70 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 289 op/s 2024-03-18T14:55:48.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 204522 10.26 MB/sec warmup 109 sec latency 30.502 ms 2024-03-18T14:55:49.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 206321 10.27 MB/sec warmup 110 sec latency 11.955 ms 2024-03-18T14:55:49.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:49 smithi174 ceph-mon[36787]: pgmap v82: 97 pgs: 97 active+clean; 67 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.2 MiB/s wr, 354 op/s 2024-03-18T14:55:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:49 smithi077 ceph-mon[35865]: pgmap v82: 97 pgs: 97 active+clean; 67 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.2 MiB/s wr, 354 op/s 2024-03-18T14:55:50.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:49 smithi076 ceph-mon[32205]: pgmap v82: 97 pgs: 97 active+clean; 67 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.2 MiB/s wr, 354 op/s 2024-03-18T14:55:50.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 208989 10.31 MB/sec warmup 111 sec latency 16.419 ms 2024-03-18T14:55:51.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 210258 10.27 MB/sec warmup 112 sec latency 16.914 ms 2024-03-18T14:55:51.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:51 smithi174 ceph-mon[36787]: pgmap v83: 97 pgs: 97 active+clean; 67 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.2 MiB/s wr, 354 op/s 2024-03-18T14:55:51.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:51 smithi077 ceph-mon[35865]: pgmap v83: 97 pgs: 97 active+clean; 67 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.2 MiB/s wr, 354 op/s 2024-03-18T14:55:52.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:51 smithi076 ceph-mon[32205]: pgmap v83: 97 pgs: 97 active+clean; 67 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.2 MiB/s wr, 354 op/s 2024-03-18T14:55:52.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 213039 10.34 MB/sec warmup 113 sec latency 26.327 ms 2024-03-18T14:55:53.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 214379 10.29 MB/sec warmup 114 sec latency 26.087 ms 2024-03-18T14:55:53.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:53 smithi174 ceph-mon[36787]: pgmap v84: 97 pgs: 97 active+clean; 72 MiB data, 891 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.2 MiB/s wr, 348 op/s 2024-03-18T14:55:53.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:53 smithi077 ceph-mon[35865]: pgmap v84: 97 pgs: 97 active+clean; 72 MiB data, 891 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.2 MiB/s wr, 348 op/s 2024-03-18T14:55:54.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:53 smithi076 ceph-mon[32205]: pgmap v84: 97 pgs: 97 active+clean; 72 MiB data, 891 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.2 MiB/s wr, 348 op/s 2024-03-18T14:55:54.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 216802 10.33 MB/sec warmup 115 sec latency 14.684 ms 2024-03-18T14:55:55.736 INFO:tasks.workunit.client.0.smithi076.stdout: 1 218114 10.28 MB/sec warmup 116 sec latency 180.013 ms 2024-03-18T14:55:55.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:55 smithi174 ceph-mon[36787]: pgmap v85: 97 pgs: 97 active+clean; 72 MiB data, 891 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 340 op/s 2024-03-18T14:55:55.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:55 smithi077 ceph-mon[35865]: pgmap v85: 97 pgs: 97 active+clean; 72 MiB data, 891 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 340 op/s 2024-03-18T14:55:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:55 smithi076 ceph-mon[32205]: pgmap v85: 97 pgs: 97 active+clean; 72 MiB data, 891 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 340 op/s 2024-03-18T14:55:56.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 220391 10.31 MB/sec warmup 117 sec latency 21.525 ms 2024-03-18T14:55:57.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 222361 10.28 MB/sec warmup 118 sec latency 27.781 ms 2024-03-18T14:55:57.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:57 smithi174 ceph-mon[36787]: pgmap v86: 97 pgs: 97 active+clean; 72 MiB data, 891 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 290 op/s 2024-03-18T14:55:57.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:57 smithi077 ceph-mon[35865]: pgmap v86: 97 pgs: 97 active+clean; 72 MiB data, 891 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 290 op/s 2024-03-18T14:55:58.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:57 smithi076 ceph-mon[32205]: pgmap v86: 97 pgs: 97 active+clean; 72 MiB data, 891 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 290 op/s 2024-03-18T14:55:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.221+0000 7fc2fbea1640 1 -- 172.21.15.76:0/2222182883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2f4105680 msgr2=0x7fc2f4105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.221+0000 7fc2fbea1640 1 --2- 172.21.15.76:0/2222182883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2f4105680 0x7fc2f4105a60 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009ca0 tx=0x7fc2f002c0f0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.222+0000 7fc2fbea1640 1 -- 172.21.15.76:0/2222182883 shutdown_connections 2024-03-18T14:55:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.222+0000 7fc2fbea1640 1 --2- 172.21.15.76:0/2222182883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2f4100930 0x7fc2f4100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.222+0000 7fc2fbea1640 1 --2- 172.21.15.76:0/2222182883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2f41012f0 0x7fc2f410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.222+0000 7fc2fbea1640 1 --2- 172.21.15.76:0/2222182883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2f4105680 0x7fc2f4105a60 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.222+0000 7fc2fbea1640 1 -- 172.21.15.76:0/2222182883 >> 172.21.15.76:0/2222182883 conn(0x7fc2f4077530 msgr2=0x7fc2f4077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.222+0000 7fc2fbea1640 1 -- 172.21.15.76:0/2222182883 shutdown_connections 2024-03-18T14:55:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.222+0000 7fc2fbea1640 1 -- 172.21.15.76:0/2222182883 wait complete. 2024-03-18T14:55:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.222+0000 7fc2fbea1640 1 Processor -- start 2024-03-18T14:55:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.222+0000 7fc2fbea1640 1 -- start start 2024-03-18T14:55:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.222+0000 7fc2fbea1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2f4100930 0x7fc2f4196fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.222+0000 7fc2fbea1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2f41012f0 0x7fc2f4197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.222+0000 7fc2fbea1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2f4105680 0x7fc2f419b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.222+0000 7fc2fbea1640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f4110ba0 con 0x7fc2f4105680 2024-03-18T14:55:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.222+0000 7fc2fbea1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f4110a20 con 0x7fc2f4100930 2024-03-18T14:55:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.222+0000 7fc2fbea1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f4110d20 con 0x7fc2f41012f0 2024-03-18T14:55:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.223+0000 7fc2f9c3d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2f4100930 0x7fc2f4196fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.223+0000 7fc2fa43e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2f4105680 0x7fc2f419b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.223+0000 7fc2f9c3d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2f4100930 0x7fc2f4196fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:38794/0 (socket says 172.21.15.76:38794) 2024-03-18T14:55:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.223+0000 7fc2f9c3d640 1 -- 172.21.15.76:0/4238798327 learned_addr learned my addr 172.21.15.76:0/4238798327 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.223+0000 7fc2f943c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2f41012f0 0x7fc2f4197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.223+0000 7fc2fa43e640 1 -- 172.21.15.76:0/4238798327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2f41012f0 msgr2=0x7fc2f4197500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.223+0000 7fc2fa43e640 1 --2- 172.21.15.76:0/4238798327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2f41012f0 0x7fc2f4197500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.223+0000 7fc2fa43e640 1 -- 172.21.15.76:0/4238798327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2f4100930 msgr2=0x7fc2f4196fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.223+0000 7fc2fa43e640 1 --2- 172.21.15.76:0/4238798327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2f4100930 0x7fc2f4196fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.223+0000 7fc2fa43e640 1 -- 172.21.15.76:0/4238798327 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f419bfb0 con 0x7fc2f4105680 2024-03-18T14:55:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.223+0000 7fc2fa43e640 1 --2- 172.21.15.76:0/4238798327 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2f4105680 0x7fc2f419b880 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec00e740 tx=0x7fc2ec00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.224+0000 7fc2eaffd640 1 -- 172.21.15.76:0/4238798327 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec00ccc0 con 0x7fc2f4105680 2024-03-18T14:55:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.224+0000 7fc2eaffd640 1 -- 172.21.15.76:0/4238798327 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2ec00ee50 con 0x7fc2f4105680 2024-03-18T14:55:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.224+0000 7fc2eaffd640 1 -- 172.21.15.76:0/4238798327 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec004a00 con 0x7fc2f4105680 2024-03-18T14:55:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.224+0000 7fc2fbea1640 1 -- 172.21.15.76:0/4238798327 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f419c2a0 con 0x7fc2f4105680 2024-03-18T14:55:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.224+0000 7fc2fbea1640 1 -- 172.21.15.76:0/4238798327 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f41a3b30 con 0x7fc2f4105680 2024-03-18T14:55:58.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.226+0000 7fc2eaffd640 1 -- 172.21.15.76:0/4238798327 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fc2ec004ba0 con 0x7fc2f4105680 2024-03-18T14:55:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.226+0000 7fc2feb6d740 1 Processor -- start 2024-03-18T14:55:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.226+0000 7fc2feb6d740 1 -- start start 2024-03-18T14:55:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.226+0000 7fc2feb6d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca8888e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.226+0000 7fc2feb6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888ea50 0x55ca888994f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.226+0000 7fc2eaffd640 1 --2- 172.21.15.76:0/4238798327 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc2e00a2a10 0x7fc2e00a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.226+0000 7fc2feb6d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca88899a30 0x55ca8889be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.226+0000 7fc2feb6d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ca887f2a40 con 0x55ca8888e130 2024-03-18T14:55:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.226+0000 7fc2feb6d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ca8879b920 con 0x55ca88899a30 2024-03-18T14:55:58.238 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.226+0000 7fc2feb6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ca887eeae0 con 0x55ca8888ea50 2024-03-18T14:55:58.238 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2f943c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca8888e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.238 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2fa43e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca88899a30 0x55ca8889be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.238 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2f9c3d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888ea50 0x55ca888994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.238 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2f943c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca8888e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56350/0 (socket says 172.21.15.76:56350) 2024-03-18T14:55:58.239 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2f943c640 1 -- 172.21.15.76:0/1115911983 learned_addr learned my addr 172.21.15.76:0/1115911983 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:58.239 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2f943c640 1 --2- 172.21.15.76:0/4238798327 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc2e00a2a10 0x7fc2e00a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.239 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2eaffd640 1 -- 172.21.15.76:0/4238798327 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8440+0+0 (secure 0 0 0) 0x7fc2ec0ced40 con 0x7fc2f4105680 2024-03-18T14:55:58.239 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2fa43e640 1 -- 172.21.15.76:0/1115911983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888ea50 msgr2=0x55ca888994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:58.239 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2fa43e640 1 --2- 172.21.15.76:0/1115911983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888ea50 0x55ca888994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.239 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2fa43e640 1 -- 172.21.15.76:0/1115911983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 msgr2=0x55ca8888e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:58.239 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2fa43e640 1 --2- 172.21.15.76:0/1115911983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca8888e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.239 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2fa43e640 1 -- 172.21.15.76:0/1115911983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca887b3460 con 0x55ca88899a30 2024-03-18T14:55:58.239 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2f943c640 1 --2- 172.21.15.76:0/1115911983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca8888e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:58.239 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2f9c3d640 1 --2- 172.21.15.76:0/1115911983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888ea50 0x55ca888994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:58.239 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2f943c640 1 --2- 172.21.15.76:0/4238798327 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc2e00a2a10 0x7fc2e00a4ed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4009710 tx=0x7fc2e4005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:58.239 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2fa43e640 1 --2- 172.21.15.76:0/1115911983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca88899a30 0x55ca8889be20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec0cc4e0 tx=0x7fc2ec00cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=205fbda859ba6f11 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:58.239 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2dffff640 1 -- 172.21.15.76:0/1115911983 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec01c020 con 0x55ca88899a30 2024-03-18T14:55:58.239 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2dffff640 1 -- 172.21.15.76:0/1115911983 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2ec0b80c0 con 0x55ca88899a30 2024-03-18T14:55:58.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.227+0000 7fc2dffff640 1 -- 172.21.15.76:0/1115911983 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec0a1870 con 0x55ca88899a30 2024-03-18T14:55:58.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.228+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1115911983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca88899a30 msgr2=0x55ca8889be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:58.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.228+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1115911983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca88899a30 0x55ca8889be20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec0cc4e0 tx=0x7fc2ec00cef0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.228+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1115911983 shutdown_connections 2024-03-18T14:55:58.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.228+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1115911983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888ea50 0x55ca888994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.228+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1115911983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca88899a30 0x55ca8889be20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.228+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1115911983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca8888e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.228+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1115911983 >> 172.21.15.76:0/1115911983 conn(0x55ca88888a50 msgr2=0x55ca88888e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:58.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.228+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1115911983 shutdown_connections 2024-03-18T14:55:58.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.228+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1115911983 wait complete. 2024-03-18T14:55:58.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.229+0000 7fc2feb6d740 1 -- start start 2024-03-18T14:55:58.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.229+0000 7fc2feb6d740 1 Processor -- start 2024-03-18T14:55:58.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.229+0000 7fc2feb6d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca88896aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.229+0000 7fc2feb6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888ea50 0x55ca88896fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.229+0000 7fc2feb6d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca88899a30 0x55ca88897520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.229+0000 7fc2feb6d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ca8889c600 con 0x55ca8888e130 2024-03-18T14:55:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.229+0000 7fc2feb6d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ca887aed70 con 0x55ca88899a30 2024-03-18T14:55:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.229+0000 7fc2feb6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ca8889c780 con 0x55ca8888ea50 2024-03-18T14:55:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.229+0000 7fc2feb6d740 10 client.0 _ll_register_callbacks cb 0x55ca88506a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:55:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.229+0000 7fc2f9c3d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca88896aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.229+0000 7fc2f9c3d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca88896aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56356/0 (socket says 172.21.15.76:56356) 2024-03-18T14:55:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.229+0000 7fc2f9c3d640 1 -- 172.21.15.76:0/1932649860 learned_addr learned my addr 172.21.15.76:0/1932649860 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.229+0000 7fc2fa43e640 1 --2- 172.21.15.76:0/1932649860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca88899a30 0x55ca88897520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.229+0000 7fc2f943c640 1 --2- 172.21.15.76:0/1932649860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888ea50 0x55ca88896fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.229+0000 7fc2f9c3d640 1 -- 172.21.15.76:0/1932649860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888ea50 msgr2=0x55ca88896fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.229+0000 7fc2f9c3d640 1 --2- 172.21.15.76:0/1932649860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888ea50 0x55ca88896fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.229+0000 7fc2f9c3d640 1 -- 172.21.15.76:0/1932649860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca88899a30 msgr2=0x55ca88897520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.230+0000 7fc2f9c3d640 1 --2- 172.21.15.76:0/1932649860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca88899a30 0x55ca88897520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.230+0000 7fc2f9c3d640 1 -- 172.21.15.76:0/1932649860 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca88507210 con 0x55ca8888e130 2024-03-18T14:55:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.230+0000 7fc2fa43e640 1 --2- 172.21.15.76:0/1932649860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca88899a30 0x55ca88897520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.230+0000 7fc2f9c3d640 1 --2- 172.21.15.76:0/1932649860 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca88896aa0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc2f002c700 tx=0x7fc2f0032340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ef47bcb6e281c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.230+0000 7fc2dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:55:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.230+0000 7fc2dd7fa640 1 -- 172.21.15.76:0/1932649860 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0044070 con 0x55ca8888e130 2024-03-18T14:55:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.230+0000 7fc2dd7fa640 1 -- 172.21.15.76:0/1932649860 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8440+0+0 (secure 0 0 0) 0x7fc2f0032790 con 0x55ca8888e130 2024-03-18T14:55:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.230+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1932649860 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2ec0b80c0 con 0x55ca8888e130 2024-03-18T14:55:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.230+0000 7fc2dd7fa640 1 -- 172.21.15.76:0/1932649860 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc2f0050050 con 0x55ca8888e130 2024-03-18T14:55:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.230+0000 7fc2feb6d740 10 client.16044 fetch_fsmap learned FSMap version 31 2024-03-18T14:55:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.230+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1932649860 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ca885d3620 con 0x55ca8888e130 2024-03-18T14:55:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.230+0000 7fc2dd7fa640 1 -- 172.21.15.76:0/1932649860 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2ec0b80c0 con 0x55ca8888e130 2024-03-18T14:55:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.230+0000 7fc2dd7fa640 1 -- 172.21.15.76:0/1932649860 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fc2f0004370 con 0x55ca8888e130 2024-03-18T14:55:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.230+0000 7fc2feb6d740 10 client.16044 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T14:55:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.231+0000 7fc2feb6d740 10 client.16044 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:55:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.231+0000 7fc2feb6d740 20 client.16044 populate_metadata read hostname 'smithi076' 2024-03-18T14:55:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.231+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1932649860 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ca888a9c10 0x55ca888ac090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.231+0000 7fc2feb6d740 4 client.16044 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:55:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.231+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1932649860 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca887fff80 con 0x55ca888a9c10 2024-03-18T14:55:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.231+0000 7fc2f943c640 1 --2- 172.21.15.76:0/1932649860 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ca888a9c10 0x55ca888ac090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.231+0000 7fc2f943c640 1 --2- 172.21.15.76:0/1932649860 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ca888a9c10 0x55ca888ac090 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5df7066a91b202ea server_cookie=da702cf48b718cba in_seq=0 out_seq=0 2024-03-18T14:55:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.231+0000 7fc2dd7fa640 10 client.16044 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.232+0000 7fc2dd7fa640 1 -- 172.21.15.76:0/1932649860 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ca887fff80 con 0x55ca888a9c10 2024-03-18T14:55:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.232+0000 7fc2dd7fa640 10 client.16044 handle_command_reply: tid=0 2024-03-18T14:55:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.232+0000 7fc2feb6d740 1 client.16044 shutdown 2024-03-18T14:55:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.233+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1932649860 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 msgr2=0x55ca88896aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.233+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1932649860 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca88896aa0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc2f002c700 tx=0x7fc2f0032340 comp rx=0 tx=0).stop 2024-03-18T14:55:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.233+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1932649860 shutdown_connections 2024-03-18T14:55:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.233+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1932649860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888ea50 0x55ca88896fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.233+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1932649860 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ca888a9c10 0x55ca888ac090 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.233+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1932649860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca88899a30 0x55ca88897520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.233+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1932649860 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca88896aa0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.233+0000 7fc2dd7fa640 0 client.16044 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.233+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1932649860 >> 172.21.15.76:0/1932649860 conn(0x55ca88888a50 msgr2=0x55ca8888a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.233+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1932649860 shutdown_connections 2024-03-18T14:55:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.233+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1932649860 wait complete. 2024-03-18T14:55:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.233+0000 7fc2feb6d740 20 client.16044 trim_cache size 0 max 16384 2024-03-18T14:55:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.248+0000 7fc2feb6d740 1 Processor -- start 2024-03-18T14:55:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.248+0000 7fc2feb6d740 1 -- start start 2024-03-18T14:55:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.248+0000 7fc2feb6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888e130 0x55ca888991d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.248+0000 7fc2feb6d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8888ea50 0x55ca88895c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.248+0000 7fc2feb6d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca88899a30 0x55ca888a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.248+0000 7fc2feb6d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f41ada60 con 0x55ca88899a30 2024-03-18T14:55:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.248+0000 7fc2feb6d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f41ab2d0 con 0x55ca8888ea50 2024-03-18T14:55:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.248+0000 7fc2feb6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f41a9140 con 0x55ca8888e130 2024-03-18T14:55:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.248+0000 7fc2fa43e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca88899a30 0x55ca888a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.248+0000 7fc2fa43e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca88899a30 0x55ca888a53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56372/0 (socket says 172.21.15.76:56372) 2024-03-18T14:55:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.248+0000 7fc2f943c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8888ea50 0x55ca88895c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.248+0000 7fc2fa43e640 1 -- 172.21.15.76:0/1335421138 learned_addr learned my addr 172.21.15.76:0/1335421138 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.248+0000 7fc2f9c3d640 1 --2- 172.21.15.76:0/1335421138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888e130 0x55ca888991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.249+0000 7fc2fa43e640 1 -- 172.21.15.76:0/1335421138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888e130 msgr2=0x55ca888991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.249+0000 7fc2fa43e640 1 --2- 172.21.15.76:0/1335421138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888e130 0x55ca888991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.249+0000 7fc2fa43e640 1 -- 172.21.15.76:0/1335421138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8888ea50 msgr2=0x55ca88895c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.249+0000 7fc2fa43e640 1 --2- 172.21.15.76:0/1335421138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8888ea50 0x55ca88895c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.249+0000 7fc2fa43e640 1 -- 172.21.15.76:0/1335421138 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca8888e880 con 0x55ca88899a30 2024-03-18T14:55:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.249+0000 7fc2f943c640 1 --2- 172.21.15.76:0/1335421138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8888ea50 0x55ca88895c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:55:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.249+0000 7fc2fa43e640 1 --2- 172.21.15.76:0/1335421138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca88899a30 0x55ca888a53a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec004fd0 tx=0x7fc2ec0a0200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30497f5d916c5e5b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.249+0000 7fc2df7fe640 1 -- 172.21.15.76:0/1335421138 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec01c030 con 0x55ca88899a30 2024-03-18T14:55:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.249+0000 7fc2df7fe640 1 -- 172.21.15.76:0/1335421138 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2ec09b3f0 con 0x55ca88899a30 2024-03-18T14:55:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.249+0000 7fc2df7fe640 1 -- 172.21.15.76:0/1335421138 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec097a60 con 0x55ca88899a30 2024-03-18T14:55:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.250+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1335421138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca88899a30 msgr2=0x55ca888a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.250+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1335421138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca88899a30 0x55ca888a53a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec004fd0 tx=0x7fc2ec0a0200 comp rx=0 tx=0).stop 2024-03-18T14:55:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.250+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1335421138 shutdown_connections 2024-03-18T14:55:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.250+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1335421138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca88899a30 0x55ca888a53a0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.250+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1335421138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8888ea50 0x55ca88895c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.250+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1335421138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888e130 0x55ca888991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.250+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1335421138 >> 172.21.15.76:0/1335421138 conn(0x55ca88888a50 msgr2=0x55ca88889e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.250+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1335421138 shutdown_connections 2024-03-18T14:55:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.250+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1335421138 wait complete. 2024-03-18T14:55:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.250+0000 7fc2feb6d740 1 -- start start 2024-03-18T14:55:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.250+0000 7fc2feb6d740 1 Processor -- start 2024-03-18T14:55:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2feb6d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca888a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2feb6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888ea50 0x55ca8889cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2feb6d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca88899a30 0x55ca8889d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2feb6d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f41a9790 con 0x55ca8888e130 2024-03-18T14:55:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2feb6d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f41aac00 con 0x55ca88899a30 2024-03-18T14:55:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2feb6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f41a9e60 con 0x55ca8888ea50 2024-03-18T14:55:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2feb6d740 10 client.0 _ll_register_callbacks cb 0x7fc2e4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:55:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2f9c3d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca888a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2f9c3d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca888a1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56386/0 (socket says 172.21.15.76:56386) 2024-03-18T14:55:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2f9c3d640 1 -- 172.21.15.76:0/1049996258 learned_addr learned my addr 172.21.15.76:0/1049996258 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:55:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2f943c640 1 --2- 172.21.15.76:0/1049996258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888ea50 0x55ca8889cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2fa43e640 1 --2- 172.21.15.76:0/1049996258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca88899a30 0x55ca8889d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2f9c3d640 1 -- 172.21.15.76:0/1049996258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888ea50 msgr2=0x55ca8889cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2f9c3d640 1 --2- 172.21.15.76:0/1049996258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888ea50 0x55ca8889cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2f9c3d640 1 -- 172.21.15.76:0/1049996258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca88899a30 msgr2=0x55ca8889d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:55:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2f9c3d640 1 --2- 172.21.15.76:0/1049996258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca88899a30 0x55ca8889d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2f9c3d640 1 -- 172.21.15.76:0/1049996258 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca8889da60 con 0x55ca8888e130 2024-03-18T14:55:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2f9c3d640 1 --2- 172.21.15.76:0/1049996258 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca888a1a10 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0006010 tx=0x7fc2f0005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa1c14e6aaf7b7a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:55:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:55:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2dd7fa640 1 -- 172.21.15.76:0/1049996258 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0044070 con 0x55ca8888e130 2024-03-18T14:55:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2dd7fa640 1 -- 172.21.15.76:0/1049996258 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8440+0+0 (secure 0 0 0) 0x7fc2f00329a0 con 0x55ca8888e130 2024-03-18T14:55:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1049996258 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ca887fa2d0 con 0x55ca8888e130 2024-03-18T14:55:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2dd7fa640 1 -- 172.21.15.76:0/1049996258 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc2f0062050 con 0x55ca8888e130 2024-03-18T14:55:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2feb6d740 10 client.16050 fetch_fsmap learned FSMap version 31 2024-03-18T14:55:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.251+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1049996258 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ca8890fd20 con 0x55ca8888e130 2024-03-18T14:55:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.252+0000 7fc2dd7fa640 1 -- 172.21.15.76:0/1049996258 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ca887fa2d0 con 0x55ca8888e130 2024-03-18T14:55:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.252+0000 7fc2dd7fa640 1 -- 172.21.15.76:0/1049996258 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fc2f00344d0 con 0x55ca8888e130 2024-03-18T14:55:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.252+0000 7fc2feb6d740 10 client.16050 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T14:55:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.252+0000 7fc2feb6d740 10 client.16050 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:55:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.252+0000 7fc2feb6d740 20 client.16050 populate_metadata read hostname 'smithi076' 2024-03-18T14:55:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.252+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1049996258 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ca88910550 0x55ca88912970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:55:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.252+0000 7fc2feb6d740 4 client.16050 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:55:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.252+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1049996258 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ca88778770 con 0x55ca88910550 2024-03-18T14:55:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.252+0000 7fc2f943c640 1 --2- 172.21.15.76:0/1049996258 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ca88910550 0x55ca88912970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:55:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.252+0000 7fc2f943c640 1 --2- 172.21.15.76:0/1049996258 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ca88910550 0x55ca88912970 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29202f7f5b9ca6d7 server_cookie=a2b5d68e155d6323 in_seq=0 out_seq=0 2024-03-18T14:55:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.252+0000 7fc2dd7fa640 10 client.16050 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:58.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.342+0000 7fc2dd7fa640 1 -- 172.21.15.76:0/1049996258 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ca88778770 con 0x55ca88910550 2024-03-18T14:55:58.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.342+0000 7fc2dd7fa640 10 client.16050 handle_command_reply: tid=0 2024-03-18T14:55:58.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.342+0000 7fc2feb6d740 1 client.16050 shutdown 2024-03-18T14:55:58.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.342+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1049996258 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 msgr2=0x55ca888a1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:58.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.342+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1049996258 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca888a1a10 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0006010 tx=0x7fc2f0005d80 comp rx=0 tx=0).stop 2024-03-18T14:55:58.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.342+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1049996258 shutdown_connections 2024-03-18T14:55:58.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.342+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1049996258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca8888ea50 0x55ca8889cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.342+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1049996258 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ca88910550 0x55ca88912970 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1049996258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca88899a30 0x55ca8889d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2feb6d740 1 --2- 172.21.15.76:0/1049996258 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca8888e130 0x55ca888a1a10 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2dd7fa640 0 client.16050 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:55:58.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1049996258 >> 172.21.15.76:0/1049996258 conn(0x55ca88888a50 msgr2=0x55ca88887bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:58.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1049996258 shutdown_connections 2024-03-18T14:55:58.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2feb6d740 1 -- 172.21.15.76:0/1049996258 wait complete. 2024-03-18T14:55:58.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2feb6d740 20 client.16050 trim_cache size 0 max 16384 2024-03-18T14:55:58.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:55:58.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:55:58.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:55:58.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:55:58.346 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2fbea1640 1 -- 172.21.15.76:0/4238798327 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc2e00a2a10 msgr2=0x7fc2e00a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:58.346 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2fbea1640 1 --2- 172.21.15.76:0/4238798327 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc2e00a2a10 0x7fc2e00a4ed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4009710 tx=0x7fc2e4005f90 comp rx=0 tx=0).stop 2024-03-18T14:55:58.346 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2fbea1640 1 -- 172.21.15.76:0/4238798327 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2f4105680 msgr2=0x7fc2f419b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:55:58.346 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2fbea1640 1 --2- 172.21.15.76:0/4238798327 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2f4105680 0x7fc2f419b880 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec00e740 tx=0x7fc2ec00ec00 comp rx=0 tx=0).stop 2024-03-18T14:55:58.346 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2fbea1640 1 -- 172.21.15.76:0/4238798327 shutdown_connections 2024-03-18T14:55:58.346 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2fbea1640 1 --2- 172.21.15.76:0/4238798327 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc2e00a2a10 0x7fc2e00a4ed0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2fbea1640 1 --2- 172.21.15.76:0/4238798327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2f41012f0 0x7fc2f4197500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2fbea1640 1 --2- 172.21.15.76:0/4238798327 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2f4105680 0x7fc2f419b880 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2fbea1640 1 --2- 172.21.15.76:0/4238798327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2f4100930 0x7fc2f4196fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:55:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2fbea1640 1 -- 172.21.15.76:0/4238798327 >> 172.21.15.76:0/4238798327 conn(0x7fc2f4077530 msgr2=0x7fc2f40fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:55:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2fbea1640 1 -- 172.21.15.76:0/4238798327 shutdown_connections 2024-03-18T14:55:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:55:58.343+0000 7fc2fbea1640 1 -- 172.21.15.76:0/4238798327 wait complete. 2024-03-18T14:55:58.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 224016 10.30 MB/sec warmup 119 sec latency 44.794 ms 2024-03-18T14:55:59.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:55:59 smithi174 ceph-mon[36787]: pgmap v87: 97 pgs: 97 active+clean; 72 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.9 MiB/s wr, 456 op/s 2024-03-18T14:55:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:55:59 smithi077 ceph-mon[35865]: pgmap v87: 97 pgs: 97 active+clean; 72 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.9 MiB/s wr, 456 op/s 2024-03-18T14:56:00.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:55:59 smithi076 ceph-mon[32205]: pgmap v87: 97 pgs: 97 active+clean; 72 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.9 MiB/s wr, 456 op/s 2024-03-18T14:56:00.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 227923 6.57 MB/sec execute 1 sec latency 37.727 ms 2024-03-18T14:56:01.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 229546 6.86 MB/sec execute 2 sec latency 23.907 ms 2024-03-18T14:56:01.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:01 smithi174 ceph-mon[36787]: pgmap v88: 97 pgs: 97 active+clean; 72 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 282 op/s 2024-03-18T14:56:01.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:01 smithi077 ceph-mon[35865]: pgmap v88: 97 pgs: 97 active+clean; 72 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 282 op/s 2024-03-18T14:56:02.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:01 smithi076 ceph-mon[32205]: pgmap v88: 97 pgs: 97 active+clean; 72 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 282 op/s 2024-03-18T14:56:02.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 231176 8.36 MB/sec execute 3 sec latency 24.529 ms 2024-03-18T14:56:03.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 233587 9.41 MB/sec execute 4 sec latency 26.424 ms 2024-03-18T14:56:03.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:03 smithi174 ceph-mon[36787]: pgmap v89: 97 pgs: 97 active+clean; 77 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 431 op/s 2024-03-18T14:56:03.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:03 smithi077 ceph-mon[35865]: pgmap v89: 97 pgs: 97 active+clean; 77 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 431 op/s 2024-03-18T14:56:04.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:03 smithi076 ceph-mon[32205]: pgmap v89: 97 pgs: 97 active+clean; 77 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 431 op/s 2024-03-18T14:56:04.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 234892 8.91 MB/sec execute 5 sec latency 14.701 ms 2024-03-18T14:56:05.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 237524 9.78 MB/sec execute 6 sec latency 12.106 ms 2024-03-18T14:56:05.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:05 smithi174 ceph-mon[36787]: pgmap v90: 97 pgs: 97 active+clean; 77 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 315 op/s 2024-03-18T14:56:05.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:05 smithi077 ceph-mon[35865]: pgmap v90: 97 pgs: 97 active+clean; 77 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 315 op/s 2024-03-18T14:56:06.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:05 smithi076 ceph-mon[32205]: pgmap v90: 97 pgs: 97 active+clean; 77 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 315 op/s 2024-03-18T14:56:06.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 238646 9.16 MB/sec execute 7 sec latency 15.807 ms 2024-03-18T14:56:07.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 241346 10.13 MB/sec execute 8 sec latency 15.247 ms 2024-03-18T14:56:07.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:07 smithi174 ceph-mon[36787]: pgmap v91: 97 pgs: 97 active+clean; 77 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 315 op/s 2024-03-18T14:56:07.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:07 smithi077 ceph-mon[35865]: pgmap v91: 97 pgs: 97 active+clean; 77 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 315 op/s 2024-03-18T14:56:08.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:07 smithi076 ceph-mon[32205]: pgmap v91: 97 pgs: 97 active+clean; 77 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 315 op/s 2024-03-18T14:56:08.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 242182 9.25 MB/sec execute 9 sec latency 85.299 ms 2024-03-18T14:56:09.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 245012 10.12 MB/sec execute 10 sec latency 11.324 ms 2024-03-18T14:56:09.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:09 smithi174 ceph-mon[36787]: pgmap v92: 97 pgs: 97 active+clean; 83 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.1 MiB/s wr, 420 op/s 2024-03-18T14:56:09.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:09 smithi077 ceph-mon[35865]: pgmap v92: 97 pgs: 97 active+clean; 83 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.1 MiB/s wr, 420 op/s 2024-03-18T14:56:10.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:09 smithi076 ceph-mon[32205]: pgmap v92: 97 pgs: 97 active+clean; 83 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.1 MiB/s wr, 420 op/s 2024-03-18T14:56:10.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 246239 9.69 MB/sec execute 11 sec latency 18.636 ms 2024-03-18T14:56:11.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.435+0000 7eff236bd640 1 -- 172.21.15.76:0/3913244348 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff1c1008a0 msgr2=0x7eff1c106330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:11.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.435+0000 7eff236bd640 1 --2- 172.21.15.76:0/3913244348 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff1c1008a0 0x7eff1c106330 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7eff0c009900 tx=0x7eff0c02bc30 comp rx=0 tx=0).stop 2024-03-18T14:56:11.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.435+0000 7eff236bd640 1 -- 172.21.15.76:0/3913244348 shutdown_connections 2024-03-18T14:56:11.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.435+0000 7eff236bd640 1 --2- 172.21.15.76:0/3913244348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff1c106a10 0x7eff1c10e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.435+0000 7eff236bd640 1 --2- 172.21.15.76:0/3913244348 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff1c1008a0 0x7eff1c106330 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.435+0000 7eff236bd640 1 --2- 172.21.15.76:0/3913244348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff1c0ffef0 0x7eff1c1002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.435+0000 7eff236bd640 1 -- 172.21.15.76:0/3913244348 >> 172.21.15.76:0/3913244348 conn(0x7eff1c076c00 msgr2=0x7eff1c0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:11.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.435+0000 7eff236bd640 1 -- 172.21.15.76:0/3913244348 shutdown_connections 2024-03-18T14:56:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.435+0000 7eff236bd640 1 -- 172.21.15.76:0/3913244348 wait complete. 2024-03-18T14:56:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.436+0000 7eff236bd640 1 Processor -- start 2024-03-18T14:56:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.436+0000 7eff236bd640 1 -- start start 2024-03-18T14:56:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.436+0000 7eff236bd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff1c0ffef0 0x7eff1c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.436+0000 7eff236bd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff1c106a10 0x7eff1c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.436+0000 7eff236bd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff1c19c920 0x7eff1c19cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.436+0000 7eff236bd640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c111be0 con 0x7eff1c0ffef0 2024-03-18T14:56:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.436+0000 7eff236bd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c111a60 con 0x7eff1c19c920 2024-03-18T14:56:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.436+0000 7eff236bd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c111d60 con 0x7eff1c106a10 2024-03-18T14:56:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.436+0000 7eff21459640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff1c0ffef0 0x7eff1c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.436+0000 7eff21459640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff1c0ffef0 0x7eff1c198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50462/0 (socket says 172.21.15.76:50462) 2024-03-18T14:56:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.436+0000 7eff21459640 1 -- 172.21.15.76:0/2285680451 learned_addr learned my addr 172.21.15.76:0/2285680451 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.436+0000 7eff21c5a640 1 --2- 172.21.15.76:0/2285680451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff1c19c920 0x7eff1c19cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.436+0000 7eff20c58640 1 --2- 172.21.15.76:0/2285680451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff1c106a10 0x7eff1c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.437+0000 7eff21c5a640 1 -- 172.21.15.76:0/2285680451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff1c106a10 msgr2=0x7eff1c1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.437+0000 7eff21c5a640 1 --2- 172.21.15.76:0/2285680451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff1c106a10 0x7eff1c1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.437+0000 7eff21c5a640 1 -- 172.21.15.76:0/2285680451 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff1c0ffef0 msgr2=0x7eff1c198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.437+0000 7eff21c5a640 1 --2- 172.21.15.76:0/2285680451 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff1c0ffef0 0x7eff1c198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.437+0000 7eff21c5a640 1 -- 172.21.15.76:0/2285680451 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1c19d410 con 0x7eff1c19c920 2024-03-18T14:56:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.437+0000 7eff21459640 1 --2- 172.21.15.76:0/2285680451 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff1c0ffef0 0x7eff1c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:56:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.437+0000 7eff21c5a640 1 --2- 172.21.15.76:0/2285680451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff1c19c920 0x7eff1c19cdd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7eff14009fd0 tx=0x7eff1400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.437+0000 7eff127fc640 1 -- 172.21.15.76:0/2285680451 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1400ce20 con 0x7eff1c19c920 2024-03-18T14:56:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.437+0000 7eff236bd640 1 -- 172.21.15.76:0/2285680451 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff1c19d700 con 0x7eff1c19c920 2024-03-18T14:56:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.437+0000 7eff127fc640 1 -- 172.21.15.76:0/2285680451 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff14005020 con 0x7eff1c19c920 2024-03-18T14:56:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.437+0000 7eff236bd640 1 -- 172.21.15.76:0/2285680451 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff1c1a4ec0 con 0x7eff1c19c920 2024-03-18T14:56:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.437+0000 7eff127fc640 1 -- 172.21.15.76:0/2285680451 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14010720 con 0x7eff1c19c920 2024-03-18T14:56:11.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.439+0000 7eff127fc640 1 -- 172.21.15.76:0/2285680451 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7eff14005540 con 0x7eff1c19c920 2024-03-18T14:56:11.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.440+0000 7eff26383740 1 Processor -- start 2024-03-18T14:56:11.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.440+0000 7eff26383740 1 -- start start 2024-03-18T14:56:11.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.440+0000 7eff26383740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039f130 0x55568039f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.440+0000 7eff26383740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55568039fa50 0x5556803a82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.440+0000 7eff26383740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556803a8820 0x5556803aac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.440+0000 7eff26383740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x555680303a40 con 0x55568039f130 2024-03-18T14:56:11.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.440+0000 7eff26383740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5556802ac920 con 0x55568039fa50 2024-03-18T14:56:11.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.440+0000 7eff26383740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5556802ffae0 con 0x5556803a8820 2024-03-18T14:56:11.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.440+0000 7eff127fc640 1 --2- 172.21.15.76:0/2285680451 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7eff080a2a10 0x7eff080a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.440+0000 7eff20c58640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039f130 0x55568039f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.440+0000 7eff21459640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55568039fa50 0x5556803a82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.440+0000 7eff21459640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55568039fa50 0x5556803a82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:57412/0 (socket says 172.21.15.76:57412) 2024-03-18T14:56:11.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.440+0000 7eff20c58640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039f130 0x55568039f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50470/0 (socket says 172.21.15.76:50470) 2024-03-18T14:56:11.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.440+0000 7eff21459640 1 -- 172.21.15.76:0/3067301149 learned_addr learned my addr 172.21.15.76:0/3067301149 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:11.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.440+0000 7eff20c58640 1 --2- 172.21.15.76:0/3067301149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556803a8820 0x5556803aac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.440+0000 7eff21c5a640 1 --2- 172.21.15.76:0/2285680451 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7eff080a2a10 0x7eff080a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.440+0000 7eff21459640 1 -- 172.21.15.76:0/3067301149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556803a8820 msgr2=0x5556803aac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.441+0000 7eff21459640 1 --2- 172.21.15.76:0/3067301149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556803a8820 0x5556803aac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.441+0000 7eff21459640 1 -- 172.21.15.76:0/3067301149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039f130 msgr2=0x55568039f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.441+0000 7eff127fc640 1 -- 172.21.15.76:0/2285680451 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8440+0+0 (secure 0 0 0) 0x7eff140cf4e0 con 0x7eff1c19c920 2024-03-18T14:56:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.441+0000 7eff21459640 1 --2- 172.21.15.76:0/3067301149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039f130 0x55568039f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.441+0000 7eff21459640 1 -- 172.21.15.76:0/3067301149 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556802c4460 con 0x55568039fa50 2024-03-18T14:56:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.441+0000 7eff21459640 1 --2- 172.21.15.76:0/3067301149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55568039fa50 0x5556803a82e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7eff18011630 tx=0x7eff18011af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49961a4014d3a894 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.441+0000 7eff21c5a640 1 --2- 172.21.15.76:0/2285680451 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7eff080a2a10 0x7eff080a4ed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7eff14097f90 tx=0x7eff14005940 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.441+0000 7eff077fe640 1 -- 172.21.15.76:0/3067301149 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1801e070 con 0x55568039fa50 2024-03-18T14:56:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.441+0000 7eff077fe640 1 -- 172.21.15.76:0/3067301149 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff1800fbd0 con 0x55568039fa50 2024-03-18T14:56:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.441+0000 7eff077fe640 1 -- 172.21.15.76:0/3067301149 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff18019d00 con 0x55568039fa50 2024-03-18T14:56:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.442+0000 7eff26383740 1 -- 172.21.15.76:0/3067301149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55568039fa50 msgr2=0x5556803a82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.442+0000 7eff26383740 1 --2- 172.21.15.76:0/3067301149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55568039fa50 0x5556803a82e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7eff18011630 tx=0x7eff18011af0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.442+0000 7eff26383740 1 -- 172.21.15.76:0/3067301149 shutdown_connections 2024-03-18T14:56:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.442+0000 7eff26383740 1 --2- 172.21.15.76:0/3067301149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556803a8820 0x5556803aac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.442+0000 7eff26383740 1 --2- 172.21.15.76:0/3067301149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55568039fa50 0x5556803a82e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.442+0000 7eff26383740 1 --2- 172.21.15.76:0/3067301149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039f130 0x55568039f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.442+0000 7eff26383740 1 -- 172.21.15.76:0/3067301149 >> 172.21.15.76:0/3067301149 conn(0x555680399a50 msgr2=0x555680399e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.442+0000 7eff26383740 1 -- 172.21.15.76:0/3067301149 shutdown_connections 2024-03-18T14:56:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.442+0000 7eff26383740 1 -- 172.21.15.76:0/3067301149 wait complete. 2024-03-18T14:56:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.442+0000 7eff26383740 1 -- start start 2024-03-18T14:56:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.442+0000 7eff26383740 1 Processor -- start 2024-03-18T14:56:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.442+0000 7eff26383740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568039f130 0x5556803a5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff26383740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff26383740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556803a8820 0x5556803a6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff26383740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5556803ab450 con 0x55568039fa50 2024-03-18T14:56:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff26383740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5556802bfd70 con 0x5556803a8820 2024-03-18T14:56:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff26383740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5556803ab5d0 con 0x55568039f130 2024-03-18T14:56:11.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff26383740 10 client.0 _ll_register_callbacks cb 0x555680017a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:56:11.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff21459640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff21459640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50476/0 (socket says 172.21.15.76:50476) 2024-03-18T14:56:11.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff20c58640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568039f130 0x5556803a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff21459640 1 -- 172.21.15.76:0/817903754 learned_addr learned my addr 172.21.15.76:0/817903754 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:11.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff20c58640 1 --2- 172.21.15.76:0/817903754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556803a8820 0x5556803a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff21459640 1 -- 172.21.15.76:0/817903754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568039f130 msgr2=0x5556803a5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:11.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff21459640 1 --2- 172.21.15.76:0/817903754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568039f130 0x5556803a5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff21459640 1 -- 172.21.15.76:0/817903754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556803a8820 msgr2=0x5556803a6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:11.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff21459640 1 --2- 172.21.15.76:0/817903754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556803a8820 0x5556803a6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff21459640 1 -- 172.21.15.76:0/817903754 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555680018210 con 0x55568039fa50 2024-03-18T14:56:11.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff20c58640 1 --2- 172.21.15.76:0/817903754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556803a8820 0x5556803a6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:56:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff21459640 1 --2- 172.21.15.76:0/817903754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a5dc0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7eff1801c040 tx=0x7eff180196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5497e1902c8055f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:56:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff04ff9640 1 -- 172.21.15.76:0/817903754 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff18026020 con 0x55568039fa50 2024-03-18T14:56:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff04ff9640 1 -- 172.21.15.76:0/817903754 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8440+0+0 (secure 0 0 0) 0x7eff1800fd80 con 0x55568039fa50 2024-03-18T14:56:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.443+0000 7eff26383740 1 -- 172.21.15.76:0/817903754 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff1800fbd0 con 0x55568039fa50 2024-03-18T14:56:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.444+0000 7eff04ff9640 1 -- 172.21.15.76:0/817903754 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7eff18031050 con 0x55568039fa50 2024-03-18T14:56:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.444+0000 7eff26383740 10 client.16062 fetch_fsmap learned FSMap version 31 2024-03-18T14:56:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.444+0000 7eff26383740 1 -- 172.21.15.76:0/817903754 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5556800e4620 con 0x55568039fa50 2024-03-18T14:56:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.444+0000 7eff04ff9640 1 -- 172.21.15.76:0/817903754 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff1800fbd0 con 0x55568039fa50 2024-03-18T14:56:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.444+0000 7eff04ff9640 1 -- 172.21.15.76:0/817903754 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7eff18021050 con 0x55568039fa50 2024-03-18T14:56:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.444+0000 7eff26383740 10 client.16062 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T14:56:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.444+0000 7eff26383740 10 client.16062 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:56:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.444+0000 7eff26383740 20 client.16062 populate_metadata read hostname 'smithi076' 2024-03-18T14:56:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.444+0000 7eff26383740 1 --2- 172.21.15.76:0/817903754 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5556803b8900 0x5556803bacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.444+0000 7eff26383740 4 client.16062 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:56:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.444+0000 7eff26383740 1 -- 172.21.15.76:0/817903754 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555680310f80 con 0x5556803b8900 2024-03-18T14:56:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.444+0000 7eff20c58640 1 --2- 172.21.15.76:0/817903754 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5556803b8900 0x5556803bacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.445+0000 7eff20c58640 1 --2- 172.21.15.76:0/817903754 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5556803b8900 0x5556803bacf0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9be5e733b5f85ab server_cookie=d5b2102fe3c6338 in_seq=0 out_seq=0 2024-03-18T14:56:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.445+0000 7eff04ff9640 10 client.16062 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:56:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.446+0000 7eff04ff9640 1 -- 172.21.15.76:0/817903754 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555680310f80 con 0x5556803b8900 2024-03-18T14:56:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.446+0000 7eff04ff9640 10 client.16062 handle_command_reply: tid=0 2024-03-18T14:56:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.446+0000 7eff26383740 1 client.16062 shutdown 2024-03-18T14:56:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.446+0000 7eff26383740 1 -- 172.21.15.76:0/817903754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 msgr2=0x5556803a5dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.446+0000 7eff26383740 1 --2- 172.21.15.76:0/817903754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a5dc0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7eff1801c040 tx=0x7eff180196a0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.446+0000 7eff26383740 1 -- 172.21.15.76:0/817903754 shutdown_connections 2024-03-18T14:56:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.446+0000 7eff26383740 1 --2- 172.21.15.76:0/817903754 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5556803b8900 0x5556803bacf0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.446+0000 7eff26383740 1 --2- 172.21.15.76:0/817903754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556803a8820 0x5556803a6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.446+0000 7eff04ff9640 0 client.16062 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:56:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.446+0000 7eff26383740 1 --2- 172.21.15.76:0/817903754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a5dc0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.446+0000 7eff26383740 1 --2- 172.21.15.76:0/817903754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568039f130 0x5556803a5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.446+0000 7eff26383740 1 -- 172.21.15.76:0/817903754 >> 172.21.15.76:0/817903754 conn(0x555680399a50 msgr2=0x55568039b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.446+0000 7eff26383740 1 -- 172.21.15.76:0/817903754 shutdown_connections 2024-03-18T14:56:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.446+0000 7eff26383740 1 -- 172.21.15.76:0/817903754 wait complete. 2024-03-18T14:56:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.446+0000 7eff26383740 20 client.16062 trim_cache size 0 max 16384 2024-03-18T14:56:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.461+0000 7eff26383740 1 Processor -- start 2024-03-18T14:56:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.461+0000 7eff26383740 1 -- start start 2024-03-18T14:56:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.461+0000 7eff26383740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568039f130 0x5556803b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.461+0000 7eff26383740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.461+0000 7eff26383740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556803a8820 0x5556803c04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.461+0000 7eff26383740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1afd10 con 0x55568039fa50 2024-03-18T14:56:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.461+0000 7eff26383740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1ad580 con 0x5556803a8820 2024-03-18T14:56:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.461+0000 7eff26383740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1ab3d0 con 0x55568039f130 2024-03-18T14:56:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.462+0000 7eff21459640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.462+0000 7eff21459640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50488/0 (socket says 172.21.15.76:50488) 2024-03-18T14:56:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.462+0000 7eff21459640 1 -- 172.21.15.76:0/3742694662 learned_addr learned my addr 172.21.15.76:0/3742694662 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.462+0000 7eff20c58640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568039f130 0x5556803b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.462+0000 7eff20c58640 1 --2- 172.21.15.76:0/3742694662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556803a8820 0x5556803c04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.462+0000 7eff21459640 1 -- 172.21.15.76:0/3742694662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568039f130 msgr2=0x5556803b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.462+0000 7eff21459640 1 --2- 172.21.15.76:0/3742694662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568039f130 0x5556803b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.462+0000 7eff21459640 1 -- 172.21.15.76:0/3742694662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556803a8820 msgr2=0x5556803c04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.462+0000 7eff21459640 1 --2- 172.21.15.76:0/3742694662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556803a8820 0x5556803c04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.462+0000 7eff21459640 1 -- 172.21.15.76:0/3742694662 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555680395310 con 0x55568039fa50 2024-03-18T14:56:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.462+0000 7eff21459640 1 --2- 172.21.15.76:0/3742694662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a76b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7eff1800a010 tx=0x7eff18004b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2f0189516198f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.462+0000 7eff06ffd640 1 -- 172.21.15.76:0/3742694662 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff18015020 con 0x55568039fa50 2024-03-18T14:56:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.462+0000 7eff06ffd640 1 -- 172.21.15.76:0/3742694662 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff18005d30 con 0x55568039fa50 2024-03-18T14:56:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.462+0000 7eff06ffd640 1 -- 172.21.15.76:0/3742694662 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff18026040 con 0x55568039fa50 2024-03-18T14:56:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.463+0000 7eff26383740 1 -- 172.21.15.76:0/3742694662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 msgr2=0x5556803a76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.463+0000 7eff26383740 1 --2- 172.21.15.76:0/3742694662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a76b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7eff1800a010 tx=0x7eff18004b80 comp rx=0 tx=0).stop 2024-03-18T14:56:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.463+0000 7eff26383740 1 -- 172.21.15.76:0/3742694662 shutdown_connections 2024-03-18T14:56:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.463+0000 7eff26383740 1 --2- 172.21.15.76:0/3742694662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556803a8820 0x5556803c04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.463+0000 7eff26383740 1 --2- 172.21.15.76:0/3742694662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a76b0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.463+0000 7eff26383740 1 --2- 172.21.15.76:0/3742694662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568039f130 0x5556803b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.463+0000 7eff26383740 1 -- 172.21.15.76:0/3742694662 >> 172.21.15.76:0/3742694662 conn(0x555680399a50 msgr2=0x5556803a6fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.463+0000 7eff26383740 1 -- 172.21.15.76:0/3742694662 shutdown_connections 2024-03-18T14:56:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.463+0000 7eff26383740 1 -- 172.21.15.76:0/3742694662 wait complete. 2024-03-18T14:56:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.463+0000 7eff26383740 1 -- start start 2024-03-18T14:56:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.463+0000 7eff26383740 1 Processor -- start 2024-03-18T14:56:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff26383740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568039f130 0x5556803b0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff26383740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff26383740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556803a8820 0x5556803a2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff26383740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1aba40 con 0x55568039fa50 2024-03-18T14:56:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff26383740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1aceb0 con 0x5556803a8820 2024-03-18T14:56:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff26383740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1ac110 con 0x55568039f130 2024-03-18T14:56:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff26383740 10 client.0 _ll_register_callbacks cb 0x7eff1801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:56:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff21459640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff21459640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50494/0 (socket says 172.21.15.76:50494) 2024-03-18T14:56:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff21459640 1 -- 172.21.15.76:0/3487644768 learned_addr learned my addr 172.21.15.76:0/3487644768 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff20c58640 1 --2- 172.21.15.76:0/3487644768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556803a8820 0x5556803a2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff20c58640 1 --2- 172.21.15.76:0/3487644768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568039f130 0x5556803b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff21459640 1 -- 172.21.15.76:0/3487644768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568039f130 msgr2=0x5556803b0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff21459640 1 --2- 172.21.15.76:0/3487644768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568039f130 0x5556803b0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff21459640 1 -- 172.21.15.76:0/3487644768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556803a8820 msgr2=0x5556803a2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff21459640 1 --2- 172.21.15.76:0/3487644768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556803a8820 0x5556803a2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff21459640 1 -- 172.21.15.76:0/3487644768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556803a2960 con 0x55568039fa50 2024-03-18T14:56:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff21459640 1 --2- 172.21.15.76:0/3487644768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a1ee0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7eff18006010 tx=0x7eff18004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba530c805baea4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.464+0000 7eff04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:56:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.465+0000 7eff04ff9640 1 -- 172.21.15.76:0/3487644768 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1801e040 con 0x55568039fa50 2024-03-18T14:56:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.465+0000 7eff04ff9640 1 -- 172.21.15.76:0/3487644768 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8440+0+0 (secure 0 0 0) 0x7eff1800fc70 con 0x55568039fa50 2024-03-18T14:56:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.465+0000 7eff26383740 1 -- 172.21.15.76:0/3487644768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff18005d30 con 0x55568039fa50 2024-03-18T14:56:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.465+0000 7eff04ff9640 1 -- 172.21.15.76:0/3487644768 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7eff18047050 con 0x55568039fa50 2024-03-18T14:56:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.465+0000 7eff26383740 10 client.16068 fetch_fsmap learned FSMap version 31 2024-03-18T14:56:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.465+0000 7eff26383740 1 -- 172.21.15.76:0/3487644768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55568041e7f0 con 0x55568039fa50 2024-03-18T14:56:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.465+0000 7eff04ff9640 1 -- 172.21.15.76:0/3487644768 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff18005d30 con 0x55568039fa50 2024-03-18T14:56:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.465+0000 7eff04ff9640 1 -- 172.21.15.76:0/3487644768 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7eff18021030 con 0x55568039fa50 2024-03-18T14:56:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.465+0000 7eff26383740 10 client.16068 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T14:56:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.465+0000 7eff26383740 10 client.16068 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:56:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.465+0000 7eff26383740 20 client.16068 populate_metadata read hostname 'smithi076' 2024-03-18T14:56:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.465+0000 7eff26383740 1 --2- 172.21.15.76:0/3487644768 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55568041eee0 0x555680421300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.465+0000 7eff26383740 4 client.16068 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:56:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.465+0000 7eff26383740 1 -- 172.21.15.76:0/3487644768 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55568030b2d0 con 0x55568041eee0 2024-03-18T14:56:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.465+0000 7eff20c58640 1 --2- 172.21.15.76:0/3487644768 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55568041eee0 0x555680421300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.466+0000 7eff04ff9640 10 client.16068 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:56:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.466+0000 7eff20c58640 1 --2- 172.21.15.76:0/3487644768 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55568041eee0 0x555680421300 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f49a28779761c7a server_cookie=5590c3e1d2324 in_seq=0 out_seq=0 2024-03-18T14:56:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.528+0000 7eff04ff9640 1 -- 172.21.15.76:0/3487644768 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55568030b2d0 con 0x55568041eee0 2024-03-18T14:56:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.528+0000 7eff04ff9640 10 client.16068 handle_command_reply: tid=0 2024-03-18T14:56:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.528+0000 7eff26383740 1 client.16068 shutdown 2024-03-18T14:56:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.529+0000 7eff26383740 1 -- 172.21.15.76:0/3487644768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 msgr2=0x5556803a1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.529+0000 7eff26383740 1 --2- 172.21.15.76:0/3487644768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a1ee0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7eff18006010 tx=0x7eff18004680 comp rx=0 tx=0).stop 2024-03-18T14:56:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.529+0000 7eff26383740 1 -- 172.21.15.76:0/3487644768 shutdown_connections 2024-03-18T14:56:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.529+0000 7eff26383740 1 --2- 172.21.15.76:0/3487644768 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55568041eee0 0x555680421300 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.529+0000 7eff26383740 1 --2- 172.21.15.76:0/3487644768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5556803a8820 0x5556803a2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.529+0000 7eff26383740 1 --2- 172.21.15.76:0/3487644768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55568039fa50 0x5556803a1ee0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.529+0000 7eff26383740 1 --2- 172.21.15.76:0/3487644768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568039f130 0x5556803b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.529+0000 7eff04ff9640 0 client.16068 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:56:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.529+0000 7eff26383740 1 -- 172.21.15.76:0/3487644768 >> 172.21.15.76:0/3487644768 conn(0x555680399a50 msgr2=0x555680398bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.529+0000 7eff26383740 1 -- 172.21.15.76:0/3487644768 shutdown_connections 2024-03-18T14:56:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.529+0000 7eff26383740 1 -- 172.21.15.76:0/3487644768 wait complete. 2024-03-18T14:56:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.529+0000 7eff26383740 20 client.16068 trim_cache size 0 max 16384 2024-03-18T14:56:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:56:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:56:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:56:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:56:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.530+0000 7eff236bd640 1 -- 172.21.15.76:0/2285680451 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7eff080a2a10 msgr2=0x7eff080a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.530+0000 7eff236bd640 1 --2- 172.21.15.76:0/2285680451 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7eff080a2a10 0x7eff080a4ed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7eff14097f90 tx=0x7eff14005940 comp rx=0 tx=0).stop 2024-03-18T14:56:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.530+0000 7eff236bd640 1 -- 172.21.15.76:0/2285680451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff1c19c920 msgr2=0x7eff1c19cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.530+0000 7eff236bd640 1 --2- 172.21.15.76:0/2285680451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff1c19c920 0x7eff1c19cdd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7eff14009fd0 tx=0x7eff1400ef10 comp rx=0 tx=0).stop 2024-03-18T14:56:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.530+0000 7eff236bd640 1 -- 172.21.15.76:0/2285680451 shutdown_connections 2024-03-18T14:56:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.530+0000 7eff236bd640 1 --2- 172.21.15.76:0/2285680451 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7eff080a2a10 0x7eff080a4ed0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.530+0000 7eff236bd640 1 --2- 172.21.15.76:0/2285680451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff1c106a10 0x7eff1c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.530+0000 7eff236bd640 1 --2- 172.21.15.76:0/2285680451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff1c19c920 0x7eff1c19cdd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.530+0000 7eff236bd640 1 --2- 172.21.15.76:0/2285680451 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff1c0ffef0 0x7eff1c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.530+0000 7eff236bd640 1 -- 172.21.15.76:0/2285680451 >> 172.21.15.76:0/2285680451 conn(0x7eff1c076c00 msgr2=0x7eff1c1051f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.530+0000 7eff236bd640 1 -- 172.21.15.76:0/2285680451 shutdown_connections 2024-03-18T14:56:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:11.530+0000 7eff236bd640 1 -- 172.21.15.76:0/2285680451 wait complete. 2024-03-18T14:56:11.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 248609 10.03 MB/sec execute 12 sec latency 37.914 ms 2024-03-18T14:56:11.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:11 smithi174 ceph-mon[36787]: pgmap v93: 97 pgs: 97 active+clean; 83 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 254 op/s 2024-03-18T14:56:11.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:11 smithi077 ceph-mon[35865]: pgmap v93: 97 pgs: 97 active+clean; 83 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 254 op/s 2024-03-18T14:56:12.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:11 smithi076 ceph-mon[32205]: pgmap v93: 97 pgs: 97 active+clean; 83 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 254 op/s 2024-03-18T14:56:12.477 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:56:12 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setuser ceph since I am not root 2024-03-18T14:56:12.477 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:56:12 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setgroup ceph since I am not root 2024-03-18T14:56:12.477 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:56:12 smithi077 ceph-mds[65404]: -- 172.21.15.77:0/3605153221 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560ce2ed7a40 con 0x560ce2eaac00 2024-03-18T14:56:12.477 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:56:12 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: starting mds.e at 2024-03-18T14:56:12.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 249951 9.68 MB/sec execute 13 sec latency 21.147 ms 2024-03-18T14:56:12.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:12 smithi174 ceph-mon[36787]: daemon mds.e restarted 2024-03-18T14:56:12.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:12 smithi077 ceph-mon[35865]: daemon mds.e restarted 2024-03-18T14:56:13.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:12 smithi076 ceph-mon[32205]: daemon mds.e restarted 2024-03-18T14:56:13.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 252203 9.96 MB/sec execute 14 sec latency 23.020 ms 2024-03-18T14:56:13.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:13 smithi174 ceph-mon[36787]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T14:56:13.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:13 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.77:6834/4153703893,v1:172.21.15.77:6835/4153703893] up:boot 2024-03-18T14:56:13.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:13 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T14:56:13.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:13 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:56:13.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:13 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:56:13.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:13 smithi174 ceph-mon[36787]: pgmap v95: 97 pgs: 97 active+clean; 78 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 321 op/s 2024-03-18T14:56:13.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:13 smithi077 ceph-mon[35865]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T14:56:13.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:13 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.77:6834/4153703893,v1:172.21.15.77:6835/4153703893] up:boot 2024-03-18T14:56:13.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:13 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T14:56:13.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:13 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:56:13.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:13 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:56:13.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:13 smithi077 ceph-mon[35865]: pgmap v95: 97 pgs: 97 active+clean; 78 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 321 op/s 2024-03-18T14:56:14.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:13 smithi076 ceph-mon[32205]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T14:56:14.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:13 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.77:6834/4153703893,v1:172.21.15.77:6835/4153703893] up:boot 2024-03-18T14:56:14.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:13 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T14:56:14.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:13 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:56:14.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:13 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:56:14.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:13 smithi076 ceph-mon[32205]: pgmap v95: 97 pgs: 97 active+clean; 78 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 321 op/s 2024-03-18T14:56:14.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 254277 9.79 MB/sec execute 15 sec latency 15.060 ms 2024-03-18T14:56:15.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 255999 9.93 MB/sec execute 16 sec latency 16.017 ms 2024-03-18T14:56:15.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:15 smithi174 ceph-mon[36787]: pgmap v96: 97 pgs: 97 active+clean; 78 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 321 op/s 2024-03-18T14:56:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:15 smithi077 ceph-mon[35865]: pgmap v96: 97 pgs: 97 active+clean; 78 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 321 op/s 2024-03-18T14:56:16.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:15 smithi076 ceph-mon[32205]: pgmap v96: 97 pgs: 97 active+clean; 78 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 321 op/s 2024-03-18T14:56:16.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 258931 10.37 MB/sec execute 17 sec latency 13.489 ms 2024-03-18T14:56:17.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 260190 10.01 MB/sec execute 18 sec latency 15.153 ms 2024-03-18T14:56:17.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:17 smithi076 ceph-mon[32205]: pgmap v97: 97 pgs: 97 active+clean; 78 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 321 op/s 2024-03-18T14:56:17.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:17 smithi174 ceph-mon[36787]: pgmap v97: 97 pgs: 97 active+clean; 78 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 321 op/s 2024-03-18T14:56:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:17 smithi077 ceph-mon[35865]: pgmap v97: 97 pgs: 97 active+clean; 78 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 321 op/s 2024-03-18T14:56:18.737 INFO:tasks.workunit.client.0.smithi076.stdout: 1 262652 10.31 MB/sec execute 19 sec latency 36.612 ms 2024-03-18T14:56:19.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 264020 10.11 MB/sec execute 20 sec latency 13.517 ms 2024-03-18T14:56:19.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:19 smithi174 ceph-mon[36787]: pgmap v98: 97 pgs: 97 active+clean; 83 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 345 op/s 2024-03-18T14:56:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:19 smithi077 ceph-mon[35865]: pgmap v98: 97 pgs: 97 active+clean; 83 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 345 op/s 2024-03-18T14:56:20.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:19 smithi076 ceph-mon[32205]: pgmap v98: 97 pgs: 97 active+clean; 83 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 345 op/s 2024-03-18T14:56:20.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 266384 10.28 MB/sec execute 21 sec latency 21.322 ms 2024-03-18T14:56:21.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 268420 10.15 MB/sec execute 22 sec latency 18.150 ms 2024-03-18T14:56:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:21 smithi174 ceph-mon[36787]: pgmap v99: 97 pgs: 97 active+clean; 83 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 345 op/s 2024-03-18T14:56:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:21 smithi077 ceph-mon[35865]: pgmap v99: 97 pgs: 97 active+clean; 83 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 345 op/s 2024-03-18T14:56:22.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:21 smithi076 ceph-mon[32205]: pgmap v99: 97 pgs: 97 active+clean; 83 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 345 op/s 2024-03-18T14:56:22.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 270069 10.23 MB/sec execute 23 sec latency 12.184 ms 2024-03-18T14:56:23.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 272093 10.13 MB/sec execute 24 sec latency 33.953 ms 2024-03-18T14:56:23.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:23 smithi174 ceph-mon[36787]: pgmap v100: 97 pgs: 97 active+clean; 90 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 436 op/s 2024-03-18T14:56:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:23 smithi077 ceph-mon[35865]: pgmap v100: 97 pgs: 97 active+clean; 90 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 436 op/s 2024-03-18T14:56:24.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:23 smithi076 ceph-mon[32205]: pgmap v100: 97 pgs: 97 active+clean; 90 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 436 op/s 2024-03-18T14:56:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.620+0000 7f028dca1640 1 -- 172.21.15.76:0/1035502095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02881077d0 msgr2=0x7f0288107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.620+0000 7f028dca1640 1 --2- 172.21.15.76:0/1035502095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02881077d0 0x7f0288107bb0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0278009900 tx=0x7f027802bc30 comp rx=0 tx=0).stop 2024-03-18T14:56:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.620+0000 7f028dca1640 1 -- 172.21.15.76:0/1035502095 shutdown_connections 2024-03-18T14:56:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.620+0000 7f028dca1640 1 --2- 172.21.15.76:0/1035502095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f028806aa00 0x7f028810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.620+0000 7f028dca1640 1 --2- 172.21.15.76:0/1035502095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028806a060 0x7f028806a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.620+0000 7f028dca1640 1 --2- 172.21.15.76:0/1035502095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02881077d0 0x7f0288107bb0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.620+0000 7f028dca1640 1 -- 172.21.15.76:0/1035502095 >> 172.21.15.76:0/1035502095 conn(0x7f0288077530 msgr2=0x7f0288077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.620+0000 7f028dca1640 1 -- 172.21.15.76:0/1035502095 shutdown_connections 2024-03-18T14:56:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.620+0000 7f028dca1640 1 -- 172.21.15.76:0/1035502095 wait complete. 2024-03-18T14:56:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.621+0000 7f028dca1640 1 Processor -- start 2024-03-18T14:56:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.621+0000 7f028dca1640 1 -- start start 2024-03-18T14:56:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.621+0000 7f028dca1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028806a060 0x7f0288197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.621+0000 7f028dca1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f028806aa00 0x7f0288198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.621+0000 7f028dca1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02881077d0 0x7f028819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.621+0000 7f028dca1640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0288111bb0 con 0x7f02881077d0 2024-03-18T14:56:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.621+0000 7f028dca1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0288111a30 con 0x7f028806a060 2024-03-18T14:56:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.621+0000 7f028dca1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0288111d30 con 0x7f028806aa00 2024-03-18T14:56:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.621+0000 7f0287fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02881077d0 0x7f028819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.621+0000 7f02877fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028806a060 0x7f0288197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.621+0000 7f0286ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f028806aa00 0x7f0288198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.621+0000 7f02877fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028806a060 0x7f0288197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:45804/0 (socket says 172.21.15.76:45804) 2024-03-18T14:56:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.621+0000 7f0287fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02881077d0 0x7f028819c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44080/0 (socket says 172.21.15.76:44080) 2024-03-18T14:56:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.621+0000 7f02877fe640 1 -- 172.21.15.76:0/1224180977 learned_addr learned my addr 172.21.15.76:0/1224180977 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.622+0000 7f0287fff640 1 -- 172.21.15.76:0/1224180977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f028806aa00 msgr2=0x7f0288198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.622+0000 7f0287fff640 1 --2- 172.21.15.76:0/1224180977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f028806aa00 0x7f0288198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.622+0000 7f0287fff640 1 -- 172.21.15.76:0/1224180977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028806a060 msgr2=0x7f0288197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.622+0000 7f0287fff640 1 --2- 172.21.15.76:0/1224180977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028806a060 0x7f0288197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.622+0000 7f0287fff640 1 -- 172.21.15.76:0/1224180977 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f028819cfc0 con 0x7f02881077d0 2024-03-18T14:56:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.622+0000 7f02877fe640 1 --2- 172.21.15.76:0/1224180977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028806a060 0x7f0288197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:56:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.622+0000 7f0287fff640 1 --2- 172.21.15.76:0/1224180977 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02881077d0 0x7f028819c890 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0280009fd0 tx=0x7f028000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.622+0000 7f0284ff9640 1 -- 172.21.15.76:0/1224180977 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028000ce20 con 0x7f02881077d0 2024-03-18T14:56:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.622+0000 7f028dca1640 1 -- 172.21.15.76:0/1224180977 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f028819d2b0 con 0x7f02881077d0 2024-03-18T14:56:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.622+0000 7f0284ff9640 1 -- 172.21.15.76:0/1224180977 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0280005020 con 0x7f02881077d0 2024-03-18T14:56:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.622+0000 7f0284ff9640 1 -- 172.21.15.76:0/1224180977 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0280010720 con 0x7f02881077d0 2024-03-18T14:56:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.622+0000 7f028dca1640 1 -- 172.21.15.76:0/1224180977 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02881a4b40 con 0x7f02881077d0 2024-03-18T14:56:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.624+0000 7f0284ff9640 1 -- 172.21.15.76:0/1224180977 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f0280005540 con 0x7f02881077d0 2024-03-18T14:56:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.624+0000 7f029096e740 1 Processor -- start 2024-03-18T14:56:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.624+0000 7f029096e740 1 -- start start 2024-03-18T14:56:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f029096e740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7ed130 0x564f2c7ed510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f029096e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7eda50 0x564f2c7f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f029096e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7f6820 0x564f2c7f8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f029096e740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564f2c751a40 con 0x564f2c7ed130 2024-03-18T14:56:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f029096e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564f2c6fa920 con 0x564f2c7f6820 2024-03-18T14:56:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f029096e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564f2c74dae0 con 0x564f2c7eda50 2024-03-18T14:56:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f0286ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7ed130 0x564f2c7ed510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f0286ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7ed130 0x564f2c7ed510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44094/0 (socket says 172.21.15.76:44094) 2024-03-18T14:56:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f0286ffd640 1 -- 172.21.15.76:0/3569855053 learned_addr learned my addr 172.21.15.76:0/3569855053 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f02877fe640 1 --2- 172.21.15.76:0/3569855053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7eda50 0x564f2c7f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f0286ffd640 1 --2- 172.21.15.76:0/3569855053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7f6820 0x564f2c7f8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f0284ff9640 1 --2- 172.21.15.76:0/1224180977 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f02700a2a10 0x7f02700a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f0286ffd640 1 -- 172.21.15.76:0/3569855053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7eda50 msgr2=0x564f2c7f62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f0286ffd640 1 --2- 172.21.15.76:0/3569855053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7eda50 0x564f2c7f62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f0286ffd640 1 -- 172.21.15.76:0/3569855053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7f6820 msgr2=0x564f2c7f8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f0286ffd640 1 --2- 172.21.15.76:0/3569855053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7f6820 0x564f2c7f8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f0286ffd640 1 -- 172.21.15.76:0/3569855053 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f2c712460 con 0x564f2c7ed130 2024-03-18T14:56:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f0287fff640 1 --2- 172.21.15.76:0/1224180977 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f02700a2a10 0x7f02700a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f0284ff9640 1 -- 172.21.15.76:0/1224180977 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7f02800cf480 con 0x7f02881077d0 2024-03-18T14:56:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f0286ffd640 1 --2- 172.21.15.76:0/3569855053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7ed130 0x564f2c7ed510 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f027c005bc0 tx=0x7f027c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1e14035aefab39d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.625+0000 7f026dffb640 1 -- 172.21.15.76:0/3569855053 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c013e50 con 0x564f2c7ed130 2024-03-18T14:56:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.626+0000 7f026dffb640 1 -- 172.21.15.76:0/3569855053 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f027c013170 con 0x564f2c7ed130 2024-03-18T14:56:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.626+0000 7f026dffb640 1 -- 172.21.15.76:0/3569855053 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c004ce0 con 0x564f2c7ed130 2024-03-18T14:56:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.626+0000 7f029096e740 1 -- 172.21.15.76:0/3569855053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7ed130 msgr2=0x564f2c7ed510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.626+0000 7f029096e740 1 --2- 172.21.15.76:0/3569855053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7ed130 0x564f2c7ed510 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f027c005bc0 tx=0x7f027c005f90 comp rx=0 tx=0).stop 2024-03-18T14:56:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.626+0000 7f0287fff640 1 --2- 172.21.15.76:0/1224180977 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f02700a2a10 0x7f02700a4ed0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f02800cd220 tx=0x7f0280066d40 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.626+0000 7f029096e740 1 -- 172.21.15.76:0/3569855053 shutdown_connections 2024-03-18T14:56:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.626+0000 7f029096e740 1 --2- 172.21.15.76:0/3569855053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7eda50 0x564f2c7f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.626+0000 7f029096e740 1 --2- 172.21.15.76:0/3569855053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7f6820 0x564f2c7f8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.626+0000 7f029096e740 1 --2- 172.21.15.76:0/3569855053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7ed130 0x564f2c7ed510 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.626+0000 7f029096e740 1 -- 172.21.15.76:0/3569855053 >> 172.21.15.76:0/3569855053 conn(0x564f2c7e7a50 msgr2=0x564f2c7e7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.626+0000 7f029096e740 1 -- 172.21.15.76:0/3569855053 shutdown_connections 2024-03-18T14:56:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.626+0000 7f029096e740 1 -- 172.21.15.76:0/3569855053 wait complete. 2024-03-18T14:56:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.626+0000 7f029096e740 1 -- start start 2024-03-18T14:56:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f029096e740 1 Processor -- start 2024-03-18T14:56:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f029096e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7ed130 0x564f2c7f3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f029096e740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7eda50 0x564f2c7f3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f029096e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7f6820 0x564f2c7f4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f029096e740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564f2c7f93f0 con 0x564f2c7eda50 2024-03-18T14:56:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f029096e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564f2c70dd70 con 0x564f2c7f6820 2024-03-18T14:56:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f029096e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564f2c7f9570 con 0x564f2c7ed130 2024-03-18T14:56:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f029096e740 10 client.0 _ll_register_callbacks cb 0x564f2c465a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:56:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f02877fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7eda50 0x564f2c7f3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f0286ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7f6820 0x564f2c7f4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f02877fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7eda50 0x564f2c7f3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44108/0 (socket says 172.21.15.76:44108) 2024-03-18T14:56:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f02877fe640 1 -- 172.21.15.76:0/381291282 learned_addr learned my addr 172.21.15.76:0/381291282 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f0286ffd640 1 --2- 172.21.15.76:0/381291282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7f6820 0x564f2c7f4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:45826/0 (socket says 172.21.15.76:45826) 2024-03-18T14:56:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f0286ffd640 1 --2- 172.21.15.76:0/381291282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7ed130 0x564f2c7f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f02877fe640 1 -- 172.21.15.76:0/381291282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7ed130 msgr2=0x564f2c7f3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f02877fe640 1 --2- 172.21.15.76:0/381291282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7ed130 0x564f2c7f3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f02877fe640 1 -- 172.21.15.76:0/381291282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7f6820 msgr2=0x564f2c7f4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f02877fe640 1 --2- 172.21.15.76:0/381291282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7f6820 0x564f2c7f4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f02877fe640 1 -- 172.21.15.76:0/381291282 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f2c466210 con 0x564f2c7eda50 2024-03-18T14:56:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.627+0000 7f0286ffd640 1 --2- 172.21.15.76:0/381291282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7f6820 0x564f2c7f4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:56:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.628+0000 7f02877fe640 1 --2- 172.21.15.76:0/381291282 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7eda50 0x564f2c7f3cd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0278036040 tx=0x7f0278004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb647efc980af98b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.628+0000 7f0266ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:56:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.628+0000 7f0266ffd640 1 -- 172.21.15.76:0/381291282 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027802cab0 con 0x564f2c7eda50 2024-03-18T14:56:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.628+0000 7f0266ffd640 1 -- 172.21.15.76:0/381291282 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7f027802cc50 con 0x564f2c7eda50 2024-03-18T14:56:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.628+0000 7f029096e740 1 -- 172.21.15.76:0/381291282 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f027c013170 con 0x564f2c7eda50 2024-03-18T14:56:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.628+0000 7f0266ffd640 1 -- 172.21.15.76:0/381291282 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f027802c6a0 con 0x564f2c7eda50 2024-03-18T14:56:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.628+0000 7f029096e740 10 client.16086 fetch_fsmap learned FSMap version 33 2024-03-18T14:56:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.628+0000 7f029096e740 1 -- 172.21.15.76:0/381291282 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564f2c532620 con 0x564f2c7eda50 2024-03-18T14:56:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.628+0000 7f0266ffd640 1 -- 172.21.15.76:0/381291282 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f027c013170 con 0x564f2c7eda50 2024-03-18T14:56:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.628+0000 7f0266ffd640 1 -- 172.21.15.76:0/381291282 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f0278046070 con 0x564f2c7eda50 2024-03-18T14:56:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.628+0000 7f029096e740 10 client.16086 fetch_fsmap finished waiting for FSMap version 33 2024-03-18T14:56:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.628+0000 7f029096e740 10 client.16086 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:56:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.628+0000 7f029096e740 20 client.16086 populate_metadata read hostname 'smithi076' 2024-03-18T14:56:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.628+0000 7f029096e740 1 --2- 172.21.15.76:0/381291282 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x564f2c8069c0 0x564f2c808e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.628+0000 7f029096e740 4 client.16086 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:56:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.628+0000 7f029096e740 1 -- 172.21.15.76:0/381291282 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f2c75ef80 con 0x564f2c8069c0 2024-03-18T14:56:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.629+0000 7f0286ffd640 1 --2- 172.21.15.76:0/381291282 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x564f2c8069c0 0x564f2c808e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.629+0000 7f0286ffd640 1 --2- 172.21.15.76:0/381291282 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x564f2c8069c0 0x564f2c808e00 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a92827fc72147f31 server_cookie=32c68139c182c026 in_seq=0 out_seq=0 2024-03-18T14:56:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.629+0000 7f0266ffd640 10 client.16086 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:56:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.629+0000 7f0266ffd640 1 -- 172.21.15.76:0/381291282 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564f2c75ef80 con 0x564f2c8069c0 2024-03-18T14:56:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.629+0000 7f0266ffd640 10 client.16086 handle_command_reply: tid=0 2024-03-18T14:56:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.629+0000 7f029096e740 1 client.16086 shutdown 2024-03-18T14:56:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.630+0000 7f029096e740 1 -- 172.21.15.76:0/381291282 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7eda50 msgr2=0x564f2c7f3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.630+0000 7f029096e740 1 --2- 172.21.15.76:0/381291282 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7eda50 0x564f2c7f3cd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0278036040 tx=0x7f0278004270 comp rx=0 tx=0).stop 2024-03-18T14:56:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.630+0000 7f029096e740 1 -- 172.21.15.76:0/381291282 shutdown_connections 2024-03-18T14:56:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.630+0000 7f029096e740 1 --2- 172.21.15.76:0/381291282 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x564f2c8069c0 0x564f2c808e00 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.630+0000 7f029096e740 1 --2- 172.21.15.76:0/381291282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7f6820 0x564f2c7f4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.630+0000 7f0266ffd640 0 client.16086 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:56:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.630+0000 7f029096e740 1 --2- 172.21.15.76:0/381291282 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7eda50 0x564f2c7f3cd0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.630+0000 7f029096e740 1 --2- 172.21.15.76:0/381291282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7ed130 0x564f2c7f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.630+0000 7f029096e740 1 -- 172.21.15.76:0/381291282 >> 172.21.15.76:0/381291282 conn(0x564f2c7e7a50 msgr2=0x564f2c7e97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.630+0000 7f029096e740 1 -- 172.21.15.76:0/381291282 shutdown_connections 2024-03-18T14:56:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.630+0000 7f029096e740 1 -- 172.21.15.76:0/381291282 wait complete. 2024-03-18T14:56:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.630+0000 7f029096e740 20 client.16086 trim_cache size 0 max 16384 2024-03-18T14:56:24.651 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.645+0000 7f029096e740 1 Processor -- start 2024-03-18T14:56:24.651 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.645+0000 7f029096e740 1 -- start start 2024-03-18T14:56:24.651 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.645+0000 7f029096e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7ed130 0x564f2c7f5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.651 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.645+0000 7f029096e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7eda50 0x564f2c7f4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.651 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.645+0000 7f029096e740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7f6820 0x564f2c80e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.651 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.645+0000 7f029096e740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f02881afb80 con 0x564f2c7f6820 2024-03-18T14:56:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.645+0000 7f029096e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f02881ad3f0 con 0x564f2c7eda50 2024-03-18T14:56:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.645+0000 7f029096e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f02881ab240 con 0x564f2c7ed130 2024-03-18T14:56:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.645+0000 7f0286ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7f6820 0x564f2c80e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.645+0000 7f0286ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7f6820 0x564f2c80e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44124/0 (socket says 172.21.15.76:44124) 2024-03-18T14:56:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.645+0000 7f0286ffd640 1 -- 172.21.15.76:0/922803900 learned_addr learned my addr 172.21.15.76:0/922803900 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.645+0000 7f02877fe640 1 --2- 172.21.15.76:0/922803900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7eda50 0x564f2c7f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.645+0000 7f0286ffd640 1 --2- 172.21.15.76:0/922803900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7ed130 0x564f2c7f5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.645+0000 7f0286ffd640 1 -- 172.21.15.76:0/922803900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7ed130 msgr2=0x564f2c7f5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.645+0000 7f0286ffd640 1 --2- 172.21.15.76:0/922803900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7ed130 0x564f2c7f5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.645+0000 7f0286ffd640 1 -- 172.21.15.76:0/922803900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7eda50 msgr2=0x564f2c7f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.645+0000 7f0286ffd640 1 --2- 172.21.15.76:0/922803900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7eda50 0x564f2c7f4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.646+0000 7f0286ffd640 1 -- 172.21.15.76:0/922803900 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f2c809480 con 0x564f2c7f6820 2024-03-18T14:56:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.646+0000 7f0286ffd640 1 --2- 172.21.15.76:0/922803900 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7f6820 0x564f2c80e710 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f027c004a30 tx=0x7f027c025040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de84aa1035630f02 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.646+0000 7f026d7fa640 1 -- 172.21.15.76:0/922803900 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c013170 con 0x564f2c7f6820 2024-03-18T14:56:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.646+0000 7f026d7fa640 1 -- 172.21.15.76:0/922803900 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f027c0133c0 con 0x564f2c7f6820 2024-03-18T14:56:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.646+0000 7f026d7fa640 1 -- 172.21.15.76:0/922803900 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c00fa30 con 0x564f2c7f6820 2024-03-18T14:56:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.646+0000 7f029096e740 1 -- 172.21.15.76:0/922803900 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7f6820 msgr2=0x564f2c80e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.646+0000 7f029096e740 1 --2- 172.21.15.76:0/922803900 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7f6820 0x564f2c80e710 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f027c004a30 tx=0x7f027c025040 comp rx=0 tx=0).stop 2024-03-18T14:56:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.646+0000 7f029096e740 1 -- 172.21.15.76:0/922803900 shutdown_connections 2024-03-18T14:56:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.646+0000 7f029096e740 1 --2- 172.21.15.76:0/922803900 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7f6820 0x564f2c80e710 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.646+0000 7f029096e740 1 --2- 172.21.15.76:0/922803900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7eda50 0x564f2c7f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.646+0000 7f029096e740 1 --2- 172.21.15.76:0/922803900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7ed130 0x564f2c7f5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.646+0000 7f029096e740 1 -- 172.21.15.76:0/922803900 >> 172.21.15.76:0/922803900 conn(0x564f2c7e7a50 msgr2=0x564f2c7e6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.646+0000 7f029096e740 1 -- 172.21.15.76:0/922803900 shutdown_connections 2024-03-18T14:56:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.646+0000 7f029096e740 1 -- 172.21.15.76:0/922803900 wait complete. 2024-03-18T14:56:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.647+0000 7f029096e740 1 -- start start 2024-03-18T14:56:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.647+0000 7f029096e740 1 Processor -- start 2024-03-18T14:56:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.647+0000 7f029096e740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7ed130 0x564f2c7fe790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.647+0000 7f029096e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7eda50 0x564f2c7fb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.647+0000 7f029096e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7f6820 0x564f2c7fbca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.647+0000 7f029096e740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f02881ab8b0 con 0x564f2c7ed130 2024-03-18T14:56:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.647+0000 7f029096e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f02881acd20 con 0x564f2c7eda50 2024-03-18T14:56:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.647+0000 7f029096e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f02881abf80 con 0x564f2c7f6820 2024-03-18T14:56:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.647+0000 7f029096e740 10 client.0 _ll_register_callbacks cb 0x7f0278009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:56:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.647+0000 7f0286ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7ed130 0x564f2c7fe790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.647+0000 7f0286ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7ed130 0x564f2c7fe790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44140/0 (socket says 172.21.15.76:44140) 2024-03-18T14:56:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.647+0000 7f0286ffd640 1 -- 172.21.15.76:0/1441224765 learned_addr learned my addr 172.21.15.76:0/1441224765 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.647+0000 7f02877fe640 1 --2- 172.21.15.76:0/1441224765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7eda50 0x564f2c7fb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.647+0000 7f0286ffd640 1 --2- 172.21.15.76:0/1441224765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7f6820 0x564f2c7fbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.647+0000 7f0286ffd640 1 -- 172.21.15.76:0/1441224765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7f6820 msgr2=0x564f2c7fbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f0286ffd640 1 --2- 172.21.15.76:0/1441224765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7f6820 0x564f2c7fbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f0286ffd640 1 -- 172.21.15.76:0/1441224765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7eda50 msgr2=0x564f2c7fb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f0286ffd640 1 --2- 172.21.15.76:0/1441224765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7eda50 0x564f2c7fb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f0286ffd640 1 -- 172.21.15.76:0/1441224765 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f2c7e7880 con 0x564f2c7ed130 2024-03-18T14:56:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f0286ffd640 1 --2- 172.21.15.76:0/1441224765 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7ed130 0x564f2c7fe790 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f027c013810 tx=0x7f027c00f830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b51f8c09812308 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f0266ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:56:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f0266ffd640 1 -- 172.21.15.76:0/1441224765 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c038040 con 0x564f2c7ed130 2024-03-18T14:56:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f0266ffd640 1 -- 172.21.15.76:0/1441224765 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7f027c004230 con 0x564f2c7ed130 2024-03-18T14:56:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f029096e740 1 -- 172.21.15.76:0/1441224765 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f027c0133c0 con 0x564f2c7ed130 2024-03-18T14:56:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f0266ffd640 1 -- 172.21.15.76:0/1441224765 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f027c004430 con 0x564f2c7ed130 2024-03-18T14:56:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f029096e740 10 client.16092 fetch_fsmap learned FSMap version 33 2024-03-18T14:56:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f029096e740 1 -- 172.21.15.76:0/1441224765 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564f2c86d6a0 con 0x564f2c7ed130 2024-03-18T14:56:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f0266ffd640 1 -- 172.21.15.76:0/1441224765 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f027c0133c0 con 0x564f2c7ed130 2024-03-18T14:56:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f0266ffd640 1 -- 172.21.15.76:0/1441224765 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f027c0035f0 con 0x564f2c7ed130 2024-03-18T14:56:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f029096e740 10 client.16092 fetch_fsmap finished waiting for FSMap version 33 2024-03-18T14:56:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f029096e740 10 client.16092 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:56:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f029096e740 20 client.16092 populate_metadata read hostname 'smithi076' 2024-03-18T14:56:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f029096e740 1 --2- 172.21.15.76:0/1441224765 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x564f2c86db80 0x564f2c86ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f029096e740 4 client.16092 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:56:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.648+0000 7f029096e740 1 -- 172.21.15.76:0/1441224765 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564f2c7592d0 con 0x564f2c86db80 2024-03-18T14:56:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.649+0000 7f02877fe640 1 --2- 172.21.15.76:0/1441224765 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x564f2c86db80 0x564f2c86ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.649+0000 7f02877fe640 1 --2- 172.21.15.76:0/1441224765 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x564f2c86db80 0x564f2c86ffa0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2078ddc13c9fd85 server_cookie=10a1ec4caab9c653 in_seq=0 out_seq=0 2024-03-18T14:56:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.649+0000 7f0266ffd640 10 client.16092 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:56:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.721+0000 7f0266ffd640 1 -- 172.21.15.76:0/1441224765 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564f2c7592d0 con 0x564f2c86db80 2024-03-18T14:56:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.721+0000 7f0266ffd640 10 client.16092 handle_command_reply: tid=0 2024-03-18T14:56:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.721+0000 7f029096e740 1 client.16092 shutdown 2024-03-18T14:56:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.722+0000 7f029096e740 1 -- 172.21.15.76:0/1441224765 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7ed130 msgr2=0x564f2c7fe790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.722+0000 7f029096e740 1 --2- 172.21.15.76:0/1441224765 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7ed130 0x564f2c7fe790 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f027c013810 tx=0x7f027c00f830 comp rx=0 tx=0).stop 2024-03-18T14:56:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.722+0000 7f029096e740 1 -- 172.21.15.76:0/1441224765 shutdown_connections 2024-03-18T14:56:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.722+0000 7f029096e740 1 --2- 172.21.15.76:0/1441224765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f2c7f6820 0x564f2c7fbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.722+0000 7f029096e740 1 --2- 172.21.15.76:0/1441224765 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x564f2c86db80 0x564f2c86ffa0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.722+0000 7f029096e740 1 --2- 172.21.15.76:0/1441224765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f2c7eda50 0x564f2c7fb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.722+0000 7f029096e740 1 --2- 172.21.15.76:0/1441224765 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564f2c7ed130 0x564f2c7fe790 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.722+0000 7f029096e740 1 -- 172.21.15.76:0/1441224765 >> 172.21.15.76:0/1441224765 conn(0x564f2c7e7a50 msgr2=0x564f2c7e6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.722+0000 7f0266ffd640 0 client.16092 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:56:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.722+0000 7f029096e740 1 -- 172.21.15.76:0/1441224765 shutdown_connections 2024-03-18T14:56:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.722+0000 7f029096e740 1 -- 172.21.15.76:0/1441224765 wait complete. 2024-03-18T14:56:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.722+0000 7f029096e740 20 client.16092 trim_cache size 0 max 16384 2024-03-18T14:56:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:56:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:56:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:56:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:56:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.723+0000 7f028dca1640 1 -- 172.21.15.76:0/1224180977 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f02700a2a10 msgr2=0x7f02700a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.723+0000 7f028dca1640 1 --2- 172.21.15.76:0/1224180977 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f02700a2a10 0x7f02700a4ed0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f02800cd220 tx=0x7f0280066d40 comp rx=0 tx=0).stop 2024-03-18T14:56:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.723+0000 7f028dca1640 1 -- 172.21.15.76:0/1224180977 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02881077d0 msgr2=0x7f028819c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.723+0000 7f028dca1640 1 --2- 172.21.15.76:0/1224180977 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02881077d0 0x7f028819c890 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0280009fd0 tx=0x7f028000ef10 comp rx=0 tx=0).stop 2024-03-18T14:56:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.723+0000 7f028dca1640 1 -- 172.21.15.76:0/1224180977 shutdown_connections 2024-03-18T14:56:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.723+0000 7f028dca1640 1 --2- 172.21.15.76:0/1224180977 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f02700a2a10 0x7f02700a4ed0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.723+0000 7f028dca1640 1 --2- 172.21.15.76:0/1224180977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f028806aa00 0x7f0288198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.723+0000 7f028dca1640 1 --2- 172.21.15.76:0/1224180977 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02881077d0 0x7f028819c890 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.723+0000 7f028dca1640 1 --2- 172.21.15.76:0/1224180977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028806a060 0x7f0288197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.723+0000 7f028dca1640 1 -- 172.21.15.76:0/1224180977 >> 172.21.15.76:0/1224180977 conn(0x7f0288077530 msgr2=0x7f02880fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.723+0000 7f028dca1640 1 -- 172.21.15.76:0/1224180977 shutdown_connections 2024-03-18T14:56:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:24.723+0000 7f028dca1640 1 -- 172.21.15.76:0/1224180977 wait complete. 2024-03-18T14:56:24.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 273619 10.18 MB/sec execute 25 sec latency 43.959 ms 2024-03-18T14:56:25.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:25 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:56:25.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:25 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:56:25.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:25 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:56:25.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 275619 10.08 MB/sec execute 26 sec latency 15.529 ms 2024-03-18T14:56:26.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:26 smithi076 ceph-mon[32205]: pgmap v101: 97 pgs: 97 active+clean; 90 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 233 op/s 2024-03-18T14:56:26.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:26 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:56:26.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:26 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:56:26.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:26 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:56:26.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:26 smithi174 ceph-mon[36787]: pgmap v101: 97 pgs: 97 active+clean; 90 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 233 op/s 2024-03-18T14:56:26.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:26 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:56:26.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:26 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:56:26.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:26 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:56:26.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:26 smithi077 ceph-mon[35865]: pgmap v101: 97 pgs: 97 active+clean; 90 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 233 op/s 2024-03-18T14:56:26.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:26 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:56:26.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:26 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:56:26.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:26 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:56:26.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 277201 10.13 MB/sec execute 27 sec latency 16.057 ms 2024-03-18T14:56:27.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 279571 10.17 MB/sec execute 28 sec latency 17.932 ms 2024-03-18T14:56:27.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:27 smithi174 ceph-mon[36787]: pgmap v102: 97 pgs: 97 active+clean; 90 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 233 op/s 2024-03-18T14:56:27.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:27 smithi077 ceph-mon[35865]: pgmap v102: 97 pgs: 97 active+clean; 90 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 233 op/s 2024-03-18T14:56:28.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:27 smithi076 ceph-mon[32205]: pgmap v102: 97 pgs: 97 active+clean; 90 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 233 op/s 2024-03-18T14:56:28.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 280798 10.10 MB/sec execute 29 sec latency 40.735 ms 2024-03-18T14:56:29.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 283567 10.24 MB/sec execute 30 sec latency 21.469 ms 2024-03-18T14:56:29.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:29 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:56:29.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:29 smithi076 ceph-mon[32205]: pgmap v103: 97 pgs: 97 active+clean; 92 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 379 op/s 2024-03-18T14:56:29.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:29 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:56:29.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:29 smithi174 ceph-mon[36787]: pgmap v103: 97 pgs: 97 active+clean; 92 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 379 op/s 2024-03-18T14:56:29.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:29 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:56:29.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:29 smithi077 ceph-mon[35865]: pgmap v103: 97 pgs: 97 active+clean; 92 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 379 op/s 2024-03-18T14:56:30.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 284763 10.09 MB/sec execute 31 sec latency 37.017 ms 2024-03-18T14:56:31.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 287558 10.33 MB/sec execute 32 sec latency 11.642 ms 2024-03-18T14:56:31.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:31 smithi174 ceph-mon[36787]: pgmap v104: 97 pgs: 97 active+clean; 92 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 254 op/s 2024-03-18T14:56:31.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:31 smithi077 ceph-mon[35865]: pgmap v104: 97 pgs: 97 active+clean; 92 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 254 op/s 2024-03-18T14:56:32.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:31 smithi076 ceph-mon[32205]: pgmap v104: 97 pgs: 97 active+clean; 92 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 254 op/s 2024-03-18T14:56:32.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 288891 10.19 MB/sec execute 33 sec latency 15.331 ms 2024-03-18T14:56:33.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 291246 10.30 MB/sec execute 34 sec latency 25.958 ms 2024-03-18T14:56:33.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:33 smithi174 ceph-mon[36787]: pgmap v105: 97 pgs: 97 active+clean; 95 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 366 op/s 2024-03-18T14:56:33.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:33 smithi077 ceph-mon[35865]: pgmap v105: 97 pgs: 97 active+clean; 95 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 366 op/s 2024-03-18T14:56:34.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:33 smithi076 ceph-mon[32205]: pgmap v105: 97 pgs: 97 active+clean; 95 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 366 op/s 2024-03-18T14:56:34.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 293221 10.20 MB/sec execute 35 sec latency 15.287 ms 2024-03-18T14:56:35.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 294901 10.26 MB/sec execute 36 sec latency 16.757 ms 2024-03-18T14:56:35.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:35 smithi174 ceph-mon[36787]: pgmap v106: 97 pgs: 97 active+clean; 95 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 258 op/s 2024-03-18T14:56:35.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:35 smithi077 ceph-mon[35865]: pgmap v106: 97 pgs: 97 active+clean; 95 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 258 op/s 2024-03-18T14:56:36.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:35 smithi076 ceph-mon[32205]: pgmap v106: 97 pgs: 97 active+clean; 95 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 258 op/s 2024-03-18T14:56:36.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 297205 10.28 MB/sec execute 37 sec latency 15.561 ms 2024-03-18T14:56:37.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 298763 10.23 MB/sec execute 38 sec latency 17.536 ms 2024-03-18T14:56:37.823 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.819+0000 7f6aa2797640 1 -- 172.21.15.76:0/984780327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a9c068e60 msgr2=0x7f6a9c069240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:37.823 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.819+0000 7f6aa2797640 1 --2- 172.21.15.76:0/984780327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a9c068e60 0x7f6a9c069240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c009860 tx=0x7f6a8c02bc00 comp rx=0 tx=0).stop 2024-03-18T14:56:37.823 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.819+0000 7f6aa2797640 1 -- 172.21.15.76:0/984780327 shutdown_connections 2024-03-18T14:56:37.823 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.819+0000 7f6aa2797640 1 --2- 172.21.15.76:0/984780327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a9c10e4e0 0x7f6a9c1108d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.823 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.819+0000 7f6aa2797640 1 --2- 172.21.15.76:0/984780327 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a9c069780 0x7f6a9c10dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.823 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.819+0000 7f6aa2797640 1 --2- 172.21.15.76:0/984780327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a9c068e60 0x7f6a9c069240 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.823 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.819+0000 7f6aa2797640 1 -- 172.21.15.76:0/984780327 >> 172.21.15.76:0/984780327 conn(0x7f6a9c077530 msgr2=0x7f6a9c077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:37.823 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.819+0000 7f6aa2797640 1 -- 172.21.15.76:0/984780327 shutdown_connections 2024-03-18T14:56:37.823 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.820+0000 7f6aa2797640 1 -- 172.21.15.76:0/984780327 wait complete. 2024-03-18T14:56:37.823 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.820+0000 7f6aa2797640 1 Processor -- start 2024-03-18T14:56:37.823 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.820+0000 7f6aa2797640 1 -- start start 2024-03-18T14:56:37.823 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.821+0000 7f6aa2797640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a9c068e60 0x7f6a9c10db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.824 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.821+0000 7f6aa2797640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a9c069780 0x7f6a9c109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.824 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.821+0000 7f6aa2797640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a9c10e4e0 0x7f6a9c10a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.824 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.821+0000 7f6aa2797640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c06db90 con 0x7f6a9c10e4e0 2024-03-18T14:56:37.824 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.821+0000 7f6aa2797640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c06da10 con 0x7f6a9c068e60 2024-03-18T14:56:37.824 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.821+0000 7f6aa2797640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c06dd10 con 0x7f6a9c069780 2024-03-18T14:56:37.824 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.821+0000 7f6a9bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a9c068e60 0x7f6a9c10db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.824 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.821+0000 7f6a9bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a9c068e60 0x7f6a9c10db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:49084/0 (socket says 172.21.15.76:49084) 2024-03-18T14:56:37.824 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.821+0000 7f6a9bfff640 1 -- 172.21.15.76:0/1493984344 learned_addr learned my addr 172.21.15.76:0/1493984344 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:37.824 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.821+0000 7f6aa0d34640 1 --2- 172.21.15.76:0/1493984344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a9c10e4e0 0x7f6a9c10a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.824 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.821+0000 7f6a9b7fe640 1 --2- 172.21.15.76:0/1493984344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a9c069780 0x7f6a9c109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.824 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.821+0000 7f6a9bfff640 1 -- 172.21.15.76:0/1493984344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a9c069780 msgr2=0x7f6a9c109ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:37.824 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.821+0000 7f6a9bfff640 1 --2- 172.21.15.76:0/1493984344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a9c069780 0x7f6a9c109ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.824 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.821+0000 7f6a9bfff640 1 -- 172.21.15.76:0/1493984344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a9c10e4e0 msgr2=0x7f6a9c10a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:37.824 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.821+0000 7f6a9bfff640 1 --2- 172.21.15.76:0/1493984344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a9c10e4e0 0x7f6a9c10a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.825 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.821+0000 7f6a9bfff640 1 -- 172.21.15.76:0/1493984344 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a9c10a6a0 con 0x7f6a9c068e60 2024-03-18T14:56:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.821+0000 7f6a9bfff640 1 --2- 172.21.15.76:0/1493984344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a9c068e60 0x7f6a9c10db10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c037040 tx=0x7f6a8c0344c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.822+0000 7f6a997fa640 1 -- 172.21.15.76:0/1493984344 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c041020 con 0x7f6a9c068e60 2024-03-18T14:56:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.822+0000 7f6a997fa640 1 -- 172.21.15.76:0/1493984344 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a8c02ca80 con 0x7f6a9c068e60 2024-03-18T14:56:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.822+0000 7f6aa2797640 1 -- 172.21.15.76:0/1493984344 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a9c10a930 con 0x7f6a9c068e60 2024-03-18T14:56:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.822+0000 7f6a997fa640 1 -- 172.21.15.76:0/1493984344 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c039040 con 0x7f6a9c068e60 2024-03-18T14:56:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.822+0000 7f6aa2797640 1 -- 172.21.15.76:0/1493984344 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a9c19e410 con 0x7f6a9c068e60 2024-03-18T14:56:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.824+0000 7f6aa5386740 1 Processor -- start 2024-03-18T14:56:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.824+0000 7f6aa5386740 1 -- start start 2024-03-18T14:56:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.824+0000 7f6aa5386740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20c130 0x5633cd20c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.824+0000 7f6aa5386740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd20ca50 0x5633cd2152e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.824+0000 7f6aa5386740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd215820 0x5633cd217c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.824+0000 7f6aa5386740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5633cd170a40 con 0x5633cd20c130 2024-03-18T14:56:37.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.824+0000 7f6aa5386740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5633cd119920 con 0x5633cd215820 2024-03-18T14:56:37.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.824+0000 7f6aa5386740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5633cd16cae0 con 0x5633cd20ca50 2024-03-18T14:56:37.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.824+0000 7f6aa0d34640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20c130 0x5633cd20c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.824+0000 7f6aa0d34640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20c130 0x5633cd20c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53074/0 (socket says 172.21.15.76:53074) 2024-03-18T14:56:37.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.824+0000 7f6aa0d34640 1 -- 172.21.15.76:0/4147152891 learned_addr learned my addr 172.21.15.76:0/4147152891 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:37.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.825+0000 7f6a997fa640 1 -- 172.21.15.76:0/1493984344 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f6a8c044020 con 0x7f6a9c068e60 2024-03-18T14:56:37.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.824+0000 7f6a9b7fe640 1 --2- 172.21.15.76:0/4147152891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd215820 0x5633cd217c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.825+0000 7f6a9bfff640 1 --2- 172.21.15.76:0/4147152891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd20ca50 0x5633cd2152e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.825+0000 7f6aa0d34640 1 -- 172.21.15.76:0/4147152891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd20ca50 msgr2=0x5633cd2152e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:37.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.825+0000 7f6aa0d34640 1 --2- 172.21.15.76:0/4147152891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd20ca50 0x5633cd2152e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.825+0000 7f6aa0d34640 1 -- 172.21.15.76:0/4147152891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd215820 msgr2=0x5633cd217c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:37.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.825+0000 7f6aa0d34640 1 --2- 172.21.15.76:0/4147152891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd215820 0x5633cd217c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.825+0000 7f6aa0d34640 1 -- 172.21.15.76:0/4147152891 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633cd131460 con 0x5633cd20c130 2024-03-18T14:56:37.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.825+0000 7f6a9bfff640 1 --2- 172.21.15.76:0/4147152891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd20ca50 0x5633cd2152e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:56:37.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.825+0000 7f6aa0d34640 1 --2- 172.21.15.76:0/4147152891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20c130 0x5633cd20c510 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f6a94003970 tx=0x7f6a94003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11388ed16b6ed060 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:37.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.825+0000 7f6a997fa640 1 --2- 172.21.15.76:0/1493984344 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f6a840a2a40 0x7f6a840a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.825+0000 7f6a827fc640 1 -- 172.21.15.76:0/4147152891 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a94004b20 con 0x5633cd20c130 2024-03-18T14:56:37.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.825+0000 7f6a9bfff640 1 --2- 172.21.15.76:0/1493984344 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f6a840a2a40 0x7f6a840a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.825+0000 7f6a827fc640 1 -- 172.21.15.76:0/4147152891 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a94004cc0 con 0x5633cd20c130 2024-03-18T14:56:37.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.825+0000 7f6a827fc640 1 -- 172.21.15.76:0/4147152891 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a94021800 con 0x5633cd20c130 2024-03-18T14:56:37.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.825+0000 7f6a997fa640 1 -- 172.21.15.76:0/1493984344 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7f6a8c0f0430 con 0x7f6a9c068e60 2024-03-18T14:56:37.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.826+0000 7f6aa5386740 1 -- 172.21.15.76:0/4147152891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20c130 msgr2=0x5633cd20c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:37.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.826+0000 7f6a9bfff640 1 --2- 172.21.15.76:0/1493984344 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f6a840a2a40 0x7f6a840a4f00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x5633cd20bbe0 tx=0x7f6a8c02bc00 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:37.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.826+0000 7f6aa5386740 1 --2- 172.21.15.76:0/4147152891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20c130 0x5633cd20c510 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f6a94003970 tx=0x7f6a94003e30 comp rx=0 tx=0).stop 2024-03-18T14:56:37.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.826+0000 7f6aa5386740 1 -- 172.21.15.76:0/4147152891 shutdown_connections 2024-03-18T14:56:37.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.826+0000 7f6aa5386740 1 --2- 172.21.15.76:0/4147152891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd20ca50 0x5633cd2152e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.826+0000 7f6aa5386740 1 --2- 172.21.15.76:0/4147152891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd215820 0x5633cd217c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.826+0000 7f6aa5386740 1 --2- 172.21.15.76:0/4147152891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20c130 0x5633cd20c510 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.826+0000 7f6aa5386740 1 -- 172.21.15.76:0/4147152891 >> 172.21.15.76:0/4147152891 conn(0x5633cd206a50 msgr2=0x5633cd206e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.826+0000 7f6aa5386740 1 -- 172.21.15.76:0/4147152891 shutdown_connections 2024-03-18T14:56:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.826+0000 7f6aa5386740 1 -- 172.21.15.76:0/4147152891 wait complete. 2024-03-18T14:56:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.826+0000 7f6aa5386740 1 -- start start 2024-03-18T14:56:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.827+0000 7f6aa5386740 1 Processor -- start 2024-03-18T14:56:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.827+0000 7f6aa5386740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd20c130 0x5633cd212900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.827+0000 7f6aa5386740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20ca50 0x5633cd212e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.827+0000 7f6aa5386740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd215820 0x5633cd213380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.827+0000 7f6aa5386740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5633cd218450 con 0x5633cd20ca50 2024-03-18T14:56:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.827+0000 7f6aa5386740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5633cd12cd70 con 0x5633cd20c130 2024-03-18T14:56:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.827+0000 7f6aa5386740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5633cd2185d0 con 0x5633cd215820 2024-03-18T14:56:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.827+0000 7f6aa5386740 10 client.0 _ll_register_callbacks cb 0x5633cce84a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:56:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.827+0000 7f6a9b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20ca50 0x5633cd212e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.827+0000 7f6aa0d34640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd20c130 0x5633cd212900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.827+0000 7f6a9b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20ca50 0x5633cd212e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53090/0 (socket says 172.21.15.76:53090) 2024-03-18T14:56:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.827+0000 7f6a9b7fe640 1 -- 172.21.15.76:0/2901842991 learned_addr learned my addr 172.21.15.76:0/2901842991 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.827+0000 7f6aa0d34640 1 --2- 172.21.15.76:0/2901842991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd215820 0x5633cd213380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.828+0000 7f6aa0d34640 1 -- 172.21.15.76:0/2901842991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd215820 msgr2=0x5633cd213380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.828+0000 7f6aa0d34640 1 --2- 172.21.15.76:0/2901842991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd215820 0x5633cd213380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.828+0000 7f6aa0d34640 1 -- 172.21.15.76:0/2901842991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20ca50 msgr2=0x5633cd212e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.828+0000 7f6aa0d34640 1 --2- 172.21.15.76:0/2901842991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20ca50 0x5633cd212e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.828+0000 7f6aa0d34640 1 -- 172.21.15.76:0/2901842991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633cce85210 con 0x5633cd20c130 2024-03-18T14:56:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.828+0000 7f6a9b7fe640 1 --2- 172.21.15.76:0/2901842991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20ca50 0x5633cd212e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:56:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.828+0000 7f6aa0d34640 1 --2- 172.21.15.76:0/2901842991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd20c130 0x5633cd212900 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6a94005b40 tx=0x7f6a94005a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd0ea69fc5832b6e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.828+0000 7f6a7b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:56:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.828+0000 7f6a7b7fe640 1 -- 172.21.15.76:0/2901842991 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9403e060 con 0x5633cd20c130 2024-03-18T14:56:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.828+0000 7f6a7b7fe640 1 -- 172.21.15.76:0/2901842991 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7f6a940054b0 con 0x5633cd20c130 2024-03-18T14:56:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.828+0000 7f6aa5386740 1 -- 172.21.15.76:0/2901842991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a94004cc0 con 0x5633cd20c130 2024-03-18T14:56:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.828+0000 7f6a7b7fe640 1 -- 172.21.15.76:0/2901842991 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f6a94004ad0 con 0x5633cd20c130 2024-03-18T14:56:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.828+0000 7f6aa5386740 10 client.25961 fetch_fsmap learned FSMap version 34 2024-03-18T14:56:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.828+0000 7f6aa5386740 1 -- 172.21.15.76:0/2901842991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5633ccf51620 con 0x5633cd20c130 2024-03-18T14:56:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.829+0000 7f6a7b7fe640 1 -- 172.21.15.76:0/2901842991 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a94004cc0 con 0x5633cd20c130 2024-03-18T14:56:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.829+0000 7f6a7b7fe640 1 -- 172.21.15.76:0/2901842991 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 34) ==== 8342+0+0 (secure 0 0 0) 0x7f6a94029070 con 0x5633cd20c130 2024-03-18T14:56:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.829+0000 7f6aa5386740 10 client.25961 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T14:56:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.829+0000 7f6aa5386740 10 client.25961 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:56:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.829+0000 7f6aa5386740 20 client.25961 populate_metadata read hostname 'smithi076' 2024-03-18T14:56:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.829+0000 7f6aa5386740 1 --2- 172.21.15.76:0/2901842991 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5633cd225960 0x5633cd227d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.829+0000 7f6aa5386740 4 client.25961 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:56:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.829+0000 7f6aa5386740 1 -- 172.21.15.76:0/2901842991 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633cd17df80 con 0x5633cd225960 2024-03-18T14:56:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.829+0000 7f6a9b7fe640 1 --2- 172.21.15.76:0/2901842991 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5633cd225960 0x5633cd227d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.830+0000 7f6a7b7fe640 10 client.25961 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:56:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.830+0000 7f6a9b7fe640 1 --2- 172.21.15.76:0/2901842991 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5633cd225960 0x5633cd227d50 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=124817ff2d1ec11d server_cookie=fba1586ccce72ac1 in_seq=0 out_seq=0 2024-03-18T14:56:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.830+0000 7f6a7b7fe640 1 -- 172.21.15.76:0/2901842991 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5633cd17df80 con 0x5633cd225960 2024-03-18T14:56:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.830+0000 7f6a7b7fe640 10 client.25961 handle_command_reply: tid=0 2024-03-18T14:56:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.830+0000 7f6aa5386740 1 client.25961 shutdown 2024-03-18T14:56:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.831+0000 7f6aa5386740 1 -- 172.21.15.76:0/2901842991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd20c130 msgr2=0x5633cd212900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.831+0000 7f6aa5386740 1 --2- 172.21.15.76:0/2901842991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd20c130 0x5633cd212900 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6a94005b40 tx=0x7f6a94005a00 comp rx=0 tx=0).stop 2024-03-18T14:56:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.831+0000 7f6aa5386740 1 -- 172.21.15.76:0/2901842991 shutdown_connections 2024-03-18T14:56:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.831+0000 7f6aa5386740 1 --2- 172.21.15.76:0/2901842991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd215820 0x5633cd213380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.831+0000 7f6aa5386740 1 --2- 172.21.15.76:0/2901842991 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5633cd225960 0x5633cd227d50 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.831+0000 7f6aa5386740 1 --2- 172.21.15.76:0/2901842991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20ca50 0x5633cd212e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.831+0000 7f6aa5386740 1 --2- 172.21.15.76:0/2901842991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd20c130 0x5633cd212900 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.831+0000 7f6a7b7fe640 0 client.25961 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:56:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.831+0000 7f6aa5386740 1 -- 172.21.15.76:0/2901842991 >> 172.21.15.76:0/2901842991 conn(0x5633cd206a50 msgr2=0x5633cd2087a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.831+0000 7f6aa5386740 1 -- 172.21.15.76:0/2901842991 shutdown_connections 2024-03-18T14:56:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.831+0000 7f6aa5386740 1 -- 172.21.15.76:0/2901842991 wait complete. 2024-03-18T14:56:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.831+0000 7f6aa5386740 20 client.25961 trim_cache size 0 max 16384 2024-03-18T14:56:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.846+0000 7f6aa5386740 1 Processor -- start 2024-03-18T14:56:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.846+0000 7f6aa5386740 1 -- start start 2024-03-18T14:56:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.846+0000 7f6aa5386740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd20c130 0x5633cd2211f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.846+0000 7f6aa5386740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20ca50 0x5633cd201a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.846+0000 7f6aa5386740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd215820 0x5633cd22d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.846+0000 7f6aa5386740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c1a96a0 con 0x5633cd20ca50 2024-03-18T14:56:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.846+0000 7f6aa5386740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c1a6f40 con 0x5633cd215820 2024-03-18T14:56:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.846+0000 7f6aa5386740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c1a4d90 con 0x5633cd20c130 2024-03-18T14:56:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.847+0000 7f6a9b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20ca50 0x5633cd201a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.847+0000 7f6a9b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20ca50 0x5633cd201a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53096/0 (socket says 172.21.15.76:53096) 2024-03-18T14:56:37.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.847+0000 7f6a9b7fe640 1 -- 172.21.15.76:0/1803615847 learned_addr learned my addr 172.21.15.76:0/1803615847 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:37.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.847+0000 7f6aa0d34640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd20c130 0x5633cd2211f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.847+0000 7f6aa0d34640 1 --2- 172.21.15.76:0/1803615847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd215820 0x5633cd22d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.847+0000 7f6a9b7fe640 1 -- 172.21.15.76:0/1803615847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd20c130 msgr2=0x5633cd2211f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:37.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.847+0000 7f6a9b7fe640 1 --2- 172.21.15.76:0/1803615847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd20c130 0x5633cd2211f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.847+0000 7f6a9b7fe640 1 -- 172.21.15.76:0/1803615847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd215820 msgr2=0x5633cd22d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:37.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.847+0000 7f6a9b7fe640 1 --2- 172.21.15.76:0/1803615847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd215820 0x5633cd22d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.847+0000 7f6a9b7fe640 1 -- 172.21.15.76:0/1803615847 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633cd218330 con 0x5633cd20ca50 2024-03-18T14:56:37.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.847+0000 7f6a9b7fe640 1 --2- 172.21.15.76:0/1803615847 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20ca50 0x5633cd201a40 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009fd0 tx=0x7f6a90009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c01278f4faffed server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:37.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.847+0000 7f6a81ffb640 1 -- 172.21.15.76:0/1803615847 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9001e070 con 0x5633cd20ca50 2024-03-18T14:56:37.855 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.847+0000 7f6a81ffb640 1 -- 172.21.15.76:0/1803615847 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a90023070 con 0x5633cd20ca50 2024-03-18T14:56:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.847+0000 7f6a81ffb640 1 -- 172.21.15.76:0/1803615847 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9001d450 con 0x5633cd20ca50 2024-03-18T14:56:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.848+0000 7f6aa5386740 1 -- 172.21.15.76:0/1803615847 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20ca50 msgr2=0x5633cd201a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.848+0000 7f6aa5386740 1 --2- 172.21.15.76:0/1803615847 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20ca50 0x5633cd201a40 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009fd0 tx=0x7f6a90009260 comp rx=0 tx=0).stop 2024-03-18T14:56:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.848+0000 7f6aa5386740 1 -- 172.21.15.76:0/1803615847 shutdown_connections 2024-03-18T14:56:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.848+0000 7f6aa5386740 1 --2- 172.21.15.76:0/1803615847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd215820 0x5633cd22d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.848+0000 7f6aa5386740 1 --2- 172.21.15.76:0/1803615847 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd20ca50 0x5633cd201a40 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.848+0000 7f6aa5386740 1 --2- 172.21.15.76:0/1803615847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd20c130 0x5633cd2211f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.848+0000 7f6aa5386740 1 -- 172.21.15.76:0/1803615847 >> 172.21.15.76:0/1803615847 conn(0x5633cd206a50 msgr2=0x5633cd206e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.848+0000 7f6aa5386740 1 -- 172.21.15.76:0/1803615847 shutdown_connections 2024-03-18T14:56:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.848+0000 7f6aa5386740 1 -- 172.21.15.76:0/1803615847 wait complete. 2024-03-18T14:56:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.848+0000 7f6aa5386740 1 -- start start 2024-03-18T14:56:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa5386740 1 Processor -- start 2024-03-18T14:56:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa5386740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd20c130 0x5633cd21d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa5386740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd20ca50 0x5633cd218e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa5386740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd215820 0x5633cd219340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa5386740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c1a5400 con 0x5633cd215820 2024-03-18T14:56:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa5386740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c1a6870 con 0x5633cd20c130 2024-03-18T14:56:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa5386740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c1a5ad0 con 0x5633cd20ca50 2024-03-18T14:56:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa5386740 10 client.0 _ll_register_callbacks cb 0x7f6a900097e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:56:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa0d34640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd215820 0x5633cd219340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6a9b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd20ca50 0x5633cd218e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa0d34640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd215820 0x5633cd219340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53102/0 (socket says 172.21.15.76:53102) 2024-03-18T14:56:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6a9b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd20ca50 0x5633cd218e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:37144/0 (socket says 172.21.15.76:37144) 2024-03-18T14:56:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa0d34640 1 -- 172.21.15.76:0/4030492249 learned_addr learned my addr 172.21.15.76:0/4030492249 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa0d34640 1 --2- 172.21.15.76:0/4030492249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd20c130 0x5633cd21d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa0d34640 1 -- 172.21.15.76:0/4030492249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd20ca50 msgr2=0x5633cd218e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa0d34640 1 --2- 172.21.15.76:0/4030492249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd20ca50 0x5633cd218e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa0d34640 1 -- 172.21.15.76:0/4030492249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd20c130 msgr2=0x5633cd21d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa0d34640 1 --2- 172.21.15.76:0/4030492249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd20c130 0x5633cd21d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa0d34640 1 -- 172.21.15.76:0/4030492249 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633cd219a80 con 0x5633cd215820 2024-03-18T14:56:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa0d34640 1 --2- 172.21.15.76:0/4030492249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd215820 0x5633cd219340 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6a940426b0 tx=0x7f6a94005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9f59ee51187d3c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6a7b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:56:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6a7b7fe640 1 -- 172.21.15.76:0/4030492249 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9403e070 con 0x5633cd215820 2024-03-18T14:56:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6a7b7fe640 1 -- 172.21.15.76:0/4030492249 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7f6a94004c70 con 0x5633cd215820 2024-03-18T14:56:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.849+0000 7f6aa5386740 1 -- 172.21.15.76:0/4030492249 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a90023070 con 0x5633cd215820 2024-03-18T14:56:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.850+0000 7f6a7b7fe640 1 -- 172.21.15.76:0/4030492249 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f6a94009c30 con 0x5633cd215820 2024-03-18T14:56:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.850+0000 7f6aa5386740 10 client.16110 fetch_fsmap learned FSMap version 34 2024-03-18T14:56:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.850+0000 7f6aa5386740 1 -- 172.21.15.76:0/4030492249 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5633cd28b9e0 con 0x5633cd215820 2024-03-18T14:56:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.850+0000 7f6a7b7fe640 1 -- 172.21.15.76:0/4030492249 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a90023070 con 0x5633cd215820 2024-03-18T14:56:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.850+0000 7f6a7b7fe640 1 -- 172.21.15.76:0/4030492249 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 34) ==== 8342+0+0 (secure 0 0 0) 0x7f6a94029070 con 0x5633cd215820 2024-03-18T14:56:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.850+0000 7f6aa5386740 10 client.16110 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T14:56:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.850+0000 7f6aa5386740 10 client.16110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:56:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.850+0000 7f6aa5386740 20 client.16110 populate_metadata read hostname 'smithi076' 2024-03-18T14:56:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.850+0000 7f6aa5386740 1 --2- 172.21.15.76:0/4030492249 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5633cd28bf90 0x5633cd28e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.850+0000 7f6aa5386740 4 client.16110 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:56:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.850+0000 7f6aa5386740 1 -- 172.21.15.76:0/4030492249 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5633cd1782d0 con 0x5633cd28bf90 2024-03-18T14:56:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.850+0000 7f6a9b7fe640 1 --2- 172.21.15.76:0/4030492249 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5633cd28bf90 0x5633cd28e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.850+0000 7f6a9b7fe640 1 --2- 172.21.15.76:0/4030492249 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5633cd28bf90 0x5633cd28e3b0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffd0c4a2c6fc18a8 server_cookie=db2c7dd69f44d675 in_seq=0 out_seq=0 2024-03-18T14:56:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.850+0000 7f6a7b7fe640 10 client.16110 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:56:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6a7b7fe640 1 -- 172.21.15.76:0/4030492249 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5633cd1782d0 con 0x5633cd28bf90 2024-03-18T14:56:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6a7b7fe640 10 client.16110 handle_command_reply: tid=0 2024-03-18T14:56:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa5386740 1 client.16110 shutdown 2024-03-18T14:56:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa5386740 1 -- 172.21.15.76:0/4030492249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd215820 msgr2=0x5633cd219340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa5386740 1 --2- 172.21.15.76:0/4030492249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd215820 0x5633cd219340 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6a940426b0 tx=0x7f6a94005a00 comp rx=0 tx=0).stop 2024-03-18T14:56:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa5386740 1 -- 172.21.15.76:0/4030492249 shutdown_connections 2024-03-18T14:56:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa5386740 1 --2- 172.21.15.76:0/4030492249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5633cd20ca50 0x5633cd218e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa5386740 1 --2- 172.21.15.76:0/4030492249 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5633cd28bf90 0x5633cd28e3b0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa5386740 1 --2- 172.21.15.76:0/4030492249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5633cd215820 0x5633cd219340 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa5386740 1 --2- 172.21.15.76:0/4030492249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633cd20c130 0x5633cd21d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6a7b7fe640 0 client.16110 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:56:37.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa5386740 1 -- 172.21.15.76:0/4030492249 >> 172.21.15.76:0/4030492249 conn(0x5633cd206a50 msgr2=0x5633cd201a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:37.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa5386740 1 -- 172.21.15.76:0/4030492249 shutdown_connections 2024-03-18T14:56:37.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa5386740 1 -- 172.21.15.76:0/4030492249 wait complete. 2024-03-18T14:56:37.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa5386740 20 client.16110 trim_cache size 0 max 16384 2024-03-18T14:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa2797640 1 -- 172.21.15.76:0/1493984344 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f6a840a2a40 msgr2=0x7f6a840a4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa2797640 1 --2- 172.21.15.76:0/1493984344 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f6a840a2a40 0x7f6a840a4f00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x5633cd20bbe0 tx=0x7f6a8c02bc00 comp rx=0 tx=0).stop 2024-03-18T14:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa2797640 1 -- 172.21.15.76:0/1493984344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a9c068e60 msgr2=0x7f6a9c10db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa2797640 1 --2- 172.21.15.76:0/1493984344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a9c068e60 0x7f6a9c10db10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c037040 tx=0x7f6a8c0344c0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa2797640 1 -- 172.21.15.76:0/1493984344 shutdown_connections 2024-03-18T14:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa2797640 1 --2- 172.21.15.76:0/1493984344 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f6a840a2a40 0x7f6a840a4f00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa2797640 1 --2- 172.21.15.76:0/1493984344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a9c069780 0x7f6a9c109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa2797640 1 --2- 172.21.15.76:0/1493984344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a9c10e4e0 0x7f6a9c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa2797640 1 --2- 172.21.15.76:0/1493984344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a9c068e60 0x7f6a9c10db10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa2797640 1 -- 172.21.15.76:0/1493984344 >> 172.21.15.76:0/1493984344 conn(0x7f6a9c077530 msgr2=0x7f6a9c10f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa2797640 1 -- 172.21.15.76:0/1493984344 shutdown_connections 2024-03-18T14:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:37.915+0000 7f6aa2797640 1 -- 172.21.15.76:0/1493984344 wait complete. 2024-03-18T14:56:37.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:37 smithi174 ceph-mon[36787]: pgmap v107: 97 pgs: 97 active+clean; 95 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 258 op/s 2024-03-18T14:56:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:37 smithi077 ceph-mon[35865]: pgmap v107: 97 pgs: 97 active+clean; 95 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 258 op/s 2024-03-18T14:56:38.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:37 smithi076 ceph-mon[32205]: pgmap v107: 97 pgs: 97 active+clean; 95 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 258 op/s 2024-03-18T14:56:38.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 301509 10.40 MB/sec execute 39 sec latency 33.195 ms 2024-03-18T14:56:39.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 302706 10.24 MB/sec execute 40 sec latency 23.757 ms 2024-03-18T14:56:39.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:39 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:56:39.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:39 smithi076 ceph-mon[32205]: pgmap v108: 97 pgs: 97 active+clean; 97 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.7 MiB/s wr, 417 op/s 2024-03-18T14:56:39.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:39 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:56:39.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:39 smithi174 ceph-mon[36787]: pgmap v108: 97 pgs: 97 active+clean; 97 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.7 MiB/s wr, 417 op/s 2024-03-18T14:56:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:39 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:56:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:39 smithi077 ceph-mon[35865]: pgmap v108: 97 pgs: 97 active+clean; 97 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.7 MiB/s wr, 417 op/s 2024-03-18T14:56:40.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 305376 10.40 MB/sec execute 41 sec latency 21.556 ms 2024-03-18T14:56:41.738 INFO:tasks.workunit.client.0.smithi076.stdout: 1 306724 10.28 MB/sec execute 42 sec latency 77.938 ms 2024-03-18T14:56:41.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:41 smithi174 ceph-mon[36787]: pgmap v109: 97 pgs: 97 active+clean; 97 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 271 op/s 2024-03-18T14:56:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:41 smithi077 ceph-mon[35865]: pgmap v109: 97 pgs: 97 active+clean; 97 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 271 op/s 2024-03-18T14:56:42.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:41 smithi076 ceph-mon[32205]: pgmap v109: 97 pgs: 97 active+clean; 97 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 271 op/s 2024-03-18T14:56:42.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 309005 10.37 MB/sec execute 43 sec latency 12.994 ms 2024-03-18T14:56:43.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 310997 10.29 MB/sec execute 44 sec latency 32.198 ms 2024-03-18T14:56:43.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:43 smithi174 ceph-mon[36787]: pgmap v110: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 373 op/s 2024-03-18T14:56:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:43 smithi077 ceph-mon[35865]: pgmap v110: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 373 op/s 2024-03-18T14:56:44.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:43 smithi076 ceph-mon[32205]: pgmap v110: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 373 op/s 2024-03-18T14:56:44.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 312664 10.33 MB/sec execute 45 sec latency 19.576 ms 2024-03-18T14:56:45.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 315075 10.38 MB/sec execute 46 sec latency 17.494 ms 2024-03-18T14:56:45.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:45 smithi174 ceph-mon[36787]: pgmap v111: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 260 op/s 2024-03-18T14:56:45.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:45 smithi077 ceph-mon[35865]: pgmap v111: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 260 op/s 2024-03-18T14:56:46.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:45 smithi076 ceph-mon[32205]: pgmap v111: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 260 op/s 2024-03-18T14:56:46.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 316459 10.30 MB/sec execute 47 sec latency 16.990 ms 2024-03-18T14:56:47.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 319192 10.44 MB/sec execute 48 sec latency 14.466 ms 2024-03-18T14:56:47.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:47 smithi076 ceph-mon[32205]: pgmap v112: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 260 op/s 2024-03-18T14:56:47.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:47 smithi174 ceph-mon[36787]: pgmap v112: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 260 op/s 2024-03-18T14:56:47.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:47 smithi077 ceph-mon[35865]: pgmap v112: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 260 op/s 2024-03-18T14:56:48.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 320306 10.31 MB/sec execute 49 sec latency 25.808 ms 2024-03-18T14:56:49.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 322937 10.42 MB/sec execute 50 sec latency 70.704 ms 2024-03-18T14:56:49.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:49 smithi174 ceph-mon[36787]: pgmap v113: 97 pgs: 97 active+clean; 99 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.4 MiB/s wr, 374 op/s 2024-03-18T14:56:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:49 smithi077 ceph-mon[35865]: pgmap v113: 97 pgs: 97 active+clean; 99 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.4 MiB/s wr, 374 op/s 2024-03-18T14:56:50.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:49 smithi076 ceph-mon[32205]: pgmap v113: 97 pgs: 97 active+clean; 99 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.4 MiB/s wr, 374 op/s 2024-03-18T14:56:50.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 324265 10.34 MB/sec execute 51 sec latency 16.739 ms 2024-03-18T14:56:51.012 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.010+0000 7fac612f8640 1 -- 172.21.15.76:0/2318278126 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac5c1012f0 msgr2=0x7fac5c10d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.010+0000 7fac612f8640 1 --2- 172.21.15.76:0/2318278126 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac5c1012f0 0x7fac5c10d510 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fac54009ca0 tx=0x7fac5402c0f0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.010+0000 7fac612f8640 1 -- 172.21.15.76:0/2318278126 shutdown_connections 2024-03-18T14:56:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.010+0000 7fac612f8640 1 --2- 172.21.15.76:0/2318278126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac5c100930 0x7fac5c100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.010+0000 7fac612f8640 1 --2- 172.21.15.76:0/2318278126 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac5c1012f0 0x7fac5c10d510 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.010+0000 7fac612f8640 1 --2- 172.21.15.76:0/2318278126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac5c105680 0x7fac5c105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.010+0000 7fac612f8640 1 -- 172.21.15.76:0/2318278126 >> 172.21.15.76:0/2318278126 conn(0x7fac5c077530 msgr2=0x7fac5c077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.010+0000 7fac612f8640 1 -- 172.21.15.76:0/2318278126 shutdown_connections 2024-03-18T14:56:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.010+0000 7fac612f8640 1 -- 172.21.15.76:0/2318278126 wait complete. 2024-03-18T14:56:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.010+0000 7fac612f8640 1 Processor -- start 2024-03-18T14:56:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.010+0000 7fac612f8640 1 -- start start 2024-03-18T14:56:51.014 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac612f8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac5c100930 0x7fac5c10e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.014 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac612f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac5c1012f0 0x7fac5c10e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.014 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac612f8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac5c105680 0x7fac5c10ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.014 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac612f8640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c111cc0 con 0x7fac5c105680 2024-03-18T14:56:51.014 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac612f8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c111b40 con 0x7fac5c1012f0 2024-03-18T14:56:51.014 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac612f8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c111e40 con 0x7fac5c100930 2024-03-18T14:56:51.014 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac5b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac5c105680 0x7fac5c10ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.014 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac5affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac5c100930 0x7fac5c10e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.014 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac5a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac5c1012f0 0x7fac5c10e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.014 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac5a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac5c1012f0 0x7fac5c10e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:60098/0 (socket says 172.21.15.76:60098) 2024-03-18T14:56:51.014 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac5b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac5c105680 0x7fac5c10ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52086/0 (socket says 172.21.15.76:52086) 2024-03-18T14:56:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac5affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac5c100930 0x7fac5c10e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:38944/0 (socket says 172.21.15.76:38944) 2024-03-18T14:56:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac5a7fc640 1 -- 172.21.15.76:0/3769807313 learned_addr learned my addr 172.21.15.76:0/3769807313 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac5a7fc640 1 -- 172.21.15.76:0/3769807313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac5c100930 msgr2=0x7fac5c10e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac5a7fc640 1 --2- 172.21.15.76:0/3769807313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac5c100930 0x7fac5c10e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac5a7fc640 1 -- 172.21.15.76:0/3769807313 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac5c105680 msgr2=0x7fac5c10ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac5a7fc640 1 --2- 172.21.15.76:0/3769807313 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac5c105680 0x7fac5c10ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac5a7fc640 1 -- 172.21.15.76:0/3769807313 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5c1a65c0 con 0x7fac5c1012f0 2024-03-18T14:56:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac5b7fe640 1 --2- 172.21.15.76:0/3769807313 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac5c105680 0x7fac5c10ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:56:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.011+0000 7fac5a7fc640 1 --2- 172.21.15.76:0/3769807313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac5c1012f0 0x7fac5c10e590 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fac5000e9f0 tx=0x7fac5000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.012+0000 7fac43fff640 1 -- 172.21.15.76:0/3769807313 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac5000cd80 con 0x7fac5c1012f0 2024-03-18T14:56:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.012+0000 7fac43fff640 1 -- 172.21.15.76:0/3769807313 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac50005500 con 0x7fac5c1012f0 2024-03-18T14:56:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.012+0000 7fac612f8640 1 -- 172.21.15.76:0/3769807313 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5c1a68b0 con 0x7fac5c1012f0 2024-03-18T14:56:51.016 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.012+0000 7fac43fff640 1 -- 172.21.15.76:0/3769807313 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50010690 con 0x7fac5c1012f0 2024-03-18T14:56:51.016 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.012+0000 7fac612f8640 1 -- 172.21.15.76:0/3769807313 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5c1a6db0 con 0x7fac5c1012f0 2024-03-18T14:56:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.014+0000 7fac43fff640 1 -- 172.21.15.76:0/3769807313 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fac500046c0 con 0x7fac5c1012f0 2024-03-18T14:56:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.014+0000 7fac63fbf740 1 Processor -- start 2024-03-18T14:56:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.014+0000 7fac63fbf740 1 -- start start 2024-03-18T14:56:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.014+0000 7fac43fff640 1 --2- 172.21.15.76:0/3769807313 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fac440a2a10 0x7fac440a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.014+0000 7fac63fbf740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bf4130 0x557ab3bf4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.014+0000 7fac63fbf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4a50 0x557ab3bfd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac63fbf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bfd820 0x557ab3bffc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac63fbf740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557ab3b58a40 con 0x557ab3bf4130 2024-03-18T14:56:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac63fbf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x557ab3b01920 con 0x557ab3bf4a50 2024-03-18T14:56:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac63fbf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557ab3b54ae0 con 0x557ab3bfd820 2024-03-18T14:56:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac5b7fe640 1 --2- 172.21.15.76:0/3769807313 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fac440a2a10 0x7fac440a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac5affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bf4130 0x557ab3bf4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac5a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4a50 0x557ab3bfd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac43fff640 1 -- 172.21.15.76:0/3769807313 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7fac500cf270 con 0x7fac5c1012f0 2024-03-18T14:56:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac5b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bfd820 0x557ab3bffc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac5a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4a50 0x557ab3bfd2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:60108/0 (socket says 172.21.15.76:60108) 2024-03-18T14:56:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac5a7fc640 1 -- 172.21.15.76:0/705596411 learned_addr learned my addr 172.21.15.76:0/705596411 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac5a7fc640 1 -- 172.21.15.76:0/705596411 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bfd820 msgr2=0x557ab3bffc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac5a7fc640 1 --2- 172.21.15.76:0/705596411 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bfd820 0x557ab3bffc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac5a7fc640 1 -- 172.21.15.76:0/705596411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bf4130 msgr2=0x557ab3bf4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac5a7fc640 1 --2- 172.21.15.76:0/705596411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bf4130 0x557ab3bf4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac5a7fc640 1 -- 172.21.15.76:0/705596411 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ab3b19460 con 0x557ab3bf4a50 2024-03-18T14:56:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac5b7fe640 1 --2- 172.21.15.76:0/3769807313 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fac440a2a10 0x7fac440a4ed0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fac54004af0 tx=0x7fac540323c0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac5a7fc640 1 --2- 172.21.15.76:0/705596411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4a50 0x557ab3bfd2e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fac500a3b30 tx=0x7fac50097c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51e00a2d4ff9943c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac417fa640 1 -- 172.21.15.76:0/705596411 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac5009b8c0 con 0x557ab3bf4a50 2024-03-18T14:56:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac417fa640 1 -- 172.21.15.76:0/705596411 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac5009fa10 con 0x557ab3bf4a50 2024-03-18T14:56:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.015+0000 7fac417fa640 1 -- 172.21.15.76:0/705596411 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac500980c0 con 0x557ab3bf4a50 2024-03-18T14:56:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.016+0000 7fac63fbf740 1 -- 172.21.15.76:0/705596411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4a50 msgr2=0x557ab3bfd2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.016+0000 7fac63fbf740 1 --2- 172.21.15.76:0/705596411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4a50 0x557ab3bfd2e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fac500a3b30 tx=0x7fac50097c90 comp rx=0 tx=0).stop 2024-03-18T14:56:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.016+0000 7fac63fbf740 1 -- 172.21.15.76:0/705596411 shutdown_connections 2024-03-18T14:56:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.016+0000 7fac63fbf740 1 --2- 172.21.15.76:0/705596411 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bfd820 0x557ab3bffc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.016+0000 7fac63fbf740 1 --2- 172.21.15.76:0/705596411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4a50 0x557ab3bfd2e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.016+0000 7fac63fbf740 1 --2- 172.21.15.76:0/705596411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bf4130 0x557ab3bf4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.016+0000 7fac63fbf740 1 -- 172.21.15.76:0/705596411 >> 172.21.15.76:0/705596411 conn(0x557ab3beea50 msgr2=0x557ab3beee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.016+0000 7fac63fbf740 1 -- 172.21.15.76:0/705596411 shutdown_connections 2024-03-18T14:56:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.016+0000 7fac63fbf740 1 -- 172.21.15.76:0/705596411 wait complete. 2024-03-18T14:56:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.016+0000 7fac63fbf740 1 -- start start 2024-03-18T14:56:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac63fbf740 1 Processor -- start 2024-03-18T14:56:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac63fbf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 0x557ab3bfa880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac63fbf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bf4a50 0x557ab3bfadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac63fbf740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bfd820 0x557ab3bfb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac63fbf740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557ab3c00450 con 0x557ab3bfd820 2024-03-18T14:56:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac63fbf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x557ab3b14d70 con 0x557ab3bf4130 2024-03-18T14:56:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac63fbf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557ab3c005d0 con 0x557ab3bf4a50 2024-03-18T14:56:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac5affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 0x557ab3bfa880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac5a7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bf4a50 0x557ab3bfadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac5affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 0x557ab3bfa880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:60118/0 (socket says 172.21.15.76:60118) 2024-03-18T14:56:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac5a7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bf4a50 0x557ab3bfadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:38966/0 (socket says 172.21.15.76:38966) 2024-03-18T14:56:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac5affd640 1 -- 172.21.15.76:0/4069137015 learned_addr learned my addr 172.21.15.76:0/4069137015 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac5b7fe640 1 --2- 172.21.15.76:0/4069137015 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bfd820 0x557ab3bfb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac63fbf740 10 client.0 _ll_register_callbacks cb 0x557ab386ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:56:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac5affd640 1 -- 172.21.15.76:0/4069137015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bf4a50 msgr2=0x557ab3bfadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac5affd640 1 --2- 172.21.15.76:0/4069137015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bf4a50 0x557ab3bfadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac5affd640 1 -- 172.21.15.76:0/4069137015 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bfd820 msgr2=0x557ab3bfb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac5affd640 1 --2- 172.21.15.76:0/4069137015 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bfd820 0x557ab3bfb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.017+0000 7fac5affd640 1 -- 172.21.15.76:0/4069137015 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ab386d210 con 0x557ab3bf4130 2024-03-18T14:56:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.018+0000 7fac5affd640 1 --2- 172.21.15.76:0/4069137015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 0x557ab3bfa880 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fac4c012770 tx=0x7fac4c012c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9058d808b03d2b90 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.018+0000 7fac3eff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:56:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.018+0000 7fac3eff5640 1 -- 172.21.15.76:0/4069137015 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c00f920 con 0x557ab3bf4130 2024-03-18T14:56:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.018+0000 7fac3eff5640 1 -- 172.21.15.76:0/4069137015 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7fac4c00fac0 con 0x557ab3bf4130 2024-03-18T14:56:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.018+0000 7fac63fbf740 1 -- 172.21.15.76:0/4069137015 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac5009fa10 con 0x557ab3bf4130 2024-03-18T14:56:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.018+0000 7fac3eff5640 1 -- 172.21.15.76:0/4069137015 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fac4c00fcc0 con 0x557ab3bf4130 2024-03-18T14:56:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.018+0000 7fac63fbf740 10 client.25979 fetch_fsmap learned FSMap version 35 2024-03-18T14:56:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.018+0000 7fac63fbf740 1 -- 172.21.15.76:0/4069137015 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557ab3939620 con 0x557ab3bf4130 2024-03-18T14:56:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.018+0000 7fac3eff5640 1 -- 172.21.15.76:0/4069137015 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac5009fa10 con 0x557ab3bf4130 2024-03-18T14:56:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.018+0000 7fac3eff5640 1 -- 172.21.15.76:0/4069137015 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 35) ==== 8338+0+0 (secure 0 0 0) 0x7fac4c01c420 con 0x557ab3bf4130 2024-03-18T14:56:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.018+0000 7fac63fbf740 10 client.25979 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T14:56:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.018+0000 7fac63fbf740 10 client.25979 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:56:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.018+0000 7fac63fbf740 20 client.25979 populate_metadata read hostname 'smithi076' 2024-03-18T14:56:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.018+0000 7fac63fbf740 1 --2- 172.21.15.76:0/4069137015 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557ab3c16270 0x557ab3c18660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.028 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.018+0000 7fac63fbf740 4 client.25979 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:56:51.028 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.018+0000 7fac63fbf740 1 -- 172.21.15.76:0/4069137015 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ab3b65f80 con 0x557ab3c16270 2024-03-18T14:56:51.028 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.018+0000 7fac5a7fc640 1 --2- 172.21.15.76:0/4069137015 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557ab3c16270 0x557ab3c18660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.028 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.019+0000 7fac3eff5640 10 client.25979 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:56:51.028 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.019+0000 7fac5a7fc640 1 --2- 172.21.15.76:0/4069137015 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557ab3c16270 0x557ab3c18660 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5b84ea8b78ab3fa server_cookie=1670f1d5486680e3 in_seq=0 out_seq=0 2024-03-18T14:56:51.028 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.019+0000 7fac3eff5640 1 -- 172.21.15.76:0/4069137015 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557ab3b65f80 con 0x557ab3c16270 2024-03-18T14:56:51.028 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.019+0000 7fac3eff5640 10 client.25979 handle_command_reply: tid=0 2024-03-18T14:56:51.028 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.019+0000 7fac63fbf740 1 client.25979 shutdown 2024-03-18T14:56:51.028 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.020+0000 7fac63fbf740 1 -- 172.21.15.76:0/4069137015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 msgr2=0x557ab3bfa880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:51.028 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.020+0000 7fac63fbf740 1 --2- 172.21.15.76:0/4069137015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 0x557ab3bfa880 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fac4c012770 tx=0x7fac4c012c30 comp rx=0 tx=0).stop 2024-03-18T14:56:51.028 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.020+0000 7fac63fbf740 1 -- 172.21.15.76:0/4069137015 shutdown_connections 2024-03-18T14:56:51.028 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.020+0000 7fac63fbf740 1 --2- 172.21.15.76:0/4069137015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bf4a50 0x557ab3bfadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.020+0000 7fac63fbf740 1 --2- 172.21.15.76:0/4069137015 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557ab3c16270 0x557ab3c18660 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.020+0000 7fac63fbf740 1 --2- 172.21.15.76:0/4069137015 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bfd820 0x557ab3bfb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.020+0000 7fac63fbf740 1 --2- 172.21.15.76:0/4069137015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 0x557ab3bfa880 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.020+0000 7fac3eff5640 0 client.25979 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:56:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.020+0000 7fac63fbf740 1 -- 172.21.15.76:0/4069137015 >> 172.21.15.76:0/4069137015 conn(0x557ab3beea50 msgr2=0x557ab3bf07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.020+0000 7fac63fbf740 1 -- 172.21.15.76:0/4069137015 shutdown_connections 2024-03-18T14:56:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.020+0000 7fac63fbf740 1 -- 172.21.15.76:0/4069137015 wait complete. 2024-03-18T14:56:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.020+0000 7fac63fbf740 20 client.25979 trim_cache size 0 max 16384 2024-03-18T14:56:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac63fbf740 1 Processor -- start 2024-03-18T14:56:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac63fbf740 1 -- start start 2024-03-18T14:56:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac63fbf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 0x557ab3bfcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac63fbf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bf4a50 0x557ab3c091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac63fbf740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bfd820 0x557ab3bf0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac63fbf740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c1ada20 con 0x557ab3bfd820 2024-03-18T14:56:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac63fbf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c1ab2f0 con 0x557ab3bf4130 2024-03-18T14:56:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac63fbf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c1a9140 con 0x557ab3bf4a50 2024-03-18T14:56:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac5b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bfd820 0x557ab3bf0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac5affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 0x557ab3bfcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac5affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 0x557ab3bfcfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:60120/0 (socket says 172.21.15.76:60120) 2024-03-18T14:56:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac5a7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bf4a50 0x557ab3c091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac5b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bfd820 0x557ab3bf0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52108/0 (socket says 172.21.15.76:52108) 2024-03-18T14:56:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac5affd640 1 -- 172.21.15.76:0/2068353627 learned_addr learned my addr 172.21.15.76:0/2068353627 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac5affd640 1 -- 172.21.15.76:0/2068353627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bf4a50 msgr2=0x557ab3c091f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac5affd640 1 --2- 172.21.15.76:0/2068353627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bf4a50 0x557ab3c091f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac5affd640 1 -- 172.21.15.76:0/2068353627 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bfd820 msgr2=0x557ab3bf0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac5affd640 1 --2- 172.21.15.76:0/2068353627 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bfd820 0x557ab3bf0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac5affd640 1 -- 172.21.15.76:0/2068353627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ab3bffc10 con 0x557ab3bf4130 2024-03-18T14:56:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.036+0000 7fac5affd640 1 --2- 172.21.15.76:0/2068353627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 0x557ab3bfcfb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fac4c01c910 tx=0x7fac4c009ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed462b7e7e8e1396 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.037+0000 7fac40ff9640 1 -- 172.21.15.76:0/2068353627 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c029070 con 0x557ab3bf4130 2024-03-18T14:56:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.037+0000 7fac40ff9640 1 -- 172.21.15.76:0/2068353627 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac4c00f920 con 0x557ab3bf4130 2024-03-18T14:56:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.037+0000 7fac40ff9640 1 -- 172.21.15.76:0/2068353627 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c005570 con 0x557ab3bf4130 2024-03-18T14:56:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.037+0000 7fac63fbf740 1 -- 172.21.15.76:0/2068353627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 msgr2=0x557ab3bfcfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.037+0000 7fac63fbf740 1 --2- 172.21.15.76:0/2068353627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 0x557ab3bfcfb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fac4c01c910 tx=0x7fac4c009ee0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.037+0000 7fac63fbf740 1 -- 172.21.15.76:0/2068353627 shutdown_connections 2024-03-18T14:56:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.037+0000 7fac63fbf740 1 --2- 172.21.15.76:0/2068353627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bf4a50 0x557ab3c091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.037+0000 7fac63fbf740 1 --2- 172.21.15.76:0/2068353627 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bfd820 0x557ab3bf0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.037+0000 7fac63fbf740 1 --2- 172.21.15.76:0/2068353627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 0x557ab3bfcfb0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.037+0000 7fac63fbf740 1 -- 172.21.15.76:0/2068353627 >> 172.21.15.76:0/2068353627 conn(0x557ab3beea50 msgr2=0x557ab3befe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.037+0000 7fac63fbf740 1 -- 172.21.15.76:0/2068353627 shutdown_connections 2024-03-18T14:56:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.037+0000 7fac63fbf740 1 -- 172.21.15.76:0/2068353627 wait complete. 2024-03-18T14:56:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac63fbf740 1 -- start start 2024-03-18T14:56:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac63fbf740 1 Processor -- start 2024-03-18T14:56:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac63fbf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 0x557ab3c01250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac63fbf740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bf4a50 0x557ab3c01790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac63fbf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bfd820 0x557ab3c01cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac63fbf740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c1a97b0 con 0x557ab3bf4a50 2024-03-18T14:56:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac63fbf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c1aac20 con 0x557ab3bf4130 2024-03-18T14:56:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac63fbf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c1a9e80 con 0x557ab3bfd820 2024-03-18T14:56:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac63fbf740 10 client.0 _ll_register_callbacks cb 0x7fac4c009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:56:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac5a7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bf4a50 0x557ab3c01790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac5a7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bf4a50 0x557ab3c01790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52124/0 (socket says 172.21.15.76:52124) 2024-03-18T14:56:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac5a7fc640 1 -- 172.21.15.76:0/973590321 learned_addr learned my addr 172.21.15.76:0/973590321 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:56:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac5b7fe640 1 --2- 172.21.15.76:0/973590321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bfd820 0x557ab3c01cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac5affd640 1 --2- 172.21.15.76:0/973590321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 0x557ab3c01250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac5a7fc640 1 -- 172.21.15.76:0/973590321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bfd820 msgr2=0x557ab3c01cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac5a7fc640 1 --2- 172.21.15.76:0/973590321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bfd820 0x557ab3c01cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac5a7fc640 1 -- 172.21.15.76:0/973590321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 msgr2=0x557ab3c01250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:56:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac5a7fc640 1 --2- 172.21.15.76:0/973590321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 0x557ab3c01250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac5a7fc640 1 -- 172.21.15.76:0/973590321 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ab3c7b6d0 con 0x557ab3bf4a50 2024-03-18T14:56:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac5a7fc640 1 --2- 172.21.15.76:0/973590321 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bf4a50 0x557ab3c01790 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fac5000c6a0 tx=0x7fac50097b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e4cf1be020f23e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:56:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.038+0000 7fac3eff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:56:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.039+0000 7fac3eff5640 1 -- 172.21.15.76:0/973590321 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50083070 con 0x557ab3bf4a50 2024-03-18T14:56:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.039+0000 7fac3eff5640 1 -- 172.21.15.76:0/973590321 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7fac500ccc30 con 0x557ab3bf4a50 2024-03-18T14:56:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.039+0000 7fac63fbf740 1 -- 172.21.15.76:0/973590321 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac4c00f920 con 0x557ab3bf4a50 2024-03-18T14:56:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.039+0000 7fac3eff5640 1 -- 172.21.15.76:0/973590321 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fac5009fa10 con 0x557ab3bf4a50 2024-03-18T14:56:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.039+0000 7fac63fbf740 10 client.16128 fetch_fsmap learned FSMap version 35 2024-03-18T14:56:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.039+0000 7fac63fbf740 1 -- 172.21.15.76:0/973590321 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557ab3c7c9c0 con 0x557ab3bf4a50 2024-03-18T14:56:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.039+0000 7fac3eff5640 1 -- 172.21.15.76:0/973590321 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac4c00f920 con 0x557ab3bf4a50 2024-03-18T14:56:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.039+0000 7fac3eff5640 1 -- 172.21.15.76:0/973590321 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 35) ==== 8338+0+0 (secure 0 0 0) 0x7fac500a3bd0 con 0x557ab3bf4a50 2024-03-18T14:56:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.039+0000 7fac63fbf740 10 client.16128 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T14:56:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.039+0000 7fac63fbf740 10 client.16128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:56:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.039+0000 7fac63fbf740 20 client.16128 populate_metadata read hostname 'smithi076' 2024-03-18T14:56:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.039+0000 7fac63fbf740 1 --2- 172.21.15.76:0/973590321 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557ab3c7cc60 0x557ab3c7f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:56:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.039+0000 7fac63fbf740 4 client.16128 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:56:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.039+0000 7fac63fbf740 1 -- 172.21.15.76:0/973590321 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557ab3b602d0 con 0x557ab3c7cc60 2024-03-18T14:56:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.039+0000 7fac5affd640 1 --2- 172.21.15.76:0/973590321 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557ab3c7cc60 0x557ab3c7f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:56:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.040+0000 7fac3eff5640 10 client.16128 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:56:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.040+0000 7fac5affd640 1 --2- 172.21.15.76:0/973590321 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557ab3c7cc60 0x557ab3c7f080 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ec757d147937f0 server_cookie=1927af8d944e82db in_seq=0 out_seq=0 2024-03-18T14:56:51.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.116+0000 7fac3eff5640 1 -- 172.21.15.76:0/973590321 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557ab3b602d0 con 0x557ab3c7cc60 2024-03-18T14:56:51.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.116+0000 7fac3eff5640 10 client.16128 handle_command_reply: tid=0 2024-03-18T14:56:51.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.116+0000 7fac63fbf740 1 client.16128 shutdown 2024-03-18T14:56:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.116+0000 7fac63fbf740 1 -- 172.21.15.76:0/973590321 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bf4a50 msgr2=0x557ab3c01790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.116+0000 7fac63fbf740 1 --2- 172.21.15.76:0/973590321 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bf4a50 0x557ab3c01790 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fac5000c6a0 tx=0x7fac50097b60 comp rx=0 tx=0).stop 2024-03-18T14:56:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.116+0000 7fac63fbf740 1 -- 172.21.15.76:0/973590321 shutdown_connections 2024-03-18T14:56:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.116+0000 7fac63fbf740 1 --2- 172.21.15.76:0/973590321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ab3bfd820 0x557ab3c01cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.116+0000 7fac63fbf740 1 --2- 172.21.15.76:0/973590321 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x557ab3c7cc60 0x557ab3c7f080 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.116+0000 7fac3eff5640 0 client.16128 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:56:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.116+0000 7fac63fbf740 1 --2- 172.21.15.76:0/973590321 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557ab3bf4a50 0x557ab3c01790 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.116+0000 7fac63fbf740 1 --2- 172.21.15.76:0/973590321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557ab3bf4130 0x557ab3c01250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.116+0000 7fac63fbf740 1 -- 172.21.15.76:0/973590321 >> 172.21.15.76:0/973590321 conn(0x557ab3beea50 msgr2=0x557ab3bedbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.116+0000 7fac63fbf740 1 -- 172.21.15.76:0/973590321 shutdown_connections 2024-03-18T14:56:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.116+0000 7fac63fbf740 1 -- 172.21.15.76:0/973590321 wait complete. 2024-03-18T14:56:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.116+0000 7fac63fbf740 20 client.16128 trim_cache size 0 max 16384 2024-03-18T14:56:51.119 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:56:51.119 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:56:51.119 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:56:51.119 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:56:51.120 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.117+0000 7fac612f8640 1 -- 172.21.15.76:0/3769807313 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fac440a2a10 msgr2=0x7fac440a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:51.120 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.117+0000 7fac612f8640 1 --2- 172.21.15.76:0/3769807313 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fac440a2a10 0x7fac440a4ed0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fac54004af0 tx=0x7fac540323c0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.120 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.117+0000 7fac612f8640 1 -- 172.21.15.76:0/3769807313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac5c1012f0 msgr2=0x7fac5c10e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:56:51.120 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.117+0000 7fac612f8640 1 --2- 172.21.15.76:0/3769807313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac5c1012f0 0x7fac5c10e590 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fac5000e9f0 tx=0x7fac5000edf0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.120 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.117+0000 7fac612f8640 1 -- 172.21.15.76:0/3769807313 shutdown_connections 2024-03-18T14:56:51.120 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.117+0000 7fac612f8640 1 --2- 172.21.15.76:0/3769807313 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fac440a2a10 0x7fac440a4ed0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.120 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.117+0000 7fac612f8640 1 --2- 172.21.15.76:0/3769807313 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac5c105680 0x7fac5c10ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.120 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.117+0000 7fac612f8640 1 --2- 172.21.15.76:0/3769807313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac5c1012f0 0x7fac5c10e590 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.120 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.117+0000 7fac612f8640 1 --2- 172.21.15.76:0/3769807313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac5c100930 0x7fac5c10e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:56:51.120 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.117+0000 7fac612f8640 1 -- 172.21.15.76:0/3769807313 >> 172.21.15.76:0/3769807313 conn(0x7fac5c077530 msgr2=0x7fac5c0fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:56:51.120 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.117+0000 7fac612f8640 1 -- 172.21.15.76:0/3769807313 shutdown_connections 2024-03-18T14:56:51.121 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:56:51.117+0000 7fac612f8640 1 -- 172.21.15.76:0/3769807313 wait complete. 2024-03-18T14:56:51.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 326591 10.40 MB/sec execute 52 sec latency 40.954 ms 2024-03-18T14:56:51.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:51 smithi174 ceph-mon[36787]: pgmap v114: 97 pgs: 97 active+clean; 99 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 215 op/s 2024-03-18T14:56:51.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:51 smithi077 ceph-mon[35865]: pgmap v114: 97 pgs: 97 active+clean; 99 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 215 op/s 2024-03-18T14:56:52.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:51 smithi076 ceph-mon[32205]: pgmap v114: 97 pgs: 97 active+clean; 99 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 215 op/s 2024-03-18T14:56:52.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 327996 10.31 MB/sec execute 53 sec latency 16.114 ms 2024-03-18T14:56:53.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 330254 10.38 MB/sec execute 54 sec latency 10.947 ms 2024-03-18T14:56:53.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:53 smithi174 ceph-mon[36787]: pgmap v115: 97 pgs: 97 active+clean; 96 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 367 op/s 2024-03-18T14:56:53.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:53 smithi077 ceph-mon[35865]: pgmap v115: 97 pgs: 97 active+clean; 96 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 367 op/s 2024-03-18T14:56:54.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:53 smithi076 ceph-mon[32205]: pgmap v115: 97 pgs: 97 active+clean; 96 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 367 op/s 2024-03-18T14:56:54.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 331948 10.29 MB/sec execute 55 sec latency 46.227 ms 2024-03-18T14:56:55.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 333893 10.35 MB/sec execute 56 sec latency 29.156 ms 2024-03-18T14:56:55.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:55 smithi174 ceph-mon[36787]: pgmap v116: 97 pgs: 97 active+clean; 96 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 265 op/s 2024-03-18T14:56:55.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:55 smithi077 ceph-mon[35865]: pgmap v116: 97 pgs: 97 active+clean; 96 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 265 op/s 2024-03-18T14:56:56.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:55 smithi076 ceph-mon[32205]: pgmap v116: 97 pgs: 97 active+clean; 96 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 265 op/s 2024-03-18T14:56:56.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 335928 10.30 MB/sec execute 57 sec latency 16.678 ms 2024-03-18T14:56:57.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 337540 10.32 MB/sec execute 58 sec latency 12.553 ms 2024-03-18T14:56:57.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:57 smithi174 ceph-mon[36787]: pgmap v117: 97 pgs: 97 active+clean; 96 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 265 op/s 2024-03-18T14:56:57.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:57 smithi077 ceph-mon[35865]: pgmap v117: 97 pgs: 97 active+clean; 96 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 265 op/s 2024-03-18T14:56:58.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:57 smithi076 ceph-mon[32205]: pgmap v117: 97 pgs: 97 active+clean; 96 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 265 op/s 2024-03-18T14:56:58.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 339974 10.36 MB/sec execute 59 sec latency 16.751 ms 2024-03-18T14:56:59.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 341138 10.30 MB/sec execute 60 sec latency 47.584 ms 2024-03-18T14:56:59.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:56:59 smithi174 ceph-mon[36787]: pgmap v118: 97 pgs: 97 active+clean; 101 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 378 op/s 2024-03-18T14:56:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:56:59 smithi077 ceph-mon[35865]: pgmap v118: 97 pgs: 97 active+clean; 101 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 378 op/s 2024-03-18T14:57:00.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:56:59 smithi076 ceph-mon[32205]: pgmap v118: 97 pgs: 97 active+clean; 101 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 378 op/s 2024-03-18T14:57:00.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 342887 10.24 MB/sec execute 61 sec latency 36.160 ms 2024-03-18T14:57:01.739 INFO:tasks.workunit.client.0.smithi076.stdout: 1 344619 10.27 MB/sec execute 62 sec latency 11.278 ms 2024-03-18T14:57:01.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:01 smithi174 ceph-mon[36787]: pgmap v119: 97 pgs: 97 active+clean; 101 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 264 op/s 2024-03-18T14:57:01.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:01 smithi077 ceph-mon[35865]: pgmap v119: 97 pgs: 97 active+clean; 101 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 264 op/s 2024-03-18T14:57:02.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:01 smithi076 ceph-mon[32205]: pgmap v119: 97 pgs: 97 active+clean; 101 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 264 op/s 2024-03-18T14:57:02.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 346846 10.28 MB/sec execute 63 sec latency 16.529 ms 2024-03-18T14:57:03.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 348401 10.26 MB/sec execute 64 sec latency 14.057 ms 2024-03-18T14:57:03.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:03 smithi174 ceph-mon[36787]: pgmap v120: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 400 op/s 2024-03-18T14:57:03.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:03 smithi077 ceph-mon[35865]: pgmap v120: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 400 op/s 2024-03-18T14:57:04.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:03 smithi076 ceph-mon[32205]: pgmap v120: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 400 op/s 2024-03-18T14:57:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.224+0000 7f736d263640 1 -- 172.21.15.76:0/296174209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7368100930 msgr2=0x7f7368100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.224+0000 7f736d263640 1 --2- 172.21.15.76:0/296174209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7368100930 0x7f7368100db0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7350009890 tx=0x7f735002bb70 comp rx=0 tx=0).stop 2024-03-18T14:57:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.224+0000 7f736d263640 1 -- 172.21.15.76:0/296174209 shutdown_connections 2024-03-18T14:57:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.224+0000 7f736d263640 1 --2- 172.21.15.76:0/296174209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73681012f0 0x7f736810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.224+0000 7f736d263640 1 --2- 172.21.15.76:0/296174209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7368100930 0x7f7368100db0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.224+0000 7f736d263640 1 --2- 172.21.15.76:0/296174209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7368105680 0x7f7368105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.224+0000 7f736d263640 1 -- 172.21.15.76:0/296174209 >> 172.21.15.76:0/296174209 conn(0x7f7368077530 msgr2=0x7f7368077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.224+0000 7f736d263640 1 -- 172.21.15.76:0/296174209 shutdown_connections 2024-03-18T14:57:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.224+0000 7f736d263640 1 -- 172.21.15.76:0/296174209 wait complete. 2024-03-18T14:57:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.224+0000 7f736d263640 1 Processor -- start 2024-03-18T14:57:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.224+0000 7f736d263640 1 -- start start 2024-03-18T14:57:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f736d263640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7368100930 0x7f7368197020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f736d263640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73681012f0 0x7f7368197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f736d263640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7368105680 0x7f736819b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f736d263640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7368110ba0 con 0x7f73681012f0 2024-03-18T14:57:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f736d263640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7368110a20 con 0x7f7368100930 2024-03-18T14:57:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f736d263640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7368110d20 con 0x7f7368105680 2024-03-18T14:57:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f735ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73681012f0 0x7f7368197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f735ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73681012f0 0x7f7368197560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33970/0 (socket says 172.21.15.76:33970) 2024-03-18T14:57:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f735ffff640 1 -- 172.21.15.76:0/2230869582 learned_addr learned my addr 172.21.15.76:0/2230869582 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f7366ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7368100930 0x7f7368197020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f73677fe640 1 --2- 172.21.15.76:0/2230869582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7368105680 0x7f736819b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f735ffff640 1 -- 172.21.15.76:0/2230869582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7368105680 msgr2=0x7f736819b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:04.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f735ffff640 1 --2- 172.21.15.76:0/2230869582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7368105680 0x7f736819b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f735ffff640 1 -- 172.21.15.76:0/2230869582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7368100930 msgr2=0x7f7368197020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:04.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f735ffff640 1 --2- 172.21.15.76:0/2230869582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7368100930 0x7f7368197020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f735ffff640 1 -- 172.21.15.76:0/2230869582 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f736819bfc0 con 0x7f73681012f0 2024-03-18T14:57:04.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f735ffff640 1 --2- 172.21.15.76:0/2230869582 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73681012f0 0x7f7368197560 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7350000c00 tx=0x7f7350004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:04.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f7364ff9640 1 -- 172.21.15.76:0/2230869582 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7350041020 con 0x7f73681012f0 2024-03-18T14:57:04.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f736d263640 1 -- 172.21.15.76:0/2230869582 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f736819c250 con 0x7f73681012f0 2024-03-18T14:57:04.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f7364ff9640 1 -- 172.21.15.76:0/2230869582 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f735002c9f0 con 0x7f73681012f0 2024-03-18T14:57:04.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f7364ff9640 1 -- 172.21.15.76:0/2230869582 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7350039040 con 0x7f73681012f0 2024-03-18T14:57:04.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.225+0000 7f736d263640 1 -- 172.21.15.76:0/2230869582 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7368068e60 con 0x7f73681012f0 2024-03-18T14:57:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.228+0000 7f7364ff9640 1 -- 172.21.15.76:0/2230869582 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f7350044020 con 0x7f73681012f0 2024-03-18T14:57:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.228+0000 7f736fe48740 1 Processor -- start 2024-03-18T14:57:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.228+0000 7f736fe48740 1 -- start start 2024-03-18T14:57:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.228+0000 7f736fe48740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6130 0x55b49d8e6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.228+0000 7f736fe48740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8e6a50 0x55b49d8ef2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f736fe48740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8ef820 0x55b49d8f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f736fe48740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b49d84aa40 con 0x55b49d8e6a50 2024-03-18T14:57:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f736fe48740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b49d7f3920 con 0x55b49d8e6130 2024-03-18T14:57:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f736fe48740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b49d846ae0 con 0x55b49d8ef820 2024-03-18T14:57:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f7364ff9640 1 --2- 172.21.15.76:0/2230869582 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f73540a2a10 0x7f73540a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f7366ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8e6a50 0x55b49d8ef2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f735ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8ef820 0x55b49d8f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f7366ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8e6a50 0x55b49d8ef2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33974/0 (socket says 172.21.15.76:33974) 2024-03-18T14:57:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f7366ffd640 1 -- 172.21.15.76:0/4177008665 learned_addr learned my addr 172.21.15.76:0/4177008665 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f73677fe640 1 --2- 172.21.15.76:0/4177008665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6130 0x55b49d8e6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f73677fe640 1 --2- 172.21.15.76:0/2230869582 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f73540a2a10 0x7f73540a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f7364ff9640 1 -- 172.21.15.76:0/2230869582 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7f73500f0500 con 0x7f73681012f0 2024-03-18T14:57:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f7366ffd640 1 -- 172.21.15.76:0/4177008665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8ef820 msgr2=0x55b49d8f1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f7366ffd640 1 --2- 172.21.15.76:0/4177008665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8ef820 0x55b49d8f1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f7366ffd640 1 -- 172.21.15.76:0/4177008665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6130 msgr2=0x55b49d8e6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f7366ffd640 1 --2- 172.21.15.76:0/4177008665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6130 0x55b49d8e6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f7366ffd640 1 -- 172.21.15.76:0/4177008665 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b49d80b460 con 0x55b49d8e6a50 2024-03-18T14:57:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f735ffff640 1 --2- 172.21.15.76:0/4177008665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8ef820 0x55b49d8f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:57:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f73677fe640 1 --2- 172.21.15.76:0/4177008665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6130 0x55b49d8e6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:57:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f7366ffd640 1 --2- 172.21.15.76:0/4177008665 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8e6a50 0x55b49d8ef2e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7358009d20 tx=0x7f7358011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=916418e0bfacb4b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f73677fe640 1 --2- 172.21.15.76:0/2230869582 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f73540a2a10 0x7f73540a4ed0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7360009fd0 tx=0x7f73600095c0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f735d7fa640 1 -- 172.21.15.76:0/4177008665 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735801e070 con 0x55b49d8e6a50 2024-03-18T14:57:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f735d7fa640 1 -- 172.21.15.76:0/4177008665 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7358009e10 con 0x55b49d8e6a50 2024-03-18T14:57:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.229+0000 7f735d7fa640 1 -- 172.21.15.76:0/4177008665 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735800e9c0 con 0x55b49d8e6a50 2024-03-18T14:57:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.230+0000 7f736fe48740 1 -- 172.21.15.76:0/4177008665 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8e6a50 msgr2=0x55b49d8ef2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.230+0000 7f736fe48740 1 --2- 172.21.15.76:0/4177008665 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8e6a50 0x55b49d8ef2e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7358009d20 tx=0x7f7358011810 comp rx=0 tx=0).stop 2024-03-18T14:57:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.230+0000 7f736fe48740 1 -- 172.21.15.76:0/4177008665 shutdown_connections 2024-03-18T14:57:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.230+0000 7f736fe48740 1 --2- 172.21.15.76:0/4177008665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8ef820 0x55b49d8f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.230+0000 7f736fe48740 1 --2- 172.21.15.76:0/4177008665 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8e6a50 0x55b49d8ef2e0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.230+0000 7f736fe48740 1 --2- 172.21.15.76:0/4177008665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6130 0x55b49d8e6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.230+0000 7f736fe48740 1 -- 172.21.15.76:0/4177008665 >> 172.21.15.76:0/4177008665 conn(0x55b49d8e0a50 msgr2=0x55b49d8e0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.230+0000 7f736fe48740 1 -- 172.21.15.76:0/4177008665 shutdown_connections 2024-03-18T14:57:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.230+0000 7f736fe48740 1 -- 172.21.15.76:0/4177008665 wait complete. 2024-03-18T14:57:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f736fe48740 1 -- start start 2024-03-18T14:57:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f736fe48740 1 Processor -- start 2024-03-18T14:57:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f736fe48740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8e6130 0x55b49d8ec880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f736fe48740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8e6a50 0x55b49d8ecdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f736fe48740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8ef820 0x55b49d8ed300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f736fe48740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b49d8f2450 con 0x55b49d8e6130 2024-03-18T14:57:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f736fe48740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b49d806d70 con 0x55b49d8ef820 2024-03-18T14:57:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f736fe48740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b49d8f25d0 con 0x55b49d8e6a50 2024-03-18T14:57:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f736fe48740 10 client.0 _ll_register_callbacks cb 0x55b49d55ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:57:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f7366ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8e6130 0x55b49d8ec880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f7366ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8e6130 0x55b49d8ec880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33980/0 (socket says 172.21.15.76:33980) 2024-03-18T14:57:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f7366ffd640 1 -- 172.21.15.76:0/1233658950 learned_addr learned my addr 172.21.15.76:0/1233658950 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f735ffff640 1 --2- 172.21.15.76:0/1233658950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8e6a50 0x55b49d8ecdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f73677fe640 1 --2- 172.21.15.76:0/1233658950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8ef820 0x55b49d8ed300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f7366ffd640 1 -- 172.21.15.76:0/1233658950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8e6a50 msgr2=0x55b49d8ecdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f7366ffd640 1 --2- 172.21.15.76:0/1233658950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8e6a50 0x55b49d8ecdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f7366ffd640 1 -- 172.21.15.76:0/1233658950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8ef820 msgr2=0x55b49d8ed300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f7366ffd640 1 --2- 172.21.15.76:0/1233658950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8ef820 0x55b49d8ed300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f7366ffd640 1 -- 172.21.15.76:0/1233658950 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b49d55f210 con 0x55b49d8e6130 2024-03-18T14:57:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f7366ffd640 1 --2- 172.21.15.76:0/1233658950 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8e6130 0x55b49d8ec880 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f7358011d10 tx=0x7f735800e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=704aa45c27b05205 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.231+0000 7f734e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:57:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.232+0000 7f734e7fc640 1 -- 172.21.15.76:0/1233658950 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735801e030 con 0x55b49d8e6130 2024-03-18T14:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.232+0000 7f734e7fc640 1 -- 172.21.15.76:0/1233658950 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7f7358005050 con 0x55b49d8e6130 2024-03-18T14:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.232+0000 7f736fe48740 1 -- 172.21.15.76:0/1233658950 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7358009e10 con 0x55b49d8e6130 2024-03-18T14:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.232+0000 7f734e7fc640 1 -- 172.21.15.76:0/1233658950 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7358031050 con 0x55b49d8e6130 2024-03-18T14:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.232+0000 7f736fe48740 10 client.16140 fetch_fsmap learned FSMap version 35 2024-03-18T14:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.232+0000 7f736fe48740 1 -- 172.21.15.76:0/1233658950 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b49d62b620 con 0x55b49d8e6130 2024-03-18T14:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.232+0000 7f734e7fc640 1 -- 172.21.15.76:0/1233658950 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7358009e10 con 0x55b49d8e6130 2024-03-18T14:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.232+0000 7f734e7fc640 1 -- 172.21.15.76:0/1233658950 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 35) ==== 8338+0+0 (secure 0 0 0) 0x7f7358004a60 con 0x55b49d8e6130 2024-03-18T14:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.232+0000 7f736fe48740 10 client.16140 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T14:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.232+0000 7f736fe48740 10 client.16140 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.232+0000 7f736fe48740 20 client.16140 populate_metadata read hostname 'smithi076' 2024-03-18T14:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.232+0000 7f736fe48740 1 --2- 172.21.15.76:0/1233658950 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55b49d8ffa00 0x55b49d901df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.232+0000 7f736fe48740 4 client.16140 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.232+0000 7f736fe48740 1 -- 172.21.15.76:0/1233658950 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b49d857f80 con 0x55b49d8ffa00 2024-03-18T14:57:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.232+0000 7f735ffff640 1 --2- 172.21.15.76:0/1233658950 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55b49d8ffa00 0x55b49d901df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.233+0000 7f735ffff640 1 --2- 172.21.15.76:0/1233658950 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55b49d8ffa00 0x55b49d901df0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c547652c8f6b9a5 server_cookie=4e1b13baeeedbbc5 in_seq=0 out_seq=0 2024-03-18T14:57:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.233+0000 7f734e7fc640 10 client.16140 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.233+0000 7f734e7fc640 1 -- 172.21.15.76:0/1233658950 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b49d857f80 con 0x55b49d8ffa00 2024-03-18T14:57:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.233+0000 7f734e7fc640 10 client.16140 handle_command_reply: tid=0 2024-03-18T14:57:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.233+0000 7f736fe48740 1 client.16140 shutdown 2024-03-18T14:57:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.233+0000 7f736fe48740 1 -- 172.21.15.76:0/1233658950 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8e6130 msgr2=0x55b49d8ec880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.233+0000 7f736fe48740 1 --2- 172.21.15.76:0/1233658950 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8e6130 0x55b49d8ec880 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f7358011d10 tx=0x7f735800e970 comp rx=0 tx=0).stop 2024-03-18T14:57:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.233+0000 7f736fe48740 1 -- 172.21.15.76:0/1233658950 shutdown_connections 2024-03-18T14:57:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.233+0000 7f736fe48740 1 --2- 172.21.15.76:0/1233658950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8e6a50 0x55b49d8ecdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.233+0000 7f736fe48740 1 --2- 172.21.15.76:0/1233658950 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55b49d8ffa00 0x55b49d901df0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.233+0000 7f736fe48740 1 --2- 172.21.15.76:0/1233658950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8ef820 0x55b49d8ed300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.233+0000 7f736fe48740 1 --2- 172.21.15.76:0/1233658950 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8e6130 0x55b49d8ec880 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.233+0000 7f734e7fc640 0 client.16140 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.233+0000 7f736fe48740 1 -- 172.21.15.76:0/1233658950 >> 172.21.15.76:0/1233658950 conn(0x55b49d8e0a50 msgr2=0x55b49d8e27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.234+0000 7f736fe48740 1 -- 172.21.15.76:0/1233658950 shutdown_connections 2024-03-18T14:57:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.234+0000 7f736fe48740 1 -- 172.21.15.76:0/1233658950 wait complete. 2024-03-18T14:57:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.234+0000 7f736fe48740 20 client.16140 trim_cache size 0 max 16384 2024-03-18T14:57:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.249+0000 7f736fe48740 1 Processor -- start 2024-03-18T14:57:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.249+0000 7f736fe48740 1 -- start start 2024-03-18T14:57:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.250+0000 7f736fe48740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8e6130 0x55b49d8fb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.250+0000 7f736fe48740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6a50 0x55b49d9074b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.250+0000 7f736fe48740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8ef820 0x55b49d8ee6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.250+0000 7f736fe48740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f73681afbd0 con 0x55b49d8ef820 2024-03-18T14:57:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.250+0000 7f736fe48740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f73681ad4a0 con 0x55b49d8e6a50 2024-03-18T14:57:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.250+0000 7f736fe48740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f73681ab310 con 0x55b49d8e6130 2024-03-18T14:57:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.250+0000 7f73677fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8ef820 0x55b49d8ee6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.250+0000 7f735ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6a50 0x55b49d9074b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.250+0000 7f73677fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8ef820 0x55b49d8ee6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33990/0 (socket says 172.21.15.76:33990) 2024-03-18T14:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.250+0000 7f73677fe640 1 -- 172.21.15.76:0/1846630765 learned_addr learned my addr 172.21.15.76:0/1846630765 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.250+0000 7f7366ffd640 1 --2- 172.21.15.76:0/1846630765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8e6130 0x55b49d8fb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.250+0000 7f735ffff640 1 -- 172.21.15.76:0/1846630765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8e6130 msgr2=0x55b49d8fb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.250+0000 7f735ffff640 1 --2- 172.21.15.76:0/1846630765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8e6130 0x55b49d8fb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.250+0000 7f735ffff640 1 -- 172.21.15.76:0/1846630765 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8ef820 msgr2=0x55b49d8ee6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.250+0000 7f735ffff640 1 --2- 172.21.15.76:0/1846630765 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8ef820 0x55b49d8ee6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.250+0000 7f735ffff640 1 -- 172.21.15.76:0/1846630765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b49d8dc310 con 0x55b49d8e6a50 2024-03-18T14:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.250+0000 7f735ffff640 1 --2- 172.21.15.76:0/1846630765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6a50 0x55b49d9074b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f73500eddb0 tx=0x7f73500b9510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab6441bd58a1b646 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.251+0000 7f735cff9640 1 -- 172.21.15.76:0/1846630765 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73500bc620 con 0x55b49d8e6a50 2024-03-18T14:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.251+0000 7f735cff9640 1 -- 172.21.15.76:0/1846630765 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7350032640 con 0x55b49d8e6a50 2024-03-18T14:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.251+0000 7f735cff9640 1 -- 172.21.15.76:0/1846630765 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73500bad60 con 0x55b49d8e6a50 2024-03-18T14:57:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.251+0000 7f736fe48740 1 -- 172.21.15.76:0/1846630765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6a50 msgr2=0x55b49d9074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.251+0000 7f736fe48740 1 --2- 172.21.15.76:0/1846630765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6a50 0x55b49d9074b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f73500eddb0 tx=0x7f73500b9510 comp rx=0 tx=0).stop 2024-03-18T14:57:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.251+0000 7f736fe48740 1 -- 172.21.15.76:0/1846630765 shutdown_connections 2024-03-18T14:57:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.251+0000 7f736fe48740 1 --2- 172.21.15.76:0/1846630765 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8ef820 0x55b49d8ee6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.251+0000 7f736fe48740 1 --2- 172.21.15.76:0/1846630765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6a50 0x55b49d9074b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f736fe48740 1 --2- 172.21.15.76:0/1846630765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8e6130 0x55b49d8fb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f736fe48740 1 -- 172.21.15.76:0/1846630765 >> 172.21.15.76:0/1846630765 conn(0x55b49d8e0a50 msgr2=0x55b49d8edfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f736fe48740 1 -- 172.21.15.76:0/1846630765 shutdown_connections 2024-03-18T14:57:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f736fe48740 1 -- 172.21.15.76:0/1846630765 wait complete. 2024-03-18T14:57:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f736fe48740 1 -- start start 2024-03-18T14:57:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f736fe48740 1 Processor -- start 2024-03-18T14:57:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f736fe48740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8e6130 0x55b49d8f3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f736fe48740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6a50 0x55b49d8f3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f736fe48740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8ef820 0x55b49d8f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f736fe48740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f73681ab960 con 0x55b49d8ef820 2024-03-18T14:57:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f736fe48740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f73681acdd0 con 0x55b49d8e6a50 2024-03-18T14:57:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f736fe48740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f73681ac030 con 0x55b49d8e6130 2024-03-18T14:57:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f736fe48740 10 client.0 _ll_register_callbacks cb 0x7f735801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:57:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f73677fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8ef820 0x55b49d8f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f73677fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8ef820 0x55b49d8f3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33994/0 (socket says 172.21.15.76:33994) 2024-03-18T14:57:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f735ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6a50 0x55b49d8f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f73677fe640 1 -- 172.21.15.76:0/1255442955 learned_addr learned my addr 172.21.15.76:0/1255442955 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f7366ffd640 1 --2- 172.21.15.76:0/1255442955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8e6130 0x55b49d8f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f73677fe640 1 -- 172.21.15.76:0/1255442955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8e6130 msgr2=0x55b49d8f3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f73677fe640 1 --2- 172.21.15.76:0/1255442955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8e6130 0x55b49d8f3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f73677fe640 1 -- 172.21.15.76:0/1255442955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6a50 msgr2=0x55b49d8f3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.252+0000 7f73677fe640 1 --2- 172.21.15.76:0/1255442955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6a50 0x55b49d8f3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f73677fe640 1 -- 172.21.15.76:0/1255442955 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b49d964db0 con 0x55b49d8ef820 2024-03-18T14:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f735ffff640 1 --2- 172.21.15.76:0/1255442955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6a50 0x55b49d8f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f73677fe640 1 --2- 172.21.15.76:0/1255442955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8ef820 0x55b49d8f3ed0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7360003cc0 tx=0x7f7360004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbbdec0c14b3c0f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f734e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f734e7fc640 1 -- 172.21.15.76:0/1255442955 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736002e030 con 0x55b49d8ef820 2024-03-18T14:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f734e7fc640 1 -- 172.21.15.76:0/1255442955 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7f7360017040 con 0x55b49d8ef820 2024-03-18T14:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f736fe48740 1 -- 172.21.15.76:0/1255442955 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b49d8522d0 con 0x55b49d8ef820 2024-03-18T14:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f734e7fc640 1 -- 172.21.15.76:0/1255442955 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7360038050 con 0x55b49d8ef820 2024-03-18T14:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f736fe48740 10 client.16146 fetch_fsmap learned FSMap version 35 2024-03-18T14:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f736fe48740 1 -- 172.21.15.76:0/1255442955 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b49d966120 con 0x55b49d8ef820 2024-03-18T14:57:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f734e7fc640 1 -- 172.21.15.76:0/1255442955 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b49d8522d0 con 0x55b49d8ef820 2024-03-18T14:57:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f734e7fc640 1 -- 172.21.15.76:0/1255442955 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 35) ==== 8338+0+0 (secure 0 0 0) 0x7f736000b040 con 0x55b49d8ef820 2024-03-18T14:57:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f736fe48740 10 client.16146 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T14:57:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f736fe48740 10 client.16146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:57:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f736fe48740 20 client.16146 populate_metadata read hostname 'smithi076' 2024-03-18T14:57:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f736fe48740 1 --2- 172.21.15.76:0/1255442955 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55b49d9668f0 0x55b49d968d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f736fe48740 4 client.16146 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:57:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f736fe48740 1 -- 172.21.15.76:0/1255442955 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b49d7d0770 con 0x55b49d9668f0 2024-03-18T14:57:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.253+0000 7f7366ffd640 1 --2- 172.21.15.76:0/1255442955 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55b49d9668f0 0x55b49d968d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.254+0000 7f734e7fc640 10 client.16146 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.254+0000 7f7366ffd640 1 --2- 172.21.15.76:0/1255442955 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55b49d9668f0 0x55b49d968d10 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cad87bd96a0dc96 server_cookie=1d561a318a4b468d in_seq=0 out_seq=0 2024-03-18T14:57:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.324+0000 7f734e7fc640 1 -- 172.21.15.76:0/1255442955 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b49d7d0770 con 0x55b49d9668f0 2024-03-18T14:57:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.324+0000 7f734e7fc640 10 client.16146 handle_command_reply: tid=0 2024-03-18T14:57:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.325+0000 7f736fe48740 1 client.16146 shutdown 2024-03-18T14:57:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.325+0000 7f736fe48740 1 -- 172.21.15.76:0/1255442955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8ef820 msgr2=0x55b49d8f3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.325+0000 7f736fe48740 1 --2- 172.21.15.76:0/1255442955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8ef820 0x55b49d8f3ed0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7360003cc0 tx=0x7f7360004a40 comp rx=0 tx=0).stop 2024-03-18T14:57:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.325+0000 7f736fe48740 1 -- 172.21.15.76:0/1255442955 shutdown_connections 2024-03-18T14:57:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.325+0000 7f736fe48740 1 --2- 172.21.15.76:0/1255442955 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55b49d9668f0 0x55b49d968d10 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.325+0000 7f736fe48740 1 --2- 172.21.15.76:0/1255442955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b49d8ef820 0x55b49d8f3ed0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.325+0000 7f734e7fc640 0 client.16146 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.325+0000 7f736fe48740 1 --2- 172.21.15.76:0/1255442955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b49d8e6a50 0x55b49d8f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.325+0000 7f736fe48740 1 --2- 172.21.15.76:0/1255442955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b49d8e6130 0x55b49d8f3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.325+0000 7f736fe48740 1 -- 172.21.15.76:0/1255442955 >> 172.21.15.76:0/1255442955 conn(0x55b49d8e0a50 msgr2=0x55b49d8dfbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.325+0000 7f736fe48740 1 -- 172.21.15.76:0/1255442955 shutdown_connections 2024-03-18T14:57:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.325+0000 7f736fe48740 1 -- 172.21.15.76:0/1255442955 wait complete. 2024-03-18T14:57:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.325+0000 7f736fe48740 20 client.16146 trim_cache size 0 max 16384 2024-03-18T14:57:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:57:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:57:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:57:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:57:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.325+0000 7f736d263640 1 -- 172.21.15.76:0/2230869582 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f73540a2a10 msgr2=0x7f73540a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.325+0000 7f736d263640 1 --2- 172.21.15.76:0/2230869582 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f73540a2a10 0x7f73540a4ed0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7360009fd0 tx=0x7f73600095c0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.325+0000 7f736d263640 1 -- 172.21.15.76:0/2230869582 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73681012f0 msgr2=0x7f7368197560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.325+0000 7f736d263640 1 --2- 172.21.15.76:0/2230869582 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73681012f0 0x7f7368197560 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7350000c00 tx=0x7f7350004550 comp rx=0 tx=0).stop 2024-03-18T14:57:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.326+0000 7f736d263640 1 -- 172.21.15.76:0/2230869582 shutdown_connections 2024-03-18T14:57:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.326+0000 7f736d263640 1 --2- 172.21.15.76:0/2230869582 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f73540a2a10 0x7f73540a4ed0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.326+0000 7f736d263640 1 --2- 172.21.15.76:0/2230869582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7368105680 0x7f736819b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.326+0000 7f736d263640 1 --2- 172.21.15.76:0/2230869582 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73681012f0 0x7f7368197560 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.326+0000 7f736d263640 1 --2- 172.21.15.76:0/2230869582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7368100930 0x7f7368197020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.326+0000 7f736d263640 1 -- 172.21.15.76:0/2230869582 >> 172.21.15.76:0/2230869582 conn(0x7f7368077530 msgr2=0x7f73680fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.326+0000 7f736d263640 1 -- 172.21.15.76:0/2230869582 shutdown_connections 2024-03-18T14:57:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:04.326+0000 7f736d263640 1 -- 172.21.15.76:0/2230869582 wait complete. 2024-03-18T14:57:04.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 350784 10.30 MB/sec execute 65 sec latency 63.577 ms 2024-03-18T14:57:05.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 352136 10.24 MB/sec execute 66 sec latency 16.438 ms 2024-03-18T14:57:05.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:05 smithi174 ceph-mon[36787]: pgmap v121: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 248 op/s 2024-03-18T14:57:05.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:05 smithi077 ceph-mon[35865]: pgmap v121: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 248 op/s 2024-03-18T14:57:06.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:05 smithi076 ceph-mon[32205]: pgmap v121: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 248 op/s 2024-03-18T14:57:06.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 354702 10.34 MB/sec execute 67 sec latency 69.213 ms 2024-03-18T14:57:07.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 355952 10.24 MB/sec execute 68 sec latency 13.823 ms 2024-03-18T14:57:07.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:07 smithi174 ceph-mon[36787]: pgmap v122: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 248 op/s 2024-03-18T14:57:07.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:07 smithi077 ceph-mon[35865]: pgmap v122: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 248 op/s 2024-03-18T14:57:08.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:07 smithi076 ceph-mon[32205]: pgmap v122: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 248 op/s 2024-03-18T14:57:08.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 358497 10.33 MB/sec execute 69 sec latency 16.864 ms 2024-03-18T14:57:09.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 359779 10.26 MB/sec execute 70 sec latency 43.572 ms 2024-03-18T14:57:09.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:09 smithi174 ceph-mon[36787]: pgmap v123: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.0 MiB/s wr, 411 op/s 2024-03-18T14:57:09.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:09 smithi077 ceph-mon[35865]: pgmap v123: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.0 MiB/s wr, 411 op/s 2024-03-18T14:57:10.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:09 smithi076 ceph-mon[32205]: pgmap v123: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.0 MiB/s wr, 411 op/s 2024-03-18T14:57:10.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 362174 10.32 MB/sec execute 71 sec latency 11.098 ms 2024-03-18T14:57:11.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 364181 10.27 MB/sec execute 72 sec latency 14.911 ms 2024-03-18T14:57:11.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:11 smithi174 ceph-mon[36787]: pgmap v124: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 298 op/s 2024-03-18T14:57:11.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:11 smithi077 ceph-mon[35865]: pgmap v124: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 298 op/s 2024-03-18T14:57:12.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:11 smithi076 ceph-mon[32205]: pgmap v124: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 298 op/s 2024-03-18T14:57:12.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 365903 10.30 MB/sec execute 73 sec latency 18.328 ms 2024-03-18T14:57:13.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 368315 10.33 MB/sec execute 74 sec latency 15.475 ms 2024-03-18T14:57:13.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:13 smithi174 ceph-mon[36787]: pgmap v125: 97 pgs: 97 active+clean; 109 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.2 MiB/s wr, 417 op/s 2024-03-18T14:57:13.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:13 smithi077 ceph-mon[35865]: pgmap v125: 97 pgs: 97 active+clean; 109 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.2 MiB/s wr, 417 op/s 2024-03-18T14:57:14.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:13 smithi076 ceph-mon[32205]: pgmap v125: 97 pgs: 97 active+clean; 109 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.2 MiB/s wr, 417 op/s 2024-03-18T14:57:14.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 369550 10.28 MB/sec execute 75 sec latency 65.302 ms 2024-03-18T14:57:15.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 372393 10.37 MB/sec execute 76 sec latency 21.719 ms 2024-03-18T14:57:15.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:15 smithi174 ceph-mon[36787]: pgmap v126: 97 pgs: 97 active+clean; 109 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 281 op/s 2024-03-18T14:57:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:15 smithi077 ceph-mon[35865]: pgmap v126: 97 pgs: 97 active+clean; 109 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 281 op/s 2024-03-18T14:57:16.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:15 smithi076 ceph-mon[32205]: pgmap v126: 97 pgs: 97 active+clean; 109 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 281 op/s 2024-03-18T14:57:16.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 373595 10.28 MB/sec execute 77 sec latency 16.717 ms 2024-03-18T14:57:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.420+0000 7fb313fff640 1 -- 172.21.15.76:0/4000627781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb314106640 msgr2=0x7fb314106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.420+0000 7fb313fff640 1 --2- 172.21.15.76:0/4000627781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb314106640 0x7fb314106a20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb304009900 tx=0x7fb30402bc30 comp rx=0 tx=0).stop 2024-03-18T14:57:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.420+0000 7fb313fff640 1 -- 172.21.15.76:0/4000627781 shutdown_connections 2024-03-18T14:57:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.420+0000 7fb313fff640 1 --2- 172.21.15.76:0/4000627781 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3140ffec0 0x7fb314100340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.420+0000 7fb313fff640 1 --2- 172.21.15.76:0/4000627781 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb314100880 0x7fb31410e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.420+0000 7fb313fff640 1 --2- 172.21.15.76:0/4000627781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb314106640 0x7fb314106a20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.420+0000 7fb313fff640 1 -- 172.21.15.76:0/4000627781 >> 172.21.15.76:0/4000627781 conn(0x7fb314075850 msgr2=0x7fb314075c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.420+0000 7fb313fff640 1 -- 172.21.15.76:0/4000627781 shutdown_connections 2024-03-18T14:57:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.420+0000 7fb313fff640 1 -- 172.21.15.76:0/4000627781 wait complete. 2024-03-18T14:57:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.421+0000 7fb313fff640 1 Processor -- start 2024-03-18T14:57:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.421+0000 7fb313fff640 1 -- start start 2024-03-18T14:57:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.421+0000 7fb313fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3140ffec0 0x7fb31419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.421+0000 7fb313fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb314100880 0x7fb31419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.421+0000 7fb313fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb314106640 0x7fb31419e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.421+0000 7fb313fff640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb314111aa0 con 0x7fb3140ffec0 2024-03-18T14:57:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.421+0000 7fb313fff640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb314111920 con 0x7fb314100880 2024-03-18T14:57:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.421+0000 7fb313fff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb314111c20 con 0x7fb314106640 2024-03-18T14:57:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.421+0000 7fb312ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3140ffec0 0x7fb31419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.421+0000 7fb312ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3140ffec0 0x7fb31419a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:36602/0 (socket says 172.21.15.76:36602) 2024-03-18T14:57:17.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.421+0000 7fb312ffd640 1 -- 172.21.15.76:0/237997024 learned_addr learned my addr 172.21.15.76:0/237997024 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:17.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.421+0000 7fb30bfff640 1 --2- 172.21.15.76:0/237997024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb314100880 0x7fb31419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.421+0000 7fb3137fe640 1 --2- 172.21.15.76:0/237997024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb314106640 0x7fb31419e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.422+0000 7fb312ffd640 1 -- 172.21.15.76:0/237997024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb314106640 msgr2=0x7fb31419e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:17.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.422+0000 7fb312ffd640 1 --2- 172.21.15.76:0/237997024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb314106640 0x7fb31419e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.422+0000 7fb312ffd640 1 -- 172.21.15.76:0/237997024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb314100880 msgr2=0x7fb31419a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:17.438 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.422+0000 7fb312ffd640 1 --2- 172.21.15.76:0/237997024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb314100880 0x7fb31419a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.422+0000 7fb312ffd640 1 -- 172.21.15.76:0/237997024 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb31419f0f0 con 0x7fb3140ffec0 2024-03-18T14:57:17.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.422+0000 7fb30bfff640 1 --2- 172.21.15.76:0/237997024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb314100880 0x7fb31419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:57:17.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.422+0000 7fb312ffd640 1 --2- 172.21.15.76:0/237997024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3140ffec0 0x7fb31419a100 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb304006010 tx=0x7fb30402cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:17.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.422+0000 7fb310ff9640 1 -- 172.21.15.76:0/237997024 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb304041070 con 0x7fb3140ffec0 2024-03-18T14:57:17.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.422+0000 7fb313fff640 1 -- 172.21.15.76:0/237997024 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb31419f380 con 0x7fb3140ffec0 2024-03-18T14:57:17.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.422+0000 7fb310ff9640 1 -- 172.21.15.76:0/237997024 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb304004450 con 0x7fb3140ffec0 2024-03-18T14:57:17.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.422+0000 7fb310ff9640 1 -- 172.21.15.76:0/237997024 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb304003d00 con 0x7fb3140ffec0 2024-03-18T14:57:17.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.422+0000 7fb313fff640 1 -- 172.21.15.76:0/237997024 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3141a6c70 con 0x7fb3140ffec0 2024-03-18T14:57:17.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.424+0000 7fb310ff9640 1 -- 172.21.15.76:0/237997024 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fb304044070 con 0x7fb3140ffec0 2024-03-18T14:57:17.439 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.425+0000 7fb31bfef740 1 Processor -- start 2024-03-18T14:57:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.425+0000 7fb31bfef740 1 -- start start 2024-03-18T14:57:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.425+0000 7fb31bfef740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ec42c0 0x561777ec46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.425+0000 7fb31bfef740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec4be0 0x561777ecd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.425+0000 7fb31bfef740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ecd9b0 0x561777ecfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.425+0000 7fb31bfef740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561777e28a40 con 0x561777ecd9b0 2024-03-18T14:57:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.425+0000 7fb31bfef740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561777dd1920 con 0x561777ec42c0 2024-03-18T14:57:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.425+0000 7fb31bfef740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561777e24ae0 con 0x561777ec4be0 2024-03-18T14:57:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.425+0000 7fb3137fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ec42c0 0x561777ec46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.425+0000 7fb310ff9640 1 --2- 172.21.15.76:0/237997024 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb3000a2960 0x7fb3000a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.425+0000 7fb30bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ecd9b0 0x561777ecfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.425+0000 7fb3137fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ec42c0 0x561777ec46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:36690/0 (socket says 172.21.15.76:36690) 2024-03-18T14:57:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.425+0000 7fb3137fe640 1 -- 172.21.15.76:0/2028225965 learned_addr learned my addr 172.21.15.76:0/2028225965 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.425+0000 7fb312ffd640 1 --2- 172.21.15.76:0/2028225965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec4be0 0x561777ecd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.425+0000 7fb3137fe640 1 --2- 172.21.15.76:0/237997024 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb3000a2960 0x7fb3000a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.426+0000 7fb3137fe640 1 -- 172.21.15.76:0/2028225965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec4be0 msgr2=0x561777ecd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.426+0000 7fb3137fe640 1 --2- 172.21.15.76:0/2028225965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec4be0 0x561777ecd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.426+0000 7fb3137fe640 1 -- 172.21.15.76:0/2028225965 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ecd9b0 msgr2=0x561777ecfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.426+0000 7fb3137fe640 1 --2- 172.21.15.76:0/2028225965 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ecd9b0 0x561777ecfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.426+0000 7fb3137fe640 1 -- 172.21.15.76:0/2028225965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561777de9460 con 0x561777ec42c0 2024-03-18T14:57:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.426+0000 7fb310ff9640 1 -- 172.21.15.76:0/237997024 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7fb3040f07d0 con 0x7fb3140ffec0 2024-03-18T14:57:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.426+0000 7fb3137fe640 1 --2- 172.21.15.76:0/2028225965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ec42c0 0x561777ec46a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb30c013b20 tx=0x7fb30c00f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cc19e8bc41b92bf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.426+0000 7fb3097fa640 1 -- 172.21.15.76:0/2028225965 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c022020 con 0x561777ec42c0 2024-03-18T14:57:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.426+0000 7fb3097fa640 1 -- 172.21.15.76:0/2028225965 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb30c010430 con 0x561777ec42c0 2024-03-18T14:57:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.426+0000 7fb3137fe640 1 --2- 172.21.15.76:0/237997024 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb3000a2960 0x7fb3000a4e20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb30c023000 tx=0x7fb30c021000 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.426+0000 7fb3097fa640 1 -- 172.21.15.76:0/2028225965 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c013560 con 0x561777ec42c0 2024-03-18T14:57:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.427+0000 7fb31bfef740 1 -- 172.21.15.76:0/2028225965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ec42c0 msgr2=0x561777ec46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.427+0000 7fb31bfef740 1 --2- 172.21.15.76:0/2028225965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ec42c0 0x561777ec46a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb30c013b20 tx=0x7fb30c00f3c0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.427+0000 7fb31bfef740 1 -- 172.21.15.76:0/2028225965 shutdown_connections 2024-03-18T14:57:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.427+0000 7fb31bfef740 1 --2- 172.21.15.76:0/2028225965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec4be0 0x561777ecd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.427+0000 7fb31bfef740 1 --2- 172.21.15.76:0/2028225965 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ecd9b0 0x561777ecfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.427+0000 7fb31bfef740 1 --2- 172.21.15.76:0/2028225965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ec42c0 0x561777ec46a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.427+0000 7fb31bfef740 1 -- 172.21.15.76:0/2028225965 >> 172.21.15.76:0/2028225965 conn(0x561777ebebe0 msgr2=0x561777ebefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.427+0000 7fb31bfef740 1 -- 172.21.15.76:0/2028225965 shutdown_connections 2024-03-18T14:57:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.427+0000 7fb31bfef740 1 -- 172.21.15.76:0/2028225965 wait complete. 2024-03-18T14:57:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.427+0000 7fb31bfef740 1 -- start start 2024-03-18T14:57:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.427+0000 7fb31bfef740 1 Processor -- start 2024-03-18T14:57:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb31bfef740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec42c0 0x561777ecaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb31bfef740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ec4be0 0x561777ecafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb31bfef740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ecd9b0 0x561777ecb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb31bfef740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561777ed05e0 con 0x561777ec4be0 2024-03-18T14:57:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb31bfef740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561777de4d70 con 0x561777ecd9b0 2024-03-18T14:57:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb31bfef740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561777ed0760 con 0x561777ec42c0 2024-03-18T14:57:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb31bfef740 10 client.0 _ll_register_callbacks cb 0x561777b3ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:57:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb30bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ec4be0 0x561777ecafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb312ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec42c0 0x561777ecaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb312ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec42c0 0x561777ecaa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:40750/0 (socket says 172.21.15.76:40750) 2024-03-18T14:57:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb3137fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ecd9b0 0x561777ecb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb30bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ec4be0 0x561777ecafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:36618/0 (socket says 172.21.15.76:36618) 2024-03-18T14:57:17.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb312ffd640 1 -- 172.21.15.76:0/2651122797 learned_addr learned my addr 172.21.15.76:0/2651122797 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:17.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb30bfff640 1 -- 172.21.15.76:0/2651122797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec42c0 msgr2=0x561777ecaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:17.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb30bfff640 1 --2- 172.21.15.76:0/2651122797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec42c0 0x561777ecaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb30bfff640 1 -- 172.21.15.76:0/2651122797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ecd9b0 msgr2=0x561777ecb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:17.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb30bfff640 1 --2- 172.21.15.76:0/2651122797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ecd9b0 0x561777ecb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb30bfff640 1 -- 172.21.15.76:0/2651122797 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561777b3d210 con 0x561777ec4be0 2024-03-18T14:57:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb30bfff640 1 --2- 172.21.15.76:0/2651122797 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ec4be0 0x561777ecafb0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x561777ebfd70 tx=0x7fb2fc00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9bd5dd117c2e3b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb2fa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:57:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb2fa7fc640 1 -- 172.21.15.76:0/2651122797 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc012110 con 0x561777ec4be0 2024-03-18T14:57:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb2fa7fc640 1 -- 172.21.15.76:0/2651122797 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7fb2fc004050 con 0x561777ec4be0 2024-03-18T14:57:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.428+0000 7fb31bfef740 1 -- 172.21.15.76:0/2651122797 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb30c010430 con 0x561777ec4be0 2024-03-18T14:57:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.429+0000 7fb2fa7fc640 1 -- 172.21.15.76:0/2651122797 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb2fc0122b0 con 0x561777ec4be0 2024-03-18T14:57:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.429+0000 7fb31bfef740 10 client.16158 fetch_fsmap learned FSMap version 35 2024-03-18T14:57:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.429+0000 7fb31bfef740 1 -- 172.21.15.76:0/2651122797 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561777c09620 con 0x561777ec4be0 2024-03-18T14:57:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.429+0000 7fb2fa7fc640 1 -- 172.21.15.76:0/2651122797 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb30c010430 con 0x561777ec4be0 2024-03-18T14:57:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.429+0000 7fb2fa7fc640 1 -- 172.21.15.76:0/2651122797 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 35) ==== 8338+0+0 (secure 0 0 0) 0x7fb2fc026480 con 0x561777ec4be0 2024-03-18T14:57:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.429+0000 7fb31bfef740 10 client.16158 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T14:57:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.429+0000 7fb31bfef740 10 client.16158 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:57:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.429+0000 7fb31bfef740 20 client.16158 populate_metadata read hostname 'smithi076' 2024-03-18T14:57:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.429+0000 7fb31bfef740 1 --2- 172.21.15.76:0/2651122797 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561777eddc20 0x561777ee0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.429+0000 7fb31bfef740 4 client.16158 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:57:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.429+0000 7fb31bfef740 1 -- 172.21.15.76:0/2651122797 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561777e35f80 con 0x561777eddc20 2024-03-18T14:57:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.429+0000 7fb312ffd640 1 --2- 172.21.15.76:0/2651122797 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561777eddc20 0x561777ee0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.429+0000 7fb2fa7fc640 10 client.16158 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.429+0000 7fb312ffd640 1 --2- 172.21.15.76:0/2651122797 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561777eddc20 0x561777ee0030 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f2ec8e49454b9d7 server_cookie=c77e78b7e2586a4f in_seq=0 out_seq=0 2024-03-18T14:57:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.432+0000 7fb2fa7fc640 1 -- 172.21.15.76:0/2651122797 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561777e35f80 con 0x561777eddc20 2024-03-18T14:57:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.432+0000 7fb2fa7fc640 10 client.16158 handle_command_reply: tid=0 2024-03-18T14:57:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.432+0000 7fb31bfef740 1 client.16158 shutdown 2024-03-18T14:57:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.432+0000 7fb31bfef740 1 -- 172.21.15.76:0/2651122797 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ec4be0 msgr2=0x561777ecafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.432+0000 7fb31bfef740 1 --2- 172.21.15.76:0/2651122797 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ec4be0 0x561777ecafb0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x561777ebfd70 tx=0x7fb2fc00f790 comp rx=0 tx=0).stop 2024-03-18T14:57:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.432+0000 7fb31bfef740 1 -- 172.21.15.76:0/2651122797 shutdown_connections 2024-03-18T14:57:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.432+0000 7fb31bfef740 1 --2- 172.21.15.76:0/2651122797 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561777eddc20 0x561777ee0030 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.432+0000 7fb31bfef740 1 --2- 172.21.15.76:0/2651122797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ecd9b0 0x561777ecb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.432+0000 7fb31bfef740 1 --2- 172.21.15.76:0/2651122797 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ec4be0 0x561777ecafb0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.432+0000 7fb31bfef740 1 --2- 172.21.15.76:0/2651122797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec42c0 0x561777ecaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.432+0000 7fb2fa7fc640 0 client.16158 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.432+0000 7fb31bfef740 1 -- 172.21.15.76:0/2651122797 >> 172.21.15.76:0/2651122797 conn(0x561777ebebe0 msgr2=0x561777ec0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.432+0000 7fb31bfef740 1 -- 172.21.15.76:0/2651122797 shutdown_connections 2024-03-18T14:57:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.432+0000 7fb31bfef740 1 -- 172.21.15.76:0/2651122797 wait complete. 2024-03-18T14:57:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.432+0000 7fb31bfef740 20 client.16158 trim_cache size 0 max 16384 2024-03-18T14:57:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.448+0000 7fb31bfef740 1 Processor -- start 2024-03-18T14:57:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.448+0000 7fb31bfef740 1 -- start start 2024-03-18T14:57:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.448+0000 7fb31bfef740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec42c0 0x561777ecd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.448+0000 7fb31bfef740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ec4be0 0x561777ed9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.448+0000 7fb31bfef740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ecd9b0 0x561777ee5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.448+0000 7fb31bfef740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb3141b1c30 con 0x561777ec4be0 2024-03-18T14:57:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.448+0000 7fb31bfef740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb3141af500 con 0x561777ecd9b0 2024-03-18T14:57:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.448+0000 7fb31bfef740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb3141ad350 con 0x561777ec42c0 2024-03-18T14:57:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.448+0000 7fb30bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ec4be0 0x561777ed9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.448+0000 7fb30bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ec4be0 0x561777ed9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:36620/0 (socket says 172.21.15.76:36620) 2024-03-18T14:57:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.448+0000 7fb30bfff640 1 -- 172.21.15.76:0/862235478 learned_addr learned my addr 172.21.15.76:0/862235478 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.448+0000 7fb3137fe640 1 --2- 172.21.15.76:0/862235478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ecd9b0 0x561777ee5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.448+0000 7fb312ffd640 1 --2- 172.21.15.76:0/862235478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec42c0 0x561777ecd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.449+0000 7fb30bfff640 1 -- 172.21.15.76:0/862235478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec42c0 msgr2=0x561777ecd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.449+0000 7fb30bfff640 1 --2- 172.21.15.76:0/862235478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec42c0 0x561777ecd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.449+0000 7fb30bfff640 1 -- 172.21.15.76:0/862235478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ecd9b0 msgr2=0x561777ee5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.449+0000 7fb30bfff640 1 --2- 172.21.15.76:0/862235478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ecd9b0 0x561777ee5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.449+0000 7fb30bfff640 1 -- 172.21.15.76:0/862235478 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561777ec8fc0 con 0x561777ec4be0 2024-03-18T14:57:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.449+0000 7fb30bfff640 1 --2- 172.21.15.76:0/862235478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ec4be0 0x561777ed9380 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc009710 tx=0x7fb2fc009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b193a69062ecda9c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.449+0000 7fb308ff9640 1 -- 172.21.15.76:0/862235478 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc026480 con 0x561777ec4be0 2024-03-18T14:57:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.449+0000 7fb308ff9640 1 -- 172.21.15.76:0/862235478 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2fc026620 con 0x561777ec4be0 2024-03-18T14:57:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.449+0000 7fb308ff9640 1 -- 172.21.15.76:0/862235478 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc003950 con 0x561777ec4be0 2024-03-18T14:57:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.450+0000 7fb31bfef740 1 -- 172.21.15.76:0/862235478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ec4be0 msgr2=0x561777ed9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.450+0000 7fb31bfef740 1 --2- 172.21.15.76:0/862235478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ec4be0 0x561777ed9380 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc009710 tx=0x7fb2fc009fd0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.450+0000 7fb31bfef740 1 -- 172.21.15.76:0/862235478 shutdown_connections 2024-03-18T14:57:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.450+0000 7fb31bfef740 1 --2- 172.21.15.76:0/862235478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ecd9b0 0x561777ee5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.450+0000 7fb31bfef740 1 --2- 172.21.15.76:0/862235478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ec4be0 0x561777ed9380 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.450+0000 7fb31bfef740 1 --2- 172.21.15.76:0/862235478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec42c0 0x561777ecd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.450+0000 7fb31bfef740 1 -- 172.21.15.76:0/862235478 >> 172.21.15.76:0/862235478 conn(0x561777ebebe0 msgr2=0x561777ecaa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.450+0000 7fb31bfef740 1 -- 172.21.15.76:0/862235478 shutdown_connections 2024-03-18T14:57:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.450+0000 7fb31bfef740 1 -- 172.21.15.76:0/862235478 wait complete. 2024-03-18T14:57:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.450+0000 7fb31bfef740 1 -- start start 2024-03-18T14:57:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.450+0000 7fb31bfef740 1 Processor -- start 2024-03-18T14:57:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.450+0000 7fb31bfef740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ec42c0 0x561777ec7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.450+0000 7fb31bfef740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec4be0 0x561777ec81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.450+0000 7fb31bfef740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ecd9b0 0x561777f42f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.450+0000 7fb31bfef740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb3141ad9c0 con 0x561777ecd9b0 2024-03-18T14:57:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.450+0000 7fb31bfef740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb3141aee30 con 0x561777ec42c0 2024-03-18T14:57:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.450+0000 7fb31bfef740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb3141ae090 con 0x561777ec4be0 2024-03-18T14:57:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb31bfef740 10 client.0 _ll_register_callbacks cb 0x7fb2fc009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:57:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb3137fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ecd9b0 0x561777f42f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb312ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ec42c0 0x561777ec7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb3137fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ecd9b0 0x561777f42f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:36622/0 (socket says 172.21.15.76:36622) 2024-03-18T14:57:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb3137fe640 1 -- 172.21.15.76:0/2164427610 learned_addr learned my addr 172.21.15.76:0/2164427610 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb30bfff640 1 --2- 172.21.15.76:0/2164427610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec4be0 0x561777ec81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb3137fe640 1 -- 172.21.15.76:0/2164427610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec4be0 msgr2=0x561777ec81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb3137fe640 1 --2- 172.21.15.76:0/2164427610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec4be0 0x561777ec81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb3137fe640 1 -- 172.21.15.76:0/2164427610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ec42c0 msgr2=0x561777ec7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb3137fe640 1 --2- 172.21.15.76:0/2164427610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ec42c0 0x561777ec7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb3137fe640 1 -- 172.21.15.76:0/2164427610 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561777f434b0 con 0x561777ecd9b0 2024-03-18T14:57:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb312ffd640 1 --2- 172.21.15.76:0/2164427610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ec42c0 0x561777ec7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:57:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb3137fe640 1 --2- 172.21.15.76:0/2164427610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ecd9b0 0x561777f42f70 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0139f0 tx=0x7fb30c005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac41157718f23ad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb2fa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:57:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb2fa7fc640 1 -- 172.21.15.76:0/2164427610 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c022040 con 0x561777ecd9b0 2024-03-18T14:57:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb2fa7fc640 1 -- 172.21.15.76:0/2164427610 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7fb30c013660 con 0x561777ecd9b0 2024-03-18T14:57:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb31bfef740 1 -- 172.21.15.76:0/2164427610 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2fc026620 con 0x561777ecd9b0 2024-03-18T14:57:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.451+0000 7fb2fa7fc640 1 -- 172.21.15.76:0/2164427610 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb30c03e050 con 0x561777ecd9b0 2024-03-18T14:57:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.452+0000 7fb31bfef740 10 client.16164 fetch_fsmap learned FSMap version 35 2024-03-18T14:57:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.452+0000 7fb31bfef740 1 -- 172.21.15.76:0/2164427610 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561777f44670 con 0x561777ecd9b0 2024-03-18T14:57:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.452+0000 7fb2fa7fc640 1 -- 172.21.15.76:0/2164427610 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2fc026620 con 0x561777ecd9b0 2024-03-18T14:57:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.452+0000 7fb2fa7fc640 1 -- 172.21.15.76:0/2164427610 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 35) ==== 8338+0+0 (secure 0 0 0) 0x7fb30c010630 con 0x561777ecd9b0 2024-03-18T14:57:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.452+0000 7fb31bfef740 10 client.16164 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T14:57:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.452+0000 7fb31bfef740 10 client.16164 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:57:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.452+0000 7fb31bfef740 20 client.16164 populate_metadata read hostname 'smithi076' 2024-03-18T14:57:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.452+0000 7fb31bfef740 1 --2- 172.21.15.76:0/2164427610 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561777f44ca0 0x561777f470c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.452+0000 7fb31bfef740 4 client.16164 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:57:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.452+0000 7fb31bfef740 1 -- 172.21.15.76:0/2164427610 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561777e302d0 con 0x561777f44ca0 2024-03-18T14:57:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.452+0000 7fb312ffd640 1 --2- 172.21.15.76:0/2164427610 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561777f44ca0 0x561777f470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.453+0000 7fb2fa7fc640 10 client.16164 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.453+0000 7fb312ffd640 1 --2- 172.21.15.76:0/2164427610 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561777f44ca0 0x561777f470c0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f08dfa9fb04dd97d server_cookie=592b72f602647f2c in_seq=0 out_seq=0 2024-03-18T14:57:17.526 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.526+0000 7fb2fa7fc640 1 -- 172.21.15.76:0/2164427610 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561777e302d0 con 0x561777f44ca0 2024-03-18T14:57:17.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.526+0000 7fb2fa7fc640 10 client.16164 handle_command_reply: tid=0 2024-03-18T14:57:17.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.526+0000 7fb31bfef740 1 client.16164 shutdown 2024-03-18T14:57:17.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.526+0000 7fb31bfef740 1 -- 172.21.15.76:0/2164427610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ecd9b0 msgr2=0x561777f42f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:17.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.526+0000 7fb31bfef740 1 --2- 172.21.15.76:0/2164427610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ecd9b0 0x561777f42f70 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0139f0 tx=0x7fb30c005f30 comp rx=0 tx=0).stop 2024-03-18T14:57:17.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.526+0000 7fb31bfef740 1 -- 172.21.15.76:0/2164427610 shutdown_connections 2024-03-18T14:57:17.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.526+0000 7fb31bfef740 1 --2- 172.21.15.76:0/2164427610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561777ec4be0 0x561777ec81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.526+0000 7fb31bfef740 1 --2- 172.21.15.76:0/2164427610 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561777f44ca0 0x561777f470c0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.526+0000 7fb31bfef740 1 --2- 172.21.15.76:0/2164427610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561777ecd9b0 0x561777f42f70 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.526+0000 7fb31bfef740 1 --2- 172.21.15.76:0/2164427610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561777ec42c0 0x561777ec7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.526+0000 7fb2fa7fc640 0 client.16164 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:17.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.526+0000 7fb31bfef740 1 -- 172.21.15.76:0/2164427610 >> 172.21.15.76:0/2164427610 conn(0x561777ebebe0 msgr2=0x561777ebdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:17.528 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.526+0000 7fb31bfef740 1 -- 172.21.15.76:0/2164427610 shutdown_connections 2024-03-18T14:57:17.529 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.527+0000 7fb31bfef740 1 -- 172.21.15.76:0/2164427610 wait complete. 2024-03-18T14:57:17.529 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.527+0000 7fb31bfef740 20 client.16164 trim_cache size 0 max 16384 2024-03-18T14:57:17.529 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:57:17.529 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:57:17.530 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:57:17.530 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:57:17.531 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.528+0000 7fb313fff640 1 -- 172.21.15.76:0/237997024 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb3000a2960 msgr2=0x7fb3000a4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:17.531 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.528+0000 7fb313fff640 1 --2- 172.21.15.76:0/237997024 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb3000a2960 0x7fb3000a4e20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb30c023000 tx=0x7fb30c021000 comp rx=0 tx=0).stop 2024-03-18T14:57:17.532 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.528+0000 7fb313fff640 1 -- 172.21.15.76:0/237997024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3140ffec0 msgr2=0x7fb31419a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:17.532 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.529+0000 7fb313fff640 1 --2- 172.21.15.76:0/237997024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3140ffec0 0x7fb31419a100 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb304006010 tx=0x7fb30402cb30 comp rx=0 tx=0).stop 2024-03-18T14:57:17.532 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.529+0000 7fb313fff640 1 -- 172.21.15.76:0/237997024 shutdown_connections 2024-03-18T14:57:17.532 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.529+0000 7fb313fff640 1 --2- 172.21.15.76:0/237997024 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb3000a2960 0x7fb3000a4e20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.532 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.529+0000 7fb313fff640 1 --2- 172.21.15.76:0/237997024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb314106640 0x7fb31419e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.532 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.529+0000 7fb313fff640 1 --2- 172.21.15.76:0/237997024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb314100880 0x7fb31419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.532 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.529+0000 7fb313fff640 1 --2- 172.21.15.76:0/237997024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3140ffec0 0x7fb31419a100 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:17.532 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.529+0000 7fb313fff640 1 -- 172.21.15.76:0/237997024 >> 172.21.15.76:0/237997024 conn(0x7fb314075850 msgr2=0x7fb31410cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:17.533 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.529+0000 7fb313fff640 1 -- 172.21.15.76:0/237997024 shutdown_connections 2024-03-18T14:57:17.533 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:17.529+0000 7fb313fff640 1 -- 172.21.15.76:0/237997024 wait complete. 2024-03-18T14:57:17.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 376219 10.37 MB/sec execute 78 sec latency 44.969 ms 2024-03-18T14:57:17.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:17 smithi076 ceph-mon[32205]: pgmap v127: 97 pgs: 97 active+clean; 109 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 281 op/s 2024-03-18T14:57:17.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:17 smithi174 ceph-mon[36787]: pgmap v127: 97 pgs: 97 active+clean; 109 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 281 op/s 2024-03-18T14:57:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:17 smithi077 ceph-mon[35865]: pgmap v127: 97 pgs: 97 active+clean; 109 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 281 op/s 2024-03-18T14:57:18.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 377583 10.31 MB/sec execute 79 sec latency 18.101 ms 2024-03-18T14:57:19.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 379830 10.35 MB/sec execute 80 sec latency 54.270 ms 2024-03-18T14:57:19.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:19 smithi174 ceph-mon[36787]: pgmap v128: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.4 MiB/s wr, 440 op/s 2024-03-18T14:57:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:19 smithi077 ceph-mon[35865]: pgmap v128: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.4 MiB/s wr, 440 op/s 2024-03-18T14:57:20.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:19 smithi076 ceph-mon[32205]: pgmap v128: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.4 MiB/s wr, 440 op/s 2024-03-18T14:57:20.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 381338 10.29 MB/sec execute 81 sec latency 33.884 ms 2024-03-18T14:57:21.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 383515 10.33 MB/sec execute 82 sec latency 11.472 ms 2024-03-18T14:57:21.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:21 smithi174 ceph-mon[36787]: pgmap v129: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 278 op/s 2024-03-18T14:57:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:21 smithi077 ceph-mon[35865]: pgmap v129: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 278 op/s 2024-03-18T14:57:22.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:21 smithi076 ceph-mon[32205]: pgmap v129: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 278 op/s 2024-03-18T14:57:22.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 385619 10.30 MB/sec execute 83 sec latency 14.279 ms 2024-03-18T14:57:23.740 INFO:tasks.workunit.client.0.smithi076.stdout: 1 387228 10.31 MB/sec execute 84 sec latency 31.708 ms 2024-03-18T14:57:23.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:23 smithi174 ceph-mon[36787]: pgmap v130: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.3 MiB/s wr, 392 op/s 2024-03-18T14:57:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:23 smithi077 ceph-mon[35865]: pgmap v130: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.3 MiB/s wr, 392 op/s 2024-03-18T14:57:24.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:23 smithi076 ceph-mon[32205]: pgmap v130: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.3 MiB/s wr, 392 op/s 2024-03-18T14:57:24.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 389438 10.32 MB/sec execute 85 sec latency 40.748 ms 2024-03-18T14:57:25.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 390813 10.30 MB/sec execute 86 sec latency 40.763 ms 2024-03-18T14:57:25.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:25 smithi174 ceph-mon[36787]: pgmap v131: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 273 op/s 2024-03-18T14:57:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:25 smithi077 ceph-mon[35865]: pgmap v131: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 273 op/s 2024-03-18T14:57:26.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:25 smithi076 ceph-mon[32205]: pgmap v131: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 273 op/s 2024-03-18T14:57:26.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 393240 10.32 MB/sec execute 87 sec latency 15.407 ms 2024-03-18T14:57:26.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:26 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:57:26.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:26 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:57:27.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:26 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:57:27.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 394640 10.29 MB/sec execute 88 sec latency 16.969 ms 2024-03-18T14:57:27.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:27 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:57:27.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:27 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:57:27.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:27 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:57:27.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:27 smithi174 ceph-mon[36787]: pgmap v132: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 273 op/s 2024-03-18T14:57:27.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:27 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:57:27.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:27 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:57:27.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:27 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:57:27.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:27 smithi077 ceph-mon[35865]: pgmap v132: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 273 op/s 2024-03-18T14:57:28.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:27 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:57:28.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:27 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:57:28.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:27 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:57:28.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:27 smithi076 ceph-mon[32205]: pgmap v132: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 273 op/s 2024-03-18T14:57:28.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 397513 10.38 MB/sec execute 89 sec latency 12.392 ms 2024-03-18T14:57:29.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 398735 10.32 MB/sec execute 90 sec latency 16.794 ms 2024-03-18T14:57:29.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:29 smithi174 ceph-mon[36787]: pgmap v133: 97 pgs: 97 active+clean; 116 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 383 op/s 2024-03-18T14:57:29.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:29 smithi077 ceph-mon[35865]: pgmap v133: 97 pgs: 97 active+clean; 116 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 383 op/s 2024-03-18T14:57:30.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:29 smithi076 ceph-mon[32205]: pgmap v133: 97 pgs: 97 active+clean; 116 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 383 op/s 2024-03-18T14:57:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.627+0000 7f7b20836640 1 -- 172.21.15.76:0/2236185867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b1c105680 msgr2=0x7f7b1c105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.627+0000 7f7b20836640 1 --2- 172.21.15.76:0/2236185867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b1c105680 0x7f7b1c105a60 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c009900 tx=0x7f7b0c02bc30 comp rx=0 tx=0).stop 2024-03-18T14:57:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.627+0000 7f7b20836640 1 -- 172.21.15.76:0/2236185867 shutdown_connections 2024-03-18T14:57:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.627+0000 7f7b20836640 1 --2- 172.21.15.76:0/2236185867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b1c100930 0x7f7b1c100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.627+0000 7f7b20836640 1 --2- 172.21.15.76:0/2236185867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b1c1012f0 0x7f7b1c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.627+0000 7f7b20836640 1 --2- 172.21.15.76:0/2236185867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b1c105680 0x7f7b1c105a60 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.627+0000 7f7b20836640 1 -- 172.21.15.76:0/2236185867 >> 172.21.15.76:0/2236185867 conn(0x7f7b1c077530 msgr2=0x7f7b1c077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.627+0000 7f7b20836640 1 -- 172.21.15.76:0/2236185867 shutdown_connections 2024-03-18T14:57:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.627+0000 7f7b20836640 1 -- 172.21.15.76:0/2236185867 wait complete. 2024-03-18T14:57:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.628+0000 7f7b20836640 1 Processor -- start 2024-03-18T14:57:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.628+0000 7f7b20836640 1 -- start start 2024-03-18T14:57:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.628+0000 7f7b20836640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b1c100930 0x7f7b1c196fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.628+0000 7f7b20836640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b1c1012f0 0x7f7b1c197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.628+0000 7f7b20836640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b1c105680 0x7f7b1c19b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.628+0000 7f7b20836640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c110ba0 con 0x7f7b1c1012f0 2024-03-18T14:57:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.628+0000 7f7b20836640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c110a20 con 0x7f7b1c100930 2024-03-18T14:57:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.628+0000 7f7b20836640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c110d20 con 0x7f7b1c105680 2024-03-18T14:57:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.628+0000 7f7b11d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b1c1012f0 0x7f7b1c197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.629+0000 7f7b1a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b1c100930 0x7f7b1c196fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.629+0000 7f7b11d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b1c1012f0 0x7f7b1c197500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34514/0 (socket says 172.21.15.76:34514) 2024-03-18T14:57:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.629+0000 7f7b11d9b640 1 -- 172.21.15.76:0/452810204 learned_addr learned my addr 172.21.15.76:0/452810204 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.629+0000 7f7b1ad9d640 1 --2- 172.21.15.76:0/452810204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b1c105680 0x7f7b1c19b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.629+0000 7f7b11d9b640 1 -- 172.21.15.76:0/452810204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b1c105680 msgr2=0x7f7b1c19b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.629+0000 7f7b11d9b640 1 --2- 172.21.15.76:0/452810204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b1c105680 0x7f7b1c19b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.629+0000 7f7b11d9b640 1 -- 172.21.15.76:0/452810204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b1c100930 msgr2=0x7f7b1c196fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.629+0000 7f7b11d9b640 1 --2- 172.21.15.76:0/452810204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b1c100930 0x7f7b1c196fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.629+0000 7f7b11d9b640 1 -- 172.21.15.76:0/452810204 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b1c19bfb0 con 0x7f7b1c1012f0 2024-03-18T14:57:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.629+0000 7f7b1a59c640 1 --2- 172.21.15.76:0/452810204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b1c100930 0x7f7b1c196fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:57:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.629+0000 7f7b11d9b640 1 --2- 172.21.15.76:0/452810204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b1c1012f0 0x7f7b1c197500 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400e9f0 tx=0x7f7b0400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.629+0000 7f7b13fff640 1 -- 172.21.15.76:0/452810204 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0400cd80 con 0x7f7b1c1012f0 2024-03-18T14:57:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.629+0000 7f7b13fff640 1 -- 172.21.15.76:0/452810204 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b04005500 con 0x7f7b1c1012f0 2024-03-18T14:57:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.629+0000 7f7b20836640 1 -- 172.21.15.76:0/452810204 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b1c19c2a0 con 0x7f7b1c1012f0 2024-03-18T14:57:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.629+0000 7f7b13fff640 1 -- 172.21.15.76:0/452810204 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b04010690 con 0x7f7b1c1012f0 2024-03-18T14:57:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.629+0000 7f7b20836640 1 -- 172.21.15.76:0/452810204 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b1c1a3b30 con 0x7f7b1c1012f0 2024-03-18T14:57:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.631+0000 7f7b13fff640 1 -- 172.21.15.76:0/452810204 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f7b040046c0 con 0x7f7b1c1012f0 2024-03-18T14:57:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.632+0000 7f7b23426740 1 Processor -- start 2024-03-18T14:57:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.632+0000 7f7b13fff640 1 --2- 172.21.15.76:0/452810204 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7b080a2a10 0x7f7b080a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.632+0000 7f7b1ad9d640 1 --2- 172.21.15.76:0/452810204 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7b080a2a10 0x7f7b080a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.632+0000 7f7b23426740 1 -- start start 2024-03-18T14:57:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.632+0000 7f7b13fff640 1 -- 172.21.15.76:0/452810204 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7f7b040cf200 con 0x7f7b1c1012f0 2024-03-18T14:57:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.632+0000 7f7b23426740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2adbd0 0x55f83e2adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.632+0000 7f7b23426740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2ae4f0 0x55f83e2b94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.632+0000 7f7b23426740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2b9a30 0x55f83e2bbe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.632+0000 7f7b23426740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f83e212a40 con 0x55f83e2ae4f0 2024-03-18T14:57:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.632+0000 7f7b23426740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f83e1bb920 con 0x55f83e2b9a30 2024-03-18T14:57:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.632+0000 7f7b23426740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f83e20eae0 con 0x55f83e2adbd0 2024-03-18T14:57:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.632+0000 7f7b11d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2ae4f0 0x55f83e2b94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.632+0000 7f7b11d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2ae4f0 0x55f83e2b94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34518/0 (socket says 172.21.15.76:34518) 2024-03-18T14:57:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.632+0000 7f7b11d9b640 1 -- 172.21.15.76:0/3405410511 learned_addr learned my addr 172.21.15.76:0/3405410511 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.632+0000 7f7b1a59c640 1 --2- 172.21.15.76:0/3405410511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2adbd0 0x55f83e2adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.632+0000 7f7b1ad9d640 1 --2- 172.21.15.76:0/452810204 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7b080a2a10 0x7f7b080a4ed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7b14005fd0 tx=0x7f7b14009500 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.632+0000 7f7b1ad9d640 1 --2- 172.21.15.76:0/3405410511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2b9a30 0x55f83e2bbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.633+0000 7f7b11d9b640 1 -- 172.21.15.76:0/3405410511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2adbd0 msgr2=0x55f83e2adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.633+0000 7f7b11d9b640 1 --2- 172.21.15.76:0/3405410511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2adbd0 0x55f83e2adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.633+0000 7f7b11d9b640 1 -- 172.21.15.76:0/3405410511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2b9a30 msgr2=0x55f83e2bbe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.633+0000 7f7b11d9b640 1 --2- 172.21.15.76:0/3405410511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2b9a30 0x55f83e2bbe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.633+0000 7f7b11d9b640 1 -- 172.21.15.76:0/3405410511 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f83e1d3460 con 0x55f83e2ae4f0 2024-03-18T14:57:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.633+0000 7f7b11d9b640 1 --2- 172.21.15.76:0/3405410511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2ae4f0 0x55f83e2b94f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f7b04009380 tx=0x7f7b0409b210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcd99543ae4a64af server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.633+0000 7f7b10d99640 1 -- 172.21.15.76:0/3405410511 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0409fa00 con 0x55f83e2ae4f0 2024-03-18T14:57:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.633+0000 7f7b10d99640 1 -- 172.21.15.76:0/3405410511 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b040a4050 con 0x55f83e2ae4f0 2024-03-18T14:57:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.633+0000 7f7b10d99640 1 -- 172.21.15.76:0/3405410511 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0409ce40 con 0x55f83e2ae4f0 2024-03-18T14:57:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.634+0000 7f7b23426740 1 -- 172.21.15.76:0/3405410511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2ae4f0 msgr2=0x55f83e2b94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.634+0000 7f7b23426740 1 --2- 172.21.15.76:0/3405410511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2ae4f0 0x55f83e2b94f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f7b04009380 tx=0x7f7b0409b210 comp rx=0 tx=0).stop 2024-03-18T14:57:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.634+0000 7f7b23426740 1 -- 172.21.15.76:0/3405410511 shutdown_connections 2024-03-18T14:57:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.634+0000 7f7b23426740 1 --2- 172.21.15.76:0/3405410511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2b9a30 0x55f83e2bbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.634+0000 7f7b23426740 1 --2- 172.21.15.76:0/3405410511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2ae4f0 0x55f83e2b94f0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.634+0000 7f7b23426740 1 --2- 172.21.15.76:0/3405410511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2adbd0 0x55f83e2adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.634+0000 7f7b23426740 1 -- 172.21.15.76:0/3405410511 >> 172.21.15.76:0/3405410511 conn(0x55f83e2a8a50 msgr2=0x55f83e2a8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.634+0000 7f7b23426740 1 -- 172.21.15.76:0/3405410511 shutdown_connections 2024-03-18T14:57:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.634+0000 7f7b23426740 1 -- 172.21.15.76:0/3405410511 wait complete. 2024-03-18T14:57:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.634+0000 7f7b23426740 1 -- start start 2024-03-18T14:57:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.634+0000 7f7b23426740 1 Processor -- start 2024-03-18T14:57:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.634+0000 7f7b23426740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2adbd0 0x55f83e2b6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.634+0000 7f7b23426740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2ae4f0 0x55f83e2b6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.634+0000 7f7b23426740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2b9a30 0x55f83e2b7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.634+0000 7f7b23426740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f83e2bc600 con 0x55f83e2ae4f0 2024-03-18T14:57:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.634+0000 7f7b23426740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f83e1ced70 con 0x55f83e2adbd0 2024-03-18T14:57:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.634+0000 7f7b23426740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f83e2bc780 con 0x55f83e2b9a30 2024-03-18T14:57:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b23426740 10 client.0 _ll_register_callbacks cb 0x55f83df26a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:57:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b11d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2ae4f0 0x55f83e2b6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b1a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2adbd0 0x55f83e2b6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b11d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2ae4f0 0x55f83e2b6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34530/0 (socket says 172.21.15.76:34530) 2024-03-18T14:57:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b11d9b640 1 -- 172.21.15.76:0/603061931 learned_addr learned my addr 172.21.15.76:0/603061931 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b1ad9d640 1 --2- 172.21.15.76:0/603061931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2b9a30 0x55f83e2b7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b11d9b640 1 -- 172.21.15.76:0/603061931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2b9a30 msgr2=0x55f83e2b7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b11d9b640 1 --2- 172.21.15.76:0/603061931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2b9a30 0x55f83e2b7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b11d9b640 1 -- 172.21.15.76:0/603061931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2adbd0 msgr2=0x55f83e2b6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b11d9b640 1 --2- 172.21.15.76:0/603061931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2adbd0 0x55f83e2b6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b11d9b640 1 -- 172.21.15.76:0/603061931 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f83df27210 con 0x55f83e2ae4f0 2024-03-18T14:57:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b11d9b640 1 --2- 172.21.15.76:0/603061931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2ae4f0 0x55f83e2b6fc0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7b04097d10 tx=0x7f7b04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf2e375969b19daa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:57:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b01ffb640 1 -- 172.21.15.76:0/603061931 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b040ba070 con 0x55f83e2ae4f0 2024-03-18T14:57:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b01ffb640 1 -- 172.21.15.76:0/603061931 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7f7b0409cde0 con 0x55f83e2ae4f0 2024-03-18T14:57:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b23426740 1 -- 172.21.15.76:0/603061931 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b040a4050 con 0x55f83e2ae4f0 2024-03-18T14:57:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b01ffb640 1 -- 172.21.15.76:0/603061931 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7b0409bb10 con 0x55f83e2ae4f0 2024-03-18T14:57:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b23426740 10 client.16176 fetch_fsmap learned FSMap version 35 2024-03-18T14:57:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.635+0000 7f7b23426740 1 -- 172.21.15.76:0/603061931 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f83dff3620 con 0x55f83e2ae4f0 2024-03-18T14:57:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.636+0000 7f7b01ffb640 1 -- 172.21.15.76:0/603061931 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b040a4050 con 0x55f83e2ae4f0 2024-03-18T14:57:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.636+0000 7f7b01ffb640 1 -- 172.21.15.76:0/603061931 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 35) ==== 8338+0+0 (secure 0 0 0) 0x7f7b040a6030 con 0x55f83e2ae4f0 2024-03-18T14:57:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.636+0000 7f7b23426740 10 client.16176 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T14:57:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.636+0000 7f7b23426740 10 client.16176 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:57:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.636+0000 7f7b23426740 20 client.16176 populate_metadata read hostname 'smithi076' 2024-03-18T14:57:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.636+0000 7f7b23426740 1 --2- 172.21.15.76:0/603061931 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f83e2b1050 0x55f83e2ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.636+0000 7f7b23426740 4 client.16176 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:57:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.636+0000 7f7b23426740 1 -- 172.21.15.76:0/603061931 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f83e21ff80 con 0x55f83e2b1050 2024-03-18T14:57:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.636+0000 7f7b1a59c640 1 --2- 172.21.15.76:0/603061931 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f83e2b1050 0x55f83e2ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.636+0000 7f7b1a59c640 1 --2- 172.21.15.76:0/603061931 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f83e2b1050 0x55f83e2ce0d0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ed30c5a085559f0 server_cookie=f9bb9ef2b2961d8b in_seq=0 out_seq=0 2024-03-18T14:57:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.636+0000 7f7b01ffb640 10 client.16176 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.637+0000 7f7b01ffb640 1 -- 172.21.15.76:0/603061931 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f83e21ff80 con 0x55f83e2b1050 2024-03-18T14:57:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.637+0000 7f7b01ffb640 10 client.16176 handle_command_reply: tid=0 2024-03-18T14:57:30.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.637+0000 7f7b23426740 1 client.16176 shutdown 2024-03-18T14:57:30.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.637+0000 7f7b23426740 1 -- 172.21.15.76:0/603061931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2ae4f0 msgr2=0x55f83e2b6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:30.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.637+0000 7f7b23426740 1 --2- 172.21.15.76:0/603061931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2ae4f0 0x55f83e2b6fc0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7b04097d10 tx=0x7f7b04004a40 comp rx=0 tx=0).stop 2024-03-18T14:57:30.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.637+0000 7f7b23426740 1 -- 172.21.15.76:0/603061931 shutdown_connections 2024-03-18T14:57:30.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.637+0000 7f7b23426740 1 --2- 172.21.15.76:0/603061931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2b9a30 0x55f83e2b7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.637+0000 7f7b23426740 1 --2- 172.21.15.76:0/603061931 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f83e2b1050 0x55f83e2ce0d0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.637+0000 7f7b23426740 1 --2- 172.21.15.76:0/603061931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2ae4f0 0x55f83e2b6fc0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.637+0000 7f7b01ffb640 0 client.16176 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:30.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.637+0000 7f7b23426740 1 --2- 172.21.15.76:0/603061931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2adbd0 0x55f83e2b6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.637+0000 7f7b23426740 1 -- 172.21.15.76:0/603061931 >> 172.21.15.76:0/603061931 conn(0x55f83e2a8a50 msgr2=0x55f83e2aa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:30.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.637+0000 7f7b23426740 1 -- 172.21.15.76:0/603061931 shutdown_connections 2024-03-18T14:57:30.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.637+0000 7f7b23426740 1 -- 172.21.15.76:0/603061931 wait complete. 2024-03-18T14:57:30.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.638+0000 7f7b23426740 20 client.16176 trim_cache size 0 max 16384 2024-03-18T14:57:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b23426740 1 Processor -- start 2024-03-18T14:57:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b23426740 1 -- start start 2024-03-18T14:57:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b23426740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2adbd0 0x55f83e2b91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b23426740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2ae4f0 0x55f83e2a7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b23426740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2b9a30 0x55f83e2d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b23426740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c1ad9e0 con 0x55f83e2adbd0 2024-03-18T14:57:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b23426740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c1ab2b0 con 0x55f83e2ae4f0 2024-03-18T14:57:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b23426740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c1a9120 con 0x55f83e2b9a30 2024-03-18T14:57:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b1a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2adbd0 0x55f83e2b91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b11d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2ae4f0 0x55f83e2a7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b11d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2ae4f0 0x55f83e2a7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:50512/0 (socket says 172.21.15.76:50512) 2024-03-18T14:57:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b11d9b640 1 -- 172.21.15.76:0/2358665876 learned_addr learned my addr 172.21.15.76:0/2358665876 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b1ad9d640 1 --2- 172.21.15.76:0/2358665876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2b9a30 0x55f83e2d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b1a59c640 1 -- 172.21.15.76:0/2358665876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2b9a30 msgr2=0x55f83e2d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b1a59c640 1 --2- 172.21.15.76:0/2358665876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2b9a30 0x55f83e2d10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b1a59c640 1 -- 172.21.15.76:0/2358665876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2ae4f0 msgr2=0x55f83e2a7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b1a59c640 1 --2- 172.21.15.76:0/2358665876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2ae4f0 0x55f83e2a7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b1a59c640 1 -- 172.21.15.76:0/2358665876 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f83e2b5a30 con 0x55f83e2adbd0 2024-03-18T14:57:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b11d9b640 1 --2- 172.21.15.76:0/2358665876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2ae4f0 0x55f83e2a7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:57:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.653+0000 7f7b1a59c640 1 --2- 172.21.15.76:0/2358665876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2adbd0 0x55f83e2b91b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c000c00 tx=0x7f7b0c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd78746ca87c5f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.654+0000 7f7b03fff640 1 -- 172.21.15.76:0/2358665876 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c04f070 con 0x55f83e2adbd0 2024-03-18T14:57:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.654+0000 7f7b03fff640 1 -- 172.21.15.76:0/2358665876 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b0c02c8b0 con 0x55f83e2adbd0 2024-03-18T14:57:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.654+0000 7f7b03fff640 1 -- 172.21.15.76:0/2358665876 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c005310 con 0x55f83e2adbd0 2024-03-18T14:57:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.654+0000 7f7b23426740 1 -- 172.21.15.76:0/2358665876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2adbd0 msgr2=0x55f83e2b91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.654+0000 7f7b23426740 1 --2- 172.21.15.76:0/2358665876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2adbd0 0x55f83e2b91b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c000c00 tx=0x7f7b0c009260 comp rx=0 tx=0).stop 2024-03-18T14:57:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.654+0000 7f7b23426740 1 -- 172.21.15.76:0/2358665876 shutdown_connections 2024-03-18T14:57:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.654+0000 7f7b23426740 1 --2- 172.21.15.76:0/2358665876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2b9a30 0x55f83e2d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.654+0000 7f7b23426740 1 --2- 172.21.15.76:0/2358665876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2ae4f0 0x55f83e2a7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.654+0000 7f7b23426740 1 --2- 172.21.15.76:0/2358665876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2adbd0 0x55f83e2b91b0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.654+0000 7f7b23426740 1 -- 172.21.15.76:0/2358665876 >> 172.21.15.76:0/2358665876 conn(0x55f83e2a8a50 msgr2=0x55f83e2a8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.655+0000 7f7b23426740 1 -- 172.21.15.76:0/2358665876 shutdown_connections 2024-03-18T14:57:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.655+0000 7f7b23426740 1 -- 172.21.15.76:0/2358665876 wait complete. 2024-03-18T14:57:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.655+0000 7f7b23426740 1 -- start start 2024-03-18T14:57:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.655+0000 7f7b23426740 1 Processor -- start 2024-03-18T14:57:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.655+0000 7f7b23426740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2adbd0 0x55f83e2cec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.655+0000 7f7b23426740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2ae4f0 0x55f83e2cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.655+0000 7f7b23426740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2b9a30 0x55f83e2cf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.655+0000 7f7b23426740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c1a9770 con 0x55f83e2b9a30 2024-03-18T14:57:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.655+0000 7f7b23426740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c1aabe0 con 0x55f83e2adbd0 2024-03-18T14:57:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.655+0000 7f7b23426740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c1a9e40 con 0x55f83e2ae4f0 2024-03-18T14:57:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.655+0000 7f7b23426740 10 client.0 _ll_register_callbacks cb 0x7f7b14009c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:57:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b1ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2b9a30 0x55f83e2cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b1a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2adbd0 0x55f83e2cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b11d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2ae4f0 0x55f83e2cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b11d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2ae4f0 0x55f83e2cf1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:49782/0 (socket says 172.21.15.76:49782) 2024-03-18T14:57:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b11d9b640 1 -- 172.21.15.76:0/1399683365 learned_addr learned my addr 172.21.15.76:0/1399683365 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b1ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2b9a30 0x55f83e2cf6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34536/0 (socket says 172.21.15.76:34536) 2024-03-18T14:57:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b1a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2adbd0 0x55f83e2cec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:50520/0 (socket says 172.21.15.76:50520) 2024-03-18T14:57:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b11d9b640 1 -- 172.21.15.76:0/1399683365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2adbd0 msgr2=0x55f83e2cec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b11d9b640 1 --2- 172.21.15.76:0/1399683365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2adbd0 0x55f83e2cec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b11d9b640 1 -- 172.21.15.76:0/1399683365 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2b9a30 msgr2=0x55f83e2cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b11d9b640 1 --2- 172.21.15.76:0/1399683365 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2b9a30 0x55f83e2cf6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b11d9b640 1 -- 172.21.15.76:0/1399683365 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f83e330ed0 con 0x55f83e2ae4f0 2024-03-18T14:57:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b1a59c640 1 --2- 172.21.15.76:0/1399683365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2adbd0 0x55f83e2cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:57:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b11d9b640 1 --2- 172.21.15.76:0/1399683365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2ae4f0 0x55f83e2cf1a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7b040a3f30 tx=0x7f7b0409bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a52ad48ca9280cb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-18T14:57:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b01ffb640 1 -- 172.21.15.76:0/1399683365 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b040ba040 con 0x55f83e2ae4f0 2024-03-18T14:57:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b01ffb640 1 -- 172.21.15.76:0/1399683365 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7f7b0409cde0 con 0x55f83e2ae4f0 2024-03-18T14:57:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b23426740 1 -- 172.21.15.76:0/1399683365 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b0c02c8b0 con 0x55f83e2ae4f0 2024-03-18T14:57:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b01ffb640 1 -- 172.21.15.76:0/1399683365 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7b040a4050 con 0x55f83e2ae4f0 2024-03-18T14:57:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b23426740 10 client.26029 fetch_fsmap learned FSMap version 35 2024-03-18T14:57:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.656+0000 7f7b23426740 1 -- 172.21.15.76:0/1399683365 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f83e330ed0 con 0x55f83e2ae4f0 2024-03-18T14:57:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.657+0000 7f7b01ffb640 1 -- 172.21.15.76:0/1399683365 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b0c02c8b0 con 0x55f83e2ae4f0 2024-03-18T14:57:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.657+0000 7f7b01ffb640 1 -- 172.21.15.76:0/1399683365 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 35) ==== 8338+0+0 (secure 0 0 0) 0x7f7b040a6040 con 0x55f83e2ae4f0 2024-03-18T14:57:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.657+0000 7f7b23426740 10 client.26029 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T14:57:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.657+0000 7f7b23426740 10 client.26029 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:57:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.657+0000 7f7b23426740 20 client.26029 populate_metadata read hostname 'smithi076' 2024-03-18T14:57:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.657+0000 7f7b23426740 1 --2- 172.21.15.76:0/1399683365 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f83e3324f0 0x55f83e334910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.657+0000 7f7b23426740 4 client.26029 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:57:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.657+0000 7f7b23426740 1 -- 172.21.15.76:0/1399683365 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f83e21a2d0 con 0x55f83e3324f0 2024-03-18T14:57:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.657+0000 7f7b1a59c640 1 --2- 172.21.15.76:0/1399683365 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f83e3324f0 0x55f83e334910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.657+0000 7f7b01ffb640 10 client.26029 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.657+0000 7f7b1a59c640 1 --2- 172.21.15.76:0/1399683365 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f83e3324f0 0x55f83e334910 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12572bc668998f58 server_cookie=cf2f49529633ce5b in_seq=0 out_seq=0 2024-03-18T14:57:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.735+0000 7f7b01ffb640 1 -- 172.21.15.76:0/1399683365 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f83e21a2d0 con 0x55f83e3324f0 2024-03-18T14:57:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.735+0000 7f7b01ffb640 10 client.26029 handle_command_reply: tid=0 2024-03-18T14:57:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.735+0000 7f7b23426740 1 client.26029 shutdown 2024-03-18T14:57:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.735+0000 7f7b23426740 1 -- 172.21.15.76:0/1399683365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2ae4f0 msgr2=0x55f83e2cf1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.735+0000 7f7b23426740 1 --2- 172.21.15.76:0/1399683365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2ae4f0 0x55f83e2cf1a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7b040a3f30 tx=0x7f7b0409bec0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.735+0000 7f7b23426740 1 -- 172.21.15.76:0/1399683365 shutdown_connections 2024-03-18T14:57:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.735+0000 7f7b23426740 1 --2- 172.21.15.76:0/1399683365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83e2ae4f0 0x55f83e2cf1a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.735+0000 7f7b23426740 1 --2- 172.21.15.76:0/1399683365 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f83e3324f0 0x55f83e334910 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.735+0000 7f7b23426740 1 --2- 172.21.15.76:0/1399683365 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f83e2b9a30 0x55f83e2cf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.735+0000 7f7b23426740 1 --2- 172.21.15.76:0/1399683365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f83e2adbd0 0x55f83e2cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.735+0000 7f7b01ffb640 0 client.26029 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.735+0000 7f7b23426740 1 -- 172.21.15.76:0/1399683365 >> 172.21.15.76:0/1399683365 conn(0x55f83e2a8a50 msgr2=0x55f83e2aa730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.735+0000 7f7b23426740 1 -- 172.21.15.76:0/1399683365 shutdown_connections 2024-03-18T14:57:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.735+0000 7f7b23426740 1 -- 172.21.15.76:0/1399683365 wait complete. 2024-03-18T14:57:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.735+0000 7f7b23426740 20 client.26029 trim_cache size 0 max 16384 2024-03-18T14:57:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:57:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:57:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:57:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:57:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.736+0000 7f7b20836640 1 -- 172.21.15.76:0/452810204 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7b080a2a10 msgr2=0x7f7b080a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.736+0000 7f7b20836640 1 --2- 172.21.15.76:0/452810204 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7b080a2a10 0x7f7b080a4ed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7b14005fd0 tx=0x7f7b14009500 comp rx=0 tx=0).stop 2024-03-18T14:57:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.736+0000 7f7b20836640 1 -- 172.21.15.76:0/452810204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b1c1012f0 msgr2=0x7f7b1c197500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.736+0000 7f7b20836640 1 --2- 172.21.15.76:0/452810204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b1c1012f0 0x7f7b1c197500 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400e9f0 tx=0x7f7b0400edf0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.736+0000 7f7b20836640 1 -- 172.21.15.76:0/452810204 shutdown_connections 2024-03-18T14:57:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.736+0000 7f7b20836640 1 --2- 172.21.15.76:0/452810204 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7b080a2a10 0x7f7b080a4ed0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.736+0000 7f7b20836640 1 --2- 172.21.15.76:0/452810204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b1c105680 0x7f7b1c19b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.736+0000 7f7b20836640 1 --2- 172.21.15.76:0/452810204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b1c1012f0 0x7f7b1c197500 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.736+0000 7f7b20836640 1 --2- 172.21.15.76:0/452810204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b1c100930 0x7f7b1c196fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.736+0000 7f7b20836640 1 -- 172.21.15.76:0/452810204 >> 172.21.15.76:0/452810204 conn(0x7f7b1c077530 msgr2=0x7f7b1c0fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.736+0000 7f7b20836640 1 -- 172.21.15.76:0/452810204 shutdown_connections 2024-03-18T14:57:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:30.736+0000 7f7b20836640 1 -- 172.21.15.76:0/452810204 wait complete. 2024-03-18T14:57:30.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 401083 10.36 MB/sec execute 91 sec latency 47.313 ms 2024-03-18T14:57:31.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 402388 10.30 MB/sec execute 92 sec latency 14.924 ms 2024-03-18T14:57:31.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:31 smithi174 ceph-mon[36787]: pgmap v134: 97 pgs: 97 active+clean; 116 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 224 op/s 2024-03-18T14:57:31.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:31 smithi077 ceph-mon[35865]: pgmap v134: 97 pgs: 97 active+clean; 116 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 224 op/s 2024-03-18T14:57:32.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:31 smithi076 ceph-mon[32205]: pgmap v134: 97 pgs: 97 active+clean; 116 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 224 op/s 2024-03-18T14:57:32.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 404792 10.34 MB/sec execute 93 sec latency 11.078 ms 2024-03-18T14:57:33.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 406652 10.30 MB/sec execute 94 sec latency 14.804 ms 2024-03-18T14:57:33.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:33 smithi174 ceph-mon[36787]: pgmap v135: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 378 op/s 2024-03-18T14:57:33.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:33 smithi077 ceph-mon[35865]: pgmap v135: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 378 op/s 2024-03-18T14:57:34.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:33 smithi076 ceph-mon[32205]: pgmap v135: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 378 op/s 2024-03-18T14:57:34.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 408425 10.33 MB/sec execute 95 sec latency 17.902 ms 2024-03-18T14:57:35.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 410544 10.30 MB/sec execute 96 sec latency 31.440 ms 2024-03-18T14:57:35.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:35 smithi174 ceph-mon[36787]: pgmap v136: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 264 op/s 2024-03-18T14:57:35.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:35 smithi077 ceph-mon[35865]: pgmap v136: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 264 op/s 2024-03-18T14:57:36.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:35 smithi076 ceph-mon[32205]: pgmap v136: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 264 op/s 2024-03-18T14:57:36.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 412172 10.31 MB/sec execute 97 sec latency 14.500 ms 2024-03-18T14:57:37.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 414502 10.34 MB/sec execute 98 sec latency 14.234 ms 2024-03-18T14:57:37.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:37 smithi174 ceph-mon[36787]: pgmap v137: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 264 op/s 2024-03-18T14:57:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:37 smithi077 ceph-mon[35865]: pgmap v137: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 264 op/s 2024-03-18T14:57:38.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:37 smithi076 ceph-mon[32205]: pgmap v137: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 264 op/s 2024-03-18T14:57:38.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 415914 10.30 MB/sec execute 99 sec latency 20.191 ms 2024-03-18T14:57:39.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 418724 10.37 MB/sec execute 100 sec latency 11.759 ms 2024-03-18T14:57:39.970 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:39 smithi174 ceph-mon[36787]: pgmap v138: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 374 op/s 2024-03-18T14:57:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:39 smithi077 ceph-mon[35865]: pgmap v138: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 374 op/s 2024-03-18T14:57:40.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:39 smithi076 ceph-mon[32205]: pgmap v138: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 374 op/s 2024-03-18T14:57:40.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 419652 10.30 MB/sec execute 101 sec latency 38.146 ms 2024-03-18T14:57:41.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 422372 10.37 MB/sec execute 102 sec latency 15.308 ms 2024-03-18T14:57:41.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:41 smithi174 ceph-mon[36787]: pgmap v139: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 263 op/s 2024-03-18T14:57:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:41 smithi077 ceph-mon[35865]: pgmap v139: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 263 op/s 2024-03-18T14:57:42.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:41 smithi076 ceph-mon[32205]: pgmap v139: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 263 op/s 2024-03-18T14:57:42.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 423791 10.32 MB/sec execute 103 sec latency 17.222 ms 2024-03-18T14:57:43.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 426068 10.35 MB/sec execute 104 sec latency 15.216 ms 2024-03-18T14:57:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.830+0000 7f7e8eaac640 1 -- 172.21.15.76:0/599273940 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e88107850 msgr2=0x7f7e88107c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.830+0000 7f7e8eaac640 1 --2- 172.21.15.76:0/599273940 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e88107850 0x7f7e88107c30 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f7e84009900 tx=0x7f7e8402bc30 comp rx=0 tx=0).stop 2024-03-18T14:57:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.830+0000 7f7e8eaac640 1 -- 172.21.15.76:0/599273940 shutdown_connections 2024-03-18T14:57:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.830+0000 7f7e8eaac640 1 --2- 172.21.15.76:0/599273940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e88069fe0 0x7f7e8806a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.830+0000 7f7e8eaac640 1 --2- 172.21.15.76:0/599273940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e8806a980 0x7f7e8810e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.830+0000 7f7e8eaac640 1 --2- 172.21.15.76:0/599273940 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e88107850 0x7f7e88107c30 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.830+0000 7f7e8eaac640 1 -- 172.21.15.76:0/599273940 >> 172.21.15.76:0/599273940 conn(0x7f7e880774b0 msgr2=0x7f7e880778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.830+0000 7f7e8eaac640 1 -- 172.21.15.76:0/599273940 shutdown_connections 2024-03-18T14:57:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.830+0000 7f7e8eaac640 1 -- 172.21.15.76:0/599273940 wait complete. 2024-03-18T14:57:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.830+0000 7f7e8eaac640 1 Processor -- start 2024-03-18T14:57:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e8eaac640 1 -- start start 2024-03-18T14:57:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e8eaac640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e88069fe0 0x7f7e88197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e8eaac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e8806a980 0x7f7e88198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e8eaac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e88107850 0x7f7e8819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e8eaac640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7e88111b30 con 0x7f7e88069fe0 2024-03-18T14:57:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e8eaac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7e881119b0 con 0x7f7e8806a980 2024-03-18T14:57:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e8eaac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7e88111cb0 con 0x7f7e88107850 2024-03-18T14:57:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e8c848640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e88069fe0 0x7f7e88197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e8c848640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e88069fe0 0x7f7e88197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53132/0 (socket says 172.21.15.76:53132) 2024-03-18T14:57:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e7ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e8806a980 0x7f7e88198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e8c848640 1 -- 172.21.15.76:0/11975161 learned_addr learned my addr 172.21.15.76:0/11975161 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e8d049640 1 --2- 172.21.15.76:0/11975161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e88107850 0x7f7e8819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e8c848640 1 -- 172.21.15.76:0/11975161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e88107850 msgr2=0x7f7e8819c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e8c848640 1 --2- 172.21.15.76:0/11975161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e88107850 0x7f7e8819c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e8c848640 1 -- 172.21.15.76:0/11975161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e8806a980 msgr2=0x7f7e88198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e8c848640 1 --2- 172.21.15.76:0/11975161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e8806a980 0x7f7e88198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e8c848640 1 -- 172.21.15.76:0/11975161 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e8819cee0 con 0x7f7e88069fe0 2024-03-18T14:57:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e7ffff640 1 --2- 172.21.15.76:0/11975161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e8806a980 0x7f7e88198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:57:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e8c848640 1 --2- 172.21.15.76:0/11975161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e88069fe0 0x7f7e88197ef0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f7e84006010 tx=0x7f7e8402cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.831+0000 7f7e7dffb640 1 -- 172.21.15.76:0/11975161 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e84041070 con 0x7f7e88069fe0 2024-03-18T14:57:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.832+0000 7f7e7dffb640 1 -- 172.21.15.76:0/11975161 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e84004450 con 0x7f7e88069fe0 2024-03-18T14:57:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.832+0000 7f7e7dffb640 1 -- 172.21.15.76:0/11975161 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e84003e00 con 0x7f7e88069fe0 2024-03-18T14:57:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.832+0000 7f7e8eaac640 1 -- 172.21.15.76:0/11975161 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e8819d170 con 0x7f7e88069fe0 2024-03-18T14:57:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.832+0000 7f7e8eaac640 1 -- 172.21.15.76:0/11975161 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e881a4a60 con 0x7f7e88069fe0 2024-03-18T14:57:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.835+0000 7f7e7dffb640 1 -- 172.21.15.76:0/11975161 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f7e84044070 con 0x7f7e88069fe0 2024-03-18T14:57:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.835+0000 7f7e91774740 1 Processor -- start 2024-03-18T14:57:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.835+0000 7f7e91774740 1 -- start start 2024-03-18T14:57:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.835+0000 7f7e7dffb640 1 --2- 172.21.15.76:0/11975161 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7e700a2a10 0x7f7e700a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.835+0000 7f7e91774740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6bfd130 0x5584e6bfd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e91774740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfda50 0x5584e6c062e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e91774740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6c06820 0x5584e6c08c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e91774740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5584e6b61a40 con 0x5584e6bfda50 2024-03-18T14:57:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e91774740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5584e6b0a920 con 0x5584e6c06820 2024-03-18T14:57:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e91774740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5584e6b5dae0 con 0x5584e6bfd130 2024-03-18T14:57:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e8c848640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6bfd130 0x5584e6bfd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e8d049640 1 --2- 172.21.15.76:0/11975161 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7e700a2a10 0x7f7e700a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e8c848640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6bfd130 0x5584e6bfd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:46604/0 (socket says 172.21.15.76:46604) 2024-03-18T14:57:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e8c848640 1 -- 172.21.15.76:0/1189207549 learned_addr learned my addr 172.21.15.76:0/1189207549 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e7ffff640 1 --2- 172.21.15.76:0/1189207549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfda50 0x5584e6c062e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e8d049640 1 --2- 172.21.15.76:0/1189207549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6c06820 0x5584e6c08c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e7dffb640 1 -- 172.21.15.76:0/11975161 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7f7e840f0730 con 0x7f7e88069fe0 2024-03-18T14:57:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e8c848640 1 -- 172.21.15.76:0/1189207549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6c06820 msgr2=0x5584e6c08c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e8c848640 1 --2- 172.21.15.76:0/1189207549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6c06820 0x5584e6c08c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e8c848640 1 -- 172.21.15.76:0/1189207549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfda50 msgr2=0x5584e6c062e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e8c848640 1 --2- 172.21.15.76:0/1189207549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfda50 0x5584e6c062e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e8c848640 1 -- 172.21.15.76:0/1189207549 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584e6b22460 con 0x5584e6bfd130 2024-03-18T14:57:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e8d049640 1 --2- 172.21.15.76:0/11975161 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7e700a2a10 0x7f7e700a4ed0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x5584e6bf8e70 tx=0x7f7e80009bf0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e8c848640 1 --2- 172.21.15.76:0/1189207549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6bfd130 0x5584e6bfd510 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7e84000c00 tx=0x7f7e8404ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66298fe74ca87ecf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.836+0000 7f7e6effd640 1 -- 172.21.15.76:0/1189207549 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8404b070 con 0x5584e6bfd130 2024-03-18T14:57:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.837+0000 7f7e6effd640 1 -- 172.21.15.76:0/1189207549 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e84039230 con 0x5584e6bfd130 2024-03-18T14:57:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.837+0000 7f7e6effd640 1 -- 172.21.15.76:0/1189207549 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e840c6550 con 0x5584e6bfd130 2024-03-18T14:57:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.837+0000 7f7e91774740 1 -- 172.21.15.76:0/1189207549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6bfd130 msgr2=0x5584e6bfd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.837+0000 7f7e91774740 1 --2- 172.21.15.76:0/1189207549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6bfd130 0x5584e6bfd510 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7e84000c00 tx=0x7f7e8404ef90 comp rx=0 tx=0).stop 2024-03-18T14:57:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.837+0000 7f7e91774740 1 -- 172.21.15.76:0/1189207549 shutdown_connections 2024-03-18T14:57:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.837+0000 7f7e91774740 1 --2- 172.21.15.76:0/1189207549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6c06820 0x5584e6c08c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.837+0000 7f7e91774740 1 --2- 172.21.15.76:0/1189207549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfda50 0x5584e6c062e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.837+0000 7f7e91774740 1 --2- 172.21.15.76:0/1189207549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6bfd130 0x5584e6bfd510 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.837+0000 7f7e91774740 1 -- 172.21.15.76:0/1189207549 >> 172.21.15.76:0/1189207549 conn(0x5584e6bf7a50 msgr2=0x5584e6bf7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.837+0000 7f7e91774740 1 -- 172.21.15.76:0/1189207549 shutdown_connections 2024-03-18T14:57:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.837+0000 7f7e91774740 1 -- 172.21.15.76:0/1189207549 wait complete. 2024-03-18T14:57:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e91774740 1 -- start start 2024-03-18T14:57:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e91774740 1 Processor -- start 2024-03-18T14:57:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e91774740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c03870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e91774740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6bfda50 0x5584e6c03db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e91774740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6c06820 0x5584e6c042f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e91774740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5584e6c09210 con 0x5584e6bfd130 2024-03-18T14:57:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e91774740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5584e6b1dd70 con 0x5584e6c06820 2024-03-18T14:57:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e91774740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5584e6c09390 con 0x5584e6bfda50 2024-03-18T14:57:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e91774740 10 client.0 _ll_register_callbacks cb 0x5584e6875a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:57:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e8c848640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c03870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e8c848640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c03870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53154/0 (socket says 172.21.15.76:53154) 2024-03-18T14:57:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e8c848640 1 -- 172.21.15.76:0/4095792956 learned_addr learned my addr 172.21.15.76:0/4095792956 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e7ffff640 1 --2- 172.21.15.76:0/4095792956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6bfda50 0x5584e6c03db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e8d049640 1 --2- 172.21.15.76:0/4095792956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6c06820 0x5584e6c042f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e8c848640 1 -- 172.21.15.76:0/4095792956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6bfda50 msgr2=0x5584e6c03db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e8c848640 1 --2- 172.21.15.76:0/4095792956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6bfda50 0x5584e6c03db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e8c848640 1 -- 172.21.15.76:0/4095792956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6c06820 msgr2=0x5584e6c042f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e8c848640 1 --2- 172.21.15.76:0/4095792956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6c06820 0x5584e6c042f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.838+0000 7f7e8c848640 1 -- 172.21.15.76:0/4095792956 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584e6876210 con 0x5584e6bfd130 2024-03-18T14:57:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.839+0000 7f7e8c848640 1 --2- 172.21.15.76:0/4095792956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c03870 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7e840bcf30 tx=0x7f7e8402ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4dcd572c91252f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.839+0000 7f7e67fff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:57:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.839+0000 7f7e67fff640 1 -- 172.21.15.76:0/4095792956 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8404b070 con 0x5584e6bfd130 2024-03-18T14:57:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.839+0000 7f7e67fff640 1 -- 172.21.15.76:0/4095792956 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7f7e840ee0c0 con 0x5584e6bfd130 2024-03-18T14:57:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.839+0000 7f7e91774740 1 -- 172.21.15.76:0/4095792956 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5584e6b6ef80 con 0x5584e6bfd130 2024-03-18T14:57:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.839+0000 7f7e67fff640 1 -- 172.21.15.76:0/4095792956 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7e840bc900 con 0x5584e6bfd130 2024-03-18T14:57:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.839+0000 7f7e91774740 10 client.16194 fetch_fsmap learned FSMap version 35 2024-03-18T14:57:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.839+0000 7f7e91774740 1 -- 172.21.15.76:0/4095792956 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5584e6942620 con 0x5584e6bfd130 2024-03-18T14:57:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.839+0000 7f7e67fff640 1 -- 172.21.15.76:0/4095792956 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5584e6b6ef80 con 0x5584e6bfd130 2024-03-18T14:57:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.839+0000 7f7e67fff640 1 -- 172.21.15.76:0/4095792956 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 35) ==== 8338+0+0 (secure 0 0 0) 0x7f7e840be220 con 0x5584e6bfd130 2024-03-18T14:57:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.839+0000 7f7e91774740 10 client.16194 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T14:57:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.839+0000 7f7e91774740 10 client.16194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:57:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.839+0000 7f7e91774740 20 client.16194 populate_metadata read hostname 'smithi076' 2024-03-18T14:57:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.839+0000 7f7e91774740 1 --2- 172.21.15.76:0/4095792956 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5584e6c16590 0x5584e6c18980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.839+0000 7f7e91774740 4 client.16194 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:57:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.839+0000 7f7e91774740 1 -- 172.21.15.76:0/4095792956 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584e6b692d0 con 0x5584e6c16590 2024-03-18T14:57:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.840+0000 7f7e7ffff640 1 --2- 172.21.15.76:0/4095792956 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5584e6c16590 0x5584e6c18980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.840+0000 7f7e67fff640 10 client.16194 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.840+0000 7f7e7ffff640 1 --2- 172.21.15.76:0/4095792956 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5584e6c16590 0x5584e6c18980 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62a6f55573cf1701 server_cookie=be75398a0a575217 in_seq=0 out_seq=0 2024-03-18T14:57:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.841+0000 7f7e67fff640 1 -- 172.21.15.76:0/4095792956 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5584e6b692d0 con 0x5584e6c16590 2024-03-18T14:57:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.841+0000 7f7e67fff640 10 client.16194 handle_command_reply: tid=0 2024-03-18T14:57:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.841+0000 7f7e91774740 1 client.16194 shutdown 2024-03-18T14:57:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.841+0000 7f7e91774740 1 -- 172.21.15.76:0/4095792956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 msgr2=0x5584e6c03870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.841+0000 7f7e91774740 1 --2- 172.21.15.76:0/4095792956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c03870 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7e840bcf30 tx=0x7f7e8402ce20 comp rx=0 tx=0).stop 2024-03-18T14:57:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.841+0000 7f7e91774740 1 -- 172.21.15.76:0/4095792956 shutdown_connections 2024-03-18T14:57:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.841+0000 7f7e91774740 1 --2- 172.21.15.76:0/4095792956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6bfda50 0x5584e6c03db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.841+0000 7f7e91774740 1 --2- 172.21.15.76:0/4095792956 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5584e6c16590 0x5584e6c18980 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.841+0000 7f7e91774740 1 --2- 172.21.15.76:0/4095792956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6c06820 0x5584e6c042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.841+0000 7f7e91774740 1 --2- 172.21.15.76:0/4095792956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c03870 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.841+0000 7f7e67fff640 0 client.16194 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.841+0000 7f7e91774740 1 -- 172.21.15.76:0/4095792956 >> 172.21.15.76:0/4095792956 conn(0x5584e6bf7a50 msgr2=0x5584e6bf97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.841+0000 7f7e91774740 1 -- 172.21.15.76:0/4095792956 shutdown_connections 2024-03-18T14:57:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.841+0000 7f7e91774740 1 -- 172.21.15.76:0/4095792956 wait complete. 2024-03-18T14:57:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.841+0000 7f7e91774740 20 client.16194 trim_cache size 0 max 16384 2024-03-18T14:57:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.856+0000 7f7e91774740 1 Processor -- start 2024-03-18T14:57:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.856+0000 7f7e91774740 1 -- start start 2024-03-18T14:57:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e91774740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c11e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e91774740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6bfda50 0x5584e6c1e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e91774740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6c06820 0x5584e6bf6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e91774740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7e881afc10 con 0x5584e6bfd130 2024-03-18T14:57:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e91774740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7e881ad4e0 con 0x5584e6bfda50 2024-03-18T14:57:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e91774740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7e881ab330 con 0x5584e6c06820 2024-03-18T14:57:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e8c848640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c11e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e8d049640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6c06820 0x5584e6bf6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e8c848640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c11e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53164/0 (socket says 172.21.15.76:53164) 2024-03-18T14:57:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e8c848640 1 -- 172.21.15.76:0/1249922793 learned_addr learned my addr 172.21.15.76:0/1249922793 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e7ffff640 1 --2- 172.21.15.76:0/1249922793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6bfda50 0x5584e6c1e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e8c848640 1 -- 172.21.15.76:0/1249922793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6c06820 msgr2=0x5584e6bf6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e8c848640 1 --2- 172.21.15.76:0/1249922793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6c06820 0x5584e6bf6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e8c848640 1 -- 172.21.15.76:0/1249922793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6bfda50 msgr2=0x5584e6c1e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e8c848640 1 --2- 172.21.15.76:0/1249922793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6bfda50 0x5584e6c1e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e8c848640 1 -- 172.21.15.76:0/1249922793 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584e6bf3310 con 0x5584e6bfd130 2024-03-18T14:57:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e8d049640 1 --2- 172.21.15.76:0/1249922793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6c06820 0x5584e6bf6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:57:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e7ffff640 1 --2- 172.21.15.76:0/1249922793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6bfda50 0x5584e6c1e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:57:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e8c848640 1 --2- 172.21.15.76:0/1249922793 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c11e30 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7e840bb250 tx=0x7f7e840bcd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46b21924925a412a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e6e7fc640 1 -- 172.21.15.76:0/1249922793 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8404b070 con 0x5584e6bfd130 2024-03-18T14:57:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e6e7fc640 1 -- 172.21.15.76:0/1249922793 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e840c1f60 con 0x5584e6bfd130 2024-03-18T14:57:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.857+0000 7f7e6e7fc640 1 -- 172.21.15.76:0/1249922793 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e840c1040 con 0x5584e6bfd130 2024-03-18T14:57:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.858+0000 7f7e91774740 1 -- 172.21.15.76:0/1249922793 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 msgr2=0x5584e6c11e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.858+0000 7f7e91774740 1 --2- 172.21.15.76:0/1249922793 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c11e30 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7e840bb250 tx=0x7f7e840bcd70 comp rx=0 tx=0).stop 2024-03-18T14:57:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.858+0000 7f7e91774740 1 -- 172.21.15.76:0/1249922793 shutdown_connections 2024-03-18T14:57:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.858+0000 7f7e91774740 1 --2- 172.21.15.76:0/1249922793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6c06820 0x5584e6bf6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.858+0000 7f7e91774740 1 --2- 172.21.15.76:0/1249922793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6bfda50 0x5584e6c1e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.858+0000 7f7e91774740 1 --2- 172.21.15.76:0/1249922793 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c11e30 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.858+0000 7f7e91774740 1 -- 172.21.15.76:0/1249922793 >> 172.21.15.76:0/1249922793 conn(0x5584e6bf7a50 msgr2=0x5584e6bf2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.858+0000 7f7e91774740 1 -- 172.21.15.76:0/1249922793 shutdown_connections 2024-03-18T14:57:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.858+0000 7f7e91774740 1 -- 172.21.15.76:0/1249922793 wait complete. 2024-03-18T14:57:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.858+0000 7f7e91774740 1 -- start start 2024-03-18T14:57:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.858+0000 7f7e91774740 1 Processor -- start 2024-03-18T14:57:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e91774740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c0e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e91774740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6bfda50 0x5584e6c09ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e91774740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6c06820 0x5584e6c0a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e91774740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7e881ab9a0 con 0x5584e6bfd130 2024-03-18T14:57:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e91774740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7e881ace10 con 0x5584e6c06820 2024-03-18T14:57:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e91774740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7e881ac070 con 0x5584e6bfda50 2024-03-18T14:57:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e91774740 10 client.0 _ll_register_callbacks cb 0x7f7e78009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:57:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e8c848640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c0e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e8c848640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c0e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53180/0 (socket says 172.21.15.76:53180) 2024-03-18T14:57:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e8d049640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6c06820 0x5584e6c0a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e8c848640 1 -- 172.21.15.76:0/836661763 learned_addr learned my addr 172.21.15.76:0/836661763 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e7ffff640 1 --2- 172.21.15.76:0/836661763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6bfda50 0x5584e6c09ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e8d049640 1 -- 172.21.15.76:0/836661763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6bfda50 msgr2=0x5584e6c09ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e8d049640 1 --2- 172.21.15.76:0/836661763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6bfda50 0x5584e6c09ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e8d049640 1 -- 172.21.15.76:0/836661763 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 msgr2=0x5584e6c0e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e8d049640 1 --2- 172.21.15.76:0/836661763 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c0e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e8d049640 1 -- 172.21.15.76:0/836661763 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584e6c0a830 con 0x5584e6c06820 2024-03-18T14:57:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.859+0000 7f7e8c848640 1 --2- 172.21.15.76:0/836661763 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c0e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:57:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.860+0000 7f7e8d049640 1 --2- 172.21.15.76:0/836661763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6c06820 0x5584e6c0a000 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009e50 tx=0x7f7e80004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47969a79c29623a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.860+0000 7f7e67fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:57:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.860+0000 7f7e67fff640 1 -- 172.21.15.76:0/836661763 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e80004b20 con 0x5584e6c06820 2024-03-18T14:57:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.860+0000 7f7e67fff640 1 -- 172.21.15.76:0/836661763 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7f7e80004cc0 con 0x5584e6c06820 2024-03-18T14:57:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.860+0000 7f7e91774740 1 -- 172.21.15.76:0/836661763 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e840c1f60 con 0x5584e6c06820 2024-03-18T14:57:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.860+0000 7f7e67fff640 1 -- 172.21.15.76:0/836661763 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7e8001f650 con 0x5584e6c06820 2024-03-18T14:57:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.860+0000 7f7e91774740 10 client.26057 fetch_fsmap learned FSMap version 35 2024-03-18T14:57:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.860+0000 7f7e91774740 1 -- 172.21.15.76:0/836661763 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5584e6c7c8f0 con 0x5584e6c06820 2024-03-18T14:57:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.860+0000 7f7e67fff640 1 -- 172.21.15.76:0/836661763 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e840c1f60 con 0x5584e6c06820 2024-03-18T14:57:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.860+0000 7f7e67fff640 1 -- 172.21.15.76:0/836661763 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 35) ==== 8338+0+0 (secure 0 0 0) 0x7f7e8002a070 con 0x5584e6c06820 2024-03-18T14:57:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.860+0000 7f7e91774740 10 client.26057 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T14:57:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.860+0000 7f7e91774740 10 client.26057 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:57:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.860+0000 7f7e91774740 20 client.26057 populate_metadata read hostname 'smithi076' 2024-03-18T14:57:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.860+0000 7f7e91774740 1 --2- 172.21.15.76:0/836661763 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5584e6c7d0c0 0x5584e6c7f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.860+0000 7f7e91774740 4 client.26057 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:57:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.860+0000 7f7e91774740 1 -- 172.21.15.76:0/836661763 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5584e6ae7770 con 0x5584e6c7d0c0 2024-03-18T14:57:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.860+0000 7f7e8c848640 1 --2- 172.21.15.76:0/836661763 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5584e6c7d0c0 0x5584e6c7f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.861+0000 7f7e67fff640 10 client.26057 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.861+0000 7f7e8c848640 1 --2- 172.21.15.76:0/836661763 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5584e6c7d0c0 0x5584e6c7f4e0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0f02dbd2cd84607 server_cookie=476ad85c519aee72 in_seq=0 out_seq=0 2024-03-18T14:57:43.928 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.927+0000 7f7e67fff640 1 -- 172.21.15.76:0/836661763 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5584e6ae7770 con 0x5584e6c7d0c0 2024-03-18T14:57:43.928 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.927+0000 7f7e67fff640 10 client.26057 handle_command_reply: tid=0 2024-03-18T14:57:43.928 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.927+0000 7f7e91774740 1 client.26057 shutdown 2024-03-18T14:57:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.928+0000 7f7e91774740 1 -- 172.21.15.76:0/836661763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6c06820 msgr2=0x5584e6c0a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.928+0000 7f7e91774740 1 --2- 172.21.15.76:0/836661763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6c06820 0x5584e6c0a000 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009e50 tx=0x7f7e80004a40 comp rx=0 tx=0).stop 2024-03-18T14:57:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.928+0000 7f7e91774740 1 -- 172.21.15.76:0/836661763 shutdown_connections 2024-03-18T14:57:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.928+0000 7f7e91774740 1 --2- 172.21.15.76:0/836661763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584e6bfda50 0x5584e6c09ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.928+0000 7f7e91774740 1 --2- 172.21.15.76:0/836661763 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5584e6c7d0c0 0x5584e6c7f4e0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.928+0000 7f7e91774740 1 --2- 172.21.15.76:0/836661763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e6c06820 0x5584e6c0a000 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.928+0000 7f7e91774740 1 --2- 172.21.15.76:0/836661763 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5584e6bfd130 0x5584e6c0e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.928+0000 7f7e91774740 1 -- 172.21.15.76:0/836661763 >> 172.21.15.76:0/836661763 conn(0x5584e6bf7a50 msgr2=0x5584e6bf8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.928+0000 7f7e67fff640 0 client.26057 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.929+0000 7f7e91774740 1 -- 172.21.15.76:0/836661763 shutdown_connections 2024-03-18T14:57:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.929+0000 7f7e91774740 1 -- 172.21.15.76:0/836661763 wait complete. 2024-03-18T14:57:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.929+0000 7f7e91774740 20 client.26057 trim_cache size 0 max 16384 2024-03-18T14:57:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:57:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:57:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:57:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:57:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.929+0000 7f7e8eaac640 1 -- 172.21.15.76:0/11975161 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7e700a2a10 msgr2=0x7f7e700a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.929+0000 7f7e8eaac640 1 --2- 172.21.15.76:0/11975161 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7e700a2a10 0x7f7e700a4ed0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x5584e6bf8e70 tx=0x7f7e80009bf0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.929+0000 7f7e8eaac640 1 -- 172.21.15.76:0/11975161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e88069fe0 msgr2=0x7f7e88197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.929+0000 7f7e8eaac640 1 --2- 172.21.15.76:0/11975161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e88069fe0 0x7f7e88197ef0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f7e84006010 tx=0x7f7e8402cb30 comp rx=0 tx=0).stop 2024-03-18T14:57:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.930+0000 7f7e8eaac640 1 -- 172.21.15.76:0/11975161 shutdown_connections 2024-03-18T14:57:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.930+0000 7f7e8eaac640 1 --2- 172.21.15.76:0/11975161 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7e700a2a10 0x7f7e700a4ed0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.930+0000 7f7e8eaac640 1 --2- 172.21.15.76:0/11975161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e88107850 0x7f7e8819c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.930+0000 7f7e8eaac640 1 --2- 172.21.15.76:0/11975161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e8806a980 0x7f7e88198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.930+0000 7f7e8eaac640 1 --2- 172.21.15.76:0/11975161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e88069fe0 0x7f7e88197ef0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.930+0000 7f7e8eaac640 1 -- 172.21.15.76:0/11975161 >> 172.21.15.76:0/11975161 conn(0x7f7e880774b0 msgr2=0x7f7e880ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.930+0000 7f7e8eaac640 1 -- 172.21.15.76:0/11975161 shutdown_connections 2024-03-18T14:57:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:43.930+0000 7f7e8eaac640 1 -- 172.21.15.76:0/11975161 wait complete. 2024-03-18T14:57:43.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:43 smithi174 ceph-mon[36787]: pgmap v140: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 377 op/s 2024-03-18T14:57:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:43 smithi077 ceph-mon[35865]: pgmap v140: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 377 op/s 2024-03-18T14:57:44.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:43 smithi076 ceph-mon[32205]: pgmap v140: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 377 op/s 2024-03-18T14:57:44.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 428076 10.32 MB/sec execute 105 sec latency 53.116 ms 2024-03-18T14:57:45.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 429685 10.34 MB/sec execute 106 sec latency 61.133 ms 2024-03-18T14:57:45.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:45 smithi174 ceph-mon[36787]: pgmap v141: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 223 op/s 2024-03-18T14:57:45.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:45 smithi077 ceph-mon[35865]: pgmap v141: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 223 op/s 2024-03-18T14:57:46.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:45 smithi076 ceph-mon[32205]: pgmap v141: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 223 op/s 2024-03-18T14:57:46.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 432203 10.36 MB/sec execute 107 sec latency 16.598 ms 2024-03-18T14:57:47.741 INFO:tasks.workunit.client.0.smithi076.stdout: 1 433550 10.33 MB/sec execute 108 sec latency 32.425 ms 2024-03-18T14:57:47.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:47 smithi076 ceph-mon[32205]: pgmap v142: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 223 op/s 2024-03-18T14:57:47.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:47 smithi174 ceph-mon[36787]: pgmap v142: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 223 op/s 2024-03-18T14:57:47.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:47 smithi077 ceph-mon[35865]: pgmap v142: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 223 op/s 2024-03-18T14:57:48.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 436158 10.36 MB/sec execute 109 sec latency 17.713 ms 2024-03-18T14:57:49.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 437439 10.33 MB/sec execute 110 sec latency 16.865 ms 2024-03-18T14:57:49.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:49 smithi174 ceph-mon[36787]: pgmap v143: 97 pgs: 97 active+clean; 118 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 384 op/s 2024-03-18T14:57:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:49 smithi077 ceph-mon[35865]: pgmap v143: 97 pgs: 97 active+clean; 118 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 384 op/s 2024-03-18T14:57:50.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:49 smithi076 ceph-mon[32205]: pgmap v143: 97 pgs: 97 active+clean; 118 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 384 op/s 2024-03-18T14:57:50.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 440084 10.39 MB/sec execute 111 sec latency 31.775 ms 2024-03-18T14:57:51.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 441477 10.34 MB/sec execute 112 sec latency 16.979 ms 2024-03-18T14:57:51.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:51 smithi174 ceph-mon[36787]: pgmap v144: 97 pgs: 97 active+clean; 118 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 275 op/s 2024-03-18T14:57:51.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:51 smithi077 ceph-mon[35865]: pgmap v144: 97 pgs: 97 active+clean; 118 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 275 op/s 2024-03-18T14:57:52.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:51 smithi076 ceph-mon[32205]: pgmap v144: 97 pgs: 97 active+clean; 118 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 275 op/s 2024-03-18T14:57:52.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 443764 10.37 MB/sec execute 113 sec latency 10.697 ms 2024-03-18T14:57:53.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 445320 10.33 MB/sec execute 114 sec latency 16.277 ms 2024-03-18T14:57:53.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:53 smithi174 ceph-mon[36787]: pgmap v145: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 388 op/s 2024-03-18T14:57:53.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:53 smithi077 ceph-mon[35865]: pgmap v145: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 388 op/s 2024-03-18T14:57:54.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:53 smithi076 ceph-mon[32205]: pgmap v145: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 388 op/s 2024-03-18T14:57:54.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 447373 10.36 MB/sec execute 115 sec latency 15.853 ms 2024-03-18T14:57:55.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 449336 10.33 MB/sec execute 116 sec latency 48.760 ms 2024-03-18T14:57:55.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:55 smithi174 ceph-mon[36787]: pgmap v146: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 275 op/s 2024-03-18T14:57:55.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:55 smithi077 ceph-mon[35865]: pgmap v146: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 275 op/s 2024-03-18T14:57:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:55 smithi076 ceph-mon[32205]: pgmap v146: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 275 op/s 2024-03-18T14:57:56.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 451000 10.35 MB/sec execute 117 sec latency 15.876 ms 2024-03-18T14:57:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.028+0000 7feb1f177640 1 -- 172.21.15.76:0/944902070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb18074530 msgr2=0x7feb1810da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.028+0000 7feb1f177640 1 --2- 172.21.15.76:0/944902070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb18074530 0x7feb1810da90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7feb04009900 tx=0x7feb0402bcd0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.028+0000 7feb1f177640 1 -- 172.21.15.76:0/944902070 shutdown_connections 2024-03-18T14:57:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.028+0000 7feb1f177640 1 --2- 172.21.15.76:0/944902070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb1810dfd0 0x7feb18110420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.028+0000 7feb1f177640 1 --2- 172.21.15.76:0/944902070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb18074530 0x7feb1810da90 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.028+0000 7feb1f177640 1 --2- 172.21.15.76:0/944902070 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb18073c10 0x7feb18073ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.028+0000 7feb1f177640 1 -- 172.21.15.76:0/944902070 >> 172.21.15.76:0/944902070 conn(0x7feb180fd260 msgr2=0x7feb180ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.028+0000 7feb1f177640 1 -- 172.21.15.76:0/944902070 shutdown_connections 2024-03-18T14:57:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.028+0000 7feb1f177640 1 -- 172.21.15.76:0/944902070 wait complete. 2024-03-18T14:57:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.029+0000 7feb1f177640 1 Processor -- start 2024-03-18T14:57:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.029+0000 7feb1f177640 1 -- start start 2024-03-18T14:57:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.029+0000 7feb1f177640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb18073c10 0x7feb1819a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.029+0000 7feb1f177640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb18074530 0x7feb1819a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.029+0000 7feb1f177640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb1810dfd0 0x7feb1819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.029+0000 7feb1f177640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7feb18113dc0 con 0x7feb18073c10 2024-03-18T14:57:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.029+0000 7feb1f177640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7feb18113c40 con 0x7feb18074530 2024-03-18T14:57:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.029+0000 7feb1f177640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feb18113f40 con 0x7feb1810dfd0 2024-03-18T14:57:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.029+0000 7feb1cf13640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb18073c10 0x7feb1819a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.029+0000 7feb1cf13640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb18073c10 0x7feb1819a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35746/0 (socket says 172.21.15.76:35746) 2024-03-18T14:57:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.029+0000 7feb1cf13640 1 -- 172.21.15.76:0/2769176990 learned_addr learned my addr 172.21.15.76:0/2769176990 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.029+0000 7feb17fff640 1 --2- 172.21.15.76:0/2769176990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb18074530 0x7feb1819a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.030+0000 7feb1d714640 1 --2- 172.21.15.76:0/2769176990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb1810dfd0 0x7feb1819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.030+0000 7feb1cf13640 1 -- 172.21.15.76:0/2769176990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb1810dfd0 msgr2=0x7feb1819eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.030+0000 7feb1cf13640 1 --2- 172.21.15.76:0/2769176990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb1810dfd0 0x7feb1819eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.030+0000 7feb1cf13640 1 -- 172.21.15.76:0/2769176990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb18074530 msgr2=0x7feb1819a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.030+0000 7feb1cf13640 1 --2- 172.21.15.76:0/2769176990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb18074530 0x7feb1819a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.030+0000 7feb1cf13640 1 -- 172.21.15.76:0/2769176990 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb1819f280 con 0x7feb18073c10 2024-03-18T14:57:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.030+0000 7feb17fff640 1 --2- 172.21.15.76:0/2769176990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb18074530 0x7feb1819a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:57:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.030+0000 7feb1cf13640 1 --2- 172.21.15.76:0/2769176990 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb18073c10 0x7feb1819a2e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7feb0c00d950 tx=0x7feb0c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.030+0000 7feb15ffb640 1 -- 172.21.15.76:0/2769176990 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c017070 con 0x7feb18073c10 2024-03-18T14:57:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.030+0000 7feb1f177640 1 -- 172.21.15.76:0/2769176990 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb1819f570 con 0x7feb18073c10 2024-03-18T14:57:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.030+0000 7feb15ffb640 1 -- 172.21.15.76:0/2769176990 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb0c009c30 con 0x7feb18073c10 2024-03-18T14:57:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.030+0000 7feb15ffb640 1 -- 172.21.15.76:0/2769176990 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c00f6b0 con 0x7feb18073c10 2024-03-18T14:57:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.031+0000 7feb1f177640 1 -- 172.21.15.76:0/2769176990 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb181a6e00 con 0x7feb18073c10 2024-03-18T14:57:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.033+0000 7feb15ffb640 1 -- 172.21.15.76:0/2769176990 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7feb0c0109d0 con 0x7feb18073c10 2024-03-18T14:57:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.033+0000 7feb21d65740 1 Processor -- start 2024-03-18T14:57:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.033+0000 7feb15ffb640 1 --2- 172.21.15.76:0/2769176990 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7feb080a2a10 0x7feb080a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.033+0000 7feb21d65740 1 -- start start 2024-03-18T14:57:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb15ffb640 1 -- 172.21.15.76:0/2769176990 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7feb0c0cea30 con 0x7feb18073c10 2024-03-18T14:57:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb1d714640 1 --2- 172.21.15.76:0/2769176990 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7feb080a2a10 0x7feb080a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb21d65740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0e0130 0x55cbef0e0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb21d65740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0e0a50 0x55cbef0f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb21d65740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0f2030 0x55cbef0f4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb21d65740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55cbef044a40 con 0x55cbef0e0a50 2024-03-18T14:57:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb21d65740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cbeefed920 con 0x55cbef0f2030 2024-03-18T14:57:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb21d65740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cbef040ae0 con 0x55cbef0e0130 2024-03-18T14:57:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb1cf13640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0e0130 0x55cbef0e0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb17fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0e0a50 0x55cbef0f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb1cf13640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0e0130 0x55cbef0e0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:56164/0 (socket says 172.21.15.76:56164) 2024-03-18T14:57:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb1cf13640 1 -- 172.21.15.76:0/2029646114 learned_addr learned my addr 172.21.15.76:0/2029646114 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb1d714640 1 --2- 172.21.15.76:0/2029646114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0f2030 0x55cbef0f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb1d714640 1 --2- 172.21.15.76:0/2769176990 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7feb080a2a10 0x7feb080a4ed0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7feb10005fd0 tx=0x7feb100107a0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb17fff640 1 -- 172.21.15.76:0/2029646114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0e0130 msgr2=0x55cbef0e0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb17fff640 1 --2- 172.21.15.76:0/2029646114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0e0130 0x55cbef0e0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb17fff640 1 -- 172.21.15.76:0/2029646114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0f2030 msgr2=0x55cbef0f4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb17fff640 1 --2- 172.21.15.76:0/2029646114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0f2030 0x55cbef0f4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb17fff640 1 -- 172.21.15.76:0/2029646114 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbef005460 con 0x55cbef0e0a50 2024-03-18T14:57:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb1d714640 1 --2- 172.21.15.76:0/2029646114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0f2030 0x55cbef0f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:57:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.034+0000 7feb17fff640 1 --2- 172.21.15.76:0/2029646114 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0e0a50 0x55cbef0f1af0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7feb0402c210 tx=0x7feb040095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b133ca1ab97d0d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.035+0000 7feb02ffd640 1 -- 172.21.15.76:0/2029646114 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb04043070 con 0x55cbef0e0a50 2024-03-18T14:57:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.035+0000 7feb02ffd640 1 -- 172.21.15.76:0/2029646114 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb04004a70 con 0x55cbef0e0a50 2024-03-18T14:57:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.035+0000 7feb02ffd640 1 -- 172.21.15.76:0/2029646114 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb04005140 con 0x55cbef0e0a50 2024-03-18T14:57:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.035+0000 7feb21d65740 1 -- 172.21.15.76:0/2029646114 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0e0a50 msgr2=0x55cbef0f1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.035+0000 7feb21d65740 1 --2- 172.21.15.76:0/2029646114 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0e0a50 0x55cbef0f1af0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7feb0402c210 tx=0x7feb040095e0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.035+0000 7feb21d65740 1 -- 172.21.15.76:0/2029646114 shutdown_connections 2024-03-18T14:57:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.035+0000 7feb21d65740 1 --2- 172.21.15.76:0/2029646114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0f2030 0x55cbef0f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.035+0000 7feb21d65740 1 --2- 172.21.15.76:0/2029646114 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0e0a50 0x55cbef0f1af0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.035+0000 7feb21d65740 1 --2- 172.21.15.76:0/2029646114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0e0130 0x55cbef0e0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.035+0000 7feb21d65740 1 -- 172.21.15.76:0/2029646114 >> 172.21.15.76:0/2029646114 conn(0x55cbef0daa50 msgr2=0x55cbef0dae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.035+0000 7feb21d65740 1 -- 172.21.15.76:0/2029646114 shutdown_connections 2024-03-18T14:57:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.036+0000 7feb21d65740 1 -- 172.21.15.76:0/2029646114 wait complete. 2024-03-18T14:57:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.036+0000 7feb21d65740 1 -- start start 2024-03-18T14:57:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.036+0000 7feb21d65740 1 Processor -- start 2024-03-18T14:57:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.036+0000 7feb21d65740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 0x55cbef0ef070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.036+0000 7feb21d65740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0e0a50 0x55cbef0ef5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.036+0000 7feb21d65740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0f2030 0x55cbef0efaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.036+0000 7feb21d65740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55cbef0f4c00 con 0x55cbef0f2030 2024-03-18T14:57:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.036+0000 7feb21d65740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cbef000d70 con 0x55cbef0e0130 2024-03-18T14:57:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.036+0000 7feb21d65740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cbef0f4d80 con 0x55cbef0e0a50 2024-03-18T14:57:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.036+0000 7feb21d65740 10 client.0 _ll_register_callbacks cb 0x55cbeed58a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:57:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.036+0000 7feb1d714640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0f2030 0x55cbef0efaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.036+0000 7feb1cf13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 0x55cbef0ef070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.036+0000 7feb1d714640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0f2030 0x55cbef0efaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35764/0 (socket says 172.21.15.76:35764) 2024-03-18T14:57:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.036+0000 7feb1d714640 1 -- 172.21.15.76:0/1238641312 learned_addr learned my addr 172.21.15.76:0/1238641312 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.036+0000 7feb17fff640 1 --2- 172.21.15.76:0/1238641312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0e0a50 0x55cbef0ef5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.037+0000 7feb1d714640 1 -- 172.21.15.76:0/1238641312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0e0a50 msgr2=0x55cbef0ef5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.037+0000 7feb1d714640 1 --2- 172.21.15.76:0/1238641312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0e0a50 0x55cbef0ef5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.037+0000 7feb1d714640 1 -- 172.21.15.76:0/1238641312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 msgr2=0x55cbef0ef070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.037+0000 7feb1d714640 1 --2- 172.21.15.76:0/1238641312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 0x55cbef0ef070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.037+0000 7feb1d714640 1 -- 172.21.15.76:0/1238641312 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbeed59210 con 0x55cbef0f2030 2024-03-18T14:57:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.037+0000 7feb1cf13640 1 --2- 172.21.15.76:0/1238641312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 0x55cbef0ef070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:57:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.037+0000 7feb1d714640 1 --2- 172.21.15.76:0/1238641312 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0f2030 0x55cbef0efaf0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7feb10013d80 tx=0x7feb10004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c79798d045d49d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.037+0000 7feaf7fff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:57:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.037+0000 7feaf7fff640 1 -- 172.21.15.76:0/1238641312 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb10003970 con 0x55cbef0f2030 2024-03-18T14:57:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.037+0000 7feaf7fff640 1 -- 172.21.15.76:0/1238641312 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7feb10003b10 con 0x55cbef0f2030 2024-03-18T14:57:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.037+0000 7feb21d65740 1 -- 172.21.15.76:0/1238641312 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb04004a70 con 0x55cbef0f2030 2024-03-18T14:57:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.037+0000 7feaf7fff640 1 -- 172.21.15.76:0/1238641312 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7feb10003d10 con 0x55cbef0f2030 2024-03-18T14:57:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.037+0000 7feb21d65740 10 client.16212 fetch_fsmap learned FSMap version 35 2024-03-18T14:57:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.037+0000 7feb21d65740 1 -- 172.21.15.76:0/1238641312 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55cbeee25620 con 0x55cbef0f2030 2024-03-18T14:57:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.037+0000 7feaf7fff640 1 -- 172.21.15.76:0/1238641312 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb04004a70 con 0x55cbef0f2030 2024-03-18T14:57:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.038+0000 7feaf7fff640 1 -- 172.21.15.76:0/1238641312 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 35) ==== 8338+0+0 (secure 0 0 0) 0x7feb10026020 con 0x55cbef0f2030 2024-03-18T14:57:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.038+0000 7feb21d65740 10 client.16212 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T14:57:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.038+0000 7feb21d65740 10 client.16212 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:57:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.038+0000 7feb21d65740 20 client.16212 populate_metadata read hostname 'smithi076' 2024-03-18T14:57:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.038+0000 7feb21d65740 1 --2- 172.21.15.76:0/1238641312 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55cbef102160 0x55cbef104570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.038+0000 7feb21d65740 4 client.16212 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:57:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.038+0000 7feb21d65740 1 -- 172.21.15.76:0/1238641312 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbef051f80 con 0x55cbef102160 2024-03-18T14:57:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.038+0000 7feb1cf13640 1 --2- 172.21.15.76:0/1238641312 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55cbef102160 0x55cbef104570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.038+0000 7feaf7fff640 10 client.16212 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.038+0000 7feb1cf13640 1 --2- 172.21.15.76:0/1238641312 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55cbef102160 0x55cbef104570 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dd729b67a8b34fc server_cookie=acf791f32f4247da in_seq=0 out_seq=0 2024-03-18T14:57:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.039+0000 7feaf7fff640 1 -- 172.21.15.76:0/1238641312 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cbef051f80 con 0x55cbef102160 2024-03-18T14:57:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.039+0000 7feaf7fff640 10 client.16212 handle_command_reply: tid=0 2024-03-18T14:57:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.039+0000 7feb21d65740 1 client.16212 shutdown 2024-03-18T14:57:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.039+0000 7feb21d65740 1 -- 172.21.15.76:0/1238641312 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0f2030 msgr2=0x55cbef0efaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.039+0000 7feb21d65740 1 --2- 172.21.15.76:0/1238641312 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0f2030 0x55cbef0efaf0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7feb10013d80 tx=0x7feb10004ab0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.039+0000 7feb21d65740 1 -- 172.21.15.76:0/1238641312 shutdown_connections 2024-03-18T14:57:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.039+0000 7feb21d65740 1 --2- 172.21.15.76:0/1238641312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0e0a50 0x55cbef0ef5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.039+0000 7feb21d65740 1 --2- 172.21.15.76:0/1238641312 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55cbef102160 0x55cbef104570 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.039+0000 7feb21d65740 1 --2- 172.21.15.76:0/1238641312 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0f2030 0x55cbef0efaf0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.039+0000 7feb21d65740 1 --2- 172.21.15.76:0/1238641312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 0x55cbef0ef070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.039+0000 7feaf7fff640 0 client.16212 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.039+0000 7feb21d65740 1 -- 172.21.15.76:0/1238641312 >> 172.21.15.76:0/1238641312 conn(0x55cbef0daa50 msgr2=0x55cbef0dc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.039+0000 7feb21d65740 1 -- 172.21.15.76:0/1238641312 shutdown_connections 2024-03-18T14:57:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.039+0000 7feb21d65740 1 -- 172.21.15.76:0/1238641312 wait complete. 2024-03-18T14:57:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.039+0000 7feb21d65740 20 client.16212 trim_cache size 0 max 16384 2024-03-18T14:57:57.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.055+0000 7feb21d65740 1 Processor -- start 2024-03-18T14:57:57.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.055+0000 7feb21d65740 1 -- start start 2024-03-18T14:57:57.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.055+0000 7feb21d65740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 0x55cbef0fd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.055+0000 7feb21d65740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0e0a50 0x55cbef0d9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.055+0000 7feb21d65740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0f2030 0x55cbef109e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.055+0000 7feb21d65740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7feb181ad9f0 con 0x55cbef0f2030 2024-03-18T14:57:57.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.055+0000 7feb21d65740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7feb181ab2c0 con 0x55cbef0e0130 2024-03-18T14:57:57.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.055+0000 7feb21d65740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feb181a9110 con 0x55cbef0e0a50 2024-03-18T14:57:57.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.056+0000 7feb1cf13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 0x55cbef0fd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.056+0000 7feb1d714640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0f2030 0x55cbef109e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.056+0000 7feb1d714640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0f2030 0x55cbef109e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35774/0 (socket says 172.21.15.76:35774) 2024-03-18T14:57:57.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.056+0000 7feb1cf13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 0x55cbef0fd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:34752/0 (socket says 172.21.15.76:34752) 2024-03-18T14:57:57.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.056+0000 7feb1d714640 1 -- 172.21.15.76:0/796765209 learned_addr learned my addr 172.21.15.76:0/796765209 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:57.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.056+0000 7feb17fff640 1 --2- 172.21.15.76:0/796765209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0e0a50 0x55cbef0d9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.056+0000 7feb1cf13640 1 -- 172.21.15.76:0/796765209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0e0a50 msgr2=0x55cbef0d9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:57.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.056+0000 7feb1cf13640 1 --2- 172.21.15.76:0/796765209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0e0a50 0x55cbef0d9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.056+0000 7feb1cf13640 1 -- 172.21.15.76:0/796765209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0f2030 msgr2=0x55cbef109e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:57.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.056+0000 7feb1cf13640 1 --2- 172.21.15.76:0/796765209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0f2030 0x55cbef109e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.056+0000 7feb1cf13640 1 -- 172.21.15.76:0/796765209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbef0db620 con 0x55cbef0e0130 2024-03-18T14:57:57.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.056+0000 7feb1d714640 1 --2- 172.21.15.76:0/796765209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0f2030 0x55cbef109e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:57:57.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.056+0000 7feb1cf13640 1 --2- 172.21.15.76:0/796765209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 0x55cbef0fd9a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feb0c097540 tx=0x7feb0c0bb040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63763b4240819558 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:57.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.056+0000 7feb027fc640 1 -- 172.21.15.76:0/796765209 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c0b3070 con 0x55cbef0e0130 2024-03-18T14:57:57.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.056+0000 7feb027fc640 1 -- 172.21.15.76:0/796765209 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb0c09b0e0 con 0x55cbef0e0130 2024-03-18T14:57:57.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.056+0000 7feb027fc640 1 -- 172.21.15.76:0/796765209 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c020040 con 0x55cbef0e0130 2024-03-18T14:57:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.057+0000 7feb21d65740 1 -- 172.21.15.76:0/796765209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 msgr2=0x55cbef0fd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.057+0000 7feb21d65740 1 --2- 172.21.15.76:0/796765209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 0x55cbef0fd9a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feb0c097540 tx=0x7feb0c0bb040 comp rx=0 tx=0).stop 2024-03-18T14:57:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.057+0000 7feb21d65740 1 -- 172.21.15.76:0/796765209 shutdown_connections 2024-03-18T14:57:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.057+0000 7feb21d65740 1 --2- 172.21.15.76:0/796765209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0e0a50 0x55cbef0d9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.057+0000 7feb21d65740 1 --2- 172.21.15.76:0/796765209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0f2030 0x55cbef109e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.057+0000 7feb21d65740 1 --2- 172.21.15.76:0/796765209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 0x55cbef0fd9a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.057+0000 7feb21d65740 1 -- 172.21.15.76:0/796765209 >> 172.21.15.76:0/796765209 conn(0x55cbef0daa50 msgr2=0x55cbef0d9320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.057+0000 7feb21d65740 1 -- 172.21.15.76:0/796765209 shutdown_connections 2024-03-18T14:57:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.057+0000 7feb21d65740 1 -- 172.21.15.76:0/796765209 wait complete. 2024-03-18T14:57:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.057+0000 7feb21d65740 1 -- start start 2024-03-18T14:57:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.057+0000 7feb21d65740 1 Processor -- start 2024-03-18T14:57:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb21d65740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 0x55cbef0f6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb21d65740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0e0a50 0x55cbef0f6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb21d65740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0f2030 0x55cbef1674b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb21d65740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7feb181a9780 con 0x55cbef0e0a50 2024-03-18T14:57:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb21d65740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7feb181aabf0 con 0x55cbef0e0130 2024-03-18T14:57:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb21d65740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feb181a9e50 con 0x55cbef0f2030 2024-03-18T14:57:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb21d65740 10 client.0 _ll_register_callbacks cb 0x7feb10013a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:57:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb17fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0e0a50 0x55cbef0f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb1cf13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 0x55cbef0f6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb1cf13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 0x55cbef0f6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:34766/0 (socket says 172.21.15.76:34766) 2024-03-18T14:57:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb1cf13640 1 -- 172.21.15.76:0/3297866945 learned_addr learned my addr 172.21.15.76:0/3297866945 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:57:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb1d714640 1 --2- 172.21.15.76:0/3297866945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0f2030 0x55cbef1674b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb1cf13640 1 -- 172.21.15.76:0/3297866945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0f2030 msgr2=0x55cbef1674b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb1cf13640 1 --2- 172.21.15.76:0/3297866945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0f2030 0x55cbef1674b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb1cf13640 1 -- 172.21.15.76:0/3297866945 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0e0a50 msgr2=0x55cbef0f6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:57:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb1cf13640 1 --2- 172.21.15.76:0/3297866945 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0e0a50 0x55cbef0f6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb1cf13640 1 -- 172.21.15.76:0/3297866945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbef167a40 con 0x55cbef0e0130 2024-03-18T14:57:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feb1cf13640 1 --2- 172.21.15.76:0/3297866945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 0x55cbef0f6040 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feb0c000c00 tx=0x7feb0c09f2b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a38d305a8696e49 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:57:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.058+0000 7feaf7fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:57:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.059+0000 7feaf7fff640 1 -- 172.21.15.76:0/3297866945 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c0b3070 con 0x55cbef0e0130 2024-03-18T14:57:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.059+0000 7feb21d65740 1 -- 172.21.15.76:0/3297866945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb0c09b0e0 con 0x55cbef0e0130 2024-03-18T14:57:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.059+0000 7feaf7fff640 1 -- 172.21.15.76:0/3297866945 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8440+0+0 (secure 0 0 0) 0x7feb0c097750 con 0x55cbef0e0130 2024-03-18T14:57:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.059+0000 7feaf7fff640 1 -- 172.21.15.76:0/3297866945 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7feb0c020050 con 0x55cbef0e0130 2024-03-18T14:57:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.059+0000 7feb21d65740 10 client.26075 fetch_fsmap learned FSMap version 35 2024-03-18T14:57:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.059+0000 7feb21d65740 1 -- 172.21.15.76:0/3297866945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55cbef168d50 con 0x55cbef0e0130 2024-03-18T14:57:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.059+0000 7feaf7fff640 1 -- 172.21.15.76:0/3297866945 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb0c09b0e0 con 0x55cbef0e0130 2024-03-18T14:57:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.059+0000 7feaf7fff640 1 -- 172.21.15.76:0/3297866945 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 35) ==== 8338+0+0 (secure 0 0 0) 0x7feb0c0b8020 con 0x55cbef0e0130 2024-03-18T14:57:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.059+0000 7feb21d65740 10 client.26075 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T14:57:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.059+0000 7feb21d65740 10 client.26075 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:57:57.065 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.059+0000 7feb21d65740 20 client.26075 populate_metadata read hostname 'smithi076' 2024-03-18T14:57:57.065 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.059+0000 7feb21d65740 1 --2- 172.21.15.76:0/3297866945 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55cbef169510 0x55cbef16b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:57:57.065 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.059+0000 7feb21d65740 4 client.26075 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:57:57.065 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.059+0000 7feb21d65740 1 -- 172.21.15.76:0/3297866945 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cbef04c2d0 con 0x55cbef169510 2024-03-18T14:57:57.065 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.059+0000 7feb17fff640 1 --2- 172.21.15.76:0/3297866945 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55cbef169510 0x55cbef16b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:57:57.065 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.060+0000 7feb17fff640 1 --2- 172.21.15.76:0/3297866945 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55cbef169510 0x55cbef16b930 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8431e357bdc3d0ef server_cookie=2fd047f06705915d in_seq=0 out_seq=0 2024-03-18T14:57:57.065 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.060+0000 7feaf7fff640 10 client.26075 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.128+0000 7feaf7fff640 1 -- 172.21.15.76:0/3297866945 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cbef04c2d0 con 0x55cbef169510 2024-03-18T14:57:57.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.128+0000 7feaf7fff640 10 client.26075 handle_command_reply: tid=0 2024-03-18T14:57:57.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.128+0000 7feb21d65740 1 client.26075 shutdown 2024-03-18T14:57:57.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.128+0000 7feb21d65740 1 -- 172.21.15.76:0/3297866945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 msgr2=0x55cbef0f6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:57.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.128+0000 7feb21d65740 1 --2- 172.21.15.76:0/3297866945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 0x55cbef0f6040 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feb0c000c00 tx=0x7feb0c09f2b0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.129+0000 7feb21d65740 1 -- 172.21.15.76:0/3297866945 shutdown_connections 2024-03-18T14:57:57.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.129+0000 7feb21d65740 1 --2- 172.21.15.76:0/3297866945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbef0f2030 0x55cbef1674b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.129+0000 7feb21d65740 1 --2- 172.21.15.76:0/3297866945 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55cbef169510 0x55cbef16b930 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.129+0000 7feaf7fff640 0 client.26075 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:57:57.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.129+0000 7feb21d65740 1 --2- 172.21.15.76:0/3297866945 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cbef0e0a50 0x55cbef0f6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.129+0000 7feb21d65740 1 --2- 172.21.15.76:0/3297866945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cbef0e0130 0x55cbef0f6040 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.129+0000 7feb21d65740 1 -- 172.21.15.76:0/3297866945 >> 172.21.15.76:0/3297866945 conn(0x55cbef0daa50 msgr2=0x55cbef0d9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:57.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.129+0000 7feb21d65740 1 -- 172.21.15.76:0/3297866945 shutdown_connections 2024-03-18T14:57:57.131 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.129+0000 7feb21d65740 1 -- 172.21.15.76:0/3297866945 wait complete. 2024-03-18T14:57:57.131 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.129+0000 7feb21d65740 20 client.26075 trim_cache size 0 max 16384 2024-03-18T14:57:57.131 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:57:57.131 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:57:57.131 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:57:57.131 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:57:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.130+0000 7feb1f177640 1 -- 172.21.15.76:0/2769176990 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7feb080a2a10 msgr2=0x7feb080a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.131+0000 7feb1f177640 1 --2- 172.21.15.76:0/2769176990 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7feb080a2a10 0x7feb080a4ed0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7feb10005fd0 tx=0x7feb100107a0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.131+0000 7feb1f177640 1 -- 172.21.15.76:0/2769176990 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb18073c10 msgr2=0x7feb1819a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:57:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.131+0000 7feb1f177640 1 --2- 172.21.15.76:0/2769176990 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb18073c10 0x7feb1819a2e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7feb0c00d950 tx=0x7feb0c00de10 comp rx=0 tx=0).stop 2024-03-18T14:57:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.132+0000 7feb1f177640 1 -- 172.21.15.76:0/2769176990 shutdown_connections 2024-03-18T14:57:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.132+0000 7feb1f177640 1 --2- 172.21.15.76:0/2769176990 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7feb080a2a10 0x7feb080a4ed0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.132+0000 7feb1f177640 1 --2- 172.21.15.76:0/2769176990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb1810dfd0 0x7feb1819eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.132+0000 7feb1f177640 1 --2- 172.21.15.76:0/2769176990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb18074530 0x7feb1819a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.132+0000 7feb1f177640 1 --2- 172.21.15.76:0/2769176990 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb18073c10 0x7feb1819a2e0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:57:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.132+0000 7feb1f177640 1 -- 172.21.15.76:0/2769176990 >> 172.21.15.76:0/2769176990 conn(0x7feb180fd260 msgr2=0x7feb180fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:57:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.132+0000 7feb1f177640 1 -- 172.21.15.76:0/2769176990 shutdown_connections 2024-03-18T14:57:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:57:57.132+0000 7feb1f177640 1 -- 172.21.15.76:0/2769176990 wait complete. 2024-03-18T14:57:57.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:57 smithi076 ceph-mon[32205]: daemon mds.e restarted 2024-03-18T14:57:57.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:57 smithi174 ceph-mon[36787]: daemon mds.e restarted 2024-03-18T14:57:57.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:57 smithi077 ceph-mon[35865]: daemon mds.e restarted 2024-03-18T14:57:57.727 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:57:57 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setuser ceph since I am not root 2024-03-18T14:57:57.727 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:57:57 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setgroup ceph since I am not root 2024-03-18T14:57:57.727 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:57:57 smithi077 ceph-mds[65404]: -- 172.21.15.77:0/1331737889 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5651cea5da40 con 0x5651cea31000 2024-03-18T14:57:57.727 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:57:57 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: starting mds.e at 2024-03-18T14:57:57.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 453053 10.33 MB/sec execute 118 sec latency 52.055 ms 2024-03-18T14:57:58.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:58 smithi076 ceph-mon[32205]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T14:57:58.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:58 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.77:6834/2498697016,v1:172.21.15.77:6835/2498697016] up:boot 2024-03-18T14:57:58.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:58 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T14:57:58.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:58 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:57:58.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:58 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:57:58.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:58 smithi076 ceph-mon[32205]: pgmap v148: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 330 op/s 2024-03-18T14:57:58.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:58 smithi174 ceph-mon[36787]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T14:57:58.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:58 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.77:6834/2498697016,v1:172.21.15.77:6835/2498697016] up:boot 2024-03-18T14:57:58.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:58 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T14:57:58.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:58 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:57:58.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:58 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:57:58.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:58 smithi174 ceph-mon[36787]: pgmap v148: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 330 op/s 2024-03-18T14:57:58.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:58 smithi077 ceph-mon[35865]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T14:57:58.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:58 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.77:6834/2498697016,v1:172.21.15.77:6835/2498697016] up:boot 2024-03-18T14:57:58.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:58 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T14:57:58.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:58 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:57:58.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:58 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:57:58.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:58 smithi077 ceph-mon[35865]: pgmap v148: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 330 op/s 2024-03-18T14:57:58.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 454620 10.33 MB/sec execute 119 sec latency 17.247 ms 2024-03-18T14:57:59.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 457010 10.35 MB/sec execute 120 sec latency 15.148 ms 2024-03-18T14:57:59.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:57:59 smithi174 ceph-mon[36787]: pgmap v149: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 322 op/s 2024-03-18T14:57:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:57:59 smithi077 ceph-mon[35865]: pgmap v149: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 322 op/s 2024-03-18T14:58:00.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:57:59 smithi076 ceph-mon[32205]: pgmap v149: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 322 op/s 2024-03-18T14:58:00.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 458205 10.32 MB/sec execute 121 sec latency 47.235 ms 2024-03-18T14:58:01.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 459439 10.43 MB/sec execute 122 sec latency 44.210 ms 2024-03-18T14:58:01.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:01 smithi174 ceph-mon[36787]: pgmap v150: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 322 op/s 2024-03-18T14:58:01.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:01 smithi077 ceph-mon[35865]: pgmap v150: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 322 op/s 2024-03-18T14:58:02.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:01 smithi076 ceph-mon[32205]: pgmap v150: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 322 op/s 2024-03-18T14:58:02.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 462156 10.48 MB/sec execute 123 sec latency 17.465 ms 2024-03-18T14:58:03.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 463859 10.44 MB/sec execute 124 sec latency 13.013 ms 2024-03-18T14:58:03.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:03 smithi174 ceph-mon[36787]: pgmap v151: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 339 op/s 2024-03-18T14:58:03.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:03 smithi077 ceph-mon[35865]: pgmap v151: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 339 op/s 2024-03-18T14:58:04.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:03 smithi076 ceph-mon[32205]: pgmap v151: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 339 op/s 2024-03-18T14:58:04.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 465926 10.47 MB/sec execute 125 sec latency 10.832 ms 2024-03-18T14:58:05.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 466041 10.38 MB/sec execute 126 sec latency 914.512 ms 2024-03-18T14:58:05.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:05 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:58:05.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:05 smithi174 ceph-mon[36787]: pgmap v152: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 339 op/s 2024-03-18T14:58:05.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:05 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:58:05.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:05 smithi077 ceph-mon[35865]: pgmap v152: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 339 op/s 2024-03-18T14:58:06.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:05 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:58:06.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:05 smithi076 ceph-mon[32205]: pgmap v152: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 339 op/s 2024-03-18T14:58:06.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 466041 10.30 MB/sec execute 127 sec latency 1914.607 ms 2024-03-18T14:58:07.742 INFO:tasks.workunit.client.0.smithi076.stdout: 1 466041 10.22 MB/sec execute 128 sec latency 2914.718 ms 2024-03-18T14:58:07.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:07 smithi174 ceph-mon[36787]: pgmap v153: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 329 op/s 2024-03-18T14:58:07.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:07 smithi077 ceph-mon[35865]: pgmap v153: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 329 op/s 2024-03-18T14:58:08.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:07 smithi076 ceph-mon[32205]: pgmap v153: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 329 op/s 2024-03-18T14:58:08.743 INFO:tasks.workunit.client.0.smithi076.stdout: 1 466041 10.14 MB/sec execute 129 sec latency 3914.847 ms 2024-03-18T14:58:09.743 INFO:tasks.workunit.client.0.smithi076.stdout: 1 466041 10.06 MB/sec execute 130 sec latency 4914.965 ms 2024-03-18T14:58:09.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:09 smithi174 ceph-mon[36787]: pgmap v154: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 329 op/s 2024-03-18T14:58:09.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:09 smithi077 ceph-mon[35865]: pgmap v154: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 329 op/s 2024-03-18T14:58:10.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:09 smithi076 ceph-mon[32205]: pgmap v154: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 329 op/s 2024-03-18T14:58:10.231 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.227+0000 7f8d20f9e640 1 -- 172.21.15.76:0/3165748753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d1c102150 msgr2=0x7f8d1c10e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:10.231 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.227+0000 7f8d20f9e640 1 --2- 172.21.15.76:0/3165748753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d1c102150 0x7f8d1c10e660 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8d14009cd0 tx=0x7f8d1402c0f0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.227+0000 7f8d20f9e640 1 -- 172.21.15.76:0/3165748753 shutdown_connections 2024-03-18T14:58:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.227+0000 7f8d20f9e640 1 --2- 172.21.15.76:0/3165748753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d1c102150 0x7f8d1c10e660 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.227+0000 7f8d20f9e640 1 --2- 172.21.15.76:0/3165748753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d1c101790 0x7f8d1c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.227+0000 7f8d20f9e640 1 --2- 172.21.15.76:0/3165748753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d1c1077a0 0x7f8d1c107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.227+0000 7f8d20f9e640 1 -- 172.21.15.76:0/3165748753 >> 172.21.15.76:0/3165748753 conn(0x7f8d1c0fd2a0 msgr2=0x7f8d1c0ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.227+0000 7f8d20f9e640 1 -- 172.21.15.76:0/3165748753 shutdown_connections 2024-03-18T14:58:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.227+0000 7f8d20f9e640 1 -- 172.21.15.76:0/3165748753 wait complete. 2024-03-18T14:58:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.228+0000 7f8d20f9e640 1 Processor -- start 2024-03-18T14:58:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.228+0000 7f8d20f9e640 1 -- start start 2024-03-18T14:58:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.228+0000 7f8d20f9e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d1c101790 0x7f8d1c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.228+0000 7f8d20f9e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d1c102150 0x7f8d1c1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.228+0000 7f8d20f9e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d1c1077a0 0x7f8d1c19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.228+0000 7f8d20f9e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c111b70 con 0x7f8d1c1077a0 2024-03-18T14:58:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.228+0000 7f8d20f9e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c1119f0 con 0x7f8d1c102150 2024-03-18T14:58:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.228+0000 7f8d20f9e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c111cf0 con 0x7f8d1c101790 2024-03-18T14:58:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.228+0000 7f8d1ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d1c1077a0 0x7f8d1c19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.228+0000 7f8d1a59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d1c101790 0x7f8d1c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.228+0000 7f8d1ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d1c1077a0 0x7f8d1c19c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52060/0 (socket says 172.21.15.76:52060) 2024-03-18T14:58:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.228+0000 7f8d1ad9d640 1 -- 172.21.15.76:0/2025155348 learned_addr learned my addr 172.21.15.76:0/2025155348 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.228+0000 7f8d19d9b640 1 --2- 172.21.15.76:0/2025155348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d1c102150 0x7f8d1c1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.229+0000 7f8d1a59c640 1 -- 172.21.15.76:0/2025155348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d1c102150 msgr2=0x7f8d1c1984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.229+0000 7f8d1a59c640 1 --2- 172.21.15.76:0/2025155348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d1c102150 0x7f8d1c1984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.229+0000 7f8d1a59c640 1 -- 172.21.15.76:0/2025155348 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d1c1077a0 msgr2=0x7f8d1c19c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.229+0000 7f8d1a59c640 1 --2- 172.21.15.76:0/2025155348 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d1c1077a0 0x7f8d1c19c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.229+0000 7f8d1a59c640 1 -- 172.21.15.76:0/2025155348 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d1c19cf80 con 0x7f8d1c101790 2024-03-18T14:58:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.229+0000 7f8d1ad9d640 1 --2- 172.21.15.76:0/2025155348 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d1c1077a0 0x7f8d1c19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:58:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.229+0000 7f8d19d9b640 1 --2- 172.21.15.76:0/2025155348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d1c102150 0x7f8d1c1984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:58:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.229+0000 7f8d1a59c640 1 --2- 172.21.15.76:0/2025155348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d1c101790 0x7f8d1c197f90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c00e9f0 tx=0x7f8d0c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.229+0000 7f8d0b7fe640 1 -- 172.21.15.76:0/2025155348 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c00ccc0 con 0x7f8d1c101790 2024-03-18T14:58:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.229+0000 7f8d20f9e640 1 -- 172.21.15.76:0/2025155348 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d1c19d270 con 0x7f8d1c101790 2024-03-18T14:58:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.229+0000 7f8d0b7fe640 1 -- 172.21.15.76:0/2025155348 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d0c010910 con 0x7f8d1c101790 2024-03-18T14:58:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.229+0000 7f8d0b7fe640 1 -- 172.21.15.76:0/2025155348 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c018960 con 0x7f8d1c101790 2024-03-18T14:58:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.229+0000 7f8d20f9e640 1 -- 172.21.15.76:0/2025155348 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d1c1a4b00 con 0x7f8d1c101790 2024-03-18T14:58:10.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.231+0000 7f8d0b7fe640 1 -- 172.21.15.76:0/2025155348 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f8d0c010ac0 con 0x7f8d1c101790 2024-03-18T14:58:10.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.232+0000 7f8d23b88740 1 Processor -- start 2024-03-18T14:58:10.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.232+0000 7f8d23b88740 1 -- start start 2024-03-18T14:58:10.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.232+0000 7f8d0b7fe640 1 --2- 172.21.15.76:0/2025155348 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f8d040a2a10 0x7f8d040a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.232+0000 7f8d23b88740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4130 0x56492d4d4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.232+0000 7f8d23b88740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4a50 0x56492d4dd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.232+0000 7f8d23b88740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4dd820 0x56492d4dfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.232+0000 7f8d23b88740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56492d438a40 con 0x56492d4d4a50 2024-03-18T14:58:10.241 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.232+0000 7f8d23b88740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56492d3e1920 con 0x56492d4d4130 2024-03-18T14:58:10.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.232+0000 7f8d23b88740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56492d434ae0 con 0x56492d4dd820 2024-03-18T14:58:10.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.233+0000 7f8d1ad9d640 1 --2- 172.21.15.76:0/2025155348 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f8d040a2a10 0x7f8d040a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.233+0000 7f8d19d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4a50 0x56492d4dd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.233+0000 7f8d19d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4a50 0x56492d4dd2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52062/0 (socket says 172.21.15.76:52062) 2024-03-18T14:58:10.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.233+0000 7f8d19d9b640 1 -- 172.21.15.76:0/2724936724 learned_addr learned my addr 172.21.15.76:0/2724936724 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:10.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.233+0000 7f8d1a59c640 1 --2- 172.21.15.76:0/2724936724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4130 0x56492d4d4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.233+0000 7f8d1ad9d640 1 --2- 172.21.15.76:0/2724936724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4dd820 0x56492d4dfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.233+0000 7f8d0b7fe640 1 -- 172.21.15.76:0/2025155348 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8440+0+0 (secure 0 0 0) 0x7f8d0c0cf190 con 0x7f8d1c101790 2024-03-18T14:58:10.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.233+0000 7f8d19d9b640 1 -- 172.21.15.76:0/2724936724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4dd820 msgr2=0x56492d4dfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:10.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.233+0000 7f8d19d9b640 1 --2- 172.21.15.76:0/2724936724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4dd820 0x56492d4dfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.233+0000 7f8d19d9b640 1 -- 172.21.15.76:0/2724936724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4130 msgr2=0x56492d4d4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:10.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.233+0000 7f8d19d9b640 1 --2- 172.21.15.76:0/2724936724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4130 0x56492d4d4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.242 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.233+0000 7f8d19d9b640 1 -- 172.21.15.76:0/2724936724 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56492d3f9460 con 0x56492d4d4a50 2024-03-18T14:58:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.233+0000 7f8d1ad9d640 1 --2- 172.21.15.76:0/2025155348 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f8d040a2a10 0x7f8d040a4ed0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8d14032530 tx=0x7f8d140098f0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.233+0000 7f8d19d9b640 1 --2- 172.21.15.76:0/2724936724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4a50 0x56492d4dd2e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8d100114f0 tx=0x7f8d100118f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc4ae947430ada18 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.233+0000 7f8d08ff9640 1 -- 172.21.15.76:0/2724936724 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d10004020 con 0x56492d4d4a50 2024-03-18T14:58:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.233+0000 7f8d08ff9640 1 -- 172.21.15.76:0/2724936724 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d100041c0 con 0x56492d4d4a50 2024-03-18T14:58:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.233+0000 7f8d08ff9640 1 -- 172.21.15.76:0/2724936724 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1000f780 con 0x56492d4d4a50 2024-03-18T14:58:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.234+0000 7f8d23b88740 1 -- 172.21.15.76:0/2724936724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4a50 msgr2=0x56492d4dd2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.234+0000 7f8d23b88740 1 --2- 172.21.15.76:0/2724936724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4a50 0x56492d4dd2e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8d100114f0 tx=0x7f8d100118f0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.234+0000 7f8d23b88740 1 -- 172.21.15.76:0/2724936724 shutdown_connections 2024-03-18T14:58:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.234+0000 7f8d23b88740 1 --2- 172.21.15.76:0/2724936724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4dd820 0x56492d4dfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.234+0000 7f8d23b88740 1 --2- 172.21.15.76:0/2724936724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4a50 0x56492d4dd2e0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.234+0000 7f8d23b88740 1 --2- 172.21.15.76:0/2724936724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4130 0x56492d4d4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.234+0000 7f8d23b88740 1 -- 172.21.15.76:0/2724936724 >> 172.21.15.76:0/2724936724 conn(0x56492d4cea50 msgr2=0x56492d4cee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.234+0000 7f8d23b88740 1 -- 172.21.15.76:0/2724936724 shutdown_connections 2024-03-18T14:58:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.234+0000 7f8d23b88740 1 -- 172.21.15.76:0/2724936724 wait complete. 2024-03-18T14:58:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.234+0000 7f8d23b88740 1 -- start start 2024-03-18T14:58:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d23b88740 1 Processor -- start 2024-03-18T14:58:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d23b88740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4130 0x56492d4da880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d23b88740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4d4a50 0x56492d4dadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d23b88740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4dd820 0x56492d4db300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d23b88740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56492d4e0450 con 0x56492d4dd820 2024-03-18T14:58:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d23b88740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56492d3f4d70 con 0x56492d4d4130 2024-03-18T14:58:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d23b88740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56492d4e05d0 con 0x56492d4d4a50 2024-03-18T14:58:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d23b88740 10 client.0 _ll_register_callbacks cb 0x56492d14ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:58:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d1ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4dd820 0x56492d4db300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d1a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4130 0x56492d4da880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d1ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4dd820 0x56492d4db300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52066/0 (socket says 172.21.15.76:52066) 2024-03-18T14:58:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d1a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4130 0x56492d4da880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:36944/0 (socket says 172.21.15.76:36944) 2024-03-18T14:58:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d1ad9d640 1 -- 172.21.15.76:0/1691975930 learned_addr learned my addr 172.21.15.76:0/1691975930 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d19d9b640 1 --2- 172.21.15.76:0/1691975930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4d4a50 0x56492d4dadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d1ad9d640 1 -- 172.21.15.76:0/1691975930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4d4a50 msgr2=0x56492d4dadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d1ad9d640 1 --2- 172.21.15.76:0/1691975930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4d4a50 0x56492d4dadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d1ad9d640 1 -- 172.21.15.76:0/1691975930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4130 msgr2=0x56492d4da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d1ad9d640 1 --2- 172.21.15.76:0/1691975930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4130 0x56492d4da880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d1ad9d640 1 -- 172.21.15.76:0/1691975930 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56492d14d210 con 0x56492d4dd820 2024-03-18T14:58:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d1a59c640 1 --2- 172.21.15.76:0/1691975930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4130 0x56492d4da880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:58:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d1ad9d640 1 --2- 172.21.15.76:0/1691975930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4dd820 0x56492d4db300 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8d14037040 tx=0x7f8d1402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ed4da5bf544e5ee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.235+0000 7f8d01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:58:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.236+0000 7f8d01ffb640 1 -- 172.21.15.76:0/1691975930 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d14053060 con 0x56492d4dd820 2024-03-18T14:58:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.236+0000 7f8d01ffb640 1 -- 172.21.15.76:0/1691975930 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8440+0+0 (secure 0 0 0) 0x7f8d14034c00 con 0x56492d4dd820 2024-03-18T14:58:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.236+0000 7f8d23b88740 1 -- 172.21.15.76:0/1691975930 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d100041c0 con 0x56492d4dd820 2024-03-18T14:58:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.236+0000 7f8d01ffb640 1 -- 172.21.15.76:0/1691975930 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8d14039040 con 0x56492d4dd820 2024-03-18T14:58:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.236+0000 7f8d23b88740 10 client.16236 fetch_fsmap learned FSMap version 38 2024-03-18T14:58:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.236+0000 7f8d23b88740 1 -- 172.21.15.76:0/1691975930 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56492d219620 con 0x56492d4dd820 2024-03-18T14:58:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.236+0000 7f8d01ffb640 1 -- 172.21.15.76:0/1691975930 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d100041c0 con 0x56492d4dd820 2024-03-18T14:58:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.236+0000 7f8d01ffb640 1 -- 172.21.15.76:0/1691975930 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f8d1403f040 con 0x56492d4dd820 2024-03-18T14:58:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.236+0000 7f8d23b88740 10 client.16236 fetch_fsmap finished waiting for FSMap version 38 2024-03-18T14:58:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.236+0000 7f8d23b88740 10 client.16236 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:58:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.236+0000 7f8d23b88740 20 client.16236 populate_metadata read hostname 'smithi076' 2024-03-18T14:58:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.236+0000 7f8d23b88740 1 --2- 172.21.15.76:0/1691975930 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56492d4ed9d0 0x56492d4efdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.236+0000 7f8d23b88740 4 client.16236 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:58:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.236+0000 7f8d23b88740 1 -- 172.21.15.76:0/1691975930 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56492d445f80 con 0x56492d4ed9d0 2024-03-18T14:58:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.236+0000 7f8d1a59c640 1 --2- 172.21.15.76:0/1691975930 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56492d4ed9d0 0x56492d4efdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.237+0000 7f8d1a59c640 1 --2- 172.21.15.76:0/1691975930 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56492d4ed9d0 0x56492d4efdc0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7c561647fe8546d server_cookie=2fe4e3a1f8a66741 in_seq=0 out_seq=0 2024-03-18T14:58:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.237+0000 7f8d01ffb640 10 client.16236 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:58:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.237+0000 7f8d01ffb640 1 -- 172.21.15.76:0/1691975930 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56492d445f80 con 0x56492d4ed9d0 2024-03-18T14:58:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.237+0000 7f8d01ffb640 10 client.16236 handle_command_reply: tid=0 2024-03-18T14:58:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.237+0000 7f8d23b88740 1 client.16236 shutdown 2024-03-18T14:58:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.237+0000 7f8d23b88740 1 -- 172.21.15.76:0/1691975930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4dd820 msgr2=0x56492d4db300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.237+0000 7f8d23b88740 1 --2- 172.21.15.76:0/1691975930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4dd820 0x56492d4db300 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8d14037040 tx=0x7f8d1402c0f0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.238+0000 7f8d23b88740 1 -- 172.21.15.76:0/1691975930 shutdown_connections 2024-03-18T14:58:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.238+0000 7f8d23b88740 1 --2- 172.21.15.76:0/1691975930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4d4a50 0x56492d4dadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.238+0000 7f8d23b88740 1 --2- 172.21.15.76:0/1691975930 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56492d4ed9d0 0x56492d4efdc0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.238+0000 7f8d23b88740 1 --2- 172.21.15.76:0/1691975930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4dd820 0x56492d4db300 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.238+0000 7f8d23b88740 1 --2- 172.21.15.76:0/1691975930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4130 0x56492d4da880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.238+0000 7f8d01ffb640 0 client.16236 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:58:10.248 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.238+0000 7f8d23b88740 1 -- 172.21.15.76:0/1691975930 >> 172.21.15.76:0/1691975930 conn(0x56492d4cea50 msgr2=0x56492d4d07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:10.248 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.238+0000 7f8d23b88740 1 -- 172.21.15.76:0/1691975930 shutdown_connections 2024-03-18T14:58:10.248 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.238+0000 7f8d23b88740 1 -- 172.21.15.76:0/1691975930 wait complete. 2024-03-18T14:58:10.248 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.238+0000 7f8d23b88740 20 client.16236 trim_cache size 0 max 16384 2024-03-18T14:58:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.252+0000 7f8d23b88740 1 Processor -- start 2024-03-18T14:58:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.252+0000 7f8d23b88740 1 -- start start 2024-03-18T14:58:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.252+0000 7f8d23b88740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4130 0x56492d4dcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.252+0000 7f8d23b88740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4a50 0x56492d4e91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.252+0000 7f8d23b88740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4dd820 0x56492d4f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.252+0000 7f8d23b88740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c1afd40 con 0x56492d4d4130 2024-03-18T14:58:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.252+0000 7f8d23b88740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c1ad5e0 con 0x56492d4d4a50 2024-03-18T14:58:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.252+0000 7f8d23b88740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c1ab430 con 0x56492d4dd820 2024-03-18T14:58:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.252+0000 7f8d1a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4130 0x56492d4dcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.252+0000 7f8d1a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4130 0x56492d4dcfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52070/0 (socket says 172.21.15.76:52070) 2024-03-18T14:58:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.252+0000 7f8d1a59c640 1 -- 172.21.15.76:0/954886622 learned_addr learned my addr 172.21.15.76:0/954886622 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.252+0000 7f8d19d9b640 1 --2- 172.21.15.76:0/954886622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4a50 0x56492d4e91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.252+0000 7f8d1ad9d640 1 --2- 172.21.15.76:0/954886622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4dd820 0x56492d4f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.253+0000 7f8d1a59c640 1 -- 172.21.15.76:0/954886622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4dd820 msgr2=0x56492d4f56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.253+0000 7f8d1a59c640 1 --2- 172.21.15.76:0/954886622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4dd820 0x56492d4f56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.253+0000 7f8d1a59c640 1 -- 172.21.15.76:0/954886622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4a50 msgr2=0x56492d4e91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.253+0000 7f8d1a59c640 1 --2- 172.21.15.76:0/954886622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4a50 0x56492d4e91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.253+0000 7f8d1a59c640 1 -- 172.21.15.76:0/954886622 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56492d4d6450 con 0x56492d4d4130 2024-03-18T14:58:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.253+0000 7f8d1a59c640 1 --2- 172.21.15.76:0/954886622 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4130 0x56492d4dcfb0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c09b820 tx=0x7f8d0c086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a63f5830788c6a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:10.266 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.253+0000 7f8d03fff640 1 -- 172.21.15.76:0/954886622 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c026040 con 0x56492d4d4130 2024-03-18T14:58:10.266 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.253+0000 7f8d03fff640 1 -- 172.21.15.76:0/954886622 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d0c0ab160 con 0x56492d4d4130 2024-03-18T14:58:10.266 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.253+0000 7f8d03fff640 1 -- 172.21.15.76:0/954886622 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c09ace0 con 0x56492d4d4130 2024-03-18T14:58:10.266 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.253+0000 7f8d23b88740 1 -- 172.21.15.76:0/954886622 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4130 msgr2=0x56492d4dcfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:10.266 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.253+0000 7f8d23b88740 1 --2- 172.21.15.76:0/954886622 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4130 0x56492d4dcfb0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c09b820 tx=0x7f8d0c086040 comp rx=0 tx=0).stop 2024-03-18T14:58:10.266 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d23b88740 1 -- 172.21.15.76:0/954886622 shutdown_connections 2024-03-18T14:58:10.267 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d23b88740 1 --2- 172.21.15.76:0/954886622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4dd820 0x56492d4f56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.267 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d23b88740 1 --2- 172.21.15.76:0/954886622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4a50 0x56492d4e91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.267 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d23b88740 1 --2- 172.21.15.76:0/954886622 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4130 0x56492d4dcfb0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.267 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d23b88740 1 -- 172.21.15.76:0/954886622 >> 172.21.15.76:0/954886622 conn(0x56492d4cea50 msgr2=0x56492d4cfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:10.267 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d23b88740 1 -- 172.21.15.76:0/954886622 shutdown_connections 2024-03-18T14:58:10.267 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d23b88740 1 -- 172.21.15.76:0/954886622 wait complete. 2024-03-18T14:58:10.267 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d23b88740 1 -- start start 2024-03-18T14:58:10.268 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d23b88740 1 Processor -- start 2024-03-18T14:58:10.268 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d23b88740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4130 0x56492d4e5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.268 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d23b88740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4a50 0x56492d4e0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.268 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d23b88740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4dd820 0x56492d4e1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.268 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d23b88740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c1abaa0 con 0x56492d4d4130 2024-03-18T14:58:10.268 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d23b88740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c1acf10 con 0x56492d4d4a50 2024-03-18T14:58:10.269 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d23b88740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c1ac170 con 0x56492d4dd820 2024-03-18T14:58:10.269 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d23b88740 10 client.0 _ll_register_callbacks cb 0x7f8d0c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:58:10.269 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d1a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4130 0x56492d4e5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.269 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d1a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4130 0x56492d4e5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52086/0 (socket says 172.21.15.76:52086) 2024-03-18T14:58:10.269 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d1a59c640 1 -- 172.21.15.76:0/830645921 learned_addr learned my addr 172.21.15.76:0/830645921 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:10.269 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d1ad9d640 1 --2- 172.21.15.76:0/830645921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4dd820 0x56492d4e1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.270 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d19d9b640 1 --2- 172.21.15.76:0/830645921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4a50 0x56492d4e0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.270 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d1a59c640 1 -- 172.21.15.76:0/830645921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4dd820 msgr2=0x56492d4e1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:10.270 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d1a59c640 1 --2- 172.21.15.76:0/830645921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4dd820 0x56492d4e1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.270 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d1a59c640 1 -- 172.21.15.76:0/830645921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4a50 msgr2=0x56492d4e0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:10.270 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.254+0000 7f8d1a59c640 1 --2- 172.21.15.76:0/830645921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4a50 0x56492d4e0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.270 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.255+0000 7f8d1a59c640 1 -- 172.21.15.76:0/830645921 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56492d4e1cc0 con 0x56492d4d4130 2024-03-18T14:58:10.271 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.255+0000 7f8d1a59c640 1 --2- 172.21.15.76:0/830645921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4130 0x56492d4e5b80 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c098260 tx=0x7f8d0c0a8c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7318eadf0565bfc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:10.271 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.255+0000 7f8d01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:58:10.271 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.255+0000 7f8d01ffb640 1 -- 172.21.15.76:0/830645921 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c0af020 con 0x56492d4d4130 2024-03-18T14:58:10.271 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.255+0000 7f8d01ffb640 1 -- 172.21.15.76:0/830645921 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8440+0+0 (secure 0 0 0) 0x7f8d0c0ccb50 con 0x56492d4d4130 2024-03-18T14:58:10.271 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.255+0000 7f8d23b88740 1 -- 172.21.15.76:0/830645921 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d0c0ab160 con 0x56492d4d4130 2024-03-18T14:58:10.271 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.255+0000 7f8d01ffb640 1 -- 172.21.15.76:0/830645921 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8d0c0ccd50 con 0x56492d4d4130 2024-03-18T14:58:10.271 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.255+0000 7f8d23b88740 10 client.16242 fetch_fsmap learned FSMap version 38 2024-03-18T14:58:10.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.255+0000 7f8d23b88740 1 -- 172.21.15.76:0/830645921 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56492d553de0 con 0x56492d4d4130 2024-03-18T14:58:10.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.255+0000 7f8d01ffb640 1 -- 172.21.15.76:0/830645921 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d0c0ab160 con 0x56492d4d4130 2024-03-18T14:58:10.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.255+0000 7f8d01ffb640 1 -- 172.21.15.76:0/830645921 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f8d0c09cb10 con 0x56492d4d4130 2024-03-18T14:58:10.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.255+0000 7f8d23b88740 10 client.16242 fetch_fsmap finished waiting for FSMap version 38 2024-03-18T14:58:10.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.255+0000 7f8d23b88740 10 client.16242 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:58:10.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.255+0000 7f8d23b88740 20 client.16242 populate_metadata read hostname 'smithi076' 2024-03-18T14:58:10.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.255+0000 7f8d23b88740 1 --2- 172.21.15.76:0/830645921 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56492d554470 0x56492d556890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:10.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.255+0000 7f8d23b88740 4 client.16242 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:58:10.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.255+0000 7f8d23b88740 1 -- 172.21.15.76:0/830645921 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56492d4402d0 con 0x56492d554470 2024-03-18T14:58:10.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.256+0000 7f8d19d9b640 1 --2- 172.21.15.76:0/830645921 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56492d554470 0x56492d556890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:10.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.256+0000 7f8d01ffb640 10 client.16242 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:58:10.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.256+0000 7f8d19d9b640 1 --2- 172.21.15.76:0/830645921 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56492d554470 0x56492d556890 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1e502531c66ef40 server_cookie=dfabddc89a2f6d06 in_seq=0 out_seq=0 2024-03-18T14:58:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.308+0000 7f8d01ffb640 1 -- 172.21.15.76:0/830645921 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56492d4402d0 con 0x56492d554470 2024-03-18T14:58:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.308+0000 7f8d01ffb640 10 client.16242 handle_command_reply: tid=0 2024-03-18T14:58:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.308+0000 7f8d23b88740 1 client.16242 shutdown 2024-03-18T14:58:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.308+0000 7f8d23b88740 1 -- 172.21.15.76:0/830645921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4130 msgr2=0x56492d4e5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.308+0000 7f8d23b88740 1 --2- 172.21.15.76:0/830645921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4130 0x56492d4e5b80 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c098260 tx=0x7f8d0c0a8c70 comp rx=0 tx=0).stop 2024-03-18T14:58:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.308+0000 7f8d23b88740 1 -- 172.21.15.76:0/830645921 shutdown_connections 2024-03-18T14:58:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.308+0000 7f8d23b88740 1 --2- 172.21.15.76:0/830645921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56492d4dd820 0x56492d4e1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.308+0000 7f8d23b88740 1 --2- 172.21.15.76:0/830645921 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56492d554470 0x56492d556890 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.308+0000 7f8d23b88740 1 --2- 172.21.15.76:0/830645921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56492d4d4a50 0x56492d4e0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.308+0000 7f8d01ffb640 0 client.16242 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:58:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.308+0000 7f8d23b88740 1 --2- 172.21.15.76:0/830645921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56492d4d4130 0x56492d4e5b80 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.308+0000 7f8d23b88740 1 -- 172.21.15.76:0/830645921 >> 172.21.15.76:0/830645921 conn(0x56492d4cea50 msgr2=0x56492d4cdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.308+0000 7f8d23b88740 1 -- 172.21.15.76:0/830645921 shutdown_connections 2024-03-18T14:58:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.308+0000 7f8d23b88740 1 -- 172.21.15.76:0/830645921 wait complete. 2024-03-18T14:58:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.308+0000 7f8d23b88740 20 client.16242 trim_cache size 0 max 16384 2024-03-18T14:58:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:58:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:58:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:58:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:58:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.309+0000 7f8d20f9e640 1 -- 172.21.15.76:0/2025155348 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f8d040a2a10 msgr2=0x7f8d040a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.309+0000 7f8d20f9e640 1 --2- 172.21.15.76:0/2025155348 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f8d040a2a10 0x7f8d040a4ed0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8d14032530 tx=0x7f8d140098f0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.309+0000 7f8d20f9e640 1 -- 172.21.15.76:0/2025155348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d1c101790 msgr2=0x7f8d1c197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.309+0000 7f8d20f9e640 1 --2- 172.21.15.76:0/2025155348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d1c101790 0x7f8d1c197f90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c00e9f0 tx=0x7f8d0c00eeb0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.309+0000 7f8d20f9e640 1 -- 172.21.15.76:0/2025155348 shutdown_connections 2024-03-18T14:58:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.309+0000 7f8d20f9e640 1 --2- 172.21.15.76:0/2025155348 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f8d040a2a10 0x7f8d040a4ed0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.309+0000 7f8d20f9e640 1 --2- 172.21.15.76:0/2025155348 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d1c1077a0 0x7f8d1c19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.309+0000 7f8d20f9e640 1 --2- 172.21.15.76:0/2025155348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d1c102150 0x7f8d1c1984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.309+0000 7f8d20f9e640 1 --2- 172.21.15.76:0/2025155348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d1c101790 0x7f8d1c197f90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.309+0000 7f8d20f9e640 1 -- 172.21.15.76:0/2025155348 >> 172.21.15.76:0/2025155348 conn(0x7f8d1c0fd2a0 msgr2=0x7f8d1c0fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.309+0000 7f8d20f9e640 1 -- 172.21.15.76:0/2025155348 shutdown_connections 2024-03-18T14:58:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:10.309+0000 7f8d20f9e640 1 -- 172.21.15.76:0/2025155348 wait complete. 2024-03-18T14:58:10.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 466041 9.99 MB/sec execute 131 sec latency 5915.066 ms 2024-03-18T14:58:11.743 INFO:tasks.workunit.client.0.smithi076.stdout: 1 466041 9.91 MB/sec execute 132 sec latency 6915.152 ms 2024-03-18T14:58:11.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:11 smithi174 ceph-mon[36787]: pgmap v155: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 2.2 MiB/s wr, 174 op/s 2024-03-18T14:58:11.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:11 smithi077 ceph-mon[35865]: pgmap v155: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 2.2 MiB/s wr, 174 op/s 2024-03-18T14:58:12.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:11 smithi076 ceph-mon[32205]: pgmap v155: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 2.2 MiB/s wr, 174 op/s 2024-03-18T14:58:12.743 INFO:tasks.workunit.client.0.smithi076.stdout: 1 466041 9.84 MB/sec execute 133 sec latency 7915.234 ms 2024-03-18T14:58:13.743 INFO:tasks.workunit.client.0.smithi076.stdout: 1 466041 9.76 MB/sec execute 134 sec latency 8915.305 ms 2024-03-18T14:58:13.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:13 smithi174 ceph-mon[36787]: pgmap v156: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 3.3 MiB/s wr, 236 op/s 2024-03-18T14:58:13.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:13 smithi077 ceph-mon[35865]: pgmap v156: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 3.3 MiB/s wr, 236 op/s 2024-03-18T14:58:14.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:13 smithi076 ceph-mon[32205]: pgmap v156: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 3.3 MiB/s wr, 236 op/s 2024-03-18T14:58:14.743 INFO:tasks.workunit.client.0.smithi076.stdout: 1 466041 9.69 MB/sec execute 135 sec latency 9915.379 ms 2024-03-18T14:58:15.743 INFO:tasks.workunit.client.0.smithi076.stdout: 1 466041 9.62 MB/sec execute 136 sec latency 10915.472 ms 2024-03-18T14:58:15.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:15 smithi174 ceph-mon[36787]: pgmap v157: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.8 MiB/s wr, 109 op/s 2024-03-18T14:58:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:15 smithi077 ceph-mon[35865]: pgmap v157: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.8 MiB/s wr, 109 op/s 2024-03-18T14:58:16.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:15 smithi076 ceph-mon[32205]: pgmap v157: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.8 MiB/s wr, 109 op/s 2024-03-18T14:58:16.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 466041 9.55 MB/sec execute 137 sec latency 11915.552 ms 2024-03-18T14:58:17.743 INFO:tasks.workunit.client.0.smithi076.stdout: 1 466041 9.48 MB/sec execute 138 sec latency 12915.638 ms 2024-03-18T14:58:17.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:17 smithi076 ceph-mon[32205]: pgmap v158: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.8 MiB/s wr, 109 op/s 2024-03-18T14:58:17.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:17 smithi174 ceph-mon[36787]: pgmap v158: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.8 MiB/s wr, 109 op/s 2024-03-18T14:58:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:17 smithi077 ceph-mon[35865]: pgmap v158: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.8 MiB/s wr, 109 op/s 2024-03-18T14:58:18.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 466041 9.41 MB/sec execute 139 sec latency 13915.690 ms 2024-03-18T14:58:19.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 468935 9.47 MB/sec execute 140 sec latency 13923.375 ms 2024-03-18T14:58:19.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:19 smithi174 ceph-mon[36787]: pgmap v159: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 2.5 MiB/s wr, 140 op/s 2024-03-18T14:58:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:19 smithi077 ceph-mon[35865]: pgmap v159: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 2.5 MiB/s wr, 140 op/s 2024-03-18T14:58:20.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:19 smithi076 ceph-mon[32205]: pgmap v159: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 2.5 MiB/s wr, 140 op/s 2024-03-18T14:58:20.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 470403 9.45 MB/sec execute 141 sec latency 17.639 ms 2024-03-18T14:58:21.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 472763 9.48 MB/sec execute 142 sec latency 59.841 ms 2024-03-18T14:58:21.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:21 smithi174 ceph-mon[36787]: pgmap v160: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-03-18T14:58:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:21 smithi077 ceph-mon[35865]: pgmap v160: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-03-18T14:58:22.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:21 smithi076 ceph-mon[32205]: pgmap v160: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-03-18T14:58:22.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 474518 9.45 MB/sec execute 143 sec latency 13.565 ms 2024-03-18T14:58:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.408+0000 7fe37ed7e640 1 -- 172.21.15.76:0/107563406 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe378100930 msgr2=0x7fe378100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.408+0000 7fe37ed7e640 1 --2- 172.21.15.76:0/107563406 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe378100930 0x7fe378100db0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe364009900 tx=0x7fe36402bc30 comp rx=0 tx=0).stop 2024-03-18T14:58:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.408+0000 7fe37ed7e640 1 -- 172.21.15.76:0/107563406 shutdown_connections 2024-03-18T14:58:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.408+0000 7fe37ed7e640 1 --2- 172.21.15.76:0/107563406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3781012f0 0x7fe37810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.408+0000 7fe37ed7e640 1 --2- 172.21.15.76:0/107563406 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe378100930 0x7fe378100db0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.408+0000 7fe37ed7e640 1 --2- 172.21.15.76:0/107563406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe378105680 0x7fe378105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.408+0000 7fe37ed7e640 1 -- 172.21.15.76:0/107563406 >> 172.21.15.76:0/107563406 conn(0x7fe378077530 msgr2=0x7fe378077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.408+0000 7fe37ed7e640 1 -- 172.21.15.76:0/107563406 shutdown_connections 2024-03-18T14:58:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.408+0000 7fe37ed7e640 1 -- 172.21.15.76:0/107563406 wait complete. 2024-03-18T14:58:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.409+0000 7fe37ed7e640 1 Processor -- start 2024-03-18T14:58:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.409+0000 7fe37ed7e640 1 -- start start 2024-03-18T14:58:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.409+0000 7fe37ed7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe378100930 0x7fe378102350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.409+0000 7fe37ed7e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe3781012f0 0x7fe378102890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.409+0000 7fe37ed7e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe378105680 0x7fe378102dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.409+0000 7fe37ed7e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe378110b60 con 0x7fe3781012f0 2024-03-18T14:58:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.409+0000 7fe37ed7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe3781109e0 con 0x7fe378100930 2024-03-18T14:58:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.409+0000 7fe37ed7e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe378110ce0 con 0x7fe378105680 2024-03-18T14:58:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.409+0000 7fe377fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe3781012f0 0x7fe378102890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.409+0000 7fe377fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe3781012f0 0x7fe378102890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56756/0 (socket says 172.21.15.76:56756) 2024-03-18T14:58:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.409+0000 7fe377fff640 1 -- 172.21.15.76:0/434913172 learned_addr learned my addr 172.21.15.76:0/434913172 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.409+0000 7fe37cb1a640 1 --2- 172.21.15.76:0/434913172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe378100930 0x7fe378102350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.409+0000 7fe37d31b640 1 --2- 172.21.15.76:0/434913172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe378105680 0x7fe378102dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.410+0000 7fe377fff640 1 -- 172.21.15.76:0/434913172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe378105680 msgr2=0x7fe378102dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.410+0000 7fe377fff640 1 --2- 172.21.15.76:0/434913172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe378105680 0x7fe378102dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.410+0000 7fe377fff640 1 -- 172.21.15.76:0/434913172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe378100930 msgr2=0x7fe378102350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.410+0000 7fe377fff640 1 --2- 172.21.15.76:0/434913172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe378100930 0x7fe378102350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.410+0000 7fe377fff640 1 -- 172.21.15.76:0/434913172 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3781a65a0 con 0x7fe3781012f0 2024-03-18T14:58:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.410+0000 7fe377fff640 1 --2- 172.21.15.76:0/434913172 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe3781012f0 0x7fe378102890 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe364005940 tx=0x7fe364004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.410+0000 7fe375ffb640 1 -- 172.21.15.76:0/434913172 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364041070 con 0x7fe3781012f0 2024-03-18T14:58:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.410+0000 7fe37ed7e640 1 -- 172.21.15.76:0/434913172 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3781a6830 con 0x7fe3781012f0 2024-03-18T14:58:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.410+0000 7fe37ed7e640 1 -- 172.21.15.76:0/434913172 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3781a6d70 con 0x7fe3781012f0 2024-03-18T14:58:23.414 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.410+0000 7fe375ffb640 1 -- 172.21.15.76:0/434913172 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3640054e0 con 0x7fe3781012f0 2024-03-18T14:58:23.414 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.410+0000 7fe375ffb640 1 -- 172.21.15.76:0/434913172 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36404b060 con 0x7fe3781012f0 2024-03-18T14:58:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.412+0000 7fe375ffb640 1 -- 172.21.15.76:0/434913172 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fe364044070 con 0x7fe3781012f0 2024-03-18T14:58:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe38196d740 1 Processor -- start 2024-03-18T14:58:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe38196d740 1 -- start start 2024-03-18T14:58:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe375ffb640 1 --2- 172.21.15.76:0/434913172 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe3680a2a10 0x7fe3680a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe38196d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474b130 0x55838474b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe38196d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474ba50 0x5583847542e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe38196d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 0x558384756c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe37d31b640 1 --2- 172.21.15.76:0/434913172 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe3680a2a10 0x7fe3680a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe38196d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5583846afa40 con 0x55838474ba50 2024-03-18T14:58:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe38196d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558384658920 con 0x55838474b130 2024-03-18T14:58:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe38196d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5583846abae0 con 0x558384754820 2024-03-18T14:58:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe375ffb640 1 -- 172.21.15.76:0/434913172 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8440+0+0 (secure 0 0 0) 0x7fe3640f0530 con 0x7fe3781012f0 2024-03-18T14:58:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe37cb1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474b130 0x55838474b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe37cb1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474b130 0x55838474b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:39046/0 (socket says 172.21.15.76:39046) 2024-03-18T14:58:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe37cb1a640 1 -- 172.21.15.76:0/4123198780 learned_addr learned my addr 172.21.15.76:0/4123198780 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe377fff640 1 --2- 172.21.15.76:0/4123198780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474ba50 0x5583847542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe37d31b640 1 --2- 172.21.15.76:0/4123198780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 0x558384756c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe37cb1a640 1 -- 172.21.15.76:0/4123198780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 msgr2=0x558384756c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe37cb1a640 1 --2- 172.21.15.76:0/4123198780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 0x558384756c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe37cb1a640 1 -- 172.21.15.76:0/4123198780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474ba50 msgr2=0x5583847542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe37cb1a640 1 --2- 172.21.15.76:0/4123198780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474ba50 0x5583847542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.413+0000 7fe37cb1a640 1 -- 172.21.15.76:0/4123198780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558384670460 con 0x55838474b130 2024-03-18T14:58:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.414+0000 7fe37d31b640 1 --2- 172.21.15.76:0/434913172 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe3680a2a10 0x7fe3680a4ed0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe370009fd0 tx=0x7fe370011bc0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.414+0000 7fe37cb1a640 1 --2- 172.21.15.76:0/4123198780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474b130 0x55838474b510 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe36c009c70 tx=0x7fe36c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7496fb0af64f76d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.414+0000 7fe362ffd640 1 -- 172.21.15.76:0/4123198780 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c01e070 con 0x55838474b130 2024-03-18T14:58:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.414+0000 7fe362ffd640 1 -- 172.21.15.76:0/4123198780 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe36c009e20 con 0x55838474b130 2024-03-18T14:58:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.414+0000 7fe362ffd640 1 -- 172.21.15.76:0/4123198780 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c019670 con 0x55838474b130 2024-03-18T14:58:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.415+0000 7fe38196d740 1 -- 172.21.15.76:0/4123198780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474b130 msgr2=0x55838474b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.415+0000 7fe38196d740 1 --2- 172.21.15.76:0/4123198780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474b130 0x55838474b510 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe36c009c70 tx=0x7fe36c011810 comp rx=0 tx=0).stop 2024-03-18T14:58:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.415+0000 7fe38196d740 1 -- 172.21.15.76:0/4123198780 shutdown_connections 2024-03-18T14:58:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.415+0000 7fe38196d740 1 --2- 172.21.15.76:0/4123198780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 0x558384756c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.415+0000 7fe38196d740 1 --2- 172.21.15.76:0/4123198780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474ba50 0x5583847542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.415+0000 7fe38196d740 1 --2- 172.21.15.76:0/4123198780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474b130 0x55838474b510 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.415+0000 7fe38196d740 1 -- 172.21.15.76:0/4123198780 >> 172.21.15.76:0/4123198780 conn(0x558384745a50 msgr2=0x558384745e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:23.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.415+0000 7fe38196d740 1 -- 172.21.15.76:0/4123198780 shutdown_connections 2024-03-18T14:58:23.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.415+0000 7fe38196d740 1 -- 172.21.15.76:0/4123198780 wait complete. 2024-03-18T14:58:23.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.415+0000 7fe38196d740 1 -- start start 2024-03-18T14:58:23.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.415+0000 7fe38196d740 1 Processor -- start 2024-03-18T14:58:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.415+0000 7fe38196d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474b130 0x558384751800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.415+0000 7fe38196d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474ba50 0x558384751d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.415+0000 7fe38196d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 0x558384752280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.415+0000 7fe38196d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x558384757450 con 0x55838474b130 2024-03-18T14:58:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.415+0000 7fe38196d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55838466bd70 con 0x55838474ba50 2024-03-18T14:58:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.415+0000 7fe38196d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5583847575d0 con 0x558384754820 2024-03-18T14:58:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe38196d740 10 client.0 _ll_register_callbacks cb 0x5583843c3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:58:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe37cb1a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474b130 0x558384751800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe377fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474ba50 0x558384751d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe377fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474ba50 0x558384751d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:39062/0 (socket says 172.21.15.76:39062) 2024-03-18T14:58:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe37cb1a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474b130 0x558384751800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56768/0 (socket says 172.21.15.76:56768) 2024-03-18T14:58:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe377fff640 1 -- 172.21.15.76:0/990820778 learned_addr learned my addr 172.21.15.76:0/990820778 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe37d31b640 1 --2- 172.21.15.76:0/990820778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 0x558384752280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe377fff640 1 -- 172.21.15.76:0/990820778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 msgr2=0x558384752280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe377fff640 1 --2- 172.21.15.76:0/990820778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 0x558384752280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe377fff640 1 -- 172.21.15.76:0/990820778 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474b130 msgr2=0x558384751800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe377fff640 1 --2- 172.21.15.76:0/990820778 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474b130 0x558384751800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe377fff640 1 -- 172.21.15.76:0/990820778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583843c4210 con 0x55838474ba50 2024-03-18T14:58:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe37cb1a640 1 --2- 172.21.15.76:0/990820778 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474b130 0x558384751800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:58:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe377fff640 1 --2- 172.21.15.76:0/990820778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474ba50 0x558384751d40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe3640b9200 tx=0x7fe3640bc830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b68d5c9b1b42de3f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe357fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:58:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe357fff640 1 -- 172.21.15.76:0/990820778 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3640c0e40 con 0x55838474ba50 2024-03-18T14:58:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe357fff640 1 -- 172.21.15.76:0/990820778 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8440+0+0 (secure 0 0 0) 0x7fe3640b9410 con 0x55838474ba50 2024-03-18T14:58:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe38196d740 1 -- 172.21.15.76:0/990820778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe36c009e20 con 0x55838474ba50 2024-03-18T14:58:23.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe357fff640 1 -- 172.21.15.76:0/990820778 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fe3640d9020 con 0x55838474ba50 2024-03-18T14:58:23.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe38196d740 10 client.26111 fetch_fsmap learned FSMap version 38 2024-03-18T14:58:23.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.416+0000 7fe38196d740 1 -- 172.21.15.76:0/990820778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x558384490620 con 0x55838474ba50 2024-03-18T14:58:23.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.417+0000 7fe357fff640 1 -- 172.21.15.76:0/990820778 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe36c009e20 con 0x55838474ba50 2024-03-18T14:58:23.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.417+0000 7fe357fff640 1 -- 172.21.15.76:0/990820778 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fe36404e060 con 0x55838474ba50 2024-03-18T14:58:23.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.417+0000 7fe38196d740 10 client.26111 fetch_fsmap finished waiting for FSMap version 38 2024-03-18T14:58:23.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.417+0000 7fe38196d740 10 client.26111 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:58:23.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.417+0000 7fe38196d740 20 client.26111 populate_metadata read hostname 'smithi076' 2024-03-18T14:58:23.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.417+0000 7fe38196d740 1 --2- 172.21.15.76:0/990820778 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558384764a90 0x558384766f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.417+0000 7fe38196d740 4 client.26111 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:58:23.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.417+0000 7fe38196d740 1 -- 172.21.15.76:0/990820778 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583846bcf80 con 0x558384764a90 2024-03-18T14:58:23.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.417+0000 7fe37cb1a640 1 --2- 172.21.15.76:0/990820778 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558384764a90 0x558384766f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.417+0000 7fe37cb1a640 1 --2- 172.21.15.76:0/990820778 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558384764a90 0x558384766f50 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd6fa78ee5076bd7 server_cookie=c8f947b4a7c37650 in_seq=0 out_seq=0 2024-03-18T14:58:23.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.417+0000 7fe357fff640 10 client.26111 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:58:23.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.418+0000 7fe357fff640 1 -- 172.21.15.76:0/990820778 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5583846bcf80 con 0x558384764a90 2024-03-18T14:58:23.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.418+0000 7fe357fff640 10 client.26111 handle_command_reply: tid=0 2024-03-18T14:58:23.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.418+0000 7fe38196d740 1 client.26111 shutdown 2024-03-18T14:58:23.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.418+0000 7fe38196d740 1 -- 172.21.15.76:0/990820778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474ba50 msgr2=0x558384751d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:23.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.418+0000 7fe38196d740 1 --2- 172.21.15.76:0/990820778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474ba50 0x558384751d40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe3640b9200 tx=0x7fe3640bc830 comp rx=0 tx=0).stop 2024-03-18T14:58:23.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.418+0000 7fe38196d740 1 -- 172.21.15.76:0/990820778 shutdown_connections 2024-03-18T14:58:23.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.418+0000 7fe38196d740 1 --2- 172.21.15.76:0/990820778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 0x558384752280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.418+0000 7fe38196d740 1 --2- 172.21.15.76:0/990820778 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558384764a90 0x558384766f50 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.418+0000 7fe38196d740 1 --2- 172.21.15.76:0/990820778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474ba50 0x558384751d40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.418+0000 7fe38196d740 1 --2- 172.21.15.76:0/990820778 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474b130 0x558384751800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.418+0000 7fe357fff640 0 client.26111 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:58:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.418+0000 7fe38196d740 1 -- 172.21.15.76:0/990820778 >> 172.21.15.76:0/990820778 conn(0x558384745a50 msgr2=0x5583847477a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.418+0000 7fe38196d740 1 -- 172.21.15.76:0/990820778 shutdown_connections 2024-03-18T14:58:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.418+0000 7fe38196d740 1 -- 172.21.15.76:0/990820778 wait complete. 2024-03-18T14:58:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.418+0000 7fe38196d740 20 client.26111 trim_cache size 0 max 16384 2024-03-18T14:58:23.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.434+0000 7fe38196d740 1 Processor -- start 2024-03-18T14:58:23.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.434+0000 7fe38196d740 1 -- start start 2024-03-18T14:58:23.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.434+0000 7fe38196d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474b130 0x558384753f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.434+0000 7fe38196d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474ba50 0x558384760220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.434+0000 7fe38196d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 0x558384750960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.434+0000 7fe38196d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe35c010f40 con 0x55838474ba50 2024-03-18T14:58:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.434+0000 7fe38196d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe35c00e7e0 con 0x55838474b130 2024-03-18T14:58:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.434+0000 7fe38196d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe35c00c630 con 0x558384754820 2024-03-18T14:58:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.435+0000 7fe377fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474ba50 0x558384760220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.435+0000 7fe377fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474ba50 0x558384760220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56782/0 (socket says 172.21.15.76:56782) 2024-03-18T14:58:23.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.435+0000 7fe377fff640 1 -- 172.21.15.76:0/1952033087 learned_addr learned my addr 172.21.15.76:0/1952033087 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:23.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.435+0000 7fe37cb1a640 1 --2- 172.21.15.76:0/1952033087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474b130 0x558384753f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.435+0000 7fe37d31b640 1 --2- 172.21.15.76:0/1952033087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 0x558384750960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.435+0000 7fe377fff640 1 -- 172.21.15.76:0/1952033087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 msgr2=0x558384750960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:23.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.435+0000 7fe377fff640 1 --2- 172.21.15.76:0/1952033087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 0x558384750960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.435+0000 7fe377fff640 1 -- 172.21.15.76:0/1952033087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474b130 msgr2=0x558384753f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:23.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.435+0000 7fe377fff640 1 --2- 172.21.15.76:0/1952033087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474b130 0x558384753f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.435+0000 7fe377fff640 1 -- 172.21.15.76:0/1952033087 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558384745880 con 0x55838474ba50 2024-03-18T14:58:23.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.435+0000 7fe37cb1a640 1 --2- 172.21.15.76:0/1952033087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474b130 0x558384753f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:58:23.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.435+0000 7fe377fff640 1 --2- 172.21.15.76:0/1952033087 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474ba50 0x558384760220 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe364009430 tx=0x7fe3640bc700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7702751da8a5f6fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:23.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.435+0000 7fe3627fc640 1 -- 172.21.15.76:0/1952033087 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3640c0e40 con 0x55838474ba50 2024-03-18T14:58:23.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.435+0000 7fe3627fc640 1 -- 172.21.15.76:0/1952033087 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3640d4070 con 0x55838474ba50 2024-03-18T14:58:23.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.435+0000 7fe3627fc640 1 -- 172.21.15.76:0/1952033087 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3640edec0 con 0x55838474ba50 2024-03-18T14:58:23.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.436+0000 7fe38196d740 1 -- 172.21.15.76:0/1952033087 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474ba50 msgr2=0x558384760220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:23.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.436+0000 7fe38196d740 1 --2- 172.21.15.76:0/1952033087 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474ba50 0x558384760220 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe364009430 tx=0x7fe3640bc700 comp rx=0 tx=0).stop 2024-03-18T14:58:23.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.436+0000 7fe38196d740 1 -- 172.21.15.76:0/1952033087 shutdown_connections 2024-03-18T14:58:23.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.436+0000 7fe38196d740 1 --2- 172.21.15.76:0/1952033087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 0x558384750960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.436+0000 7fe38196d740 1 --2- 172.21.15.76:0/1952033087 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474ba50 0x558384760220 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.436+0000 7fe38196d740 1 --2- 172.21.15.76:0/1952033087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474b130 0x558384753f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.436+0000 7fe38196d740 1 -- 172.21.15.76:0/1952033087 >> 172.21.15.76:0/1952033087 conn(0x558384745a50 msgr2=0x558384746ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:23.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.436+0000 7fe38196d740 1 -- 172.21.15.76:0/1952033087 shutdown_connections 2024-03-18T14:58:23.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.436+0000 7fe38196d740 1 -- 172.21.15.76:0/1952033087 wait complete. 2024-03-18T14:58:23.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.436+0000 7fe38196d740 1 -- start start 2024-03-18T14:58:23.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.436+0000 7fe38196d740 1 Processor -- start 2024-03-18T14:58:23.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe38196d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474b130 0x55838474e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe38196d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474ba50 0x55838474e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe38196d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 0x55838474ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe38196d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe35c00cca0 con 0x55838474b130 2024-03-18T14:58:23.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe38196d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe35c00e110 con 0x55838474ba50 2024-03-18T14:58:23.451 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe38196d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe35c00d370 con 0x558384754820 2024-03-18T14:58:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe38196d740 10 client.0 _ll_register_callbacks cb 0x7fe3640b9040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:58:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe37cb1a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474b130 0x55838474e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe37cb1a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474b130 0x55838474e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56798/0 (socket says 172.21.15.76:56798) 2024-03-18T14:58:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe37cb1a640 1 -- 172.21.15.76:0/3897964439 learned_addr learned my addr 172.21.15.76:0/3897964439 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe377fff640 1 --2- 172.21.15.76:0/3897964439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474ba50 0x55838474e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe37d31b640 1 --2- 172.21.15.76:0/3897964439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 0x55838474ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe37cb1a640 1 -- 172.21.15.76:0/3897964439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 msgr2=0x55838474ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe37cb1a640 1 --2- 172.21.15.76:0/3897964439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 0x55838474ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe37cb1a640 1 -- 172.21.15.76:0/3897964439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474ba50 msgr2=0x55838474e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe37cb1a640 1 --2- 172.21.15.76:0/3897964439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474ba50 0x55838474e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe37cb1a640 1 -- 172.21.15.76:0/3897964439 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583847c9e80 con 0x55838474b130 2024-03-18T14:58:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe37cb1a640 1 --2- 172.21.15.76:0/3897964439 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474b130 0x55838474e3b0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe36c012fd0 tx=0x7fe36c0093c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5812f6722999358a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe357fff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:58:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.437+0000 7fe357fff640 1 -- 172.21.15.76:0/3897964439 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c01e040 con 0x55838474b130 2024-03-18T14:58:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.438+0000 7fe357fff640 1 -- 172.21.15.76:0/3897964439 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8440+0+0 (secure 0 0 0) 0x7fe36c0050c0 con 0x55838474b130 2024-03-18T14:58:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.438+0000 7fe38196d740 1 -- 172.21.15.76:0/3897964439 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3640d4070 con 0x55838474b130 2024-03-18T14:58:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.438+0000 7fe357fff640 1 -- 172.21.15.76:0/3897964439 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fe36c03a050 con 0x55838474b130 2024-03-18T14:58:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.438+0000 7fe38196d740 10 client.16260 fetch_fsmap learned FSMap version 38 2024-03-18T14:58:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.438+0000 7fe38196d740 1 -- 172.21.15.76:0/3897964439 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5583847cb1a0 con 0x55838474b130 2024-03-18T14:58:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.438+0000 7fe357fff640 1 -- 172.21.15.76:0/3897964439 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3640d4070 con 0x55838474b130 2024-03-18T14:58:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.438+0000 7fe357fff640 1 -- 172.21.15.76:0/3897964439 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fe36c027440 con 0x55838474b130 2024-03-18T14:58:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.438+0000 7fe38196d740 10 client.16260 fetch_fsmap finished waiting for FSMap version 38 2024-03-18T14:58:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.438+0000 7fe38196d740 10 client.16260 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:58:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.438+0000 7fe38196d740 20 client.16260 populate_metadata read hostname 'smithi076' 2024-03-18T14:58:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.438+0000 7fe38196d740 1 --2- 172.21.15.76:0/3897964439 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5583847cb810 0x5583847cdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.438+0000 7fe38196d740 4 client.16260 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:58:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.438+0000 7fe38196d740 1 -- 172.21.15.76:0/3897964439 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5583846b72d0 con 0x5583847cb810 2024-03-18T14:58:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.438+0000 7fe377fff640 1 --2- 172.21.15.76:0/3897964439 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5583847cb810 0x5583847cdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.439+0000 7fe357fff640 10 client.16260 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:58:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.439+0000 7fe377fff640 1 --2- 172.21.15.76:0/3897964439 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5583847cb810 0x5583847cdc30 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c6b751dd3ffec21 server_cookie=44f447cf6f5d3dcf in_seq=0 out_seq=0 2024-03-18T14:58:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.488+0000 7fe357fff640 1 -- 172.21.15.76:0/3897964439 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5583846b72d0 con 0x5583847cb810 2024-03-18T14:58:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.488+0000 7fe357fff640 10 client.16260 handle_command_reply: tid=0 2024-03-18T14:58:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.488+0000 7fe38196d740 1 client.16260 shutdown 2024-03-18T14:58:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.488+0000 7fe38196d740 1 -- 172.21.15.76:0/3897964439 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474b130 msgr2=0x55838474e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.488+0000 7fe38196d740 1 --2- 172.21.15.76:0/3897964439 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474b130 0x55838474e3b0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe36c012fd0 tx=0x7fe36c0093c0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.488+0000 7fe38196d740 1 -- 172.21.15.76:0/3897964439 shutdown_connections 2024-03-18T14:58:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.488+0000 7fe38196d740 1 --2- 172.21.15.76:0/3897964439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558384754820 0x55838474ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.488+0000 7fe38196d740 1 --2- 172.21.15.76:0/3897964439 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5583847cb810 0x5583847cdc30 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.488+0000 7fe38196d740 1 --2- 172.21.15.76:0/3897964439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55838474ba50 0x55838474e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.488+0000 7fe357fff640 0 client.16260 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:58:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.488+0000 7fe38196d740 1 --2- 172.21.15.76:0/3897964439 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55838474b130 0x55838474e3b0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.488+0000 7fe38196d740 1 -- 172.21.15.76:0/3897964439 >> 172.21.15.76:0/3897964439 conn(0x558384745a50 msgr2=0x558384744320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.488+0000 7fe38196d740 1 -- 172.21.15.76:0/3897964439 shutdown_connections 2024-03-18T14:58:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.488+0000 7fe38196d740 1 -- 172.21.15.76:0/3897964439 wait complete. 2024-03-18T14:58:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.488+0000 7fe38196d740 20 client.16260 trim_cache size 0 max 16384 2024-03-18T14:58:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:58:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:58:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:58:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:58:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.489+0000 7fe37ed7e640 1 -- 172.21.15.76:0/434913172 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe3680a2a10 msgr2=0x7fe3680a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.489+0000 7fe37ed7e640 1 --2- 172.21.15.76:0/434913172 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe3680a2a10 0x7fe3680a4ed0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe370009fd0 tx=0x7fe370011bc0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.489+0000 7fe37ed7e640 1 -- 172.21.15.76:0/434913172 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe3781012f0 msgr2=0x7fe378102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.489+0000 7fe37ed7e640 1 --2- 172.21.15.76:0/434913172 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe3781012f0 0x7fe378102890 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe364005940 tx=0x7fe364004270 comp rx=0 tx=0).stop 2024-03-18T14:58:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.489+0000 7fe37ed7e640 1 -- 172.21.15.76:0/434913172 shutdown_connections 2024-03-18T14:58:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.489+0000 7fe37ed7e640 1 --2- 172.21.15.76:0/434913172 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe3680a2a10 0x7fe3680a4ed0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.489+0000 7fe37ed7e640 1 --2- 172.21.15.76:0/434913172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe378105680 0x7fe378102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.489+0000 7fe37ed7e640 1 --2- 172.21.15.76:0/434913172 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe3781012f0 0x7fe378102890 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.489+0000 7fe37ed7e640 1 --2- 172.21.15.76:0/434913172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe378100930 0x7fe378102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.489+0000 7fe37ed7e640 1 -- 172.21.15.76:0/434913172 >> 172.21.15.76:0/434913172 conn(0x7fe378077530 msgr2=0x7fe3780690a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.489+0000 7fe37ed7e640 1 -- 172.21.15.76:0/434913172 shutdown_connections 2024-03-18T14:58:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:23.489+0000 7fe37ed7e640 1 -- 172.21.15.76:0/434913172 wait complete. 2024-03-18T14:58:23.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 476529 9.48 MB/sec execute 144 sec latency 30.059 ms 2024-03-18T14:58:23.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:23 smithi174 ceph-mon[36787]: pgmap v161: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 3.3 MiB/s wr, 240 op/s 2024-03-18T14:58:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:23 smithi077 ceph-mon[35865]: pgmap v161: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 3.3 MiB/s wr, 240 op/s 2024-03-18T14:58:23.977 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:58:23 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setuser ceph since I am not root 2024-03-18T14:58:23.977 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:58:23 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setgroup ceph since I am not root 2024-03-18T14:58:23.977 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:58:23 smithi077 ceph-mds[65404]: -- 172.21.15.77:0/1890213849 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564eb6b57a40 con 0x564eb6b2ac00 2024-03-18T14:58:23.978 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 14:58:23 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: starting mds.e at 2024-03-18T14:58:24.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:23 smithi076 ceph-mon[32205]: pgmap v161: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 3.3 MiB/s wr, 240 op/s 2024-03-18T14:58:24.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 478582 9.47 MB/sec execute 145 sec latency 22.343 ms 2024-03-18T14:58:24.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:24 smithi174 ceph-mon[36787]: daemon mds.e restarted 2024-03-18T14:58:24.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:24 smithi077 ceph-mon[35865]: daemon mds.e restarted 2024-03-18T14:58:25.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:24 smithi076 ceph-mon[32205]: daemon mds.e restarted 2024-03-18T14:58:25.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 480262 9.48 MB/sec execute 146 sec latency 20.671 ms 2024-03-18T14:58:25.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:25 smithi174 ceph-mon[36787]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T14:58:25.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:25 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.77:6834/3976391263,v1:172.21.15.77:6835/3976391263] up:boot 2024-03-18T14:58:25.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:25 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T14:58:25.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:25 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:58:25.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:25 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:58:25.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:25 smithi174 ceph-mon[36787]: pgmap v163: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 2.7 MiB/s wr, 213 op/s 2024-03-18T14:58:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:25 smithi077 ceph-mon[35865]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T14:58:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:25 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.77:6834/3976391263,v1:172.21.15.77:6835/3976391263] up:boot 2024-03-18T14:58:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:25 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T14:58:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:25 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:58:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:25 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:58:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:25 smithi077 ceph-mon[35865]: pgmap v163: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 2.7 MiB/s wr, 213 op/s 2024-03-18T14:58:26.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:25 smithi076 ceph-mon[32205]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T14:58:26.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:25 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.77:6834/3976391263,v1:172.21.15.77:6835/3976391263] up:boot 2024-03-18T14:58:26.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:25 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T14:58:26.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:25 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T14:58:26.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:25 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:58:26.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:25 smithi076 ceph-mon[32205]: pgmap v163: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 2.7 MiB/s wr, 213 op/s 2024-03-18T14:58:26.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 482122 9.47 MB/sec execute 147 sec latency 25.978 ms 2024-03-18T14:58:27.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:27 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:58:27.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:27 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:58:27.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:27 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:58:27.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 483764 9.49 MB/sec execute 148 sec latency 20.143 ms 2024-03-18T14:58:28.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:28 smithi174 ceph-mon[36787]: pgmap v164: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 2.7 MiB/s wr, 213 op/s 2024-03-18T14:58:28.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:28 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:58:28.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:28 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:58:28.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:28 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:58:28.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:28 smithi077 ceph-mon[35865]: pgmap v164: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 2.7 MiB/s wr, 213 op/s 2024-03-18T14:58:28.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:28 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:58:28.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:28 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:58:28.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:28 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:58:28.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:28 smithi076 ceph-mon[32205]: pgmap v164: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 2.7 MiB/s wr, 213 op/s 2024-03-18T14:58:28.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:28 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:58:28.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:28 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:58:28.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:28 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:58:28.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 485814 9.48 MB/sec execute 149 sec latency 19.686 ms 2024-03-18T14:58:29.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 487483 9.49 MB/sec execute 150 sec latency 12.546 ms 2024-03-18T14:58:29.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:29 smithi174 ceph-mon[36787]: pgmap v165: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 308 op/s 2024-03-18T14:58:29.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:29 smithi077 ceph-mon[35865]: pgmap v165: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 308 op/s 2024-03-18T14:58:30.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:29 smithi076 ceph-mon[32205]: pgmap v165: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 308 op/s 2024-03-18T14:58:30.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 489928 9.51 MB/sec execute 151 sec latency 19.810 ms 2024-03-18T14:58:31.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 491089 9.49 MB/sec execute 152 sec latency 41.047 ms 2024-03-18T14:58:31.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:31 smithi174 ceph-mon[36787]: pgmap v166: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 308 op/s 2024-03-18T14:58:31.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:31 smithi077 ceph-mon[35865]: pgmap v166: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 308 op/s 2024-03-18T14:58:32.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:31 smithi076 ceph-mon[32205]: pgmap v166: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 308 op/s 2024-03-18T14:58:32.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 493503 9.51 MB/sec execute 153 sec latency 21.353 ms 2024-03-18T14:58:33.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 494921 9.50 MB/sec execute 154 sec latency 17.234 ms 2024-03-18T14:58:33.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:33 smithi174 ceph-mon[36787]: pgmap v167: 97 pgs: 97 active+clean; 142 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 253 op/s 2024-03-18T14:58:33.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:33 smithi077 ceph-mon[35865]: pgmap v167: 97 pgs: 97 active+clean; 142 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 253 op/s 2024-03-18T14:58:34.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:33 smithi076 ceph-mon[32205]: pgmap v167: 97 pgs: 97 active+clean; 142 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 253 op/s 2024-03-18T14:58:34.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 497546 9.54 MB/sec execute 155 sec latency 56.264 ms 2024-03-18T14:58:35.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 498959 9.52 MB/sec execute 156 sec latency 18.613 ms 2024-03-18T14:58:35.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:35 smithi174 ceph-mon[36787]: pgmap v168: 97 pgs: 97 active+clean; 142 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 232 op/s 2024-03-18T14:58:35.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:35 smithi077 ceph-mon[35865]: pgmap v168: 97 pgs: 97 active+clean; 142 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 232 op/s 2024-03-18T14:58:36.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:35 smithi076 ceph-mon[32205]: pgmap v168: 97 pgs: 97 active+clean; 142 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 232 op/s 2024-03-18T14:58:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.592+0000 7f7245952640 1 -- 172.21.15.76:0/3117446427 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7240069800 msgr2=0x7f724010e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.592+0000 7f7245952640 1 --2- 172.21.15.76:0/3117446427 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7240069800 0x7f724010e670 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f7238009ca0 tx=0x7f723802c0f0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.592+0000 7f7245952640 1 -- 172.21.15.76:0/3117446427 shutdown_connections 2024-03-18T14:58:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.592+0000 7f7245952640 1 --2- 172.21.15.76:0/3117446427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7240068e60 0x7f72400692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.592+0000 7f7245952640 1 --2- 172.21.15.76:0/3117446427 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7240069800 0x7f724010e670 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.592+0000 7f7245952640 1 --2- 172.21.15.76:0/3117446427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72401078d0 0x7f7240107cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.592+0000 7f7245952640 1 -- 172.21.15.76:0/3117446427 >> 172.21.15.76:0/3117446427 conn(0x7f7240077530 msgr2=0x7f7240077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.592+0000 7f7245952640 1 -- 172.21.15.76:0/3117446427 shutdown_connections 2024-03-18T14:58:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.592+0000 7f7245952640 1 -- 172.21.15.76:0/3117446427 wait complete. 2024-03-18T14:58:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.592+0000 7f7245952640 1 Processor -- start 2024-03-18T14:58:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f7245952640 1 -- start start 2024-03-18T14:58:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f7245952640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7240068e60 0x7f7240197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f7245952640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7240069800 0x7f7240198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f7245952640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72401078d0 0x7f724019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f7245952640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7240111c50 con 0x7f7240068e60 2024-03-18T14:58:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f7245952640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7240111ad0 con 0x7f7240069800 2024-03-18T14:58:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f7245952640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7240111dd0 con 0x7f72401078d0 2024-03-18T14:58:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f723effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7240068e60 0x7f7240197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f723effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7240068e60 0x7f7240197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34050/0 (socket says 172.21.15.76:34050) 2024-03-18T14:58:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f723effd640 1 -- 172.21.15.76:0/3885667604 learned_addr learned my addr 172.21.15.76:0/3885667604 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f723e7fc640 1 --2- 172.21.15.76:0/3885667604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7240069800 0x7f7240198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f723f7fe640 1 --2- 172.21.15.76:0/3885667604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72401078d0 0x7f724019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f723effd640 1 -- 172.21.15.76:0/3885667604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72401078d0 msgr2=0x7f724019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f723effd640 1 --2- 172.21.15.76:0/3885667604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72401078d0 0x7f724019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f723effd640 1 -- 172.21.15.76:0/3885667604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7240069800 msgr2=0x7f7240198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f723effd640 1 --2- 172.21.15.76:0/3885667604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7240069800 0x7f7240198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f723effd640 1 -- 172.21.15.76:0/3885667604 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f724019cf40 con 0x7f7240068e60 2024-03-18T14:58:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f723e7fc640 1 --2- 172.21.15.76:0/3885667604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7240069800 0x7f7240198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:58:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f723effd640 1 --2- 172.21.15.76:0/3885667604 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7240068e60 0x7f7240197f50 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f723000ea60 tx=0x7f723000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.593+0000 7f7244950640 1 -- 172.21.15.76:0/3885667604 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723000cde0 con 0x7f7240068e60 2024-03-18T14:58:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.594+0000 7f7245952640 1 -- 172.21.15.76:0/3885667604 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f724019d230 con 0x7f7240068e60 2024-03-18T14:58:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.594+0000 7f7244950640 1 -- 172.21.15.76:0/3885667604 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72300035f0 con 0x7f7240068e60 2024-03-18T14:58:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.594+0000 7f7244950640 1 -- 172.21.15.76:0/3885667604 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230010670 con 0x7f7240068e60 2024-03-18T14:58:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.594+0000 7f7245952640 1 -- 172.21.15.76:0/3885667604 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72401a4ac0 con 0x7f7240068e60 2024-03-18T14:58:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.596+0000 7f7244950640 1 -- 172.21.15.76:0/3885667604 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f72300037a0 con 0x7f7240068e60 2024-03-18T14:58:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.596+0000 7f724860e740 1 Processor -- start 2024-03-18T14:58:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.596+0000 7f7244950640 1 --2- 172.21.15.76:0/3885667604 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f72280a2a10 0x7f72280a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.596+0000 7f724860e740 1 -- start start 2024-03-18T14:58:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.596+0000 7f723f7fe640 1 --2- 172.21.15.76:0/3885667604 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f72280a2a10 0x7f72280a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.596+0000 7f7244950640 1 -- 172.21.15.76:0/3885667604 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7f72300cf100 con 0x7f7240068e60 2024-03-18T14:58:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.596+0000 7f724860e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be552e130 0x555be552e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.596+0000 7f724860e740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552ea50 0x555be55372e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.596+0000 7f724860e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be5537820 0x555be5539c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f724860e740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x555be5492a40 con 0x555be552ea50 2024-03-18T14:58:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f724860e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555be543b920 con 0x555be5537820 2024-03-18T14:58:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f724860e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555be548eae0 con 0x555be552e130 2024-03-18T14:58:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f723e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552ea50 0x555be55372e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f723e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552ea50 0x555be55372e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34058/0 (socket says 172.21.15.76:34058) 2024-03-18T14:58:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f723effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be552e130 0x555be552e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f723e7fc640 1 -- 172.21.15.76:0/2692586998 learned_addr learned my addr 172.21.15.76:0/2692586998 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f723f7fe640 1 --2- 172.21.15.76:0/2692586998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be5537820 0x555be5539c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f723effd640 1 -- 172.21.15.76:0/2692586998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be5537820 msgr2=0x555be5539c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f723effd640 1 --2- 172.21.15.76:0/2692586998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be5537820 0x555be5539c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f723effd640 1 -- 172.21.15.76:0/2692586998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552ea50 msgr2=0x555be55372e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f723effd640 1 --2- 172.21.15.76:0/2692586998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552ea50 0x555be55372e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f723effd640 1 -- 172.21.15.76:0/2692586998 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555be5453460 con 0x555be552e130 2024-03-18T14:58:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f723e7fc640 1 --2- 172.21.15.76:0/2692586998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552ea50 0x555be55372e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:58:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f723f7fe640 1 --2- 172.21.15.76:0/3885667604 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f72280a2a10 0x7f72280a4ed0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7238004af0 tx=0x7f723802cea0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f723effd640 1 --2- 172.21.15.76:0/2692586998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be552e130 0x555be552e510 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f72300cc8a0 tx=0x7f723009b3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67aea6d9d7df39f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f7225ffb640 1 -- 172.21.15.76:0/2692586998 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723009f900 con 0x555be552e130 2024-03-18T14:58:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f7225ffb640 1 -- 172.21.15.76:0/2692586998 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72300a3f50 con 0x555be552e130 2024-03-18T14:58:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.597+0000 7f7225ffb640 1 -- 172.21.15.76:0/2692586998 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230097e20 con 0x555be552e130 2024-03-18T14:58:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.598+0000 7f724860e740 1 -- 172.21.15.76:0/2692586998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be552e130 msgr2=0x555be552e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.598+0000 7f724860e740 1 --2- 172.21.15.76:0/2692586998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be552e130 0x555be552e510 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f72300cc8a0 tx=0x7f723009b3b0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.598+0000 7f724860e740 1 -- 172.21.15.76:0/2692586998 shutdown_connections 2024-03-18T14:58:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.598+0000 7f724860e740 1 --2- 172.21.15.76:0/2692586998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be5537820 0x555be5539c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.598+0000 7f724860e740 1 --2- 172.21.15.76:0/2692586998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552ea50 0x555be55372e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.598+0000 7f724860e740 1 --2- 172.21.15.76:0/2692586998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be552e130 0x555be552e510 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.598+0000 7f724860e740 1 -- 172.21.15.76:0/2692586998 >> 172.21.15.76:0/2692586998 conn(0x555be5528a50 msgr2=0x555be5528e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.598+0000 7f724860e740 1 -- 172.21.15.76:0/2692586998 shutdown_connections 2024-03-18T14:58:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.598+0000 7f724860e740 1 -- 172.21.15.76:0/2692586998 wait complete. 2024-03-18T14:58:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.598+0000 7f724860e740 1 -- start start 2024-03-18T14:58:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f724860e740 1 Processor -- start 2024-03-18T14:58:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f724860e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be552e130 0x555be55348c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f724860e740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552ea50 0x555be5534e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f724860e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be5537820 0x555be5535340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f724860e740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x555be553a3f0 con 0x555be552ea50 2024-03-18T14:58:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f724860e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555be544ed70 con 0x555be5537820 2024-03-18T14:58:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f724860e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555be553a570 con 0x555be552e130 2024-03-18T14:58:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f724860e740 10 client.0 _ll_register_callbacks cb 0x555be51a6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:58:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f723e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552ea50 0x555be5534e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f723e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552ea50 0x555be5534e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34064/0 (socket says 172.21.15.76:34064) 2024-03-18T14:58:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f723e7fc640 1 -- 172.21.15.76:0/3496780602 learned_addr learned my addr 172.21.15.76:0/3496780602 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f723f7fe640 1 --2- 172.21.15.76:0/3496780602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be5537820 0x555be5535340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f723effd640 1 --2- 172.21.15.76:0/3496780602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be552e130 0x555be55348c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f723e7fc640 1 -- 172.21.15.76:0/3496780602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be552e130 msgr2=0x555be55348c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f723e7fc640 1 --2- 172.21.15.76:0/3496780602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be552e130 0x555be55348c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f723e7fc640 1 -- 172.21.15.76:0/3496780602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be5537820 msgr2=0x555be5535340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f723e7fc640 1 --2- 172.21.15.76:0/3496780602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be5537820 0x555be5535340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f723e7fc640 1 -- 172.21.15.76:0/3496780602 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555be51a7210 con 0x555be552ea50 2024-03-18T14:58:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.599+0000 7f723f7fe640 1 --2- 172.21.15.76:0/3496780602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be5537820 0x555be5535340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:58:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.600+0000 7f723e7fc640 1 --2- 172.21.15.76:0/3496780602 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552ea50 0x555be5534e00 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f7234012d20 tx=0x7f723400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a8331e0ec733cf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.600+0000 7f721effd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:58:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.600+0000 7f721effd640 1 -- 172.21.15.76:0/3496780602 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723400fe60 con 0x555be552ea50 2024-03-18T14:58:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.600+0000 7f721effd640 1 -- 172.21.15.76:0/3496780602 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7f72340120d0 con 0x555be552ea50 2024-03-18T14:58:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.600+0000 7f724860e740 1 -- 172.21.15.76:0/3496780602 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72300a3f50 con 0x555be552ea50 2024-03-18T14:58:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.600+0000 7f721effd640 1 -- 172.21.15.76:0/3496780602 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7234004020 con 0x555be552ea50 2024-03-18T14:58:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.600+0000 7f724860e740 10 client.16275 fetch_fsmap learned FSMap version 40 2024-03-18T14:58:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.600+0000 7f724860e740 1 -- 172.21.15.76:0/3496780602 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x555be5273620 con 0x555be552ea50 2024-03-18T14:58:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.600+0000 7f721effd640 1 -- 172.21.15.76:0/3496780602 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72300a3f50 con 0x555be552ea50 2024-03-18T14:58:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.600+0000 7f721effd640 1 -- 172.21.15.76:0/3496780602 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 40) ==== 8342+0+0 (secure 0 0 0) 0x7f7234026480 con 0x555be552ea50 2024-03-18T14:58:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.600+0000 7f724860e740 10 client.16275 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T14:58:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.600+0000 7f724860e740 10 client.16275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:58:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.600+0000 7f724860e740 20 client.16275 populate_metadata read hostname 'smithi076' 2024-03-18T14:58:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.600+0000 7f724860e740 1 --2- 172.21.15.76:0/3496780602 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x555be5547a00 0x555be5549e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.600+0000 7f724860e740 4 client.16275 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:58:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.600+0000 7f724860e740 1 -- 172.21.15.76:0/3496780602 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555be549ff80 con 0x555be5547a00 2024-03-18T14:58:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.600+0000 7f723effd640 1 --2- 172.21.15.76:0/3496780602 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x555be5547a00 0x555be5549e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.601+0000 7f721effd640 10 client.16275 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:58:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.601+0000 7f723effd640 1 --2- 172.21.15.76:0/3496780602 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x555be5547a00 0x555be5549e10 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33fcda8d6aee4e07 server_cookie=ee2e4ac789a97a9e in_seq=0 out_seq=0 2024-03-18T14:58:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.601+0000 7f721effd640 1 -- 172.21.15.76:0/3496780602 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555be549ff80 con 0x555be5547a00 2024-03-18T14:58:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.601+0000 7f721effd640 10 client.16275 handle_command_reply: tid=0 2024-03-18T14:58:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.602+0000 7f724860e740 1 client.16275 shutdown 2024-03-18T14:58:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.602+0000 7f724860e740 1 -- 172.21.15.76:0/3496780602 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552ea50 msgr2=0x555be5534e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.602+0000 7f724860e740 1 --2- 172.21.15.76:0/3496780602 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552ea50 0x555be5534e00 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f7234012d20 tx=0x7f723400f790 comp rx=0 tx=0).stop 2024-03-18T14:58:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.602+0000 7f724860e740 1 -- 172.21.15.76:0/3496780602 shutdown_connections 2024-03-18T14:58:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.602+0000 7f724860e740 1 --2- 172.21.15.76:0/3496780602 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x555be5547a00 0x555be5549e10 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.602+0000 7f724860e740 1 --2- 172.21.15.76:0/3496780602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be5537820 0x555be5535340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.602+0000 7f721effd640 0 client.16275 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:58:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.602+0000 7f724860e740 1 --2- 172.21.15.76:0/3496780602 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552ea50 0x555be5534e00 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.602+0000 7f724860e740 1 --2- 172.21.15.76:0/3496780602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be552e130 0x555be55348c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.602+0000 7f724860e740 1 -- 172.21.15.76:0/3496780602 >> 172.21.15.76:0/3496780602 conn(0x555be5528a50 msgr2=0x555be552a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.602+0000 7f724860e740 1 -- 172.21.15.76:0/3496780602 shutdown_connections 2024-03-18T14:58:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.602+0000 7f724860e740 1 -- 172.21.15.76:0/3496780602 wait complete. 2024-03-18T14:58:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.602+0000 7f724860e740 20 client.16275 trim_cache size 0 max 16384 2024-03-18T14:58:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.617+0000 7f724860e740 1 Processor -- start 2024-03-18T14:58:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.617+0000 7f724860e740 1 -- start start 2024-03-18T14:58:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.617+0000 7f724860e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be552e130 0x555be5543190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.617+0000 7f724860e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be552ea50 0x555be55366f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.617+0000 7f724860e740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be5537820 0x555be554f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.617+0000 7f724860e740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f72401afbb0 con 0x555be5537820 2024-03-18T14:58:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.617+0000 7f724860e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f72401ad450 con 0x555be552ea50 2024-03-18T14:58:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.617+0000 7f724860e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f72401ab2a0 con 0x555be552e130 2024-03-18T14:58:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.617+0000 7f723f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be5537820 0x555be554f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.617+0000 7f723f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be5537820 0x555be554f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34072/0 (socket says 172.21.15.76:34072) 2024-03-18T14:58:36.621 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.617+0000 7f723f7fe640 1 -- 172.21.15.76:0/4049983055 learned_addr learned my addr 172.21.15.76:0/4049983055 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:36.621 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.617+0000 7f723e7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be552ea50 0x555be55366f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.621 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.617+0000 7f723effd640 1 --2- 172.21.15.76:0/4049983055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be552e130 0x555be5543190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.621 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.618+0000 7f723f7fe640 1 -- 172.21.15.76:0/4049983055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be552e130 msgr2=0x555be5543190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:36.621 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.618+0000 7f723f7fe640 1 --2- 172.21.15.76:0/4049983055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be552e130 0x555be5543190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.621 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.618+0000 7f723f7fe640 1 -- 172.21.15.76:0/4049983055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be552ea50 msgr2=0x555be55366f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:36.621 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.618+0000 7f723f7fe640 1 --2- 172.21.15.76:0/4049983055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be552ea50 0x555be55366f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.621 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.618+0000 7f723f7fe640 1 -- 172.21.15.76:0/4049983055 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555be552d410 con 0x555be5537820 2024-03-18T14:58:36.621 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.618+0000 7f723f7fe640 1 --2- 172.21.15.76:0/4049983055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be5537820 0x555be554f720 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7238037040 tx=0x7f723802e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63008719e38438f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:36.621 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.618+0000 7f72257fa640 1 -- 172.21.15.76:0/4049983055 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238052070 con 0x555be5537820 2024-03-18T14:58:36.621 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.618+0000 7f72257fa640 1 -- 172.21.15.76:0/4049983055 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72380328a0 con 0x555be5537820 2024-03-18T14:58:36.621 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.618+0000 7f72257fa640 1 -- 172.21.15.76:0/4049983055 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238039040 con 0x555be5537820 2024-03-18T14:58:36.621 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.619+0000 7f724860e740 1 -- 172.21.15.76:0/4049983055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be5537820 msgr2=0x555be554f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:36.621 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.619+0000 7f724860e740 1 --2- 172.21.15.76:0/4049983055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be5537820 0x555be554f720 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7238037040 tx=0x7f723802e050 comp rx=0 tx=0).stop 2024-03-18T14:58:36.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.619+0000 7f724860e740 1 -- 172.21.15.76:0/4049983055 shutdown_connections 2024-03-18T14:58:36.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.619+0000 7f724860e740 1 --2- 172.21.15.76:0/4049983055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be5537820 0x555be554f720 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.619+0000 7f724860e740 1 --2- 172.21.15.76:0/4049983055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be552ea50 0x555be55366f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.619+0000 7f724860e740 1 --2- 172.21.15.76:0/4049983055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be552e130 0x555be5543190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.619+0000 7f724860e740 1 -- 172.21.15.76:0/4049983055 >> 172.21.15.76:0/4049983055 conn(0x555be5528a50 msgr2=0x555be5535fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:36.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.619+0000 7f724860e740 1 -- 172.21.15.76:0/4049983055 shutdown_connections 2024-03-18T14:58:36.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.619+0000 7f724860e740 1 -- 172.21.15.76:0/4049983055 wait complete. 2024-03-18T14:58:36.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.619+0000 7f724860e740 1 -- start start 2024-03-18T14:58:36.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.619+0000 7f724860e740 1 Processor -- start 2024-03-18T14:58:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f724860e740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552e130 0x555be5531720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f724860e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be552ea50 0x555be5531c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f724860e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be5537820 0x555be55321a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f724860e740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f72401ab910 con 0x555be552e130 2024-03-18T14:58:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f724860e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f72401acd80 con 0x555be552ea50 2024-03-18T14:58:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f724860e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f72401abfe0 con 0x555be5537820 2024-03-18T14:58:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f724860e740 10 client.0 _ll_register_callbacks cb 0x7f7234009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:58:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f723effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552e130 0x555be5531720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f723effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552e130 0x555be5531720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34084/0 (socket says 172.21.15.76:34084) 2024-03-18T14:58:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f723effd640 1 -- 172.21.15.76:0/2293544363 learned_addr learned my addr 172.21.15.76:0/2293544363 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f723e7fc640 1 --2- 172.21.15.76:0/2293544363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be552ea50 0x555be5531c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f723f7fe640 1 --2- 172.21.15.76:0/2293544363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be5537820 0x555be55321a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f723effd640 1 -- 172.21.15.76:0/2293544363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be5537820 msgr2=0x555be55321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f723effd640 1 --2- 172.21.15.76:0/2293544363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be5537820 0x555be55321a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f723effd640 1 -- 172.21.15.76:0/2293544363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be552ea50 msgr2=0x555be5531c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f723effd640 1 --2- 172.21.15.76:0/2293544363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be552ea50 0x555be5531c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f723effd640 1 -- 172.21.15.76:0/2293544363 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555be55acff0 con 0x555be552e130 2024-03-18T14:58:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f723effd640 1 --2- 172.21.15.76:0/2293544363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552e130 0x555be5531720 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f72300cc950 tx=0x7f7230009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229559b56c143417 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f721effd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:58:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f721effd640 1 -- 172.21.15.76:0/2293544363 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72300a3f50 con 0x555be552e130 2024-03-18T14:58:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f721effd640 1 -- 172.21.15.76:0/2293544363 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7f72300860b0 con 0x555be552e130 2024-03-18T14:58:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.620+0000 7f724860e740 1 -- 172.21.15.76:0/2293544363 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72380328a0 con 0x555be552e130 2024-03-18T14:58:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.621+0000 7f721effd640 1 -- 172.21.15.76:0/2293544363 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f72300ccb20 con 0x555be552e130 2024-03-18T14:58:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.621+0000 7f724860e740 10 client.16281 fetch_fsmap learned FSMap version 40 2024-03-18T14:58:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.621+0000 7f724860e740 1 -- 172.21.15.76:0/2293544363 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x555be55ae280 con 0x555be552e130 2024-03-18T14:58:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.621+0000 7f721effd640 1 -- 172.21.15.76:0/2293544363 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72380328a0 con 0x555be552e130 2024-03-18T14:58:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.621+0000 7f721effd640 1 -- 172.21.15.76:0/2293544363 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 40) ==== 8342+0+0 (secure 0 0 0) 0x7f723009ac00 con 0x555be552e130 2024-03-18T14:58:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.621+0000 7f724860e740 10 client.16281 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T14:58:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.621+0000 7f724860e740 10 client.16281 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:58:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.621+0000 7f724860e740 20 client.16281 populate_metadata read hostname 'smithi076' 2024-03-18T14:58:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.621+0000 7f724860e740 1 --2- 172.21.15.76:0/2293544363 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x555be5547a00 0x555be55b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.621+0000 7f724860e740 4 client.16281 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:58:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.621+0000 7f724860e740 1 -- 172.21.15.76:0/2293544363 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555be549a2d0 con 0x555be5547a00 2024-03-18T14:58:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.621+0000 7f723e7fc640 1 --2- 172.21.15.76:0/2293544363 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x555be5547a00 0x555be55b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.621+0000 7f721effd640 10 client.16281 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:58:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.621+0000 7f723e7fc640 1 --2- 172.21.15.76:0/2293544363 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x555be5547a00 0x555be55b05d0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8e576661dee1c1a server_cookie=cc02a3aa5286f08a in_seq=0 out_seq=0 2024-03-18T14:58:36.697 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.696+0000 7f721effd640 1 -- 172.21.15.76:0/2293544363 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555be549a2d0 con 0x555be5547a00 2024-03-18T14:58:36.697 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.696+0000 7f721effd640 10 client.16281 handle_command_reply: tid=0 2024-03-18T14:58:36.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.696+0000 7f724860e740 1 client.16281 shutdown 2024-03-18T14:58:36.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.696+0000 7f724860e740 1 -- 172.21.15.76:0/2293544363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552e130 msgr2=0x555be5531720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:36.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.696+0000 7f724860e740 1 --2- 172.21.15.76:0/2293544363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552e130 0x555be5531720 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f72300cc950 tx=0x7f7230009520 comp rx=0 tx=0).stop 2024-03-18T14:58:36.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.697+0000 7f724860e740 1 -- 172.21.15.76:0/2293544363 shutdown_connections 2024-03-18T14:58:36.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.697+0000 7f724860e740 1 --2- 172.21.15.76:0/2293544363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555be5537820 0x555be55321a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.697+0000 7f724860e740 1 --2- 172.21.15.76:0/2293544363 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x555be5547a00 0x555be55b05d0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.697+0000 7f724860e740 1 --2- 172.21.15.76:0/2293544363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555be552ea50 0x555be5531c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.697+0000 7f724860e740 1 --2- 172.21.15.76:0/2293544363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555be552e130 0x555be5531720 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.697+0000 7f724860e740 1 -- 172.21.15.76:0/2293544363 >> 172.21.15.76:0/2293544363 conn(0x555be5528a50 msgr2=0x555be5527bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:36.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.697+0000 7f721effd640 0 client.16281 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:58:36.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.697+0000 7f724860e740 1 -- 172.21.15.76:0/2293544363 shutdown_connections 2024-03-18T14:58:36.698 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.697+0000 7f724860e740 1 -- 172.21.15.76:0/2293544363 wait complete. 2024-03-18T14:58:36.699 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.698+0000 7f724860e740 20 client.16281 trim_cache size 0 max 16384 2024-03-18T14:58:36.699 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:58:36.699 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:58:36.699 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:58:36.699 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:58:36.700 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.699+0000 7f7245952640 1 -- 172.21.15.76:0/3885667604 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f72280a2a10 msgr2=0x7f72280a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:36.700 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.699+0000 7f7245952640 1 --2- 172.21.15.76:0/3885667604 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f72280a2a10 0x7f72280a4ed0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7238004af0 tx=0x7f723802cea0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.700 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.699+0000 7f7245952640 1 -- 172.21.15.76:0/3885667604 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7240068e60 msgr2=0x7f7240197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:36.700 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.699+0000 7f7245952640 1 --2- 172.21.15.76:0/3885667604 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7240068e60 0x7f7240197f50 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f723000ea60 tx=0x7f723000ef20 comp rx=0 tx=0).stop 2024-03-18T14:58:36.700 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.699+0000 7f7245952640 1 -- 172.21.15.76:0/3885667604 shutdown_connections 2024-03-18T14:58:36.700 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.699+0000 7f7245952640 1 --2- 172.21.15.76:0/3885667604 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f72280a2a10 0x7f72280a4ed0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.701 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.699+0000 7f7245952640 1 --2- 172.21.15.76:0/3885667604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72401078d0 0x7f724019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.701 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.699+0000 7f7245952640 1 --2- 172.21.15.76:0/3885667604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7240069800 0x7f7240198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:36.701 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.699+0000 7f7245952640 1 --2- 172.21.15.76:0/3885667604 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7240068e60 0x7f7240197f50 secure :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f723000ea60 tx=0x7f723000ef20 comp rx=0 tx=0).stop 2024-03-18T14:58:36.701 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.699+0000 7f7245952640 1 -- 172.21.15.76:0/3885667604 >> 172.21.15.76:0/3885667604 conn(0x7f7240077530 msgr2=0x7f72400fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:36.701 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.699+0000 7f7245952640 1 -- 172.21.15.76:0/3885667604 shutdown_connections 2024-03-18T14:58:36.701 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:36.700+0000 7f7245952640 1 -- 172.21.15.76:0/3885667604 wait complete. 2024-03-18T14:58:36.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 501158 9.55 MB/sec execute 157 sec latency 70.572 ms 2024-03-18T14:58:37.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 502447 9.52 MB/sec execute 158 sec latency 55.242 ms 2024-03-18T14:58:37.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:37 smithi174 ceph-mon[36787]: pgmap v169: 97 pgs: 97 active+clean; 142 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 210 op/s 2024-03-18T14:58:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:37 smithi077 ceph-mon[35865]: pgmap v169: 97 pgs: 97 active+clean; 142 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 210 op/s 2024-03-18T14:58:38.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:37 smithi076 ceph-mon[32205]: pgmap v169: 97 pgs: 97 active+clean; 142 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 210 op/s 2024-03-18T14:58:38.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 504788 9.55 MB/sec execute 159 sec latency 14.492 ms 2024-03-18T14:58:39.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 506184 9.52 MB/sec execute 160 sec latency 20.552 ms 2024-03-18T14:58:39.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:39 smithi174 ceph-mon[36787]: pgmap v170: 97 pgs: 97 active+clean; 152 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 353 op/s 2024-03-18T14:58:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:39 smithi077 ceph-mon[35865]: pgmap v170: 97 pgs: 97 active+clean; 152 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 353 op/s 2024-03-18T14:58:40.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:39 smithi076 ceph-mon[32205]: pgmap v170: 97 pgs: 97 active+clean; 152 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 353 op/s 2024-03-18T14:58:40.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 508423 9.55 MB/sec execute 161 sec latency 13.751 ms 2024-03-18T14:58:41.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 510235 9.53 MB/sec execute 162 sec latency 18.535 ms 2024-03-18T14:58:41.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:41 smithi174 ceph-mon[36787]: pgmap v171: 97 pgs: 97 active+clean; 152 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 243 op/s 2024-03-18T14:58:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:41 smithi077 ceph-mon[35865]: pgmap v171: 97 pgs: 97 active+clean; 152 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 243 op/s 2024-03-18T14:58:42.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:41 smithi076 ceph-mon[32205]: pgmap v171: 97 pgs: 97 active+clean; 152 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 243 op/s 2024-03-18T14:58:42.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 512115 9.55 MB/sec execute 163 sec latency 13.016 ms 2024-03-18T14:58:43.744 INFO:tasks.workunit.client.0.smithi076.stdout: 1 514177 9.54 MB/sec execute 164 sec latency 22.766 ms 2024-03-18T14:58:43.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:43 smithi174 ceph-mon[36787]: pgmap v172: 97 pgs: 97 active+clean; 159 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 327 op/s 2024-03-18T14:58:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:43 smithi077 ceph-mon[35865]: pgmap v172: 97 pgs: 97 active+clean; 159 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 327 op/s 2024-03-18T14:58:44.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:43 smithi076 ceph-mon[32205]: pgmap v172: 97 pgs: 97 active+clean; 159 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 327 op/s 2024-03-18T14:58:44.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 515710 9.55 MB/sec execute 165 sec latency 21.581 ms 2024-03-18T14:58:45.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 517896 9.56 MB/sec execute 166 sec latency 21.685 ms 2024-03-18T14:58:45.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:45 smithi174 ceph-mon[36787]: pgmap v173: 97 pgs: 97 active+clean; 159 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-03-18T14:58:45.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:45 smithi077 ceph-mon[35865]: pgmap v173: 97 pgs: 97 active+clean; 159 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-03-18T14:58:46.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:45 smithi076 ceph-mon[32205]: pgmap v173: 97 pgs: 97 active+clean; 159 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-03-18T14:58:46.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 519383 9.55 MB/sec execute 167 sec latency 23.736 ms 2024-03-18T14:58:47.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 521581 9.57 MB/sec execute 168 sec latency 16.581 ms 2024-03-18T14:58:47.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:47 smithi076 ceph-mon[32205]: pgmap v174: 97 pgs: 97 active+clean; 159 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-03-18T14:58:47.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:47 smithi174 ceph-mon[36787]: pgmap v174: 97 pgs: 97 active+clean; 159 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-03-18T14:58:47.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:47 smithi077 ceph-mon[35865]: pgmap v174: 97 pgs: 97 active+clean; 159 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-03-18T14:58:48.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 522924 9.55 MB/sec execute 169 sec latency 22.148 ms 2024-03-18T14:58:49.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 525388 9.57 MB/sec execute 170 sec latency 17.954 ms 2024-03-18T14:58:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.800+0000 7fc790705640 1 -- 172.21.15.76:0/3171379901 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7881078b0 msgr2=0x7fc788107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.800+0000 7fc790705640 1 --2- 172.21.15.76:0/3171379901 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7881078b0 0x7fc788107c90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc784009930 tx=0x7fc78402bc30 comp rx=0 tx=0).stop 2024-03-18T14:58:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.800+0000 7fc790705640 1 -- 172.21.15.76:0/3171379901 shutdown_connections 2024-03-18T14:58:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.800+0000 7fc790705640 1 --2- 172.21.15.76:0/3171379901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc788069800 0x7fc78810e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.800+0000 7fc790705640 1 --2- 172.21.15.76:0/3171379901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc788068e60 0x7fc7880692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.800+0000 7fc790705640 1 --2- 172.21.15.76:0/3171379901 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7881078b0 0x7fc788107c90 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.800+0000 7fc790705640 1 -- 172.21.15.76:0/3171379901 >> 172.21.15.76:0/3171379901 conn(0x7fc788077530 msgr2=0x7fc788077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.800+0000 7fc790705640 1 -- 172.21.15.76:0/3171379901 shutdown_connections 2024-03-18T14:58:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.800+0000 7fc790705640 1 -- 172.21.15.76:0/3171379901 wait complete. 2024-03-18T14:58:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.801+0000 7fc790705640 1 Processor -- start 2024-03-18T14:58:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.801+0000 7fc790705640 1 -- start start 2024-03-18T14:58:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.801+0000 7fc790705640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc788068e60 0x7fc788197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.801+0000 7fc790705640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc788069800 0x7fc7881984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.801+0000 7fc790705640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7881078b0 0x7fc78819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.801+0000 7fc790705640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc788111b60 con 0x7fc7881078b0 2024-03-18T14:58:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.801+0000 7fc790705640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc7881119e0 con 0x7fc788068e60 2024-03-18T14:58:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.801+0000 7fc790705640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc788111ce0 con 0x7fc788069800 2024-03-18T14:58:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.801+0000 7fc78eca2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7881078b0 0x7fc78819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.801+0000 7fc78eca2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7881078b0 0x7fc78819c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58298/0 (socket says 172.21.15.76:58298) 2024-03-18T14:58:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.801+0000 7fc78eca2640 1 -- 172.21.15.76:0/4027683524 learned_addr learned my addr 172.21.15.76:0/4027683524 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.801+0000 7fc78e4a1640 1 --2- 172.21.15.76:0/4027683524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc788068e60 0x7fc788197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.801+0000 7fc78dca0640 1 --2- 172.21.15.76:0/4027683524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc788069800 0x7fc7881984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.802+0000 7fc78eca2640 1 -- 172.21.15.76:0/4027683524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc788069800 msgr2=0x7fc7881984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.802+0000 7fc78eca2640 1 --2- 172.21.15.76:0/4027683524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc788069800 0x7fc7881984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.802+0000 7fc78eca2640 1 -- 172.21.15.76:0/4027683524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc788068e60 msgr2=0x7fc788197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.802+0000 7fc78eca2640 1 --2- 172.21.15.76:0/4027683524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc788068e60 0x7fc788197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.802+0000 7fc78eca2640 1 -- 172.21.15.76:0/4027683524 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc78819cf90 con 0x7fc7881078b0 2024-03-18T14:58:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.802+0000 7fc78eca2640 1 --2- 172.21.15.76:0/4027683524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7881078b0 0x7fc78819c860 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc78000ea60 tx=0x7fc78000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.802+0000 7fc77f7fe640 1 -- 172.21.15.76:0/4027683524 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78000ce40 con 0x7fc7881078b0 2024-03-18T14:58:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.802+0000 7fc77f7fe640 1 -- 172.21.15.76:0/4027683524 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc780005020 con 0x7fc7881078b0 2024-03-18T14:58:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.802+0000 7fc790705640 1 -- 172.21.15.76:0/4027683524 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc78819d280 con 0x7fc7881078b0 2024-03-18T14:58:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.802+0000 7fc77f7fe640 1 -- 172.21.15.76:0/4027683524 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc780010690 con 0x7fc7881078b0 2024-03-18T14:58:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.802+0000 7fc790705640 1 -- 172.21.15.76:0/4027683524 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7881a4b10 con 0x7fc7881078b0 2024-03-18T14:58:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.804+0000 7fc77f7fe640 1 -- 172.21.15.76:0/4027683524 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fc780005520 con 0x7fc7881078b0 2024-03-18T14:58:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.805+0000 7fc77f7fe640 1 --2- 172.21.15.76:0/4027683524 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc7740a2a90 0x7fc7740a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.805+0000 7fc78e4a1640 1 --2- 172.21.15.76:0/4027683524 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc7740a2a90 0x7fc7740a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.805+0000 7fc7933c5740 1 Processor -- start 2024-03-18T14:58:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.805+0000 7fc77f7fe640 1 -- 172.21.15.76:0/4027683524 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7fc7800cf2b0 con 0x7fc7881078b0 2024-03-18T14:58:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.805+0000 7fc7933c5740 1 -- start start 2024-03-18T14:58:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.805+0000 7fc7933c5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c0376f130 0x562c0376f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.805+0000 7fc7933c5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376fa50 0x562c037782e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.805+0000 7fc7933c5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c03778820 0x562c0377ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.805+0000 7fc7933c5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562c036d3a40 con 0x562c0376fa50 2024-03-18T14:58:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.805+0000 7fc7933c5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562c0367c920 con 0x562c0376f130 2024-03-18T14:58:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.805+0000 7fc7933c5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562c036cfae0 con 0x562c03778820 2024-03-18T14:58:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.806+0000 7fc78e4a1640 1 --2- 172.21.15.76:0/4027683524 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc7740a2a90 0x7fc7740a4f50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc784005ab0 tx=0x7fc784000bc0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.806+0000 7fc78e4a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c0376f130 0x562c0376f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.806+0000 7fc78dca0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376fa50 0x562c037782e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.806+0000 7fc78eca2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c03778820 0x562c0377ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.806+0000 7fc78e4a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c0376f130 0x562c0376f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:38932/0 (socket says 172.21.15.76:38932) 2024-03-18T14:58:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.806+0000 7fc78eca2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c03778820 0x562c0377ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:35488/0 (socket says 172.21.15.76:35488) 2024-03-18T14:58:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.806+0000 7fc78e4a1640 1 -- 172.21.15.76:0/2614474570 learned_addr learned my addr 172.21.15.76:0/2614474570 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.806+0000 7fc78dca0640 1 -- 172.21.15.76:0/2614474570 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c03778820 msgr2=0x562c0377ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.806+0000 7fc78dca0640 1 --2- 172.21.15.76:0/2614474570 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c03778820 0x562c0377ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.806+0000 7fc78dca0640 1 -- 172.21.15.76:0/2614474570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c0376f130 msgr2=0x562c0376f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.806+0000 7fc78dca0640 1 --2- 172.21.15.76:0/2614474570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c0376f130 0x562c0376f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.806+0000 7fc78dca0640 1 -- 172.21.15.76:0/2614474570 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c03694460 con 0x562c0376fa50 2024-03-18T14:58:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.806+0000 7fc78e4a1640 1 --2- 172.21.15.76:0/2614474570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c0376f130 0x562c0376f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:58:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.806+0000 7fc78dca0640 1 --2- 172.21.15.76:0/2614474570 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376fa50 0x562c037782e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc778009bf0 tx=0x7fc778011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d391eef0d577ce2c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.806+0000 7fc77cff9640 1 -- 172.21.15.76:0/2614474570 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc778004050 con 0x562c0376fa50 2024-03-18T14:58:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.806+0000 7fc77cff9640 1 -- 172.21.15.76:0/2614474570 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc778005500 con 0x562c0376fa50 2024-03-18T14:58:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.806+0000 7fc77cff9640 1 -- 172.21.15.76:0/2614474570 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc778004de0 con 0x562c0376fa50 2024-03-18T14:58:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.807+0000 7fc7933c5740 1 -- 172.21.15.76:0/2614474570 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376fa50 msgr2=0x562c037782e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.807+0000 7fc7933c5740 1 --2- 172.21.15.76:0/2614474570 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376fa50 0x562c037782e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc778009bf0 tx=0x7fc778011810 comp rx=0 tx=0).stop 2024-03-18T14:58:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.807+0000 7fc7933c5740 1 -- 172.21.15.76:0/2614474570 shutdown_connections 2024-03-18T14:58:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.807+0000 7fc7933c5740 1 --2- 172.21.15.76:0/2614474570 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c03778820 0x562c0377ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.807+0000 7fc7933c5740 1 --2- 172.21.15.76:0/2614474570 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376fa50 0x562c037782e0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.807+0000 7fc7933c5740 1 --2- 172.21.15.76:0/2614474570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c0376f130 0x562c0376f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.807+0000 7fc7933c5740 1 -- 172.21.15.76:0/2614474570 >> 172.21.15.76:0/2614474570 conn(0x562c03769a50 msgr2=0x562c03769e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.807+0000 7fc7933c5740 1 -- 172.21.15.76:0/2614474570 shutdown_connections 2024-03-18T14:58:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.807+0000 7fc7933c5740 1 -- 172.21.15.76:0/2614474570 wait complete. 2024-03-18T14:58:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.807+0000 7fc7933c5740 1 -- start start 2024-03-18T14:58:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.807+0000 7fc7933c5740 1 Processor -- start 2024-03-18T14:58:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc7933c5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376f130 0x562c03775830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc7933c5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c0376fa50 0x562c03775d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc7933c5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c03778820 0x562c037762b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc7933c5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562c0377b450 con 0x562c0376f130 2024-03-18T14:58:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc7933c5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562c0368fd70 con 0x562c0376fa50 2024-03-18T14:58:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc7933c5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562c0377b5d0 con 0x562c03778820 2024-03-18T14:58:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc7933c5740 10 client.0 _ll_register_callbacks cb 0x562c033e7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:58:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc78e4a1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376f130 0x562c03775830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc78dca0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c0376fa50 0x562c03775d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc78e4a1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376f130 0x562c03775830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58320/0 (socket says 172.21.15.76:58320) 2024-03-18T14:58:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc78e4a1640 1 -- 172.21.15.76:0/459343267 learned_addr learned my addr 172.21.15.76:0/459343267 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc78eca2640 1 --2- 172.21.15.76:0/459343267 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c03778820 0x562c037762b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc78e4a1640 1 -- 172.21.15.76:0/459343267 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c03778820 msgr2=0x562c037762b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc78e4a1640 1 --2- 172.21.15.76:0/459343267 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c03778820 0x562c037762b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc78e4a1640 1 -- 172.21.15.76:0/459343267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c0376fa50 msgr2=0x562c03775d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc78e4a1640 1 --2- 172.21.15.76:0/459343267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c0376fa50 0x562c03775d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc78e4a1640 1 -- 172.21.15.76:0/459343267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c033e8210 con 0x562c0376f130 2024-03-18T14:58:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc78e4a1640 1 --2- 172.21.15.76:0/459343267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376f130 0x562c03775830 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc78402c170 tx=0x7fc78402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78ee3804dbc653 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc771ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:58:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc771ffb640 1 -- 172.21.15.76:0/459343267 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc784050070 con 0x562c0376f130 2024-03-18T14:58:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc771ffb640 1 -- 172.21.15.76:0/459343267 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7fc7840046c0 con 0x562c0376f130 2024-03-18T14:58:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.808+0000 7fc7933c5740 1 -- 172.21.15.76:0/459343267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc778005500 con 0x562c0376f130 2024-03-18T14:58:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.809+0000 7fc771ffb640 1 -- 172.21.15.76:0/459343267 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc7840048c0 con 0x562c0376f130 2024-03-18T14:58:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.809+0000 7fc7933c5740 10 client.16293 fetch_fsmap learned FSMap version 40 2024-03-18T14:58:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.809+0000 7fc7933c5740 1 -- 172.21.15.76:0/459343267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x562c034b4620 con 0x562c0376f130 2024-03-18T14:58:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.809+0000 7fc771ffb640 1 -- 172.21.15.76:0/459343267 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc778005500 con 0x562c0376f130 2024-03-18T14:58:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.809+0000 7fc771ffb640 1 -- 172.21.15.76:0/459343267 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 40) ==== 8342+0+0 (secure 0 0 0) 0x7fc7840035f0 con 0x562c0376f130 2024-03-18T14:58:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.809+0000 7fc7933c5740 10 client.16293 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T14:58:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.809+0000 7fc7933c5740 10 client.16293 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:58:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.809+0000 7fc7933c5740 20 client.16293 populate_metadata read hostname 'smithi076' 2024-03-18T14:58:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.809+0000 7fc7933c5740 1 --2- 172.21.15.76:0/459343267 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x562c03788980 0x562c0378ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.809+0000 7fc7933c5740 4 client.16293 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:58:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.809+0000 7fc7933c5740 1 -- 172.21.15.76:0/459343267 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c036e0f80 con 0x562c03788980 2024-03-18T14:58:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.809+0000 7fc78dca0640 1 --2- 172.21.15.76:0/459343267 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x562c03788980 0x562c0378ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.809+0000 7fc78dca0640 1 --2- 172.21.15.76:0/459343267 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x562c03788980 0x562c0378ad70 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1e05e35db13f352 server_cookie=60d1e3c8e43d1d03 in_seq=0 out_seq=0 2024-03-18T14:58:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.809+0000 7fc771ffb640 10 client.16293 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:58:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.810+0000 7fc771ffb640 1 -- 172.21.15.76:0/459343267 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562c036e0f80 con 0x562c03788980 2024-03-18T14:58:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.810+0000 7fc771ffb640 10 client.16293 handle_command_reply: tid=0 2024-03-18T14:58:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.811+0000 7fc7933c5740 1 client.16293 shutdown 2024-03-18T14:58:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.811+0000 7fc7933c5740 1 -- 172.21.15.76:0/459343267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376f130 msgr2=0x562c03775830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.811+0000 7fc7933c5740 1 --2- 172.21.15.76:0/459343267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376f130 0x562c03775830 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc78402c170 tx=0x7fc78402bc30 comp rx=0 tx=0).stop 2024-03-18T14:58:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.811+0000 7fc7933c5740 1 -- 172.21.15.76:0/459343267 shutdown_connections 2024-03-18T14:58:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.811+0000 7fc7933c5740 1 --2- 172.21.15.76:0/459343267 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c03778820 0x562c037762b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.811+0000 7fc7933c5740 1 --2- 172.21.15.76:0/459343267 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x562c03788980 0x562c0378ad70 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.811+0000 7fc7933c5740 1 --2- 172.21.15.76:0/459343267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c0376fa50 0x562c03775d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.811+0000 7fc7933c5740 1 --2- 172.21.15.76:0/459343267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376f130 0x562c03775830 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.811+0000 7fc771ffb640 0 client.16293 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:58:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.811+0000 7fc7933c5740 1 -- 172.21.15.76:0/459343267 >> 172.21.15.76:0/459343267 conn(0x562c03769a50 msgr2=0x562c0376b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.811+0000 7fc7933c5740 1 -- 172.21.15.76:0/459343267 shutdown_connections 2024-03-18T14:58:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.811+0000 7fc7933c5740 1 -- 172.21.15.76:0/459343267 wait complete. 2024-03-18T14:58:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.811+0000 7fc7933c5740 20 client.16293 trim_cache size 0 max 16384 2024-03-18T14:58:49.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.826+0000 7fc7933c5740 1 Processor -- start 2024-03-18T14:58:49.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.826+0000 7fc7933c5740 1 -- start start 2024-03-18T14:58:49.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc7933c5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0376f130 0x562c03777f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc7933c5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376fa50 0x562c0376ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc7933c5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c03778820 0x562c037841f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc7933c5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc7881afce0 con 0x562c0376fa50 2024-03-18T14:58:49.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc7933c5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc7881ad580 con 0x562c03778820 2024-03-18T14:58:49.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc7933c5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7881ab3f0 con 0x562c0376f130 2024-03-18T14:58:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc78dca0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376fa50 0x562c0376ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc78eca2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c03778820 0x562c037841f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc78dca0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376fa50 0x562c0376ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58324/0 (socket says 172.21.15.76:58324) 2024-03-18T14:58:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc78eca2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c03778820 0x562c037841f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:38946/0 (socket says 172.21.15.76:38946) 2024-03-18T14:58:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc78dca0640 1 -- 172.21.15.76:0/4231438118 learned_addr learned my addr 172.21.15.76:0/4231438118 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc78e4a1640 1 --2- 172.21.15.76:0/4231438118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0376f130 0x562c03777f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc78dca0640 1 -- 172.21.15.76:0/4231438118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0376f130 msgr2=0x562c03777f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc78dca0640 1 --2- 172.21.15.76:0/4231438118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0376f130 0x562c03777f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc78dca0640 1 -- 172.21.15.76:0/4231438118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c03778820 msgr2=0x562c037841f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc78dca0640 1 --2- 172.21.15.76:0/4231438118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c03778820 0x562c037841f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc78dca0640 1 -- 172.21.15.76:0/4231438118 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c03774820 con 0x562c0376fa50 2024-03-18T14:58:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc78eca2640 1 --2- 172.21.15.76:0/4231438118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c03778820 0x562c037841f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:58:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc78dca0640 1 --2- 172.21.15.76:0/4231438118 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376fa50 0x562c0376ffb0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc778009bf0 tx=0x7fc778009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d59e533dc4732f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.827+0000 7fc773fff640 1 -- 172.21.15.76:0/4231438118 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc778033070 con 0x562c0376fa50 2024-03-18T14:58:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.828+0000 7fc773fff640 1 -- 172.21.15.76:0/4231438118 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7780048f0 con 0x562c0376fa50 2024-03-18T14:58:49.830 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.828+0000 7fc773fff640 1 -- 172.21.15.76:0/4231438118 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7780035f0 con 0x562c0376fa50 2024-03-18T14:58:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.828+0000 7fc7933c5740 1 -- 172.21.15.76:0/4231438118 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376fa50 msgr2=0x562c0376ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.828+0000 7fc7933c5740 1 --2- 172.21.15.76:0/4231438118 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376fa50 0x562c0376ffb0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc778009bf0 tx=0x7fc778009680 comp rx=0 tx=0).stop 2024-03-18T14:58:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.828+0000 7fc7933c5740 1 -- 172.21.15.76:0/4231438118 shutdown_connections 2024-03-18T14:58:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.828+0000 7fc7933c5740 1 --2- 172.21.15.76:0/4231438118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c03778820 0x562c037841f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.828+0000 7fc7933c5740 1 --2- 172.21.15.76:0/4231438118 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376fa50 0x562c0376ffb0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.828+0000 7fc7933c5740 1 --2- 172.21.15.76:0/4231438118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0376f130 0x562c03777f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.828+0000 7fc7933c5740 1 -- 172.21.15.76:0/4231438118 >> 172.21.15.76:0/4231438118 conn(0x562c03769a50 msgr2=0x562c03768bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.828+0000 7fc7933c5740 1 -- 172.21.15.76:0/4231438118 shutdown_connections 2024-03-18T14:58:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.828+0000 7fc7933c5740 1 -- 172.21.15.76:0/4231438118 wait complete. 2024-03-18T14:58:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc7933c5740 1 -- start start 2024-03-18T14:58:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc7933c5740 1 Processor -- start 2024-03-18T14:58:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc7933c5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376f130 0x562c03780820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc7933c5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c0376fa50 0x562c0377d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc7933c5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c03778820 0x562c0377dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc7933c5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc7881aba40 con 0x562c0376f130 2024-03-18T14:58:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc7933c5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc7881aceb0 con 0x562c0376fa50 2024-03-18T14:58:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc7933c5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7881ac110 con 0x562c03778820 2024-03-18T14:58:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc78e4a1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376f130 0x562c03780820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc7933c5740 10 client.0 _ll_register_callbacks cb 0x7fc7800093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:58:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc78e4a1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376f130 0x562c03780820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58332/0 (socket says 172.21.15.76:58332) 2024-03-18T14:58:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc78e4a1640 1 -- 172.21.15.76:0/976874104 learned_addr learned my addr 172.21.15.76:0/976874104 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:58:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc78eca2640 1 --2- 172.21.15.76:0/976874104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c03778820 0x562c0377dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc78dca0640 1 --2- 172.21.15.76:0/976874104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c0376fa50 0x562c0377d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc78e4a1640 1 -- 172.21.15.76:0/976874104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c03778820 msgr2=0x562c0377dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc78e4a1640 1 --2- 172.21.15.76:0/976874104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c03778820 0x562c0377dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc78e4a1640 1 -- 172.21.15.76:0/976874104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c0376fa50 msgr2=0x562c0377d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:58:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc78e4a1640 1 --2- 172.21.15.76:0/976874104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c0376fa50 0x562c0377d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc78e4a1640 1 -- 172.21.15.76:0/976874104 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c0377e270 con 0x562c0376f130 2024-03-18T14:58:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc78e4a1640 1 --2- 172.21.15.76:0/976874104 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376f130 0x562c03780820 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc784006010 tx=0x7fc784004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e565e17171daa7a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:58:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.829+0000 7fc771ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:58:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.830+0000 7fc771ffb640 1 -- 172.21.15.76:0/976874104 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc784050040 con 0x562c0376f130 2024-03-18T14:58:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.830+0000 7fc771ffb640 1 -- 172.21.15.76:0/976874104 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7fc784003e90 con 0x562c0376f130 2024-03-18T14:58:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.830+0000 7fc7933c5740 1 -- 172.21.15.76:0/976874104 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7780048f0 con 0x562c0376f130 2024-03-18T14:58:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.830+0000 7fc771ffb640 1 -- 172.21.15.76:0/976874104 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc784004f10 con 0x562c0376f130 2024-03-18T14:58:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.830+0000 7fc7933c5740 10 client.16299 fetch_fsmap learned FSMap version 40 2024-03-18T14:58:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.830+0000 7fc7933c5740 1 -- 172.21.15.76:0/976874104 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x562c037ef950 con 0x562c0376f130 2024-03-18T14:58:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.830+0000 7fc771ffb640 1 -- 172.21.15.76:0/976874104 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7780048f0 con 0x562c0376f130 2024-03-18T14:58:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.830+0000 7fc771ffb640 1 -- 172.21.15.76:0/976874104 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 40) ==== 8342+0+0 (secure 0 0 0) 0x7fc78404c5f0 con 0x562c0376f130 2024-03-18T14:58:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.830+0000 7fc7933c5740 10 client.16299 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T14:58:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.830+0000 7fc7933c5740 10 client.16299 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:58:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.830+0000 7fc7933c5740 20 client.16299 populate_metadata read hostname 'smithi076' 2024-03-18T14:58:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.830+0000 7fc7933c5740 1 --2- 172.21.15.76:0/976874104 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x562c03788980 0x562c037f20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:58:49.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.830+0000 7fc7933c5740 4 client.16299 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:58:49.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.830+0000 7fc7933c5740 1 -- 172.21.15.76:0/976874104 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562c036db2d0 con 0x562c03788980 2024-03-18T14:58:49.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.830+0000 7fc78dca0640 1 --2- 172.21.15.76:0/976874104 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x562c03788980 0x562c037f20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:58:49.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.831+0000 7fc771ffb640 10 client.16299 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:58:49.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.831+0000 7fc78dca0640 1 --2- 172.21.15.76:0/976874104 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x562c03788980 0x562c037f20b0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a11206e2c8e02f6 server_cookie=963bd4b90b288e1f in_seq=0 out_seq=0 2024-03-18T14:58:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.911+0000 7fc771ffb640 1 -- 172.21.15.76:0/976874104 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562c036db2d0 con 0x562c03788980 2024-03-18T14:58:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.911+0000 7fc771ffb640 10 client.16299 handle_command_reply: tid=0 2024-03-18T14:58:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.911+0000 7fc7933c5740 1 client.16299 shutdown 2024-03-18T14:58:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.912+0000 7fc7933c5740 1 -- 172.21.15.76:0/976874104 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376f130 msgr2=0x562c03780820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.912+0000 7fc7933c5740 1 --2- 172.21.15.76:0/976874104 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376f130 0x562c03780820 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc784006010 tx=0x7fc784004d10 comp rx=0 tx=0).stop 2024-03-18T14:58:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.912+0000 7fc7933c5740 1 -- 172.21.15.76:0/976874104 shutdown_connections 2024-03-18T14:58:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.912+0000 7fc7933c5740 1 --2- 172.21.15.76:0/976874104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c03778820 0x562c0377dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.912+0000 7fc7933c5740 1 --2- 172.21.15.76:0/976874104 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x562c03788980 0x562c037f20b0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.912+0000 7fc7933c5740 1 --2- 172.21.15.76:0/976874104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c0376fa50 0x562c0377d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.912+0000 7fc771ffb640 0 client.16299 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:58:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.912+0000 7fc7933c5740 1 --2- 172.21.15.76:0/976874104 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562c0376f130 0x562c03780820 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.912+0000 7fc7933c5740 1 -- 172.21.15.76:0/976874104 >> 172.21.15.76:0/976874104 conn(0x562c03769a50 msgr2=0x562c037757f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.912+0000 7fc7933c5740 1 -- 172.21.15.76:0/976874104 shutdown_connections 2024-03-18T14:58:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.912+0000 7fc7933c5740 1 -- 172.21.15.76:0/976874104 wait complete. 2024-03-18T14:58:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.912+0000 7fc7933c5740 20 client.16299 trim_cache size 0 max 16384 2024-03-18T14:58:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:58:49.915 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:58:49.915 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:58:49.915 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:58:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.912+0000 7fc790705640 1 -- 172.21.15.76:0/4027683524 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc7740a2a90 msgr2=0x7fc7740a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.912+0000 7fc790705640 1 --2- 172.21.15.76:0/4027683524 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc7740a2a90 0x7fc7740a4f50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc784005ab0 tx=0x7fc784000bc0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.912+0000 7fc790705640 1 -- 172.21.15.76:0/4027683524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7881078b0 msgr2=0x7fc78819c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:58:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.912+0000 7fc790705640 1 --2- 172.21.15.76:0/4027683524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7881078b0 0x7fc78819c860 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc78000ea60 tx=0x7fc78000ef20 comp rx=0 tx=0).stop 2024-03-18T14:58:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.913+0000 7fc790705640 1 -- 172.21.15.76:0/4027683524 shutdown_connections 2024-03-18T14:58:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.913+0000 7fc790705640 1 --2- 172.21.15.76:0/4027683524 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc7740a2a90 0x7fc7740a4f50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.913+0000 7fc790705640 1 --2- 172.21.15.76:0/4027683524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc788069800 0x7fc7881984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.913+0000 7fc790705640 1 --2- 172.21.15.76:0/4027683524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7881078b0 0x7fc78819c860 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.913+0000 7fc790705640 1 --2- 172.21.15.76:0/4027683524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc788068e60 0x7fc788197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:58:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.913+0000 7fc790705640 1 -- 172.21.15.76:0/4027683524 >> 172.21.15.76:0/4027683524 conn(0x7fc788077530 msgr2=0x7fc7880fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:58:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.913+0000 7fc790705640 1 -- 172.21.15.76:0/4027683524 shutdown_connections 2024-03-18T14:58:49.916 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:58:49.913+0000 7fc790705640 1 -- 172.21.15.76:0/4027683524 wait complete. 2024-03-18T14:58:49.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:49 smithi174 ceph-mon[36787]: pgmap v175: 97 pgs: 97 active+clean; 166 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 307 op/s 2024-03-18T14:58:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:49 smithi077 ceph-mon[35865]: pgmap v175: 97 pgs: 97 active+clean; 166 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 307 op/s 2024-03-18T14:58:50.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:49 smithi076 ceph-mon[32205]: pgmap v175: 97 pgs: 97 active+clean; 166 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 307 op/s 2024-03-18T14:58:50.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 526545 9.55 MB/sec execute 171 sec latency 75.005 ms 2024-03-18T14:58:51.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 528936 9.57 MB/sec execute 172 sec latency 39.761 ms 2024-03-18T14:58:51.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:51 smithi174 ceph-mon[36787]: pgmap v176: 97 pgs: 97 active+clean; 166 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-03-18T14:58:51.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:51 smithi077 ceph-mon[35865]: pgmap v176: 97 pgs: 97 active+clean; 166 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-03-18T14:58:52.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:51 smithi076 ceph-mon[32205]: pgmap v176: 97 pgs: 97 active+clean; 166 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-03-18T14:58:52.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 530180 9.56 MB/sec execute 173 sec latency 31.541 ms 2024-03-18T14:58:53.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 532829 9.60 MB/sec execute 174 sec latency 22.108 ms 2024-03-18T14:58:53.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:53 smithi077 ceph-mon[35865]: pgmap v177: 97 pgs: 97 active+clean; 161 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 318 op/s 2024-03-18T14:58:54.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:53 smithi076 ceph-mon[32205]: pgmap v177: 97 pgs: 97 active+clean; 161 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 318 op/s 2024-03-18T14:58:54.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:53 smithi174 ceph-mon[36787]: pgmap v177: 97 pgs: 97 active+clean; 161 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 318 op/s 2024-03-18T14:58:54.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 534042 9.56 MB/sec execute 175 sec latency 20.623 ms 2024-03-18T14:58:55.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 536515 9.60 MB/sec execute 176 sec latency 26.143 ms 2024-03-18T14:58:55.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:55 smithi174 ceph-mon[36787]: pgmap v178: 97 pgs: 97 active+clean; 161 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 234 op/s 2024-03-18T14:58:55.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:55 smithi077 ceph-mon[35865]: pgmap v178: 97 pgs: 97 active+clean; 161 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 234 op/s 2024-03-18T14:58:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:55 smithi076 ceph-mon[32205]: pgmap v178: 97 pgs: 97 active+clean; 161 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 234 op/s 2024-03-18T14:58:56.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 537840 9.58 MB/sec execute 177 sec latency 22.776 ms 2024-03-18T14:58:57.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 540048 9.60 MB/sec execute 178 sec latency 53.246 ms 2024-03-18T14:58:57.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:57 smithi174 ceph-mon[36787]: pgmap v179: 97 pgs: 97 active+clean; 161 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 234 op/s 2024-03-18T14:58:57.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:57 smithi077 ceph-mon[35865]: pgmap v179: 97 pgs: 97 active+clean; 161 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 234 op/s 2024-03-18T14:58:58.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:57 smithi076 ceph-mon[32205]: pgmap v179: 97 pgs: 97 active+clean; 161 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 234 op/s 2024-03-18T14:58:58.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 541389 9.58 MB/sec execute 179 sec latency 24.084 ms 2024-03-18T14:58:59.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 543766 9.60 MB/sec execute 180 sec latency 16.663 ms 2024-03-18T14:58:59.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:58:59 smithi174 ceph-mon[36787]: pgmap v180: 97 pgs: 97 active+clean; 166 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 342 op/s 2024-03-18T14:58:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:58:59 smithi077 ceph-mon[35865]: pgmap v180: 97 pgs: 97 active+clean; 166 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 342 op/s 2024-03-18T14:59:00.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:58:59 smithi076 ceph-mon[32205]: pgmap v180: 97 pgs: 97 active+clean; 166 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 342 op/s 2024-03-18T14:59:00.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 544494 9.56 MB/sec execute 181 sec latency 347.666 ms 2024-03-18T14:59:01.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 544494 9.50 MB/sec execute 182 sec latency 1347.776 ms 2024-03-18T14:59:01.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:01 smithi174 ceph-mon[36787]: pgmap v181: 97 pgs: 97 active+clean; 166 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 262 op/s 2024-03-18T14:59:01.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:01 smithi077 ceph-mon[35865]: pgmap v181: 97 pgs: 97 active+clean; 166 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 262 op/s 2024-03-18T14:59:02.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:01 smithi076 ceph-mon[32205]: pgmap v181: 97 pgs: 97 active+clean; 166 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 262 op/s 2024-03-18T14:59:02.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 545381 9.48 MB/sec execute 183 sec latency 1553.509 ms 2024-03-18T14:59:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.003+0000 7f43ea0c6640 1 -- 172.21.15.76:0/1557716830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43e4101790 msgr2=0x7f43e4101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.003+0000 7f43ea0c6640 1 --2- 172.21.15.76:0/1557716830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43e4101790 0x7f43e4101c10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009930 tx=0x7f43d402bc30 comp rx=0 tx=0).stop 2024-03-18T14:59:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.003+0000 7f43ea0c6640 1 -- 172.21.15.76:0/1557716830 shutdown_connections 2024-03-18T14:59:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.003+0000 7f43ea0c6640 1 --2- 172.21.15.76:0/1557716830 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43e4102150 0x7f43e410e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.003+0000 7f43ea0c6640 1 --2- 172.21.15.76:0/1557716830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43e4101790 0x7f43e4101c10 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.003+0000 7f43ea0c6640 1 --2- 172.21.15.76:0/1557716830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43e41077a0 0x7f43e4107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.003+0000 7f43ea0c6640 1 -- 172.21.15.76:0/1557716830 >> 172.21.15.76:0/1557716830 conn(0x7f43e40fd2a0 msgr2=0x7f43e40ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.003+0000 7f43ea0c6640 1 -- 172.21.15.76:0/1557716830 shutdown_connections 2024-03-18T14:59:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.003+0000 7f43ea0c6640 1 -- 172.21.15.76:0/1557716830 wait complete. 2024-03-18T14:59:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.003+0000 7f43ea0c6640 1 Processor -- start 2024-03-18T14:59:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.003+0000 7f43ea0c6640 1 -- start start 2024-03-18T14:59:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43ea0c6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43e4101790 0x7f43e4073ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43ea0c6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43e4102150 0x7f43e4074020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43ea0c6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43e41077a0 0x7f43e4074560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43ea0c6640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f43e4111bb0 con 0x7f43e41077a0 2024-03-18T14:59:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43ea0c6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f43e4111a30 con 0x7f43e4102150 2024-03-18T14:59:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43ea0c6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f43e4111d30 con 0x7f43e4101790 2024-03-18T14:59:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43e3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43e41077a0 0x7f43e4074560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43e37fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43e4101790 0x7f43e4073ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43e2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43e4102150 0x7f43e4074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43e2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43e4102150 0x7f43e4074020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:46942/0 (socket says 172.21.15.76:46942) 2024-03-18T14:59:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43e3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43e41077a0 0x7f43e4074560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57218/0 (socket says 172.21.15.76:57218) 2024-03-18T14:59:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43e37fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43e4101790 0x7f43e4073ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:53300/0 (socket says 172.21.15.76:53300) 2024-03-18T14:59:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43e3fff640 1 -- 172.21.15.76:0/2854224886 learned_addr learned my addr 172.21.15.76:0/2854224886 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43e3fff640 1 -- 172.21.15.76:0/2854224886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43e4101790 msgr2=0x7f43e4073ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43e3fff640 1 --2- 172.21.15.76:0/2854224886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43e4101790 0x7f43e4073ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43e3fff640 1 -- 172.21.15.76:0/2854224886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43e4102150 msgr2=0x7f43e4074020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43e3fff640 1 --2- 172.21.15.76:0/2854224886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43e4102150 0x7f43e4074020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43e3fff640 1 -- 172.21.15.76:0/2854224886 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e41a6510 con 0x7f43e41077a0 2024-03-18T14:59:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43e37fe640 1 --2- 172.21.15.76:0/2854224886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43e4101790 0x7f43e4073ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:59:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43e2ffd640 1 --2- 172.21.15.76:0/2854224886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43e4102150 0x7f43e4074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:59:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43e3fff640 1 --2- 172.21.15.76:0/2854224886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43e41077a0 0x7f43e4074560 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f43dc00eb10 tx=0x7f43dc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.004+0000 7f43e0ff9640 1 -- 172.21.15.76:0/2854224886 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43dc004b20 con 0x7f43e41077a0 2024-03-18T14:59:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.005+0000 7f43e0ff9640 1 -- 172.21.15.76:0/2854224886 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43dc004cc0 con 0x7f43e41077a0 2024-03-18T14:59:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.005+0000 7f43e0ff9640 1 -- 172.21.15.76:0/2854224886 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43dc010760 con 0x7f43e41077a0 2024-03-18T14:59:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.005+0000 7f43ea0c6640 1 -- 172.21.15.76:0/2854224886 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43e41a6800 con 0x7f43e41077a0 2024-03-18T14:59:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.005+0000 7f43ea0c6640 1 -- 172.21.15.76:0/2854224886 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43e41a6d70 con 0x7f43e41077a0 2024-03-18T14:59:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.007+0000 7f43e0ff9640 1 -- 172.21.15.76:0/2854224886 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f43dc005020 con 0x7f43e41077a0 2024-03-18T14:59:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.007+0000 7f43ecd8f740 1 Processor -- start 2024-03-18T14:59:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.007+0000 7f43ecd8f740 1 -- start start 2024-03-18T14:59:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.007+0000 7f43ecd8f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c7343130 0x5606c7343510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.007+0000 7f43ecd8f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343a50 0x5606c734c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.007+0000 7f43ecd8f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c734c820 0x5606c734ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.007+0000 7f43ecd8f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5606c72a7a40 con 0x5606c7343a50 2024-03-18T14:59:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.007+0000 7f43ecd8f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5606c7250920 con 0x5606c734c820 2024-03-18T14:59:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.007+0000 7f43ecd8f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5606c72a3ae0 con 0x5606c7343130 2024-03-18T14:59:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.007+0000 7f43e0ff9640 1 --2- 172.21.15.76:0/2854224886 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f43cc0a2a90 0x7f43cc0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.007+0000 7f43e37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343a50 0x5606c734c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.007+0000 7f43e37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343a50 0x5606c734c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57232/0 (socket says 172.21.15.76:57232) 2024-03-18T14:59:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.007+0000 7f43e2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c734c820 0x5606c734ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.007+0000 7f43e37fe640 1 -- 172.21.15.76:0/3763045283 learned_addr learned my addr 172.21.15.76:0/3763045283 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.007+0000 7f43e2ffd640 1 --2- 172.21.15.76:0/3763045283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c7343130 0x5606c7343510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.007+0000 7f43e3fff640 1 --2- 172.21.15.76:0/2854224886 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f43cc0a2a90 0x7f43cc0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.008+0000 7f43e0ff9640 1 -- 172.21.15.76:0/2854224886 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7f43dc0cf470 con 0x7f43e41077a0 2024-03-18T14:59:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.008+0000 7f43e2ffd640 1 -- 172.21.15.76:0/3763045283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c7343130 msgr2=0x5606c7343510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.008+0000 7f43e2ffd640 1 --2- 172.21.15.76:0/3763045283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c7343130 0x5606c7343510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.008+0000 7f43e2ffd640 1 -- 172.21.15.76:0/3763045283 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343a50 msgr2=0x5606c734c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.008+0000 7f43e2ffd640 1 --2- 172.21.15.76:0/3763045283 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343a50 0x5606c734c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.008+0000 7f43e2ffd640 1 -- 172.21.15.76:0/3763045283 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606c7268460 con 0x5606c734c820 2024-03-18T14:59:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.008+0000 7f43e37fe640 1 --2- 172.21.15.76:0/3763045283 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343a50 0x5606c734c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:59:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.008+0000 7f43e2ffd640 1 --2- 172.21.15.76:0/3763045283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c734c820 0x5606c734ec10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009230 tx=0x7f43d4005b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78d1137ce96a5d4e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.008+0000 7f43e3fff640 1 --2- 172.21.15.76:0/2854224886 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f43cc0a2a90 0x7f43cc0a4f50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f43dc09bab0 tx=0x7f43dc00cee0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.008+0000 7f43c9ffb640 1 -- 172.21.15.76:0/3763045283 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d402ca80 con 0x5606c734c820 2024-03-18T14:59:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.008+0000 7f43c9ffb640 1 -- 172.21.15.76:0/3763045283 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43d4004ca0 con 0x5606c734c820 2024-03-18T14:59:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.008+0000 7f43c9ffb640 1 -- 172.21.15.76:0/3763045283 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d4005570 con 0x5606c734c820 2024-03-18T14:59:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.009+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3763045283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c734c820 msgr2=0x5606c734ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.009+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3763045283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c734c820 0x5606c734ec10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009230 tx=0x7f43d4005b70 comp rx=0 tx=0).stop 2024-03-18T14:59:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.009+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3763045283 shutdown_connections 2024-03-18T14:59:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.009+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3763045283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c734c820 0x5606c734ec10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.009+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3763045283 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343a50 0x5606c734c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.009+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3763045283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c7343130 0x5606c7343510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.009+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3763045283 >> 172.21.15.76:0/3763045283 conn(0x5606c733da50 msgr2=0x5606c733de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.009+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3763045283 shutdown_connections 2024-03-18T14:59:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.009+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3763045283 wait complete. 2024-03-18T14:59:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.009+0000 7f43ecd8f740 1 -- start start 2024-03-18T14:59:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.009+0000 7f43ecd8f740 1 Processor -- start 2024-03-18T14:59:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43ecd8f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c7343130 0x5606c7349870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43ecd8f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c7343a50 0x5606c7349db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43ecd8f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c734c820 0x5606c734a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43ecd8f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5606c734f3f0 con 0x5606c734c820 2024-03-18T14:59:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43ecd8f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5606c7263d70 con 0x5606c7343130 2024-03-18T14:59:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43ecd8f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5606c734f570 con 0x5606c7343a50 2024-03-18T14:59:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43ecd8f740 10 client.0 _ll_register_callbacks cb 0x5606c6fbba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:59:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43e2ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c734c820 0x5606c734a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43e2ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c734c820 0x5606c734a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57234/0 (socket says 172.21.15.76:57234) 2024-03-18T14:59:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43e2ffd640 1 -- 172.21.15.76:0/3067575436 learned_addr learned my addr 172.21.15.76:0/3067575436 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43e37fe640 1 --2- 172.21.15.76:0/3067575436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c7343a50 0x5606c7349db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43e2ffd640 1 --2- 172.21.15.76:0/3067575436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c7343130 0x5606c7349870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43e2ffd640 1 -- 172.21.15.76:0/3067575436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c7343a50 msgr2=0x5606c7349db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43e2ffd640 1 --2- 172.21.15.76:0/3067575436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c7343a50 0x5606c7349db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43e2ffd640 1 -- 172.21.15.76:0/3067575436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c7343130 msgr2=0x5606c7349870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43e2ffd640 1 --2- 172.21.15.76:0/3067575436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c7343130 0x5606c7349870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43e2ffd640 1 -- 172.21.15.76:0/3067575436 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606c6fbc210 con 0x5606c734c820 2024-03-18T14:59:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43e2ffd640 1 --2- 172.21.15.76:0/3067575436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c734c820 0x5606c734a2f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f43d402c170 tx=0x7f43d4043040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a65057c730400de server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:59:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43c2ffd640 1 -- 172.21.15.76:0/3067575436 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d4050070 con 0x5606c734c820 2024-03-18T14:59:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43c2ffd640 1 -- 172.21.15.76:0/3067575436 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7f43d402ca80 con 0x5606c734c820 2024-03-18T14:59:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3067575436 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43d4004ca0 con 0x5606c734c820 2024-03-18T14:59:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43c2ffd640 1 -- 172.21.15.76:0/3067575436 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f43d402cc80 con 0x5606c734c820 2024-03-18T14:59:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43ecd8f740 10 client.16311 fetch_fsmap learned FSMap version 40 2024-03-18T14:59:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.010+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3067575436 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5606c7088620 con 0x5606c734c820 2024-03-18T14:59:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.011+0000 7f43c2ffd640 1 -- 172.21.15.76:0/3067575436 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43d4004ca0 con 0x5606c734c820 2024-03-18T14:59:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.011+0000 7f43c2ffd640 1 -- 172.21.15.76:0/3067575436 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 40) ==== 8342+0+0 (secure 0 0 0) 0x7f43d4003bc0 con 0x5606c734c820 2024-03-18T14:59:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.011+0000 7f43ecd8f740 10 client.16311 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T14:59:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.011+0000 7f43ecd8f740 10 client.16311 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:59:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.011+0000 7f43ecd8f740 20 client.16311 populate_metadata read hostname 'smithi076' 2024-03-18T14:59:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.011+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3067575436 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5606c735c7f0 0x5606c735ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.011+0000 7f43ecd8f740 4 client.16311 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:59:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.011+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3067575436 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606c72b4f80 con 0x5606c735c7f0 2024-03-18T14:59:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.011+0000 7f43e37fe640 1 --2- 172.21.15.76:0/3067575436 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5606c735c7f0 0x5606c735ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.011+0000 7f43e37fe640 1 --2- 172.21.15.76:0/3067575436 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5606c735c7f0 0x5606c735ebe0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16274e2484599c6d server_cookie=ffe9edb82e47f979 in_seq=0 out_seq=0 2024-03-18T14:59:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.011+0000 7f43c2ffd640 10 client.16311 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.013+0000 7f43c2ffd640 1 -- 172.21.15.76:0/3067575436 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5606c72b4f80 con 0x5606c735c7f0 2024-03-18T14:59:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.013+0000 7f43c2ffd640 10 client.16311 handle_command_reply: tid=0 2024-03-18T14:59:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.013+0000 7f43ecd8f740 1 client.16311 shutdown 2024-03-18T14:59:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.014+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3067575436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c734c820 msgr2=0x5606c734a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.014+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3067575436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c734c820 0x5606c734a2f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f43d402c170 tx=0x7f43d4043040 comp rx=0 tx=0).stop 2024-03-18T14:59:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.014+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3067575436 shutdown_connections 2024-03-18T14:59:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.014+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3067575436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c7343a50 0x5606c7349db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.014+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3067575436 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5606c735c7f0 0x5606c735ebe0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.014+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3067575436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c734c820 0x5606c734a2f0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.021 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.014+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3067575436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c7343130 0x5606c7349870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.021 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.014+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3067575436 >> 172.21.15.76:0/3067575436 conn(0x5606c733da50 msgr2=0x5606c733f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:03.021 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.014+0000 7f43c2ffd640 0 client.16311 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:03.021 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.014+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3067575436 shutdown_connections 2024-03-18T14:59:03.021 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.014+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3067575436 wait complete. 2024-03-18T14:59:03.021 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.014+0000 7f43ecd8f740 20 client.16311 trim_cache size 0 max 16384 2024-03-18T14:59:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.029+0000 7f43ecd8f740 1 Processor -- start 2024-03-18T14:59:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.029+0000 7f43ecd8f740 1 -- start start 2024-03-18T14:59:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43ecd8f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343130 0x5606c734bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43ecd8f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c7343a50 0x5606c733f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43ecd8f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c734c820 0x5606c7361820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43ecd8f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f43e41b1f40 con 0x5606c7343130 2024-03-18T14:59:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43ecd8f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f43e41af7e0 con 0x5606c7343a50 2024-03-18T14:59:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43ecd8f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f43e41ad630 con 0x5606c734c820 2024-03-18T14:59:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43e2ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343130 0x5606c734bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43e2ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343130 0x5606c734bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57240/0 (socket says 172.21.15.76:57240) 2024-03-18T14:59:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43e2ffd640 1 -- 172.21.15.76:0/3551783865 learned_addr learned my addr 172.21.15.76:0/3551783865 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43e37fe640 1 --2- 172.21.15.76:0/3551783865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c7343a50 0x5606c733f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43e2ffd640 1 --2- 172.21.15.76:0/3551783865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c734c820 0x5606c7361820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43e2ffd640 1 -- 172.21.15.76:0/3551783865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c734c820 msgr2=0x5606c7361820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43e2ffd640 1 --2- 172.21.15.76:0/3551783865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c734c820 0x5606c7361820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43e2ffd640 1 -- 172.21.15.76:0/3551783865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c7343a50 msgr2=0x5606c733f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43e2ffd640 1 --2- 172.21.15.76:0/3551783865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c7343a50 0x5606c733f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43e2ffd640 1 -- 172.21.15.76:0/3551783865 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606c7339310 con 0x5606c7343130 2024-03-18T14:59:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43e2ffd640 1 --2- 172.21.15.76:0/3551783865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343130 0x5606c734bfa0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009850 tx=0x7f43d402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e03d024a09ec27 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43c97fa640 1 -- 172.21.15.76:0/3551783865 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d4050030 con 0x5606c7343130 2024-03-18T14:59:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43c97fa640 1 -- 172.21.15.76:0/3551783865 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43d4051040 con 0x5606c7343130 2024-03-18T14:59:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.030+0000 7f43c97fa640 1 -- 172.21.15.76:0/3551783865 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d4004000 con 0x5606c7343130 2024-03-18T14:59:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.031+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3551783865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343130 msgr2=0x5606c734bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.031+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3551783865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343130 0x5606c734bfa0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009850 tx=0x7f43d402bc30 comp rx=0 tx=0).stop 2024-03-18T14:59:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.031+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3551783865 shutdown_connections 2024-03-18T14:59:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.031+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3551783865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c734c820 0x5606c7361820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.031+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3551783865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c7343a50 0x5606c733f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.031+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3551783865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343130 0x5606c734bfa0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.031+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3551783865 >> 172.21.15.76:0/3551783865 conn(0x5606c733da50 msgr2=0x5606c733ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.031+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3551783865 shutdown_connections 2024-03-18T14:59:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.031+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3551783865 wait complete. 2024-03-18T14:59:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.031+0000 7f43ecd8f740 1 -- start start 2024-03-18T14:59:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43ecd8f740 1 Processor -- start 2024-03-18T14:59:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43ecd8f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343130 0x5606c73bab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43ecd8f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c7343a50 0x5606c73bb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43ecd8f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c734c820 0x5606c735ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43ecd8f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f43e41adca0 con 0x5606c7343130 2024-03-18T14:59:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43ecd8f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f43e41af110 con 0x5606c734c820 2024-03-18T14:59:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43ecd8f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f43e41ae370 con 0x5606c7343a50 2024-03-18T14:59:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43ecd8f740 10 client.0 _ll_register_callbacks cb 0x7f43d8009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:59:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43e2ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343130 0x5606c73bab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43e2ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343130 0x5606c73bab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57242/0 (socket says 172.21.15.76:57242) 2024-03-18T14:59:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43e2ffd640 1 -- 172.21.15.76:0/3416131619 learned_addr learned my addr 172.21.15.76:0/3416131619 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43e37fe640 1 --2- 172.21.15.76:0/3416131619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c7343a50 0x5606c73bb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43e2ffd640 1 --2- 172.21.15.76:0/3416131619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c734c820 0x5606c735ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43e37fe640 1 -- 172.21.15.76:0/3416131619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c734c820 msgr2=0x5606c735ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43e37fe640 1 --2- 172.21.15.76:0/3416131619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c734c820 0x5606c735ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43e37fe640 1 -- 172.21.15.76:0/3416131619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343130 msgr2=0x5606c73bab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43e37fe640 1 --2- 172.21.15.76:0/3416131619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343130 0x5606c73bab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43e37fe640 1 -- 172.21.15.76:0/3416131619 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606c735f110 con 0x5606c7343a50 2024-03-18T14:59:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.032+0000 7f43e2ffd640 1 --2- 172.21.15.76:0/3416131619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343130 0x5606c73bab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:59:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.033+0000 7f43e37fe640 1 --2- 172.21.15.76:0/3416131619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c7343a50 0x5606c73bb0b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x5606c734f590 tx=0x7f43d801f770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dcdbdfb05b10dd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.033+0000 7f43c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-18T14:59:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.033+0000 7f43c2ffd640 1 -- 172.21.15.76:0/3416131619 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d8004220 con 0x5606c7343a50 2024-03-18T14:59:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.033+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3416131619 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43d4051040 con 0x5606c7343a50 2024-03-18T14:59:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.033+0000 7f43c2ffd640 1 -- 172.21.15.76:0/3416131619 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7f43d80043c0 con 0x5606c7343a50 2024-03-18T14:59:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.033+0000 7f43c2ffd640 1 -- 172.21.15.76:0/3416131619 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f43d802b020 con 0x5606c7343a50 2024-03-18T14:59:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.033+0000 7f43ecd8f740 10 client.26167 fetch_fsmap learned FSMap version 40 2024-03-18T14:59:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.033+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3416131619 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5606c73c23e0 con 0x5606c7343a50 2024-03-18T14:59:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.033+0000 7f43c2ffd640 1 -- 172.21.15.76:0/3416131619 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43d4051040 con 0x5606c7343a50 2024-03-18T14:59:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.033+0000 7f43c2ffd640 1 -- 172.21.15.76:0/3416131619 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 40) ==== 8342+0+0 (secure 0 0 0) 0x7f43d8004b40 con 0x5606c7343a50 2024-03-18T14:59:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.033+0000 7f43ecd8f740 10 client.26167 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T14:59:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.033+0000 7f43ecd8f740 10 client.26167 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:59:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.033+0000 7f43ecd8f740 20 client.26167 populate_metadata read hostname 'smithi076' 2024-03-18T14:59:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.033+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3416131619 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5606c73c2b10 0x5606c73c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.033+0000 7f43ecd8f740 4 client.26167 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:59:03.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.033+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3416131619 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5606c72af2d0 con 0x5606c73c2b10 2024-03-18T14:59:03.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.034+0000 7f43e2ffd640 1 --2- 172.21.15.76:0/3416131619 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5606c73c2b10 0x5606c73c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:03.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.034+0000 7f43c2ffd640 10 client.26167 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:03.040 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.034+0000 7f43e2ffd640 1 --2- 172.21.15.76:0/3416131619 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5606c73c2b10 0x5606c73c4f30 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf631a1008fecbc7 server_cookie=cd36da5d639c5a84 in_seq=0 out_seq=0 2024-03-18T14:59:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.105+0000 7f43c2ffd640 1 -- 172.21.15.76:0/3416131619 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5606c72af2d0 con 0x5606c73c2b10 2024-03-18T14:59:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.105+0000 7f43c2ffd640 10 client.26167 handle_command_reply: tid=0 2024-03-18T14:59:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.105+0000 7f43ecd8f740 1 client.26167 shutdown 2024-03-18T14:59:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.105+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3416131619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c7343a50 msgr2=0x5606c73bb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.105+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3416131619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c7343a50 0x5606c73bb0b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x5606c734f590 tx=0x7f43d801f770 comp rx=0 tx=0).stop 2024-03-18T14:59:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.105+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3416131619 shutdown_connections 2024-03-18T14:59:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.105+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3416131619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606c7343a50 0x5606c73bb0b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.105+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3416131619 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5606c73c2b10 0x5606c73c4f30 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.105+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3416131619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5606c734c820 0x5606c735ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.105+0000 7f43c2ffd640 0 client.26167 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.105+0000 7f43ecd8f740 1 --2- 172.21.15.76:0/3416131619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606c7343130 0x5606c73bab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.105+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3416131619 >> 172.21.15.76:0/3416131619 conn(0x5606c733da50 msgr2=0x5606c7358050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.105+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3416131619 shutdown_connections 2024-03-18T14:59:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.105+0000 7f43ecd8f740 1 -- 172.21.15.76:0/3416131619 wait complete. 2024-03-18T14:59:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.105+0000 7f43ecd8f740 20 client.26167 trim_cache size 0 max 16384 2024-03-18T14:59:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:59:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:59:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:59:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:59:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.106+0000 7f43ea0c6640 1 -- 172.21.15.76:0/2854224886 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f43cc0a2a90 msgr2=0x7f43cc0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.106+0000 7f43ea0c6640 1 --2- 172.21.15.76:0/2854224886 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f43cc0a2a90 0x7f43cc0a4f50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f43dc09bab0 tx=0x7f43dc00cee0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.106+0000 7f43ea0c6640 1 -- 172.21.15.76:0/2854224886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43e41077a0 msgr2=0x7f43e4074560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.106+0000 7f43ea0c6640 1 --2- 172.21.15.76:0/2854224886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43e41077a0 0x7f43e4074560 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f43dc00eb10 tx=0x7f43dc00c2d0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.106+0000 7f43ea0c6640 1 -- 172.21.15.76:0/2854224886 shutdown_connections 2024-03-18T14:59:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.106+0000 7f43ea0c6640 1 --2- 172.21.15.76:0/2854224886 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f43cc0a2a90 0x7f43cc0a4f50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.106+0000 7f43ea0c6640 1 --2- 172.21.15.76:0/2854224886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43e41077a0 0x7f43e4074560 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.106+0000 7f43ea0c6640 1 --2- 172.21.15.76:0/2854224886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43e4102150 0x7f43e4074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.106+0000 7f43ea0c6640 1 --2- 172.21.15.76:0/2854224886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43e4101790 0x7f43e4073ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.106+0000 7f43ea0c6640 1 -- 172.21.15.76:0/2854224886 >> 172.21.15.76:0/2854224886 conn(0x7f43e40fd2a0 msgr2=0x7f43e40fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.106+0000 7f43ea0c6640 1 -- 172.21.15.76:0/2854224886 shutdown_connections 2024-03-18T14:59:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:03.107+0000 7f43ea0c6640 1 -- 172.21.15.76:0/2854224886 wait complete. 2024-03-18T14:59:03.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 547442 9.50 MB/sec execute 184 sec latency 16.241 ms 2024-03-18T14:59:03.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:03 smithi077 ceph-mon[35865]: pgmap v182: 97 pgs: 97 active+clean; 162 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 408 op/s 2024-03-18T14:59:04.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:03 smithi076 ceph-mon[32205]: pgmap v182: 97 pgs: 97 active+clean; 162 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 408 op/s 2024-03-18T14:59:04.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:03 smithi174 ceph-mon[36787]: pgmap v182: 97 pgs: 97 active+clean; 162 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 408 op/s 2024-03-18T14:59:04.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 548985 9.48 MB/sec execute 185 sec latency 22.210 ms 2024-03-18T14:59:05.745 INFO:tasks.workunit.client.0.smithi076.stdout: 1 551003 9.50 MB/sec execute 186 sec latency 14.573 ms 2024-03-18T14:59:05.966 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:05 smithi174 ceph-mon[36787]: pgmap v183: 97 pgs: 97 active+clean; 162 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 254 op/s 2024-03-18T14:59:05.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:05 smithi077 ceph-mon[35865]: pgmap v183: 97 pgs: 97 active+clean; 162 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 254 op/s 2024-03-18T14:59:06.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:05 smithi076 ceph-mon[32205]: pgmap v183: 97 pgs: 97 active+clean; 162 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 254 op/s 2024-03-18T14:59:06.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 552474 9.48 MB/sec execute 187 sec latency 23.984 ms 2024-03-18T14:59:07.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 554414 9.50 MB/sec execute 188 sec latency 48.700 ms 2024-03-18T14:59:07.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:07 smithi077 ceph-mon[35865]: pgmap v184: 97 pgs: 97 active+clean; 162 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 254 op/s 2024-03-18T14:59:08.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:07 smithi076 ceph-mon[32205]: pgmap v184: 97 pgs: 97 active+clean; 162 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 254 op/s 2024-03-18T14:59:08.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:07 smithi174 ceph-mon[36787]: pgmap v184: 97 pgs: 97 active+clean; 162 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 254 op/s 2024-03-18T14:59:08.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 555669 9.48 MB/sec execute 189 sec latency 25.572 ms 2024-03-18T14:59:09.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 558001 9.50 MB/sec execute 190 sec latency 14.967 ms 2024-03-18T14:59:09.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:09 smithi077 ceph-mon[35865]: pgmap v185: 97 pgs: 97 active+clean; 170 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 358 op/s 2024-03-18T14:59:10.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:09 smithi076 ceph-mon[32205]: pgmap v185: 97 pgs: 97 active+clean; 170 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 358 op/s 2024-03-18T14:59:10.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:09 smithi174 ceph-mon[36787]: pgmap v185: 97 pgs: 97 active+clean; 170 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 358 op/s 2024-03-18T14:59:10.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 559531 9.48 MB/sec execute 191 sec latency 20.425 ms 2024-03-18T14:59:11.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 561667 9.50 MB/sec execute 192 sec latency 13.215 ms 2024-03-18T14:59:11.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:11 smithi174 ceph-mon[36787]: pgmap v186: 97 pgs: 97 active+clean; 170 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 250 op/s 2024-03-18T14:59:11.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:11 smithi077 ceph-mon[35865]: pgmap v186: 97 pgs: 97 active+clean; 170 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 250 op/s 2024-03-18T14:59:12.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:11 smithi076 ceph-mon[32205]: pgmap v186: 97 pgs: 97 active+clean; 170 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 250 op/s 2024-03-18T14:59:12.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 562946 9.48 MB/sec execute 193 sec latency 115.454 ms 2024-03-18T14:59:13.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 565178 9.50 MB/sec execute 194 sec latency 17.768 ms 2024-03-18T14:59:14.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:14 smithi076 ceph-mon[32205]: pgmap v187: 97 pgs: 97 active+clean; 177 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 MiB/s wr, 357 op/s 2024-03-18T14:59:14.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:14 smithi174 ceph-mon[36787]: pgmap v187: 97 pgs: 97 active+clean; 177 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 MiB/s wr, 357 op/s 2024-03-18T14:59:14.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:14 smithi077 ceph-mon[35865]: pgmap v187: 97 pgs: 97 active+clean; 177 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 MiB/s wr, 357 op/s 2024-03-18T14:59:14.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 566857 9.48 MB/sec execute 195 sec latency 19.078 ms 2024-03-18T14:59:15.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 568776 9.50 MB/sec execute 196 sec latency 14.725 ms 2024-03-18T14:59:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:15 smithi077 ceph-mon[35865]: pgmap v188: 97 pgs: 97 active+clean; 177 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 211 op/s 2024-03-18T14:59:16.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:15 smithi076 ceph-mon[32205]: pgmap v188: 97 pgs: 97 active+clean; 177 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 211 op/s 2024-03-18T14:59:16.202 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.199+0000 7fc24b036640 1 -- 172.21.15.76:0/3985395177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2441067f0 msgr2=0x7fc244106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:16.202 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.199+0000 7fc24b036640 1 --2- 172.21.15.76:0/3985395177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2441067f0 0x7fc244106bd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc238009900 tx=0x7fc23802bc30 comp rx=0 tx=0).stop 2024-03-18T14:59:16.202 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.199+0000 7fc24b036640 1 -- 172.21.15.76:0/3985395177 shutdown_connections 2024-03-18T14:59:16.202 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.199+0000 7fc24b036640 1 --2- 172.21.15.76:0/3985395177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc24406a9e0 0x7fc24410e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.202 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.199+0000 7fc24b036640 1 --2- 172.21.15.76:0/3985395177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc24406a020 0x7fc24406a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.202 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.199+0000 7fc24b036640 1 --2- 172.21.15.76:0/3985395177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2441067f0 0x7fc244106bd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.202 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.199+0000 7fc24b036640 1 -- 172.21.15.76:0/3985395177 >> 172.21.15.76:0/3985395177 conn(0x7fc2440774b0 msgr2=0x7fc2440778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:16.202 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.199+0000 7fc24b036640 1 -- 172.21.15.76:0/3985395177 shutdown_connections 2024-03-18T14:59:16.203 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.199+0000 7fc24b036640 1 -- 172.21.15.76:0/3985395177 wait complete. 2024-03-18T14:59:16.203 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.199+0000 7fc24b036640 1 Processor -- start 2024-03-18T14:59:16.203 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc24b036640 1 -- start start 2024-03-18T14:59:16.203 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc24b036640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24406a020 0x7fc244197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.203 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc24b036640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc24406a9e0 0x7fc244198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.203 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc24b036640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2441067f0 0x7fc24419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.203 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc24b036640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc244111ba0 con 0x7fc2441067f0 2024-03-18T14:59:16.203 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc24b036640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc244111a20 con 0x7fc24406a020 2024-03-18T14:59:16.203 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc24b036640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc244111d20 con 0x7fc24406a9e0 2024-03-18T14:59:16.203 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc248dd2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24406a020 0x7fc244197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.203 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc2495d3640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2441067f0 0x7fc24419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc248dd2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24406a020 0x7fc244197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:53262/0 (socket says 172.21.15.76:53262) 2024-03-18T14:59:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc248dd2640 1 -- 172.21.15.76:0/1078851173 learned_addr learned my addr 172.21.15.76:0/1078851173 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc243fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc24406a9e0 0x7fc244198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc248dd2640 1 -- 172.21.15.76:0/1078851173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc24406a9e0 msgr2=0x7fc244198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc248dd2640 1 --2- 172.21.15.76:0/1078851173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc24406a9e0 0x7fc244198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc248dd2640 1 -- 172.21.15.76:0/1078851173 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2441067f0 msgr2=0x7fc24419c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc248dd2640 1 --2- 172.21.15.76:0/1078851173 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2441067f0 0x7fc24419c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc248dd2640 1 -- 172.21.15.76:0/1078851173 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24419cf60 con 0x7fc24406a020 2024-03-18T14:59:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc2495d3640 1 --2- 172.21.15.76:0/1078851173 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2441067f0 0x7fc24419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:59:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc248dd2640 1 --2- 172.21.15.76:0/1078851173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24406a020 0x7fc244197fc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc238005940 tx=0x7fc238004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.200+0000 7fc241ffb640 1 -- 172.21.15.76:0/1078851173 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc238041070 con 0x7fc24406a020 2024-03-18T14:59:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.201+0000 7fc24b036640 1 -- 172.21.15.76:0/1078851173 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc24419d1f0 con 0x7fc24406a020 2024-03-18T14:59:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.201+0000 7fc241ffb640 1 -- 172.21.15.76:0/1078851173 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2380054e0 con 0x7fc24406a020 2024-03-18T14:59:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.201+0000 7fc241ffb640 1 -- 172.21.15.76:0/1078851173 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc238003d00 con 0x7fc24406a020 2024-03-18T14:59:16.205 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.201+0000 7fc24b036640 1 -- 172.21.15.76:0/1078851173 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2441a4ae0 con 0x7fc24406a020 2024-03-18T14:59:16.211 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.203+0000 7fc241ffb640 1 -- 172.21.15.76:0/1078851173 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fc238044070 con 0x7fc24406a020 2024-03-18T14:59:16.211 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.203+0000 7fc24dc28740 1 Processor -- start 2024-03-18T14:59:16.211 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.203+0000 7fc24dc28740 1 -- start start 2024-03-18T14:59:16.212 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc24dc28740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee568130 0x55dbee568510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.212 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc24dc28740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568a50 0x55dbee5712e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.212 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc24dc28740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee571820 0x55dbee573c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.212 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc24dc28740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55dbee4cca40 con 0x55dbee571820 2024-03-18T14:59:16.212 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc24dc28740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55dbee475920 con 0x55dbee568a50 2024-03-18T14:59:16.212 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc24dc28740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55dbee4c8ae0 con 0x55dbee568130 2024-03-18T14:59:16.212 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc241ffb640 1 --2- 172.21.15.76:0/1078851173 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc2340a2a10 0x7fc2340a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.212 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc2495d3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee568130 0x55dbee568510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.212 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc243fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee571820 0x55dbee573c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.212 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc248dd2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568a50 0x55dbee5712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.212 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc2495d3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee568130 0x55dbee568510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:38402/0 (socket says 172.21.15.76:38402) 2024-03-18T14:59:16.212 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc2495d3640 1 -- 172.21.15.76:0/3011729561 learned_addr learned my addr 172.21.15.76:0/3011729561 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:16.212 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc2495d3640 1 --2- 172.21.15.76:0/1078851173 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc2340a2a10 0x7fc2340a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.213 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc241ffb640 1 -- 172.21.15.76:0/1078851173 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7fc2380f07a0 con 0x7fc24406a020 2024-03-18T14:59:16.213 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc2495d3640 1 -- 172.21.15.76:0/3011729561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568a50 msgr2=0x55dbee5712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:16.213 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc2495d3640 1 --2- 172.21.15.76:0/3011729561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568a50 0x55dbee5712e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.213 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc2495d3640 1 -- 172.21.15.76:0/3011729561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee571820 msgr2=0x55dbee573c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:16.213 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc2495d3640 1 --2- 172.21.15.76:0/3011729561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee571820 0x55dbee573c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.213 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc2495d3640 1 -- 172.21.15.76:0/3011729561 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbee48d460 con 0x55dbee568130 2024-03-18T14:59:16.213 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc243fff640 1 --2- 172.21.15.76:0/3011729561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee571820 0x55dbee573c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:59:16.213 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc2495d3640 1 --2- 172.21.15.76:0/1078851173 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc2340a2a10 0x7fc2340a4ed0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc23c00ff50 tx=0x7fc23c018000 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:16.213 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.204+0000 7fc2495d3640 1 --2- 172.21.15.76:0/3011729561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee568130 0x55dbee568510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc23c009930 tx=0x7fc23c00fa50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46ae385715321d3f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:16.213 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.205+0000 7fc22effd640 1 -- 172.21.15.76:0/3011729561 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c022070 con 0x55dbee568130 2024-03-18T14:59:16.213 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.205+0000 7fc22effd640 1 -- 172.21.15.76:0/3011729561 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc23c009570 con 0x55dbee568130 2024-03-18T14:59:16.213 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.205+0000 7fc22effd640 1 -- 172.21.15.76:0/3011729561 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c00edb0 con 0x55dbee568130 2024-03-18T14:59:16.213 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.205+0000 7fc24dc28740 1 -- 172.21.15.76:0/3011729561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee568130 msgr2=0x55dbee568510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.205+0000 7fc24dc28740 1 --2- 172.21.15.76:0/3011729561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee568130 0x55dbee568510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc23c009930 tx=0x7fc23c00fa50 comp rx=0 tx=0).stop 2024-03-18T14:59:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.205+0000 7fc24dc28740 1 -- 172.21.15.76:0/3011729561 shutdown_connections 2024-03-18T14:59:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.205+0000 7fc24dc28740 1 --2- 172.21.15.76:0/3011729561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee571820 0x55dbee573c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.205+0000 7fc24dc28740 1 --2- 172.21.15.76:0/3011729561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568a50 0x55dbee5712e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.205+0000 7fc24dc28740 1 --2- 172.21.15.76:0/3011729561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee568130 0x55dbee568510 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.205+0000 7fc24dc28740 1 -- 172.21.15.76:0/3011729561 >> 172.21.15.76:0/3011729561 conn(0x55dbee562a50 msgr2=0x55dbee562e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.205+0000 7fc24dc28740 1 -- 172.21.15.76:0/3011729561 shutdown_connections 2024-03-18T14:59:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.205+0000 7fc24dc28740 1 -- 172.21.15.76:0/3011729561 wait complete. 2024-03-18T14:59:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.206+0000 7fc24dc28740 1 -- start start 2024-03-18T14:59:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.206+0000 7fc24dc28740 1 Processor -- start 2024-03-18T14:59:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.206+0000 7fc24dc28740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568130 0x55dbee56e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.206+0000 7fc24dc28740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee568a50 0x55dbee56ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.206+0000 7fc24dc28740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee571820 0x55dbee56f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.206+0000 7fc24dc28740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55dbee574390 con 0x55dbee571820 2024-03-18T14:59:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.206+0000 7fc24dc28740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55dbee488d70 con 0x55dbee568130 2024-03-18T14:59:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.206+0000 7fc24dc28740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55dbee574510 con 0x55dbee568a50 2024-03-18T14:59:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.206+0000 7fc24dc28740 10 client.0 _ll_register_callbacks cb 0x55dbee1e0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:59:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.206+0000 7fc2495d3640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee571820 0x55dbee56f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.206+0000 7fc248dd2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568130 0x55dbee56e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.206+0000 7fc2495d3640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee571820 0x55dbee56f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50802/0 (socket says 172.21.15.76:50802) 2024-03-18T14:59:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.206+0000 7fc2495d3640 1 -- 172.21.15.76:0/4277900037 learned_addr learned my addr 172.21.15.76:0/4277900037 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.206+0000 7fc248dd2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568130 0x55dbee56e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:53282/0 (socket says 172.21.15.76:53282) 2024-03-18T14:59:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.206+0000 7fc243fff640 1 --2- 172.21.15.76:0/4277900037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee568a50 0x55dbee56ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc2495d3640 1 -- 172.21.15.76:0/4277900037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee568a50 msgr2=0x55dbee56ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc2495d3640 1 --2- 172.21.15.76:0/4277900037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee568a50 0x55dbee56ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc2495d3640 1 -- 172.21.15.76:0/4277900037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568130 msgr2=0x55dbee56e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc2495d3640 1 --2- 172.21.15.76:0/4277900037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568130 0x55dbee56e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc2495d3640 1 -- 172.21.15.76:0/4277900037 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbee1e1210 con 0x55dbee571820 2024-03-18T14:59:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc248dd2640 1 --2- 172.21.15.76:0/4277900037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568130 0x55dbee56e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:59:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc2495d3640 1 --2- 172.21.15.76:0/4277900037 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee571820 0x55dbee56f340 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc23c009900 tx=0x7fc23c003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=755ce0bcf3db6e43 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc223fff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:59:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc223fff640 1 -- 172.21.15.76:0/4277900037 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c033020 con 0x55dbee571820 2024-03-18T14:59:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc24dc28740 1 -- 172.21.15.76:0/4277900037 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc23c009570 con 0x55dbee571820 2024-03-18T14:59:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc223fff640 1 -- 172.21.15.76:0/4277900037 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7fc23c005240 con 0x55dbee571820 2024-03-18T14:59:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc223fff640 1 -- 172.21.15.76:0/4277900037 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc23c03e050 con 0x55dbee571820 2024-03-18T14:59:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc24dc28740 10 client.16329 fetch_fsmap learned FSMap version 40 2024-03-18T14:59:16.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc24dc28740 1 -- 172.21.15.76:0/4277900037 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55dbee2ad620 con 0x55dbee571820 2024-03-18T14:59:16.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc223fff640 1 -- 172.21.15.76:0/4277900037 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc23c009570 con 0x55dbee571820 2024-03-18T14:59:16.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc223fff640 1 -- 172.21.15.76:0/4277900037 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 40) ==== 8342+0+0 (secure 0 0 0) 0x7fc23c026710 con 0x55dbee571820 2024-03-18T14:59:16.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc24dc28740 10 client.16329 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T14:59:16.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc24dc28740 10 client.16329 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:59:16.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc24dc28740 20 client.16329 populate_metadata read hostname 'smithi076' 2024-03-18T14:59:16.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.207+0000 7fc24dc28740 1 --2- 172.21.15.76:0/4277900037 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55dbee581940 0x55dbee583d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.208+0000 7fc24dc28740 4 client.16329 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:59:16.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.208+0000 7fc24dc28740 1 -- 172.21.15.76:0/4277900037 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbee4d9f80 con 0x55dbee581940 2024-03-18T14:59:16.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.208+0000 7fc248dd2640 1 --2- 172.21.15.76:0/4277900037 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55dbee581940 0x55dbee583d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.208+0000 7fc248dd2640 1 --2- 172.21.15.76:0/4277900037 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55dbee581940 0x55dbee583d30 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d06fdc8dfe35d157 server_cookie=f5b268cbd1367d21 in_seq=0 out_seq=0 2024-03-18T14:59:16.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.208+0000 7fc223fff640 10 client.16329 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:16.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.209+0000 7fc223fff640 1 -- 172.21.15.76:0/4277900037 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dbee4d9f80 con 0x55dbee581940 2024-03-18T14:59:16.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.209+0000 7fc223fff640 10 client.16329 handle_command_reply: tid=0 2024-03-18T14:59:16.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.209+0000 7fc24dc28740 1 client.16329 shutdown 2024-03-18T14:59:16.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.209+0000 7fc24dc28740 1 -- 172.21.15.76:0/4277900037 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee571820 msgr2=0x55dbee56f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:16.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.209+0000 7fc24dc28740 1 --2- 172.21.15.76:0/4277900037 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee571820 0x55dbee56f340 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc23c009900 tx=0x7fc23c003f90 comp rx=0 tx=0).stop 2024-03-18T14:59:16.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.209+0000 7fc24dc28740 1 -- 172.21.15.76:0/4277900037 shutdown_connections 2024-03-18T14:59:16.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.209+0000 7fc24dc28740 1 --2- 172.21.15.76:0/4277900037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee568a50 0x55dbee56ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.209+0000 7fc24dc28740 1 --2- 172.21.15.76:0/4277900037 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55dbee581940 0x55dbee583d30 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.209+0000 7fc24dc28740 1 --2- 172.21.15.76:0/4277900037 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee571820 0x55dbee56f340 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.209+0000 7fc24dc28740 1 --2- 172.21.15.76:0/4277900037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568130 0x55dbee56e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.209+0000 7fc223fff640 0 client.16329 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:16.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.209+0000 7fc24dc28740 1 -- 172.21.15.76:0/4277900037 >> 172.21.15.76:0/4277900037 conn(0x55dbee562a50 msgr2=0x55dbee5647a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:16.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.209+0000 7fc24dc28740 1 -- 172.21.15.76:0/4277900037 shutdown_connections 2024-03-18T14:59:16.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.209+0000 7fc24dc28740 1 -- 172.21.15.76:0/4277900037 wait complete. 2024-03-18T14:59:16.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.209+0000 7fc24dc28740 20 client.16329 trim_cache size 0 max 16384 2024-03-18T14:59:16.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:15 smithi174 ceph-mon[36787]: pgmap v188: 97 pgs: 97 active+clean; 177 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 211 op/s 2024-03-18T14:59:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc24dc28740 1 Processor -- start 2024-03-18T14:59:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc24dc28740 1 -- start start 2024-03-18T14:59:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc24dc28740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee568130 0x55dbee570ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc24dc28740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568a50 0x55dbee57d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc24dc28740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee571820 0x55dbee589510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc24dc28740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc2441ada20 con 0x55dbee568130 2024-03-18T14:59:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc24dc28740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc2441ab2c0 con 0x55dbee568a50 2024-03-18T14:59:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc24dc28740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc2441a9130 con 0x55dbee571820 2024-03-18T14:59:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc248dd2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee568130 0x55dbee570ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc248dd2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee568130 0x55dbee570ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50804/0 (socket says 172.21.15.76:50804) 2024-03-18T14:59:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc248dd2640 1 -- 172.21.15.76:0/3668472661 learned_addr learned my addr 172.21.15.76:0/3668472661 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc2495d3640 1 --2- 172.21.15.76:0/3668472661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee571820 0x55dbee589510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc243fff640 1 --2- 172.21.15.76:0/3668472661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568a50 0x55dbee57d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc248dd2640 1 -- 172.21.15.76:0/3668472661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee571820 msgr2=0x55dbee589510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc248dd2640 1 --2- 172.21.15.76:0/3668472661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee571820 0x55dbee589510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc248dd2640 1 -- 172.21.15.76:0/3668472661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568a50 msgr2=0x55dbee57d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:16.230 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc248dd2640 1 --2- 172.21.15.76:0/3668472661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568a50 0x55dbee57d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.230 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc248dd2640 1 -- 172.21.15.76:0/3668472661 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbee56a490 con 0x55dbee568130 2024-03-18T14:59:16.230 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc248dd2640 1 --2- 172.21.15.76:0/3668472661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee568130 0x55dbee570ff0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc2380bccb0 tx=0x7fc23808ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9716b997cd8cd476 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:16.230 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.224+0000 7fc22e7fc640 1 -- 172.21.15.76:0/3668472661 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23804b040 con 0x55dbee568130 2024-03-18T14:59:16.230 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.225+0000 7fc22e7fc640 1 -- 172.21.15.76:0/3668472661 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc23802ccd0 con 0x55dbee568130 2024-03-18T14:59:16.230 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.225+0000 7fc22e7fc640 1 -- 172.21.15.76:0/3668472661 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2380bb070 con 0x55dbee568130 2024-03-18T14:59:16.230 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.225+0000 7fc24dc28740 1 -- 172.21.15.76:0/3668472661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee568130 msgr2=0x55dbee570ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:16.230 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.225+0000 7fc24dc28740 1 --2- 172.21.15.76:0/3668472661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee568130 0x55dbee570ff0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc2380bccb0 tx=0x7fc23808ebe0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.230 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.225+0000 7fc24dc28740 1 -- 172.21.15.76:0/3668472661 shutdown_connections 2024-03-18T14:59:16.230 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.225+0000 7fc24dc28740 1 --2- 172.21.15.76:0/3668472661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee571820 0x55dbee589510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.230 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.225+0000 7fc24dc28740 1 --2- 172.21.15.76:0/3668472661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568a50 0x55dbee57d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.230 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.225+0000 7fc24dc28740 1 --2- 172.21.15.76:0/3668472661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee568130 0x55dbee570ff0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.230 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.225+0000 7fc24dc28740 1 -- 172.21.15.76:0/3668472661 >> 172.21.15.76:0/3668472661 conn(0x55dbee562a50 msgr2=0x55dbee561bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:16.230 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.225+0000 7fc24dc28740 1 -- 172.21.15.76:0/3668472661 shutdown_connections 2024-03-18T14:59:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.225+0000 7fc24dc28740 1 -- 172.21.15.76:0/3668472661 wait complete. 2024-03-18T14:59:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.225+0000 7fc24dc28740 1 -- start start 2024-03-18T14:59:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc24dc28740 1 Processor -- start 2024-03-18T14:59:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc24dc28740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568130 0x55dbee575090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc24dc28740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee568a50 0x55dbee5755d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc24dc28740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee571820 0x55dbee575b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc24dc28740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc2441a9780 con 0x55dbee568a50 2024-03-18T14:59:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc24dc28740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc2441aabf0 con 0x55dbee568130 2024-03-18T14:59:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc24dc28740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc2441a9e50 con 0x55dbee571820 2024-03-18T14:59:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc24dc28740 10 client.0 _ll_register_callbacks cb 0x7fc230009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:59:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc243fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee568a50 0x55dbee5755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc243fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee568a50 0x55dbee5755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50812/0 (socket says 172.21.15.76:50812) 2024-03-18T14:59:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc243fff640 1 -- 172.21.15.76:0/3579677779 learned_addr learned my addr 172.21.15.76:0/3579677779 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc248dd2640 1 --2- 172.21.15.76:0/3579677779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568130 0x55dbee575090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc2495d3640 1 --2- 172.21.15.76:0/3579677779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee571820 0x55dbee575b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc243fff640 1 -- 172.21.15.76:0/3579677779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee571820 msgr2=0x55dbee575b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc243fff640 1 --2- 172.21.15.76:0/3579677779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee571820 0x55dbee575b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc243fff640 1 -- 172.21.15.76:0/3579677779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568130 msgr2=0x55dbee575090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc243fff640 1 --2- 172.21.15.76:0/3579677779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568130 0x55dbee575090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.226+0000 7fc243fff640 1 -- 172.21.15.76:0/3579677779 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbee5e6b40 con 0x55dbee568a50 2024-03-18T14:59:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.227+0000 7fc243fff640 1 --2- 172.21.15.76:0/3579677779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee568a50 0x55dbee5755d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x55dbee563be0 tx=0x7fc23000e3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17c55069572f9508 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.227+0000 7fc223fff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:59:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.227+0000 7fc223fff640 1 -- 172.21.15.76:0/3579677779 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc230004050 con 0x55dbee568a50 2024-03-18T14:59:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.227+0000 7fc223fff640 1 -- 172.21.15.76:0/3579677779 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7fc2300041f0 con 0x55dbee568a50 2024-03-18T14:59:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.227+0000 7fc24dc28740 1 -- 172.21.15.76:0/3579677779 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dbee4d42d0 con 0x55dbee568a50 2024-03-18T14:59:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.227+0000 7fc223fff640 1 -- 172.21.15.76:0/3579677779 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc23002d050 con 0x55dbee568a50 2024-03-18T14:59:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.227+0000 7fc24dc28740 10 client.16335 fetch_fsmap learned FSMap version 40 2024-03-18T14:59:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.227+0000 7fc24dc28740 1 -- 172.21.15.76:0/3579677779 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55dbee5e7eb0 con 0x55dbee568a50 2024-03-18T14:59:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.227+0000 7fc223fff640 1 -- 172.21.15.76:0/3579677779 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55dbee4d42d0 con 0x55dbee568a50 2024-03-18T14:59:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.227+0000 7fc223fff640 1 -- 172.21.15.76:0/3579677779 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 40) ==== 8342+0+0 (secure 0 0 0) 0x7fc23001f640 con 0x55dbee568a50 2024-03-18T14:59:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.227+0000 7fc24dc28740 10 client.16335 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T14:59:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.227+0000 7fc24dc28740 10 client.16335 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:59:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.227+0000 7fc24dc28740 20 client.16335 populate_metadata read hostname 'smithi076' 2024-03-18T14:59:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.227+0000 7fc24dc28740 1 --2- 172.21.15.76:0/3579677779 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55dbee5e86b0 0x55dbee5eaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.227+0000 7fc24dc28740 4 client.16335 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:59:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.227+0000 7fc24dc28740 1 -- 172.21.15.76:0/3579677779 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dbee452770 con 0x55dbee5e86b0 2024-03-18T14:59:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.227+0000 7fc248dd2640 1 --2- 172.21.15.76:0/3579677779 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55dbee5e86b0 0x55dbee5eaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.228+0000 7fc223fff640 10 client.16335 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.228+0000 7fc248dd2640 1 --2- 172.21.15.76:0/3579677779 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55dbee5e86b0 0x55dbee5eaad0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0897c60c2660741 server_cookie=5a107b0b77bbd6ce in_seq=0 out_seq=0 2024-03-18T14:59:16.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.307+0000 7fc223fff640 1 -- 172.21.15.76:0/3579677779 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dbee452770 con 0x55dbee5e86b0 2024-03-18T14:59:16.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.307+0000 7fc223fff640 10 client.16335 handle_command_reply: tid=0 2024-03-18T14:59:16.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.307+0000 7fc24dc28740 1 client.16335 shutdown 2024-03-18T14:59:16.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.307+0000 7fc24dc28740 1 -- 172.21.15.76:0/3579677779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee568a50 msgr2=0x55dbee5755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:16.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.307+0000 7fc24dc28740 1 --2- 172.21.15.76:0/3579677779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee568a50 0x55dbee5755d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x55dbee563be0 tx=0x7fc23000e3b0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.307+0000 7fc24dc28740 1 -- 172.21.15.76:0/3579677779 shutdown_connections 2024-03-18T14:59:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.307+0000 7fc24dc28740 1 --2- 172.21.15.76:0/3579677779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dbee571820 0x55dbee575b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.307+0000 7fc24dc28740 1 --2- 172.21.15.76:0/3579677779 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55dbee5e86b0 0x55dbee5eaad0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.307+0000 7fc24dc28740 1 --2- 172.21.15.76:0/3579677779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dbee568a50 0x55dbee5755d0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.307+0000 7fc223fff640 0 client.16335 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.307+0000 7fc24dc28740 1 --2- 172.21.15.76:0/3579677779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbee568130 0x55dbee575090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.307+0000 7fc24dc28740 1 -- 172.21.15.76:0/3579677779 >> 172.21.15.76:0/3579677779 conn(0x55dbee562a50 msgr2=0x55dbee563ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.307+0000 7fc24dc28740 1 -- 172.21.15.76:0/3579677779 shutdown_connections 2024-03-18T14:59:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.307+0000 7fc24dc28740 1 -- 172.21.15.76:0/3579677779 wait complete. 2024-03-18T14:59:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.307+0000 7fc24dc28740 20 client.16335 trim_cache size 0 max 16384 2024-03-18T14:59:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:59:16.310 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:59:16.310 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:59:16.310 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:59:16.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.308+0000 7fc24b036640 1 -- 172.21.15.76:0/1078851173 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc2340a2a10 msgr2=0x7fc2340a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:16.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.308+0000 7fc24b036640 1 --2- 172.21.15.76:0/1078851173 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc2340a2a10 0x7fc2340a4ed0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc23c00ff50 tx=0x7fc23c018000 comp rx=0 tx=0).stop 2024-03-18T14:59:16.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.308+0000 7fc24b036640 1 -- 172.21.15.76:0/1078851173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24406a020 msgr2=0x7fc244197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:16.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.308+0000 7fc24b036640 1 --2- 172.21.15.76:0/1078851173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24406a020 0x7fc244197fc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc238005940 tx=0x7fc238004270 comp rx=0 tx=0).stop 2024-03-18T14:59:16.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.308+0000 7fc24b036640 1 -- 172.21.15.76:0/1078851173 shutdown_connections 2024-03-18T14:59:16.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.308+0000 7fc24b036640 1 --2- 172.21.15.76:0/1078851173 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc2340a2a10 0x7fc2340a4ed0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.308+0000 7fc24b036640 1 --2- 172.21.15.76:0/1078851173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc24406a9e0 0x7fc244198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.308+0000 7fc24b036640 1 --2- 172.21.15.76:0/1078851173 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2441067f0 0x7fc24419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.308+0000 7fc24b036640 1 --2- 172.21.15.76:0/1078851173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24406a020 0x7fc244197fc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:16.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.308+0000 7fc24b036640 1 -- 172.21.15.76:0/1078851173 >> 172.21.15.76:0/1078851173 conn(0x7fc2440774b0 msgr2=0x7fc2440fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:16.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.308+0000 7fc24b036640 1 -- 172.21.15.76:0/1078851173 shutdown_connections 2024-03-18T14:59:16.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:16.308+0000 7fc24b036640 1 -- 172.21.15.76:0/1078851173 wait complete. 2024-03-18T14:59:16.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 570484 9.49 MB/sec execute 197 sec latency 39.476 ms 2024-03-18T14:59:17.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 572295 9.50 MB/sec execute 198 sec latency 40.044 ms 2024-03-18T14:59:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:17 smithi077 ceph-mon[35865]: pgmap v189: 97 pgs: 97 active+clean; 177 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 211 op/s 2024-03-18T14:59:18.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:17 smithi076 ceph-mon[32205]: pgmap v189: 97 pgs: 97 active+clean; 177 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 211 op/s 2024-03-18T14:59:18.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:17 smithi174 ceph-mon[36787]: pgmap v189: 97 pgs: 97 active+clean; 177 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 211 op/s 2024-03-18T14:59:18.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 574277 9.49 MB/sec execute 199 sec latency 27.646 ms 2024-03-18T14:59:19.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 576027 9.51 MB/sec execute 200 sec latency 13.457 ms 2024-03-18T14:59:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:19 smithi077 ceph-mon[35865]: pgmap v190: 97 pgs: 97 active+clean; 172 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 361 op/s 2024-03-18T14:59:20.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:19 smithi076 ceph-mon[32205]: pgmap v190: 97 pgs: 97 active+clean; 172 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 361 op/s 2024-03-18T14:59:20.218 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:19 smithi174 ceph-mon[36787]: pgmap v190: 97 pgs: 97 active+clean; 172 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 361 op/s 2024-03-18T14:59:20.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 578415 9.52 MB/sec execute 201 sec latency 17.690 ms 2024-03-18T14:59:21.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 579890 9.51 MB/sec execute 202 sec latency 13.833 ms 2024-03-18T14:59:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:21 smithi077 ceph-mon[35865]: pgmap v191: 97 pgs: 97 active+clean; 172 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 256 op/s 2024-03-18T14:59:22.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:21 smithi076 ceph-mon[32205]: pgmap v191: 97 pgs: 97 active+clean; 172 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 256 op/s 2024-03-18T14:59:22.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:21 smithi174 ceph-mon[36787]: pgmap v191: 97 pgs: 97 active+clean; 172 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 256 op/s 2024-03-18T14:59:22.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 582276 9.53 MB/sec execute 203 sec latency 40.242 ms 2024-03-18T14:59:23.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 583439 9.51 MB/sec execute 204 sec latency 34.304 ms 2024-03-18T14:59:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:23 smithi077 ceph-mon[35865]: pgmap v192: 97 pgs: 97 active+clean; 181 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 356 op/s 2024-03-18T14:59:24.018 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:23 smithi174 ceph-mon[36787]: pgmap v192: 97 pgs: 97 active+clean; 181 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 356 op/s 2024-03-18T14:59:24.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:23 smithi076 ceph-mon[32205]: pgmap v192: 97 pgs: 97 active+clean; 181 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 356 op/s 2024-03-18T14:59:24.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 585739 9.53 MB/sec execute 205 sec latency 76.379 ms 2024-03-18T14:59:25.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 587061 9.51 MB/sec execute 206 sec latency 16.835 ms 2024-03-18T14:59:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:25 smithi077 ceph-mon[35865]: pgmap v193: 97 pgs: 97 active+clean; 181 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 249 op/s 2024-03-18T14:59:26.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:25 smithi076 ceph-mon[32205]: pgmap v193: 97 pgs: 97 active+clean; 181 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 249 op/s 2024-03-18T14:59:26.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:25 smithi174 ceph-mon[36787]: pgmap v193: 97 pgs: 97 active+clean; 181 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 249 op/s 2024-03-18T14:59:26.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 589763 9.55 MB/sec execute 207 sec latency 21.467 ms 2024-03-18T14:59:27.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 591005 9.53 MB/sec execute 208 sec latency 52.056 ms 2024-03-18T14:59:27.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:27 smithi077 ceph-mon[35865]: pgmap v194: 97 pgs: 97 active+clean; 181 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 249 op/s 2024-03-18T14:59:28.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:27 smithi076 ceph-mon[32205]: pgmap v194: 97 pgs: 97 active+clean; 181 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 249 op/s 2024-03-18T14:59:28.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:27 smithi174 ceph-mon[36787]: pgmap v194: 97 pgs: 97 active+clean; 181 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 249 op/s 2024-03-18T14:59:28.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 593242 9.55 MB/sec execute 209 sec latency 31.814 ms 2024-03-18T14:59:28.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:28 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:59:29.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:28 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:59:29.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:28 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T14:59:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.399+0000 7feb2da52640 1 -- 172.21.15.76:0/2495616436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb281078b0 msgr2=0x7feb28107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.399+0000 7feb2da52640 1 --2- 172.21.15.76:0/2495616436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb281078b0 0x7feb28107c90 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7feb18009930 tx=0x7feb1802bc30 comp rx=0 tx=0).stop 2024-03-18T14:59:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.399+0000 7feb2da52640 1 -- 172.21.15.76:0/2495616436 shutdown_connections 2024-03-18T14:59:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.399+0000 7feb2da52640 1 --2- 172.21.15.76:0/2495616436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb28068e60 0x7feb280692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.399+0000 7feb2da52640 1 --2- 172.21.15.76:0/2495616436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb28069800 0x7feb2810e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.399+0000 7feb2da52640 1 --2- 172.21.15.76:0/2495616436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb281078b0 0x7feb28107c90 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.399+0000 7feb2da52640 1 -- 172.21.15.76:0/2495616436 >> 172.21.15.76:0/2495616436 conn(0x7feb28077530 msgr2=0x7feb28077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.399+0000 7feb2da52640 1 -- 172.21.15.76:0/2495616436 shutdown_connections 2024-03-18T14:59:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.399+0000 7feb2da52640 1 -- 172.21.15.76:0/2495616436 wait complete. 2024-03-18T14:59:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.399+0000 7feb2da52640 1 Processor -- start 2024-03-18T14:59:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.399+0000 7feb2da52640 1 -- start start 2024-03-18T14:59:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb2da52640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb28068e60 0x7feb28197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb2da52640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb28069800 0x7feb28198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb2da52640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb281078b0 0x7feb2819c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb2da52640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7feb28111ba0 con 0x7feb28068e60 2024-03-18T14:59:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb2da52640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7feb28111a20 con 0x7feb281078b0 2024-03-18T14:59:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb2da52640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feb28111d20 con 0x7feb28069800 2024-03-18T14:59:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb26ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb28068e60 0x7feb28197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb26ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb28068e60 0x7feb28197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53658/0 (socket says 172.21.15.76:53658) 2024-03-18T14:59:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb1ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb28069800 0x7feb28198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb26ffd640 1 -- 172.21.15.76:0/331051673 learned_addr learned my addr 172.21.15.76:0/331051673 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb277fe640 1 --2- 172.21.15.76:0/331051673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb281078b0 0x7feb2819c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb26ffd640 1 -- 172.21.15.76:0/331051673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb28069800 msgr2=0x7feb28198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb26ffd640 1 --2- 172.21.15.76:0/331051673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb28069800 0x7feb28198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.404 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb26ffd640 1 -- 172.21.15.76:0/331051673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb281078b0 msgr2=0x7feb2819c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:29.404 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb26ffd640 1 --2- 172.21.15.76:0/331051673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb281078b0 0x7feb2819c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.404 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb26ffd640 1 -- 172.21.15.76:0/331051673 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb2819cfd0 con 0x7feb28068e60 2024-03-18T14:59:29.404 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb1ffff640 1 --2- 172.21.15.76:0/331051673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb28069800 0x7feb28198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:59:29.404 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb26ffd640 1 --2- 172.21.15.76:0/331051673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb28068e60 0x7feb28197fe0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7feb18009230 tx=0x7feb18004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:29.404 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb2ca50640 1 -- 172.21.15.76:0/331051673 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb18041070 con 0x7feb28068e60 2024-03-18T14:59:29.404 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb2ca50640 1 -- 172.21.15.76:0/331051673 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb180054e0 con 0x7feb28068e60 2024-03-18T14:59:29.404 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb2da52640 1 -- 172.21.15.76:0/331051673 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb2819d260 con 0x7feb28068e60 2024-03-18T14:59:29.404 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.400+0000 7feb2ca50640 1 -- 172.21.15.76:0/331051673 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb18003c90 con 0x7feb28068e60 2024-03-18T14:59:29.404 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.401+0000 7feb2da52640 1 -- 172.21.15.76:0/331051673 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb281a4b50 con 0x7feb28068e60 2024-03-18T14:59:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.403+0000 7feb2ca50640 1 -- 172.21.15.76:0/331051673 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7feb18044070 con 0x7feb28068e60 2024-03-18T14:59:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.403+0000 7feb3063f740 1 Processor -- start 2024-03-18T14:59:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.403+0000 7feb3063f740 1 -- start start 2024-03-18T14:59:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.403+0000 7feb2ca50640 1 --2- 172.21.15.76:0/331051673 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7feb140a2a10 0x7feb140a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.403+0000 7feb3063f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090130 0x56267b090510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.403+0000 7feb2ca50640 1 -- 172.21.15.76:0/331051673 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7feb180f08f0 con 0x7feb28068e60 2024-03-18T14:59:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.403+0000 7feb277fe640 1 --2- 172.21.15.76:0/331051673 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7feb140a2a10 0x7feb140a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.403+0000 7feb3063f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b090a50 0x56267b0992e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.403+0000 7feb3063f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b099820 0x56267b09bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.403+0000 7feb3063f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56267aff4a40 con 0x56267b099820 2024-03-18T14:59:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.403+0000 7feb3063f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56267af9d920 con 0x56267b090130 2024-03-18T14:59:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.403+0000 7feb3063f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56267aff0ae0 con 0x56267b090a50 2024-03-18T14:59:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.404+0000 7feb26ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090130 0x56267b090510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.404+0000 7feb26ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090130 0x56267b090510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:35692/0 (socket says 172.21.15.76:35692) 2024-03-18T14:59:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.404+0000 7feb26ffd640 1 -- 172.21.15.76:0/611718907 learned_addr learned my addr 172.21.15.76:0/611718907 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.404+0000 7feb1ffff640 1 --2- 172.21.15.76:0/611718907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b090a50 0x56267b0992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.404+0000 7feb277fe640 1 --2- 172.21.15.76:0/611718907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b099820 0x56267b09bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.404+0000 7feb26ffd640 1 -- 172.21.15.76:0/611718907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b090a50 msgr2=0x56267b0992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.404+0000 7feb26ffd640 1 --2- 172.21.15.76:0/611718907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b090a50 0x56267b0992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.404+0000 7feb26ffd640 1 -- 172.21.15.76:0/611718907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b099820 msgr2=0x56267b09bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.404+0000 7feb26ffd640 1 --2- 172.21.15.76:0/611718907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b099820 0x56267b09bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.404+0000 7feb26ffd640 1 -- 172.21.15.76:0/611718907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56267afb5460 con 0x56267b090130 2024-03-18T14:59:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.404+0000 7feb277fe640 1 --2- 172.21.15.76:0/611718907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b099820 0x56267b09bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:59:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.404+0000 7feb277fe640 1 --2- 172.21.15.76:0/331051673 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7feb140a2a10 0x7feb140a4ed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7feb20005fd0 tx=0x7feb20013b60 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.404+0000 7feb26ffd640 1 --2- 172.21.15.76:0/611718907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090130 0x56267b090510 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7feb180b95c0 tx=0x7feb180bcff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d806b24d5f30a0fb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.404+0000 7feb1dffb640 1 -- 172.21.15.76:0/611718907 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1804b070 con 0x56267b090130 2024-03-18T14:59:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.404+0000 7feb1dffb640 1 -- 172.21.15.76:0/611718907 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb1802ccd0 con 0x56267b090130 2024-03-18T14:59:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.404+0000 7feb1dffb640 1 -- 172.21.15.76:0/611718907 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb180c5850 con 0x56267b090130 2024-03-18T14:59:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.405+0000 7feb3063f740 1 -- 172.21.15.76:0/611718907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090130 msgr2=0x56267b090510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.405+0000 7feb3063f740 1 --2- 172.21.15.76:0/611718907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090130 0x56267b090510 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7feb180b95c0 tx=0x7feb180bcff0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.405+0000 7feb3063f740 1 -- 172.21.15.76:0/611718907 shutdown_connections 2024-03-18T14:59:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.405+0000 7feb3063f740 1 --2- 172.21.15.76:0/611718907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b090a50 0x56267b0992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.405+0000 7feb3063f740 1 --2- 172.21.15.76:0/611718907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b099820 0x56267b09bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.405+0000 7feb3063f740 1 --2- 172.21.15.76:0/611718907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090130 0x56267b090510 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.405+0000 7feb3063f740 1 -- 172.21.15.76:0/611718907 >> 172.21.15.76:0/611718907 conn(0x56267b08aa50 msgr2=0x56267b08ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.405+0000 7feb3063f740 1 -- 172.21.15.76:0/611718907 shutdown_connections 2024-03-18T14:59:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.405+0000 7feb3063f740 1 -- 172.21.15.76:0/611718907 wait complete. 2024-03-18T14:59:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.405+0000 7feb3063f740 1 -- start start 2024-03-18T14:59:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.405+0000 7feb3063f740 1 Processor -- start 2024-03-18T14:59:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb3063f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090130 0x56267b0968c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb3063f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b090a50 0x56267b096e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb3063f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b099820 0x56267b097340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb3063f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56267b09c210 con 0x56267b099820 2024-03-18T14:59:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb3063f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56267afb0d70 con 0x56267b090130 2024-03-18T14:59:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb3063f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56267b09c390 con 0x56267b090a50 2024-03-18T14:59:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb3063f740 10 client.0 _ll_register_callbacks cb 0x56267ad08a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:59:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb26ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090130 0x56267b0968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb1ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b090a50 0x56267b096e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb277fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b099820 0x56267b097340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb26ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090130 0x56267b0968c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:35704/0 (socket says 172.21.15.76:35704) 2024-03-18T14:59:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb1ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b090a50 0x56267b096e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:54276/0 (socket says 172.21.15.76:54276) 2024-03-18T14:59:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb26ffd640 1 -- 172.21.15.76:0/559366553 learned_addr learned my addr 172.21.15.76:0/559366553 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb26ffd640 1 -- 172.21.15.76:0/559366553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b090a50 msgr2=0x56267b096e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb26ffd640 1 --2- 172.21.15.76:0/559366553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b090a50 0x56267b096e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb26ffd640 1 -- 172.21.15.76:0/559366553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b099820 msgr2=0x56267b097340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb26ffd640 1 --2- 172.21.15.76:0/559366553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b099820 0x56267b097340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb26ffd640 1 -- 172.21.15.76:0/559366553 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56267ad09210 con 0x56267b090130 2024-03-18T14:59:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb277fe640 1 --2- 172.21.15.76:0/559366553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b099820 0x56267b097340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:59:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb26ffd640 1 --2- 172.21.15.76:0/559366553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090130 0x56267b0968c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7feb180ee250 tx=0x7feb180bcbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e05d60fca3a2f6c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.406+0000 7feb06ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:59:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.407+0000 7feb06ffd640 1 -- 172.21.15.76:0/559366553 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb180d7020 con 0x56267b090130 2024-03-18T14:59:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.407+0000 7feb06ffd640 1 -- 172.21.15.76:0/559366553 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7feb180c53c0 con 0x56267b090130 2024-03-18T14:59:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.407+0000 7feb3063f740 1 -- 172.21.15.76:0/559366553 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56267b001f80 con 0x56267b090130 2024-03-18T14:59:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.407+0000 7feb06ffd640 1 -- 172.21.15.76:0/559366553 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7feb180d5070 con 0x56267b090130 2024-03-18T14:59:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.407+0000 7feb3063f740 10 client.26207 fetch_fsmap learned FSMap version 40 2024-03-18T14:59:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.407+0000 7feb3063f740 1 -- 172.21.15.76:0/559366553 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56267add5620 con 0x56267b090130 2024-03-18T14:59:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.407+0000 7feb06ffd640 1 -- 172.21.15.76:0/559366553 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56267b001f80 con 0x56267b090130 2024-03-18T14:59:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.407+0000 7feb06ffd640 1 -- 172.21.15.76:0/559366553 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 40) ==== 8342+0+0 (secure 0 0 0) 0x7feb180dbbb0 con 0x56267b090130 2024-03-18T14:59:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.407+0000 7feb3063f740 10 client.26207 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T14:59:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.407+0000 7feb3063f740 10 client.26207 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:59:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.407+0000 7feb3063f740 20 client.26207 populate_metadata read hostname 'smithi076' 2024-03-18T14:59:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.407+0000 7feb3063f740 1 --2- 172.21.15.76:0/559366553 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56267b0a95c0 0x56267b0ab9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.407+0000 7feb3063f740 4 client.26207 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:59:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.407+0000 7feb3063f740 1 -- 172.21.15.76:0/559366553 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56267affc2d0 con 0x56267b0a95c0 2024-03-18T14:59:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.407+0000 7feb1ffff640 1 --2- 172.21.15.76:0/559366553 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56267b0a95c0 0x56267b0ab9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.408+0000 7feb06ffd640 10 client.26207 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.408+0000 7feb1ffff640 1 --2- 172.21.15.76:0/559366553 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56267b0a95c0 0x56267b0ab9b0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6addd37c67862c3 server_cookie=355979e935407405 in_seq=0 out_seq=0 2024-03-18T14:59:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.410+0000 7feb06ffd640 1 -- 172.21.15.76:0/559366553 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56267affc2d0 con 0x56267b0a95c0 2024-03-18T14:59:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.410+0000 7feb06ffd640 10 client.26207 handle_command_reply: tid=0 2024-03-18T14:59:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.410+0000 7feb3063f740 1 client.26207 shutdown 2024-03-18T14:59:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.410+0000 7feb3063f740 1 -- 172.21.15.76:0/559366553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090130 msgr2=0x56267b0968c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.410+0000 7feb3063f740 1 --2- 172.21.15.76:0/559366553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090130 0x56267b0968c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7feb180ee250 tx=0x7feb180bcbc0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.410+0000 7feb3063f740 1 -- 172.21.15.76:0/559366553 shutdown_connections 2024-03-18T14:59:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.410+0000 7feb3063f740 1 --2- 172.21.15.76:0/559366553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b090a50 0x56267b096e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.410+0000 7feb3063f740 1 --2- 172.21.15.76:0/559366553 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56267b0a95c0 0x56267b0ab9b0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.410+0000 7feb3063f740 1 --2- 172.21.15.76:0/559366553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b099820 0x56267b097340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.410+0000 7feb3063f740 1 --2- 172.21.15.76:0/559366553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090130 0x56267b0968c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.410+0000 7feb06ffd640 0 client.26207 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.410+0000 7feb3063f740 1 -- 172.21.15.76:0/559366553 >> 172.21.15.76:0/559366553 conn(0x56267b08aa50 msgr2=0x56267b08c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.410+0000 7feb3063f740 1 -- 172.21.15.76:0/559366553 shutdown_connections 2024-03-18T14:59:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.410+0000 7feb3063f740 1 -- 172.21.15.76:0/559366553 wait complete. 2024-03-18T14:59:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.410+0000 7feb3063f740 20 client.26207 trim_cache size 0 max 16384 2024-03-18T14:59:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb3063f740 1 Processor -- start 2024-03-18T14:59:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb3063f740 1 -- start start 2024-03-18T14:59:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb3063f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b090130 0x56267b098ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb3063f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b090a50 0x56267b0a4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb3063f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b099820 0x56267b089a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb3063f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7feb281afbc0 con 0x56267b090130 2024-03-18T14:59:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb3063f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7feb281ad460 con 0x56267b099820 2024-03-18T14:59:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb3063f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feb281ab2b0 con 0x56267b090a50 2024-03-18T14:59:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb26ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b090130 0x56267b098ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb26ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b090130 0x56267b098ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53674/0 (socket says 172.21.15.76:53674) 2024-03-18T14:59:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb26ffd640 1 -- 172.21.15.76:0/2037182254 learned_addr learned my addr 172.21.15.76:0/2037182254 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb277fe640 1 --2- 172.21.15.76:0/2037182254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b099820 0x56267b089a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb1ffff640 1 --2- 172.21.15.76:0/2037182254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b090a50 0x56267b0a4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb26ffd640 1 -- 172.21.15.76:0/2037182254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b090a50 msgr2=0x56267b0a4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb26ffd640 1 --2- 172.21.15.76:0/2037182254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b090a50 0x56267b0a4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb26ffd640 1 -- 172.21.15.76:0/2037182254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b099820 msgr2=0x56267b089a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb26ffd640 1 --2- 172.21.15.76:0/2037182254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b099820 0x56267b089a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb26ffd640 1 -- 172.21.15.76:0/2037182254 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56267b0abf90 con 0x56267b090130 2024-03-18T14:59:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb26ffd640 1 --2- 172.21.15.76:0/2037182254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b090130 0x56267b098ff0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7feb180b95c0 tx=0x7feb18003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702987baac7766b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb1d7fa640 1 -- 172.21.15.76:0/2037182254 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb180d7070 con 0x56267b090130 2024-03-18T14:59:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb1d7fa640 1 -- 172.21.15.76:0/2037182254 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb180c5850 con 0x56267b090130 2024-03-18T14:59:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.426+0000 7feb1d7fa640 1 -- 172.21.15.76:0/2037182254 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1804b070 con 0x56267b090130 2024-03-18T14:59:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.427+0000 7feb3063f740 1 -- 172.21.15.76:0/2037182254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b090130 msgr2=0x56267b098ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.427+0000 7feb3063f740 1 --2- 172.21.15.76:0/2037182254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b090130 0x56267b098ff0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7feb180b95c0 tx=0x7feb18003760 comp rx=0 tx=0).stop 2024-03-18T14:59:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.427+0000 7feb3063f740 1 -- 172.21.15.76:0/2037182254 shutdown_connections 2024-03-18T14:59:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.427+0000 7feb3063f740 1 --2- 172.21.15.76:0/2037182254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b090a50 0x56267b0a4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.427+0000 7feb3063f740 1 --2- 172.21.15.76:0/2037182254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b099820 0x56267b089a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.427+0000 7feb3063f740 1 --2- 172.21.15.76:0/2037182254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b090130 0x56267b098ff0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.427+0000 7feb3063f740 1 -- 172.21.15.76:0/2037182254 >> 172.21.15.76:0/2037182254 conn(0x56267b08aa50 msgr2=0x56267b089320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.427+0000 7feb3063f740 1 -- 172.21.15.76:0/2037182254 shutdown_connections 2024-03-18T14:59:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.427+0000 7feb3063f740 1 -- 172.21.15.76:0/2037182254 wait complete. 2024-03-18T14:59:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.427+0000 7feb3063f740 1 -- start start 2024-03-18T14:59:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb3063f740 1 Processor -- start 2024-03-18T14:59:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb3063f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b090130 0x56267b093610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb3063f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090a50 0x56267b093b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb3063f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b099820 0x56267b094090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb3063f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7feb281ab920 con 0x56267b090130 2024-03-18T14:59:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb3063f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7feb281acd90 con 0x56267b090a50 2024-03-18T14:59:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb3063f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feb281abff0 con 0x56267b099820 2024-03-18T14:59:29.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb3063f740 10 client.0 _ll_register_callbacks cb 0x7feb10009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:59:29.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb26ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b090130 0x56267b093610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb26ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b090130 0x56267b093610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53688/0 (socket says 172.21.15.76:53688) 2024-03-18T14:59:29.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb26ffd640 1 -- 172.21.15.76:0/3949878193 learned_addr learned my addr 172.21.15.76:0/3949878193 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:29.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb1ffff640 1 --2- 172.21.15.76:0/3949878193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090a50 0x56267b093b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb277fe640 1 --2- 172.21.15.76:0/3949878193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b099820 0x56267b094090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb26ffd640 1 -- 172.21.15.76:0/3949878193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b099820 msgr2=0x56267b094090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:29.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb26ffd640 1 --2- 172.21.15.76:0/3949878193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b099820 0x56267b094090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb26ffd640 1 -- 172.21.15.76:0/3949878193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090a50 msgr2=0x56267b093b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:29.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb26ffd640 1 --2- 172.21.15.76:0/3949878193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090a50 0x56267b093b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb26ffd640 1 -- 172.21.15.76:0/3949878193 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56267b0a5460 con 0x56267b090130 2024-03-18T14:59:29.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb26ffd640 1 --2- 172.21.15.76:0/3949878193 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b090130 0x56267b093610 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7feb18006010 tx=0x7feb180c5730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710614fe4882093f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:29.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb06ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:59:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb06ffd640 1 -- 172.21.15.76:0/3949878193 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1804b070 con 0x56267b090130 2024-03-18T14:59:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb06ffd640 1 -- 172.21.15.76:0/3949878193 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7feb180bc810 con 0x56267b090130 2024-03-18T14:59:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.428+0000 7feb3063f740 1 -- 172.21.15.76:0/3949878193 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb180c5850 con 0x56267b090130 2024-03-18T14:59:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.429+0000 7feb06ffd640 1 -- 172.21.15.76:0/3949878193 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7feb180c9030 con 0x56267b090130 2024-03-18T14:59:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.429+0000 7feb3063f740 10 client.16353 fetch_fsmap learned FSMap version 40 2024-03-18T14:59:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.429+0000 7feb3063f740 1 -- 172.21.15.76:0/3949878193 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56267b0a5460 con 0x56267b090130 2024-03-18T14:59:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.429+0000 7feb06ffd640 1 -- 172.21.15.76:0/3949878193 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb180c5850 con 0x56267b090130 2024-03-18T14:59:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.429+0000 7feb06ffd640 1 -- 172.21.15.76:0/3949878193 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 40) ==== 8342+0+0 (secure 0 0 0) 0x7feb180cc070 con 0x56267b090130 2024-03-18T14:59:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.429+0000 7feb3063f740 10 client.16353 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T14:59:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.429+0000 7feb3063f740 10 client.16353 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:59:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.429+0000 7feb3063f740 20 client.16353 populate_metadata read hostname 'smithi076' 2024-03-18T14:59:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.429+0000 7feb3063f740 1 --2- 172.21.15.76:0/3949878193 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56267b10fe50 0x56267b112240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.429+0000 7feb3063f740 4 client.16353 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:59:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.429+0000 7feb3063f740 1 -- 172.21.15.76:0/3949878193 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56267af7a770 con 0x56267b10fe50 2024-03-18T14:59:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.429+0000 7feb1ffff640 1 --2- 172.21.15.76:0/3949878193 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56267b10fe50 0x56267b112240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.429+0000 7feb1ffff640 1 --2- 172.21.15.76:0/3949878193 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56267b10fe50 0x56267b112240 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7cdc9f36fd7cb35 server_cookie=55b30f2ae686c1ef in_seq=0 out_seq=0 2024-03-18T14:59:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.429+0000 7feb06ffd640 10 client.16353 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:29.509 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.507+0000 7feb06ffd640 1 -- 172.21.15.76:0/3949878193 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56267af7a770 con 0x56267b10fe50 2024-03-18T14:59:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.507+0000 7feb06ffd640 10 client.16353 handle_command_reply: tid=0 2024-03-18T14:59:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.507+0000 7feb3063f740 1 client.16353 shutdown 2024-03-18T14:59:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.507+0000 7feb3063f740 1 -- 172.21.15.76:0/3949878193 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b090130 msgr2=0x56267b093610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.507+0000 7feb3063f740 1 --2- 172.21.15.76:0/3949878193 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b090130 0x56267b093610 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7feb18006010 tx=0x7feb180c5730 comp rx=0 tx=0).stop 2024-03-18T14:59:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb3063f740 1 -- 172.21.15.76:0/3949878193 shutdown_connections 2024-03-18T14:59:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb3063f740 1 --2- 172.21.15.76:0/3949878193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56267b099820 0x56267b094090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb3063f740 1 --2- 172.21.15.76:0/3949878193 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56267b10fe50 0x56267b112240 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb3063f740 1 --2- 172.21.15.76:0/3949878193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56267b090a50 0x56267b093b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb3063f740 1 --2- 172.21.15.76:0/3949878193 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56267b090130 0x56267b093610 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb06ffd640 0 client.16353 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb3063f740 1 -- 172.21.15.76:0/3949878193 >> 172.21.15.76:0/3949878193 conn(0x56267b08aa50 msgr2=0x56267b08c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb3063f740 1 -- 172.21.15.76:0/3949878193 shutdown_connections 2024-03-18T14:59:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb3063f740 1 -- 172.21.15.76:0/3949878193 wait complete. 2024-03-18T14:59:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb3063f740 20 client.16353 trim_cache size 0 max 16384 2024-03-18T14:59:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:59:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:59:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:59:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:59:29.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb2da52640 1 -- 172.21.15.76:0/331051673 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7feb140a2a10 msgr2=0x7feb140a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:29.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb2da52640 1 --2- 172.21.15.76:0/331051673 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7feb140a2a10 0x7feb140a4ed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7feb20005fd0 tx=0x7feb20013b60 comp rx=0 tx=0).stop 2024-03-18T14:59:29.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb2da52640 1 -- 172.21.15.76:0/331051673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb28068e60 msgr2=0x7feb28197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:29.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb2da52640 1 --2- 172.21.15.76:0/331051673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb28068e60 0x7feb28197fe0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7feb18009230 tx=0x7feb18004270 comp rx=0 tx=0).stop 2024-03-18T14:59:29.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb2da52640 1 -- 172.21.15.76:0/331051673 shutdown_connections 2024-03-18T14:59:29.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb2da52640 1 --2- 172.21.15.76:0/331051673 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7feb140a2a10 0x7feb140a4ed0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb2da52640 1 --2- 172.21.15.76:0/331051673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb28069800 0x7feb28198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb2da52640 1 --2- 172.21.15.76:0/331051673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb281078b0 0x7feb2819c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb2da52640 1 --2- 172.21.15.76:0/331051673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb28068e60 0x7feb28197fe0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:29.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.508+0000 7feb2da52640 1 -- 172.21.15.76:0/331051673 >> 172.21.15.76:0/331051673 conn(0x7feb28077530 msgr2=0x7feb280fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:29.513 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.509+0000 7feb2da52640 1 -- 172.21.15.76:0/331051673 shutdown_connections 2024-03-18T14:59:29.513 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:29.509+0000 7feb2da52640 1 -- 172.21.15.76:0/331051673 wait complete. 2024-03-18T14:59:29.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 594233 9.52 MB/sec execute 210 sec latency 58.852 ms 2024-03-18T14:59:29.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:29 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:59:29.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:29 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:59:29.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:29 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:59:29.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:29 smithi077 ceph-mon[35865]: pgmap v195: 97 pgs: 97 active+clean; 191 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 359 op/s 2024-03-18T14:59:30.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:29 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:59:30.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:29 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:59:30.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:29 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:59:30.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:29 smithi076 ceph-mon[32205]: pgmap v195: 97 pgs: 97 active+clean; 191 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 359 op/s 2024-03-18T14:59:30.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:29 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T14:59:30.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:29 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T14:59:30.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:29 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T14:59:30.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:29 smithi174 ceph-mon[36787]: pgmap v195: 97 pgs: 97 active+clean; 191 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 359 op/s 2024-03-18T14:59:30.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 596788 9.55 MB/sec execute 211 sec latency 20.635 ms 2024-03-18T14:59:31.746 INFO:tasks.workunit.client.0.smithi076.stdout: 1 598121 9.53 MB/sec execute 212 sec latency 20.818 ms 2024-03-18T14:59:31.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:31 smithi077 ceph-mon[35865]: pgmap v196: 97 pgs: 97 active+clean; 191 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 209 op/s 2024-03-18T14:59:32.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:31 smithi076 ceph-mon[32205]: pgmap v196: 97 pgs: 97 active+clean; 191 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 209 op/s 2024-03-18T14:59:32.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:31 smithi174 ceph-mon[36787]: pgmap v196: 97 pgs: 97 active+clean; 191 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 209 op/s 2024-03-18T14:59:32.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 600469 9.55 MB/sec execute 213 sec latency 26.980 ms 2024-03-18T14:59:33.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 601680 9.53 MB/sec execute 214 sec latency 29.530 ms 2024-03-18T14:59:33.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:33 smithi077 ceph-mon[35865]: pgmap v197: 97 pgs: 97 active+clean; 184 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 340 op/s 2024-03-18T14:59:34.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:33 smithi076 ceph-mon[32205]: pgmap v197: 97 pgs: 97 active+clean; 184 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 340 op/s 2024-03-18T14:59:34.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:33 smithi174 ceph-mon[36787]: pgmap v197: 97 pgs: 97 active+clean; 184 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 340 op/s 2024-03-18T14:59:34.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 604022 9.55 MB/sec execute 215 sec latency 16.789 ms 2024-03-18T14:59:35.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 605471 9.53 MB/sec execute 216 sec latency 20.709 ms 2024-03-18T14:59:35.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:35 smithi077 ceph-mon[35865]: pgmap v198: 97 pgs: 97 active+clean; 184 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-03-18T14:59:36.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:35 smithi076 ceph-mon[32205]: pgmap v198: 97 pgs: 97 active+clean; 184 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-03-18T14:59:36.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:35 smithi174 ceph-mon[36787]: pgmap v198: 97 pgs: 97 active+clean; 184 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-03-18T14:59:36.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 607709 9.55 MB/sec execute 217 sec latency 18.832 ms 2024-03-18T14:59:37.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 609259 9.54 MB/sec execute 218 sec latency 35.896 ms 2024-03-18T14:59:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:37 smithi077 ceph-mon[35865]: pgmap v199: 97 pgs: 97 active+clean; 184 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-03-18T14:59:38.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:37 smithi076 ceph-mon[32205]: pgmap v199: 97 pgs: 97 active+clean; 184 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-03-18T14:59:38.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:37 smithi174 ceph-mon[36787]: pgmap v199: 97 pgs: 97 active+clean; 184 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-03-18T14:59:38.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 611267 9.55 MB/sec execute 219 sec latency 50.637 ms 2024-03-18T14:59:39.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 612755 9.54 MB/sec execute 220 sec latency 19.906 ms 2024-03-18T14:59:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:39 smithi077 ceph-mon[35865]: pgmap v200: 97 pgs: 97 active+clean; 197 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 344 op/s 2024-03-18T14:59:40.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:39 smithi076 ceph-mon[32205]: pgmap v200: 97 pgs: 97 active+clean; 197 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 344 op/s 2024-03-18T14:59:40.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:39 smithi174 ceph-mon[36787]: pgmap v200: 97 pgs: 97 active+clean; 197 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 344 op/s 2024-03-18T14:59:40.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 614826 9.55 MB/sec execute 221 sec latency 14.188 ms 2024-03-18T14:59:41.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 616417 9.54 MB/sec execute 222 sec latency 23.494 ms 2024-03-18T14:59:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:41 smithi077 ceph-mon[35865]: pgmap v201: 97 pgs: 97 active+clean; 197 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 235 op/s 2024-03-18T14:59:42.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:41 smithi076 ceph-mon[32205]: pgmap v201: 97 pgs: 97 active+clean; 197 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 235 op/s 2024-03-18T14:59:42.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:41 smithi174 ceph-mon[36787]: pgmap v201: 97 pgs: 97 active+clean; 197 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 235 op/s 2024-03-18T14:59:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.602+0000 7fe83d5df640 1 -- 172.21.15.76:0/1117376554 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe838105660 msgr2=0x7fe838105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.602+0000 7fe83d5df640 1 --2- 172.21.15.76:0/1117376554 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe838105660 0x7fe838105a40 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fe828009900 tx=0x7fe82802bc30 comp rx=0 tx=0).stop 2024-03-18T14:59:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.602+0000 7fe83d5df640 1 -- 172.21.15.76:0/1117376554 shutdown_connections 2024-03-18T14:59:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.602+0000 7fe83d5df640 1 --2- 172.21.15.76:0/1117376554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8381012d0 0x7fe83810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.602+0000 7fe83d5df640 1 --2- 172.21.15.76:0/1117376554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe838100930 0x7fe838100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.602+0000 7fe83d5df640 1 --2- 172.21.15.76:0/1117376554 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe838105660 0x7fe838105a40 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.602+0000 7fe83d5df640 1 -- 172.21.15.76:0/1117376554 >> 172.21.15.76:0/1117376554 conn(0x7fe8380774b0 msgr2=0x7fe8380778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.602+0000 7fe83d5df640 1 -- 172.21.15.76:0/1117376554 shutdown_connections 2024-03-18T14:59:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.602+0000 7fe83d5df640 1 -- 172.21.15.76:0/1117376554 wait complete. 2024-03-18T14:59:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.603+0000 7fe83d5df640 1 Processor -- start 2024-03-18T14:59:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.603+0000 7fe83d5df640 1 -- start start 2024-03-18T14:59:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.603+0000 7fe83d5df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe838100930 0x7fe838196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.603+0000 7fe83d5df640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8381012d0 0x7fe8381971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.603+0000 7fe83d5df640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe838105660 0x7fe838197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.603+0000 7fe83d5df640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe838110b70 con 0x7fe838105660 2024-03-18T14:59:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.603+0000 7fe83d5df640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe8381109f0 con 0x7fe838100930 2024-03-18T14:59:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.603+0000 7fe83d5df640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe838110cf0 con 0x7fe8381012d0 2024-03-18T14:59:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.603+0000 7fe8377fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe838105660 0x7fe838197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.603+0000 7fe8377fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe838105660 0x7fe838197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48798/0 (socket says 172.21.15.76:48798) 2024-03-18T14:59:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.603+0000 7fe8377fe640 1 -- 172.21.15.76:0/2835364186 learned_addr learned my addr 172.21.15.76:0/2835364186 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.603+0000 7fe836ffd640 1 --2- 172.21.15.76:0/2835364186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe838100930 0x7fe838196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.603+0000 7fe8367fc640 1 --2- 172.21.15.76:0/2835364186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8381012d0 0x7fe8381971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.604+0000 7fe8377fe640 1 -- 172.21.15.76:0/2835364186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8381012d0 msgr2=0x7fe8381971c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.604+0000 7fe8377fe640 1 --2- 172.21.15.76:0/2835364186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8381012d0 0x7fe8381971c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.604+0000 7fe8377fe640 1 -- 172.21.15.76:0/2835364186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe838100930 msgr2=0x7fe838196c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.604+0000 7fe8377fe640 1 --2- 172.21.15.76:0/2835364186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe838100930 0x7fe838196c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.604+0000 7fe8377fe640 1 -- 172.21.15.76:0/2835364186 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8381a64f0 con 0x7fe838105660 2024-03-18T14:59:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.604+0000 7fe8377fe640 1 --2- 172.21.15.76:0/2835364186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe838105660 0x7fe838197700 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fe83000eb10 tx=0x7fe83000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.604+0000 7fe81ffff640 1 -- 172.21.15.76:0/2835364186 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe830004b20 con 0x7fe838105660 2024-03-18T14:59:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.604+0000 7fe83d5df640 1 -- 172.21.15.76:0/2835364186 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8381a67e0 con 0x7fe838105660 2024-03-18T14:59:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.604+0000 7fe81ffff640 1 -- 172.21.15.76:0/2835364186 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe830004cc0 con 0x7fe838105660 2024-03-18T14:59:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.604+0000 7fe81ffff640 1 -- 172.21.15.76:0/2835364186 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe830010760 con 0x7fe838105660 2024-03-18T14:59:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.604+0000 7fe83d5df640 1 -- 172.21.15.76:0/2835364186 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8381a6d30 con 0x7fe838105660 2024-03-18T14:59:42.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.606+0000 7fe81ffff640 1 -- 172.21.15.76:0/2835364186 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fe830005020 con 0x7fe838105660 2024-03-18T14:59:42.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe8401c9740 1 Processor -- start 2024-03-18T14:59:42.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe8401c9740 1 -- start start 2024-03-18T14:59:42.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe8401c9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32f9c130 0x55ed32f9c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe8401c9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32f9ca50 0x55ed32fa52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe81ffff640 1 --2- 172.21.15.76:0/2835364186 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe8200a2a10 0x7fe8200a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe8401c9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32fa5820 0x55ed32fa7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe8401c9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ed32f00a40 con 0x55ed32f9c130 2024-03-18T14:59:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe8401c9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ed32ea9920 con 0x55ed32fa5820 2024-03-18T14:59:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe8401c9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ed32efcae0 con 0x55ed32f9ca50 2024-03-18T14:59:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe8367fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32f9c130 0x55ed32f9c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe836ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32f9ca50 0x55ed32fa52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe8377fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32fa5820 0x55ed32fa7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe8367fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32f9c130 0x55ed32f9c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48812/0 (socket says 172.21.15.76:48812) 2024-03-18T14:59:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe836ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32f9ca50 0x55ed32fa52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:39484/0 (socket says 172.21.15.76:39484) 2024-03-18T14:59:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe8367fc640 1 -- 172.21.15.76:0/4278415243 learned_addr learned my addr 172.21.15.76:0/4278415243 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe8377fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32fa5820 0x55ed32fa7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:36016/0 (socket says 172.21.15.76:36016) 2024-03-18T14:59:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe8367fc640 1 --2- 172.21.15.76:0/2835364186 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe8200a2a10 0x7fe8200a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.607+0000 7fe81ffff640 1 -- 172.21.15.76:0/2835364186 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7fe8300cf560 con 0x7fe838105660 2024-03-18T14:59:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.608+0000 7fe8377fe640 1 -- 172.21.15.76:0/4278415243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32f9ca50 msgr2=0x55ed32fa52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.608+0000 7fe8377fe640 1 --2- 172.21.15.76:0/4278415243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32f9ca50 0x55ed32fa52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.608+0000 7fe8377fe640 1 -- 172.21.15.76:0/4278415243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32f9c130 msgr2=0x55ed32f9c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.608+0000 7fe8377fe640 1 --2- 172.21.15.76:0/4278415243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32f9c130 0x55ed32f9c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.608+0000 7fe8377fe640 1 -- 172.21.15.76:0/4278415243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed32ec1460 con 0x55ed32fa5820 2024-03-18T14:59:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.608+0000 7fe8377fe640 1 --2- 172.21.15.76:0/4278415243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32fa5820 0x55ed32fa7c10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe8300ccca0 tx=0x7fe8300058d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7f43aa2e69dca1f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.608+0000 7fe8367fc640 1 --2- 172.21.15.76:0/2835364186 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe8200a2a10 0x7fe8200a4ed0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe82c009fd0 tx=0x7fe82c005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.608+0000 7fe81d7fa640 1 -- 172.21.15.76:0/4278415243 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83009b980 con 0x55ed32fa5820 2024-03-18T14:59:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.608+0000 7fe81d7fa640 1 -- 172.21.15.76:0/4278415243 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe83009c2b0 con 0x55ed32fa5820 2024-03-18T14:59:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.608+0000 7fe81d7fa640 1 -- 172.21.15.76:0/4278415243 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe830099200 con 0x55ed32fa5820 2024-03-18T14:59:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 1 -- 172.21.15.76:0/4278415243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32fa5820 msgr2=0x55ed32fa7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 1 --2- 172.21.15.76:0/4278415243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32fa5820 0x55ed32fa7c10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe8300ccca0 tx=0x7fe8300058d0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 1 -- 172.21.15.76:0/4278415243 shutdown_connections 2024-03-18T14:59:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 1 --2- 172.21.15.76:0/4278415243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32f9ca50 0x55ed32fa52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 1 --2- 172.21.15.76:0/4278415243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32fa5820 0x55ed32fa7c10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 1 --2- 172.21.15.76:0/4278415243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32f9c130 0x55ed32f9c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 1 -- 172.21.15.76:0/4278415243 >> 172.21.15.76:0/4278415243 conn(0x55ed32f96a50 msgr2=0x55ed32f96e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 1 -- 172.21.15.76:0/4278415243 shutdown_connections 2024-03-18T14:59:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 1 -- 172.21.15.76:0/4278415243 wait complete. 2024-03-18T14:59:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 1 -- start start 2024-03-18T14:59:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 1 Processor -- start 2024-03-18T14:59:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32f9c130 0x55ed32fa2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 0x55ed32fa2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32fa5820 0x55ed32fa32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ed32fa8450 con 0x55ed32f9c130 2024-03-18T14:59:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ed32ebcd70 con 0x55ed32f9ca50 2024-03-18T14:59:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ed32fa85d0 con 0x55ed32fa5820 2024-03-18T14:59:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe8401c9740 10 client.0 _ll_register_callbacks cb 0x55ed32c14a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:59:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.609+0000 7fe836ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32f9c130 0x55ed32fa2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe8367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 0x55ed32fa2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe836ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32f9c130 0x55ed32fa2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48822/0 (socket says 172.21.15.76:48822) 2024-03-18T14:59:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe836ffd640 1 -- 172.21.15.76:0/3553623850 learned_addr learned my addr 172.21.15.76:0/3553623850 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe8377fe640 1 --2- 172.21.15.76:0/3553623850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32fa5820 0x55ed32fa32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe836ffd640 1 -- 172.21.15.76:0/3553623850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32fa5820 msgr2=0x55ed32fa32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe836ffd640 1 --2- 172.21.15.76:0/3553623850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32fa5820 0x55ed32fa32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe836ffd640 1 -- 172.21.15.76:0/3553623850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 msgr2=0x55ed32fa2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe836ffd640 1 --2- 172.21.15.76:0/3553623850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 0x55ed32fa2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe836ffd640 1 -- 172.21.15.76:0/3553623850 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed32c15210 con 0x55ed32f9c130 2024-03-18T14:59:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe8367fc640 1 --2- 172.21.15.76:0/3553623850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 0x55ed32fa2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:59:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe836ffd640 1 --2- 172.21.15.76:0/3553623850 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32f9c130 0x55ed32fa2870 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fe82802c170 tx=0x7fe82802ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4dbb98037742df2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe81aff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:59:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe81aff5640 1 -- 172.21.15.76:0/3553623850 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8280046c0 con 0x55ed32f9c130 2024-03-18T14:59:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe81aff5640 1 -- 172.21.15.76:0/3553623850 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7fe828004860 con 0x55ed32f9c130 2024-03-18T14:59:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe8401c9740 1 -- 172.21.15.76:0/3553623850 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe83009c2b0 con 0x55ed32f9c130 2024-03-18T14:59:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe81aff5640 1 -- 172.21.15.76:0/3553623850 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe828004a60 con 0x55ed32f9c130 2024-03-18T14:59:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe8401c9740 10 client.16365 fetch_fsmap learned FSMap version 40 2024-03-18T14:59:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe8401c9740 1 -- 172.21.15.76:0/3553623850 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ed32ce1620 con 0x55ed32f9c130 2024-03-18T14:59:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.610+0000 7fe81aff5640 1 -- 172.21.15.76:0/3553623850 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe83009c2b0 con 0x55ed32f9c130 2024-03-18T14:59:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.611+0000 7fe81aff5640 1 -- 172.21.15.76:0/3553623850 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 40) ==== 8342+0+0 (secure 0 0 0) 0x7fe828003850 con 0x55ed32f9c130 2024-03-18T14:59:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.611+0000 7fe8401c9740 10 client.16365 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T14:59:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.611+0000 7fe8401c9740 10 client.16365 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:59:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.611+0000 7fe8401c9740 20 client.16365 populate_metadata read hostname 'smithi076' 2024-03-18T14:59:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.611+0000 7fe8401c9740 1 --2- 172.21.15.76:0/3553623850 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ed32fb5970 0x55ed32fb7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.611+0000 7fe8401c9740 4 client.16365 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:59:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.611+0000 7fe8401c9740 1 -- 172.21.15.76:0/3553623850 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed32f0df80 con 0x55ed32fb5970 2024-03-18T14:59:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.611+0000 7fe8367fc640 1 --2- 172.21.15.76:0/3553623850 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ed32fb5970 0x55ed32fb7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.611+0000 7fe8367fc640 1 --2- 172.21.15.76:0/3553623850 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ed32fb5970 0x55ed32fb7e00 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e9c13c3373d2f40 server_cookie=2ce65e3f5cd26e36 in_seq=0 out_seq=0 2024-03-18T14:59:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.611+0000 7fe81aff5640 10 client.16365 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.612+0000 7fe81aff5640 1 -- 172.21.15.76:0/3553623850 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ed32f0df80 con 0x55ed32fb5970 2024-03-18T14:59:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.612+0000 7fe81aff5640 10 client.16365 handle_command_reply: tid=0 2024-03-18T14:59:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.612+0000 7fe8401c9740 1 client.16365 shutdown 2024-03-18T14:59:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.612+0000 7fe8401c9740 1 -- 172.21.15.76:0/3553623850 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32f9c130 msgr2=0x55ed32fa2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.612+0000 7fe8401c9740 1 --2- 172.21.15.76:0/3553623850 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32f9c130 0x55ed32fa2870 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fe82802c170 tx=0x7fe82802ce30 comp rx=0 tx=0).stop 2024-03-18T14:59:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.612+0000 7fe8401c9740 1 -- 172.21.15.76:0/3553623850 shutdown_connections 2024-03-18T14:59:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.612+0000 7fe8401c9740 1 --2- 172.21.15.76:0/3553623850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32fa5820 0x55ed32fa32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.612+0000 7fe8401c9740 1 --2- 172.21.15.76:0/3553623850 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ed32fb5970 0x55ed32fb7e00 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.612+0000 7fe8401c9740 1 --2- 172.21.15.76:0/3553623850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 0x55ed32fa2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.612+0000 7fe8401c9740 1 --2- 172.21.15.76:0/3553623850 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32f9c130 0x55ed32fa2870 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.612+0000 7fe81aff5640 0 client.16365 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.612+0000 7fe8401c9740 1 -- 172.21.15.76:0/3553623850 >> 172.21.15.76:0/3553623850 conn(0x55ed32f96a50 msgr2=0x55ed32f987a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.613+0000 7fe8401c9740 1 -- 172.21.15.76:0/3553623850 shutdown_connections 2024-03-18T14:59:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.613+0000 7fe8401c9740 1 -- 172.21.15.76:0/3553623850 wait complete. 2024-03-18T14:59:42.621 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.613+0000 7fe8401c9740 20 client.16365 trim_cache size 0 max 16384 2024-03-18T14:59:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.629+0000 7fe8401c9740 1 Processor -- start 2024-03-18T14:59:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.629+0000 7fe8401c9740 1 -- start start 2024-03-18T14:59:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.629+0000 7fe8401c9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32f9c130 0x55ed32fa4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.629+0000 7fe8401c9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 0x55ed32fb11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe8401c9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32fa5820 0x55ed32fbd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe8401c9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe8381b1e50 con 0x55ed32fa5820 2024-03-18T14:59:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe8401c9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe8381af6f0 con 0x55ed32f9ca50 2024-03-18T14:59:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe8401c9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe8381ad540 con 0x55ed32f9c130 2024-03-18T14:59:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe8377fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32fa5820 0x55ed32fbd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe836ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32f9c130 0x55ed32fa4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe8367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 0x55ed32fb11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe8377fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32fa5820 0x55ed32fbd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48834/0 (socket says 172.21.15.76:48834) 2024-03-18T14:59:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe8367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 0x55ed32fb11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:36028/0 (socket says 172.21.15.76:36028) 2024-03-18T14:59:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe8377fe640 1 -- 172.21.15.76:0/2653061156 learned_addr learned my addr 172.21.15.76:0/2653061156 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe836ffd640 1 -- 172.21.15.76:0/2653061156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 msgr2=0x55ed32fb11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe836ffd640 1 --2- 172.21.15.76:0/2653061156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 0x55ed32fb11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe836ffd640 1 -- 172.21.15.76:0/2653061156 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32fa5820 msgr2=0x55ed32fbd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe836ffd640 1 --2- 172.21.15.76:0/2653061156 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32fa5820 0x55ed32fbd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe836ffd640 1 -- 172.21.15.76:0/2653061156 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed32fb8480 con 0x55ed32f9c130 2024-03-18T14:59:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe8377fe640 1 --2- 172.21.15.76:0/2653061156 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32fa5820 0x55ed32fbd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:59:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe836ffd640 1 --2- 172.21.15.76:0/2653061156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32f9c130 0x55ed32fa4fa0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe828009850 tx=0x7fe82802bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e663a161946aed7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe81cff9640 1 -- 172.21.15.76:0/2653061156 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe828057070 con 0x55ed32f9c130 2024-03-18T14:59:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe81cff9640 1 -- 172.21.15.76:0/2653061156 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe828043070 con 0x55ed32f9c130 2024-03-18T14:59:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.630+0000 7fe81cff9640 1 -- 172.21.15.76:0/2653061156 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe828052570 con 0x55ed32f9c130 2024-03-18T14:59:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.631+0000 7fe8401c9740 1 -- 172.21.15.76:0/2653061156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32f9c130 msgr2=0x55ed32fa4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.631+0000 7fe8401c9740 1 --2- 172.21.15.76:0/2653061156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32f9c130 0x55ed32fa4fa0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe828009850 tx=0x7fe82802bc30 comp rx=0 tx=0).stop 2024-03-18T14:59:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.631+0000 7fe8401c9740 1 -- 172.21.15.76:0/2653061156 shutdown_connections 2024-03-18T14:59:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.631+0000 7fe8401c9740 1 --2- 172.21.15.76:0/2653061156 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32fa5820 0x55ed32fbd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.631+0000 7fe8401c9740 1 --2- 172.21.15.76:0/2653061156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 0x55ed32fb11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.631+0000 7fe8401c9740 1 --2- 172.21.15.76:0/2653061156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32f9c130 0x55ed32fa4fa0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.631+0000 7fe8401c9740 1 -- 172.21.15.76:0/2653061156 >> 172.21.15.76:0/2653061156 conn(0x55ed32f96a50 msgr2=0x55ed32f91a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.631+0000 7fe8401c9740 1 -- 172.21.15.76:0/2653061156 shutdown_connections 2024-03-18T14:59:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.631+0000 7fe8401c9740 1 -- 172.21.15.76:0/2653061156 wait complete. 2024-03-18T14:59:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.631+0000 7fe8401c9740 1 -- start start 2024-03-18T14:59:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8401c9740 1 Processor -- start 2024-03-18T14:59:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8401c9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32f9c130 0x55ed32f9fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8401c9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 0x55ed32fa0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8401c9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32fa5820 0x55ed33014d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8401c9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe8381adbb0 con 0x55ed32fa5820 2024-03-18T14:59:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8401c9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe8381af020 con 0x55ed32f9ca50 2024-03-18T14:59:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8401c9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe8381ae280 con 0x55ed32f9c130 2024-03-18T14:59:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8401c9740 10 client.0 _ll_register_callbacks cb 0x7fe828009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:59:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8377fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32fa5820 0x55ed33014d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 0x55ed32fa0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8377fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32fa5820 0x55ed33014d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48840/0 (socket says 172.21.15.76:48840) 2024-03-18T14:59:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 0x55ed32fa0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:36030/0 (socket says 172.21.15.76:36030) 2024-03-18T14:59:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8377fe640 1 -- 172.21.15.76:0/153108436 learned_addr learned my addr 172.21.15.76:0/153108436 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe836ffd640 1 --2- 172.21.15.76:0/153108436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32f9c130 0x55ed32f9fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8377fe640 1 -- 172.21.15.76:0/153108436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32f9c130 msgr2=0x55ed32f9fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8377fe640 1 --2- 172.21.15.76:0/153108436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32f9c130 0x55ed32f9fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8377fe640 1 -- 172.21.15.76:0/153108436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 msgr2=0x55ed32fa0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8377fe640 1 --2- 172.21.15.76:0/153108436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 0x55ed32fa0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8377fe640 1 -- 172.21.15.76:0/153108436 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed32f9d090 con 0x55ed32fa5820 2024-03-18T14:59:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe8377fe640 1 --2- 172.21.15.76:0/153108436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32fa5820 0x55ed33014d10 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fe830099e60 tx=0x7fe83009c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b62626942ccccbd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe81aff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:59:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe81aff5640 1 -- 172.21.15.76:0/153108436 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83009b510 con 0x55ed32fa5820 2024-03-18T14:59:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.632+0000 7fe81aff5640 1 -- 172.21.15.76:0/153108436 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7fe8300cf2e0 con 0x55ed32fa5820 2024-03-18T14:59:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.633+0000 7fe8401c9740 1 -- 172.21.15.76:0/153108436 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe828043070 con 0x55ed32fa5820 2024-03-18T14:59:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.633+0000 7fe81aff5640 1 -- 172.21.15.76:0/153108436 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe83009b6f0 con 0x55ed32fa5820 2024-03-18T14:59:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.633+0000 7fe8401c9740 10 client.16371 fetch_fsmap learned FSMap version 40 2024-03-18T14:59:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.633+0000 7fe8401c9740 1 -- 172.21.15.76:0/153108436 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ed3301c480 con 0x55ed32fa5820 2024-03-18T14:59:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.633+0000 7fe81aff5640 1 -- 172.21.15.76:0/153108436 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe828043070 con 0x55ed32fa5820 2024-03-18T14:59:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.633+0000 7fe81aff5640 1 -- 172.21.15.76:0/153108436 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 40) ==== 8342+0+0 (secure 0 0 0) 0x7fe8300b8070 con 0x55ed32fa5820 2024-03-18T14:59:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.633+0000 7fe8401c9740 10 client.16371 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T14:59:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.633+0000 7fe8401c9740 10 client.16371 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:59:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.633+0000 7fe8401c9740 20 client.16371 populate_metadata read hostname 'smithi076' 2024-03-18T14:59:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.633+0000 7fe8401c9740 1 --2- 172.21.15.76:0/153108436 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ed3301c710 0x55ed3301eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.633+0000 7fe8401c9740 4 client.16371 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:59:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.633+0000 7fe8401c9740 1 -- 172.21.15.76:0/153108436 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ed32f082d0 con 0x55ed3301c710 2024-03-18T14:59:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.633+0000 7fe836ffd640 1 --2- 172.21.15.76:0/153108436 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ed3301c710 0x55ed3301eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.633+0000 7fe836ffd640 1 --2- 172.21.15.76:0/153108436 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ed3301c710 0x55ed3301eb00 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7486f569be99ee6a server_cookie=48a1ef575c4a053f in_seq=0 out_seq=0 2024-03-18T14:59:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.633+0000 7fe81aff5640 10 client.16371 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.705+0000 7fe81aff5640 1 -- 172.21.15.76:0/153108436 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ed32f082d0 con 0x55ed3301c710 2024-03-18T14:59:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.705+0000 7fe81aff5640 10 client.16371 handle_command_reply: tid=0 2024-03-18T14:59:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.705+0000 7fe8401c9740 1 client.16371 shutdown 2024-03-18T14:59:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.705+0000 7fe8401c9740 1 -- 172.21.15.76:0/153108436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32fa5820 msgr2=0x55ed33014d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.705+0000 7fe8401c9740 1 --2- 172.21.15.76:0/153108436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32fa5820 0x55ed33014d10 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fe830099e60 tx=0x7fe83009c330 comp rx=0 tx=0).stop 2024-03-18T14:59:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.705+0000 7fe8401c9740 1 -- 172.21.15.76:0/153108436 shutdown_connections 2024-03-18T14:59:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.705+0000 7fe8401c9740 1 --2- 172.21.15.76:0/153108436 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ed3301c710 0x55ed3301eb00 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.705+0000 7fe81aff5640 0 client.16371 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.705+0000 7fe8401c9740 1 --2- 172.21.15.76:0/153108436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed32fa5820 0x55ed33014d10 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.705+0000 7fe8401c9740 1 --2- 172.21.15.76:0/153108436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ed32f9ca50 0x55ed32fa0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.705+0000 7fe8401c9740 1 --2- 172.21.15.76:0/153108436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed32f9c130 0x55ed32f9fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.705+0000 7fe8401c9740 1 -- 172.21.15.76:0/153108436 >> 172.21.15.76:0/153108436 conn(0x55ed32f96a50 msgr2=0x55ed32fa2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.705+0000 7fe8401c9740 1 -- 172.21.15.76:0/153108436 shutdown_connections 2024-03-18T14:59:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.705+0000 7fe8401c9740 1 -- 172.21.15.76:0/153108436 wait complete. 2024-03-18T14:59:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.705+0000 7fe8401c9740 20 client.16371 trim_cache size 0 max 16384 2024-03-18T14:59:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:59:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:59:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:59:42.708 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:59:42.708 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.706+0000 7fe83d5df640 1 -- 172.21.15.76:0/2835364186 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe8200a2a10 msgr2=0x7fe8200a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:42.708 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.706+0000 7fe83d5df640 1 --2- 172.21.15.76:0/2835364186 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe8200a2a10 0x7fe8200a4ed0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe82c009fd0 tx=0x7fe82c005f90 comp rx=0 tx=0).stop 2024-03-18T14:59:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.706+0000 7fe83d5df640 1 -- 172.21.15.76:0/2835364186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe838105660 msgr2=0x7fe838197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.706+0000 7fe83d5df640 1 --2- 172.21.15.76:0/2835364186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe838105660 0x7fe838197700 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fe83000eb10 tx=0x7fe83000c2d0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.706+0000 7fe83d5df640 1 -- 172.21.15.76:0/2835364186 shutdown_connections 2024-03-18T14:59:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.706+0000 7fe83d5df640 1 --2- 172.21.15.76:0/2835364186 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe8200a2a10 0x7fe8200a4ed0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.706+0000 7fe83d5df640 1 --2- 172.21.15.76:0/2835364186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8381012d0 0x7fe8381971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.706+0000 7fe83d5df640 1 --2- 172.21.15.76:0/2835364186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe838105660 0x7fe838197700 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.706+0000 7fe83d5df640 1 --2- 172.21.15.76:0/2835364186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe838100930 0x7fe838196c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.706+0000 7fe83d5df640 1 -- 172.21.15.76:0/2835364186 >> 172.21.15.76:0/2835364186 conn(0x7fe8380774b0 msgr2=0x7fe8380fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.706+0000 7fe83d5df640 1 -- 172.21.15.76:0/2835364186 shutdown_connections 2024-03-18T14:59:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:42.706+0000 7fe83d5df640 1 -- 172.21.15.76:0/2835364186 wait complete. 2024-03-18T14:59:42.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 618367 9.55 MB/sec execute 223 sec latency 57.623 ms 2024-03-18T14:59:43.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 619678 9.53 MB/sec execute 224 sec latency 41.304 ms 2024-03-18T14:59:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:43 smithi077 ceph-mon[35865]: pgmap v202: 97 pgs: 97 active+clean; 190 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 381 op/s 2024-03-18T14:59:44.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:43 smithi076 ceph-mon[32205]: pgmap v202: 97 pgs: 97 active+clean; 190 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 381 op/s 2024-03-18T14:59:44.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:43 smithi174 ceph-mon[36787]: pgmap v202: 97 pgs: 97 active+clean; 190 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 381 op/s 2024-03-18T14:59:44.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 621839 9.55 MB/sec execute 225 sec latency 45.291 ms 2024-03-18T14:59:45.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 623231 9.54 MB/sec execute 226 sec latency 55.273 ms 2024-03-18T14:59:45.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:45 smithi077 ceph-mon[35865]: pgmap v203: 97 pgs: 97 active+clean; 190 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 251 op/s 2024-03-18T14:59:46.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:45 smithi076 ceph-mon[32205]: pgmap v203: 97 pgs: 97 active+clean; 190 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 251 op/s 2024-03-18T14:59:46.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:45 smithi174 ceph-mon[36787]: pgmap v203: 97 pgs: 97 active+clean; 190 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 251 op/s 2024-03-18T14:59:46.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 625454 9.55 MB/sec execute 227 sec latency 15.341 ms 2024-03-18T14:59:47.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 626992 9.54 MB/sec execute 228 sec latency 24.241 ms 2024-03-18T14:59:47.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:47 smithi077 ceph-mon[35865]: pgmap v204: 97 pgs: 97 active+clean; 190 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 251 op/s 2024-03-18T14:59:48.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:47 smithi076 ceph-mon[32205]: pgmap v204: 97 pgs: 97 active+clean; 190 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 251 op/s 2024-03-18T14:59:48.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:47 smithi174 ceph-mon[36787]: pgmap v204: 97 pgs: 97 active+clean; 190 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 251 op/s 2024-03-18T14:59:48.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 629003 9.55 MB/sec execute 229 sec latency 21.653 ms 2024-03-18T14:59:49.747 INFO:tasks.workunit.client.0.smithi076.stdout: 1 630507 9.54 MB/sec execute 230 sec latency 21.232 ms 2024-03-18T14:59:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:49 smithi077 ceph-mon[35865]: pgmap v205: 97 pgs: 97 active+clean; 198 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 351 op/s 2024-03-18T14:59:50.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:49 smithi076 ceph-mon[32205]: pgmap v205: 97 pgs: 97 active+clean; 198 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 351 op/s 2024-03-18T14:59:50.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:49 smithi174 ceph-mon[36787]: pgmap v205: 97 pgs: 97 active+clean; 198 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 351 op/s 2024-03-18T14:59:50.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 632598 9.55 MB/sec execute 231 sec latency 14.320 ms 2024-03-18T14:59:51.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 634504 9.54 MB/sec execute 232 sec latency 22.079 ms 2024-03-18T14:59:51.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:51 smithi077 ceph-mon[35865]: pgmap v206: 97 pgs: 97 active+clean; 198 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 247 op/s 2024-03-18T14:59:52.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:51 smithi076 ceph-mon[32205]: pgmap v206: 97 pgs: 97 active+clean; 198 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 247 op/s 2024-03-18T14:59:52.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:51 smithi174 ceph-mon[36787]: pgmap v206: 97 pgs: 97 active+clean; 198 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 247 op/s 2024-03-18T14:59:52.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 636217 9.55 MB/sec execute 233 sec latency 12.777 ms 2024-03-18T14:59:53.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 637970 9.54 MB/sec execute 234 sec latency 30.175 ms 2024-03-18T14:59:53.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:53 smithi077 ceph-mon[35865]: pgmap v207: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.2 MiB/s wr, 359 op/s 2024-03-18T14:59:54.018 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:53 smithi174 ceph-mon[36787]: pgmap v207: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.2 MiB/s wr, 359 op/s 2024-03-18T14:59:54.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:53 smithi076 ceph-mon[32205]: pgmap v207: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.2 MiB/s wr, 359 op/s 2024-03-18T14:59:54.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 639712 9.55 MB/sec execute 235 sec latency 20.368 ms 2024-03-18T14:59:55.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 641603 9.54 MB/sec execute 236 sec latency 19.963 ms 2024-03-18T14:59:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.801+0000 7f09f03ab640 1 -- 172.21.15.76:0/2120306135 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e806a980 msgr2=0x7f09e810e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.801+0000 7f09f03ab640 1 --2- 172.21.15.76:0/2120306135 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e806a980 0x7f09e810e600 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f09e0009ca0 tx=0x7f09e002c0f0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.801+0000 7f09f03ab640 1 -- 172.21.15.76:0/2120306135 shutdown_connections 2024-03-18T14:59:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.801+0000 7f09f03ab640 1 --2- 172.21.15.76:0/2120306135 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e806a980 0x7f09e810e600 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.801+0000 7f09f03ab640 1 --2- 172.21.15.76:0/2120306135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09e8069fe0 0x7f09e806a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.801+0000 7f09f03ab640 1 --2- 172.21.15.76:0/2120306135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09e81077b0 0x7f09e8107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.801+0000 7f09f03ab640 1 -- 172.21.15.76:0/2120306135 >> 172.21.15.76:0/2120306135 conn(0x7f09e80774b0 msgr2=0x7f09e80778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.801+0000 7f09f03ab640 1 -- 172.21.15.76:0/2120306135 shutdown_connections 2024-03-18T14:59:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.801+0000 7f09f03ab640 1 -- 172.21.15.76:0/2120306135 wait complete. 2024-03-18T14:59:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.801+0000 7f09f03ab640 1 Processor -- start 2024-03-18T14:59:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09f03ab640 1 -- start start 2024-03-18T14:59:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09f03ab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09e8069fe0 0x7f09e819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09f03ab640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09e806a980 0x7f09e819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09f03ab640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e81077b0 0x7f09e819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09f03ab640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f09e8111b80 con 0x7f09e81077b0 2024-03-18T14:59:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09f03ab640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f09e8111a00 con 0x7f09e8069fe0 2024-03-18T14:59:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09f03ab640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f09e8111d00 con 0x7f09e806a980 2024-03-18T14:59:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09ee948640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e81077b0 0x7f09e819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09ee948640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e81077b0 0x7f09e819eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39492/0 (socket says 172.21.15.76:39492) 2024-03-18T14:59:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09ee948640 1 -- 172.21.15.76:0/3672854412 learned_addr learned my addr 172.21.15.76:0/3672854412 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09ee147640 1 --2- 172.21.15.76:0/3672854412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09e8069fe0 0x7f09e819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09ed946640 1 --2- 172.21.15.76:0/3672854412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09e806a980 0x7f09e819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09ee948640 1 -- 172.21.15.76:0/3672854412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09e806a980 msgr2=0x7f09e819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09ee948640 1 --2- 172.21.15.76:0/3672854412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09e806a980 0x7f09e819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09ee948640 1 -- 172.21.15.76:0/3672854412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09e8069fe0 msgr2=0x7f09e819a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09ee948640 1 --2- 172.21.15.76:0/3672854412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09e8069fe0 0x7f09e819a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09ee948640 1 -- 172.21.15.76:0/3672854412 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e819f1f0 con 0x7f09e81077b0 2024-03-18T14:59:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09ee948640 1 --2- 172.21.15.76:0/3672854412 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e81077b0 0x7f09e819eac0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f09e0037040 tx=0x7f09e0032cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.802+0000 7f09df7fe640 1 -- 172.21.15.76:0/3672854412 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0041020 con 0x7f09e81077b0 2024-03-18T14:59:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.803+0000 7f09f03ab640 1 -- 172.21.15.76:0/3672854412 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09e819f480 con 0x7f09e81077b0 2024-03-18T14:59:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.803+0000 7f09df7fe640 1 -- 172.21.15.76:0/3672854412 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09e0034cd0 con 0x7f09e81077b0 2024-03-18T14:59:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.803+0000 7f09df7fe640 1 -- 172.21.15.76:0/3672854412 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0039040 con 0x7f09e81077b0 2024-03-18T14:59:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.803+0000 7f09f03ab640 1 -- 172.21.15.76:0/3672854412 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09e81a6d70 con 0x7f09e81077b0 2024-03-18T14:59:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.806+0000 7f09df7fe640 1 -- 172.21.15.76:0/3672854412 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f09e0044020 con 0x7f09e81077b0 2024-03-18T14:59:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.807+0000 7f09f2f8b740 1 Processor -- start 2024-03-18T14:59:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.807+0000 7f09df7fe640 1 --2- 172.21.15.76:0/3672854412 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f09d40a2a90 0x7f09d40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.807+0000 7f09df7fe640 1 -- 172.21.15.76:0/3672854412 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7f09e00f0a40 con 0x7f09e81077b0 2024-03-18T14:59:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.807+0000 7f09f2f8b740 1 -- start start 2024-03-18T14:59:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.807+0000 7f09ed946640 1 --2- 172.21.15.76:0/3672854412 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f09d40a2a90 0x7f09d40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09f2f8b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38130 0x558e2fe38510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09f2f8b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe38a50 0x558e2fe49af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09f2f8b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe4a030 0x558e2fe4c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09f2f8b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x558e2fd9ca40 con 0x558e2fe38a50 2024-03-18T14:59:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09f2f8b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558e2fd45920 con 0x558e2fe38130 2024-03-18T14:59:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09f2f8b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558e2fd98ae0 con 0x558e2fe4a030 2024-03-18T14:59:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09ee948640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe4a030 0x558e2fe4c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09ee147640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38130 0x558e2fe38510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09ee948640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe4a030 0x558e2fe4c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:60318/0 (socket says 172.21.15.76:60318) 2024-03-18T14:59:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09ee948640 1 -- 172.21.15.76:0/3658491332 learned_addr learned my addr 172.21.15.76:0/3658491332 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09ed946640 1 --2- 172.21.15.76:0/3658491332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe38a50 0x558e2fe49af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09ee147640 1 -- 172.21.15.76:0/3658491332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe4a030 msgr2=0x558e2fe4c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09ee147640 1 --2- 172.21.15.76:0/3658491332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe4a030 0x558e2fe4c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09ee147640 1 -- 172.21.15.76:0/3658491332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe38a50 msgr2=0x558e2fe49af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09ee147640 1 --2- 172.21.15.76:0/3658491332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe38a50 0x558e2fe49af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09ee147640 1 -- 172.21.15.76:0/3658491332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e2fd5d460 con 0x558e2fe38130 2024-03-18T14:59:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09ed946640 1 --2- 172.21.15.76:0/3658491332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe38a50 0x558e2fe49af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:59:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.808+0000 7f09ed946640 1 --2- 172.21.15.76:0/3672854412 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f09d40a2a90 0x7f09d40a4f50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f09e810ea40 tx=0x7f09d8005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.809+0000 7f09ee147640 1 --2- 172.21.15.76:0/3658491332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38130 0x558e2fe38510 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f09e40045b0 tx=0x7f09e400bb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fcf5dabeee0cc0f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.809+0000 7f09dcbfe640 1 -- 172.21.15.76:0/3658491332 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e4003bd0 con 0x558e2fe38130 2024-03-18T14:59:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.809+0000 7f09dcbfe640 1 -- 172.21.15.76:0/3658491332 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09e4015070 con 0x558e2fe38130 2024-03-18T14:59:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.809+0000 7f09dcbfe640 1 -- 172.21.15.76:0/3658491332 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e4004d70 con 0x558e2fe38130 2024-03-18T14:59:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.810+0000 7f09f2f8b740 1 -- 172.21.15.76:0/3658491332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38130 msgr2=0x558e2fe38510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.810+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/3658491332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38130 0x558e2fe38510 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f09e40045b0 tx=0x7f09e400bb20 comp rx=0 tx=0).stop 2024-03-18T14:59:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.811+0000 7f09f2f8b740 1 -- 172.21.15.76:0/3658491332 shutdown_connections 2024-03-18T14:59:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.811+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/3658491332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe4a030 0x558e2fe4c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.811+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/3658491332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe38a50 0x558e2fe49af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.811+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/3658491332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38130 0x558e2fe38510 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.811+0000 7f09f2f8b740 1 -- 172.21.15.76:0/3658491332 >> 172.21.15.76:0/3658491332 conn(0x558e2fe32a50 msgr2=0x558e2fe32e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.811+0000 7f09f2f8b740 1 -- 172.21.15.76:0/3658491332 shutdown_connections 2024-03-18T14:59:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.811+0000 7f09f2f8b740 1 -- 172.21.15.76:0/3658491332 wait complete. 2024-03-18T14:59:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.811+0000 7f09f2f8b740 1 -- start start 2024-03-18T14:59:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.811+0000 7f09f2f8b740 1 Processor -- start 2024-03-18T14:59:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.812+0000 7f09f2f8b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe38130 0x558e2fe470b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.812+0000 7f09f2f8b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38a50 0x558e2fe475f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.812+0000 7f09f2f8b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe4a030 0x558e2fe47b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.812+0000 7f09f2f8b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x558e2fe4cc60 con 0x558e2fe38130 2024-03-18T14:59:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.812+0000 7f09f2f8b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558e2fd58d70 con 0x558e2fe38a50 2024-03-18T14:59:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.812+0000 7f09f2f8b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558e2fe4cde0 con 0x558e2fe4a030 2024-03-18T14:59:55.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.812+0000 7f09f2f8b740 10 client.0 _ll_register_callbacks cb 0x558e2fab0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:59:55.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.812+0000 7f09ed946640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38a50 0x558e2fe475f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:55.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.812+0000 7f09ed946640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38a50 0x558e2fe475f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:38756/0 (socket says 172.21.15.76:38756) 2024-03-18T14:59:55.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.812+0000 7f09ed946640 1 -- 172.21.15.76:0/803929879 learned_addr learned my addr 172.21.15.76:0/803929879 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:55.814 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.812+0000 7f09ee948640 1 --2- 172.21.15.76:0/803929879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe4a030 0x558e2fe47b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.812+0000 7f09ed946640 1 -- 172.21.15.76:0/803929879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe4a030 msgr2=0x558e2fe47b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.812+0000 7f09ed946640 1 --2- 172.21.15.76:0/803929879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe4a030 0x558e2fe47b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.812+0000 7f09ed946640 1 -- 172.21.15.76:0/803929879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe38130 msgr2=0x558e2fe470b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.812+0000 7f09ed946640 1 --2- 172.21.15.76:0/803929879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe38130 0x558e2fe470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.812+0000 7f09ed946640 1 -- 172.21.15.76:0/803929879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e2fab1210 con 0x558e2fe38a50 2024-03-18T14:59:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.813+0000 7f09ed946640 1 --2- 172.21.15.76:0/803929879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38a50 0x558e2fe475f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f09d801c950 tx=0x7f09d8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=543ebf0ff5e9c7a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.813+0000 7f09d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T14:59:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.813+0000 7f09d1ffb640 1 -- 172.21.15.76:0/803929879 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d802a060 con 0x558e2fe38a50 2024-03-18T14:59:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.813+0000 7f09d1ffb640 1 -- 172.21.15.76:0/803929879 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7f09d800eb40 con 0x558e2fe38a50 2024-03-18T14:59:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.813+0000 7f09f2f8b740 1 -- 172.21.15.76:0/803929879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09e4015070 con 0x558e2fe38a50 2024-03-18T14:59:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.814+0000 7f09d1ffb640 1 -- 172.21.15.76:0/803929879 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f09d800ed40 con 0x558e2fe38a50 2024-03-18T14:59:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.814+0000 7f09f2f8b740 10 client.26243 fetch_fsmap learned FSMap version 41 2024-03-18T14:59:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.814+0000 7f09f2f8b740 1 -- 172.21.15.76:0/803929879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x558e2fb7d620 con 0x558e2fe38a50 2024-03-18T14:59:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.814+0000 7f09d1ffb640 1 -- 172.21.15.76:0/803929879 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09d80035f0 con 0x558e2fe38a50 2024-03-18T14:59:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.814+0000 7f09d1ffb640 1 -- 172.21.15.76:0/803929879 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f09d802f460 con 0x558e2fe38a50 2024-03-18T14:59:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.814+0000 7f09f2f8b740 10 client.26243 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T14:59:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.814+0000 7f09f2f8b740 10 client.26243 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:59:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.814+0000 7f09f2f8b740 20 client.26243 populate_metadata read hostname 'smithi076' 2024-03-18T14:59:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.814+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/803929879 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558e2fe5a2b0 0x558e2fe5c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.814+0000 7f09f2f8b740 4 client.26243 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:59:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.814+0000 7f09f2f8b740 1 -- 172.21.15.76:0/803929879 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f09e4015070 con 0x558e2fe5a2b0 2024-03-18T14:59:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.814+0000 7f09ee147640 1 --2- 172.21.15.76:0/803929879 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558e2fe5a2b0 0x558e2fe5c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.815+0000 7f09d1ffb640 10 client.26243 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.815+0000 7f09ee147640 1 --2- 172.21.15.76:0/803929879 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558e2fe5a2b0 0x558e2fe5c740 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9481c4e9d5490d84 server_cookie=3f0f67769c669162 in_seq=0 out_seq=0 2024-03-18T14:59:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.817+0000 7f09d1ffb640 1 -- 172.21.15.76:0/803929879 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f09e4015070 con 0x558e2fe5a2b0 2024-03-18T14:59:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.817+0000 7f09d1ffb640 10 client.26243 handle_command_reply: tid=0 2024-03-18T14:59:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.817+0000 7f09f2f8b740 1 client.26243 shutdown 2024-03-18T14:59:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.817+0000 7f09f2f8b740 1 -- 172.21.15.76:0/803929879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38a50 msgr2=0x558e2fe475f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.817+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/803929879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38a50 0x558e2fe475f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f09d801c950 tx=0x7f09d8004730 comp rx=0 tx=0).stop 2024-03-18T14:59:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.818+0000 7f09f2f8b740 1 -- 172.21.15.76:0/803929879 shutdown_connections 2024-03-18T14:59:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.818+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/803929879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe4a030 0x558e2fe47b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.818+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/803929879 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558e2fe5a2b0 0x558e2fe5c740 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.818+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/803929879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38a50 0x558e2fe475f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.818+0000 7f09d1ffb640 0 client.26243 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.818+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/803929879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe38130 0x558e2fe470b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.818+0000 7f09f2f8b740 1 -- 172.21.15.76:0/803929879 >> 172.21.15.76:0/803929879 conn(0x558e2fe32a50 msgr2=0x558e2fe347a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.818+0000 7f09f2f8b740 1 -- 172.21.15.76:0/803929879 shutdown_connections 2024-03-18T14:59:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.818+0000 7f09f2f8b740 1 -- 172.21.15.76:0/803929879 wait complete. 2024-03-18T14:59:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.818+0000 7f09f2f8b740 20 client.26243 trim_cache size 0 max 16384 2024-03-18T14:59:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09f2f8b740 1 Processor -- start 2024-03-18T14:59:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09f2f8b740 1 -- start start 2024-03-18T14:59:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09f2f8b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38130 0x558e2fe497e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09f2f8b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe38a50 0x558e2fe2da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09f2f8b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe4a030 0x558e2fe55a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09f2f8b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f09e81ad9c0 con 0x558e2fe4a030 2024-03-18T14:59:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09f2f8b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f09e81ab290 con 0x558e2fe38130 2024-03-18T14:59:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09f2f8b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f09e81a9120 con 0x558e2fe38a50 2024-03-18T14:59:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09ee948640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe4a030 0x558e2fe55a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09ee948640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe4a030 0x558e2fe55a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39520/0 (socket says 172.21.15.76:39520) 2024-03-18T14:59:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09ee948640 1 -- 172.21.15.76:0/2874719489 learned_addr learned my addr 172.21.15.76:0/2874719489 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09ee147640 1 --2- 172.21.15.76:0/2874719489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38130 0x558e2fe497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09ed946640 1 --2- 172.21.15.76:0/2874719489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe38a50 0x558e2fe2da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09ee948640 1 -- 172.21.15.76:0/2874719489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe38a50 msgr2=0x558e2fe2da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09ee948640 1 --2- 172.21.15.76:0/2874719489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe38a50 0x558e2fe2da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09ee948640 1 -- 172.21.15.76:0/2874719489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38130 msgr2=0x558e2fe497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09ee948640 1 --2- 172.21.15.76:0/2874719489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38130 0x558e2fe497e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.834+0000 7f09ee948640 1 -- 172.21.15.76:0/2874719489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e2fe46060 con 0x558e2fe4a030 2024-03-18T14:59:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.835+0000 7f09ee948640 1 --2- 172.21.15.76:0/2874719489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe4a030 0x558e2fe55a00 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f09e00bcea0 tx=0x7f09e00dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=883540bd9f880689 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.835+0000 7f09d3fff640 1 -- 172.21.15.76:0/2874719489 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e00bcb00 con 0x558e2fe4a030 2024-03-18T14:59:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.835+0000 7f09d3fff640 1 -- 172.21.15.76:0/2874719489 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09e0034920 con 0x558e2fe4a030 2024-03-18T14:59:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.835+0000 7f09d3fff640 1 -- 172.21.15.76:0/2874719489 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e00bb4a0 con 0x558e2fe4a030 2024-03-18T14:59:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.835+0000 7f09f2f8b740 1 -- 172.21.15.76:0/2874719489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe4a030 msgr2=0x558e2fe55a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.835+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/2874719489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe4a030 0x558e2fe55a00 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f09e00bcea0 tx=0x7f09e00dd040 comp rx=0 tx=0).stop 2024-03-18T14:59:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09f2f8b740 1 -- 172.21.15.76:0/2874719489 shutdown_connections 2024-03-18T14:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/2874719489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe38a50 0x558e2fe2da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/2874719489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe4a030 0x558e2fe55a00 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/2874719489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38130 0x558e2fe497e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09f2f8b740 1 -- 172.21.15.76:0/2874719489 >> 172.21.15.76:0/2874719489 conn(0x558e2fe32a50 msgr2=0x558e2fe31bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09f2f8b740 1 -- 172.21.15.76:0/2874719489 shutdown_connections 2024-03-18T14:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09f2f8b740 1 -- 172.21.15.76:0/2874719489 wait complete. 2024-03-18T14:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09f2f8b740 1 -- start start 2024-03-18T14:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09f2f8b740 1 Processor -- start 2024-03-18T14:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09f2f8b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38130 0x558e2fe52030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09f2f8b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe38a50 0x558e2fe4f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09f2f8b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe4a030 0x558e2fe4f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09f2f8b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f09e81a9770 con 0x558e2fe4a030 2024-03-18T14:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09f2f8b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f09e81aabc0 con 0x558e2fe38130 2024-03-18T14:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09f2f8b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f09e81a9e20 con 0x558e2fe38a50 2024-03-18T14:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09f2f8b740 10 client.0 _ll_register_callbacks cb 0x7f09d80099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:59:55.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09ee147640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38130 0x558e2fe52030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:55.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09ee948640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe4a030 0x558e2fe4f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:55.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09ed946640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe38a50 0x558e2fe4f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:55.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09ee948640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe4a030 0x558e2fe4f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39536/0 (socket says 172.21.15.76:39536) 2024-03-18T14:59:55.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.836+0000 7f09ee948640 1 -- 172.21.15.76:0/65952364 learned_addr learned my addr 172.21.15.76:0/65952364 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T14:59:55.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.837+0000 7f09ee948640 1 -- 172.21.15.76:0/65952364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe38a50 msgr2=0x558e2fe4f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:55.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.837+0000 7f09ee948640 1 --2- 172.21.15.76:0/65952364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe38a50 0x558e2fe4f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.837+0000 7f09ee948640 1 -- 172.21.15.76:0/65952364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38130 msgr2=0x558e2fe52030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:59:55.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.837+0000 7f09ee948640 1 --2- 172.21.15.76:0/65952364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38130 0x558e2fe52030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.837+0000 7f09ee948640 1 -- 172.21.15.76:0/65952364 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e2fe4fab0 con 0x558e2fe4a030 2024-03-18T14:59:55.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.837+0000 7f09ee948640 1 --2- 172.21.15.76:0/65952364 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe4a030 0x558e2fe4f540 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f09e00c5410 tx=0x7f09e0003bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c117e23a89aeaaf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:59:55.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.837+0000 7f09d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T14:59:55.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.837+0000 7f09d1ffb640 1 -- 172.21.15.76:0/65952364 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e00bcb00 con 0x558e2fe4a030 2024-03-18T14:59:55.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.837+0000 7f09f2f8b740 1 -- 172.21.15.76:0/65952364 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09e0034920 con 0x558e2fe4a030 2024-03-18T14:59:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.837+0000 7f09d1ffb640 1 -- 172.21.15.76:0/65952364 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8440+0+0 (secure 0 0 0) 0x7f09e00ee400 con 0x558e2fe4a030 2024-03-18T14:59:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.837+0000 7f09d1ffb640 1 -- 172.21.15.76:0/65952364 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f09e00a0020 con 0x558e2fe4a030 2024-03-18T14:59:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.837+0000 7f09f2f8b740 10 client.16386 fetch_fsmap learned FSMap version 41 2024-03-18T14:59:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.837+0000 7f09f2f8b740 1 -- 172.21.15.76:0/65952364 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x558e2fec1470 con 0x558e2fe4a030 2024-03-18T14:59:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.837+0000 7f09d1ffb640 1 -- 172.21.15.76:0/65952364 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09e0034920 con 0x558e2fe4a030 2024-03-18T14:59:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.837+0000 7f09d1ffb640 1 -- 172.21.15.76:0/65952364 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f09e003c030 con 0x558e2fe4a030 2024-03-18T14:59:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.838+0000 7f09f2f8b740 10 client.16386 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T14:59:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.838+0000 7f09f2f8b740 10 client.16386 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T14:59:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.838+0000 7f09f2f8b740 20 client.16386 populate_metadata read hostname 'smithi076' 2024-03-18T14:59:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.838+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/65952364 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558e2fe5a2b0 0x558e2fec3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:59:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.838+0000 7f09f2f8b740 4 client.16386 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T14:59:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.838+0000 7f09f2f8b740 1 -- 172.21.15.76:0/65952364 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558e2fda9f80 con 0x558e2fe5a2b0 2024-03-18T14:59:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.838+0000 7f09ee147640 1 --2- 172.21.15.76:0/65952364 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558e2fe5a2b0 0x558e2fec3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:59:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.838+0000 7f09ee147640 1 --2- 172.21.15.76:0/65952364 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558e2fe5a2b0 0x558e2fec3b50 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b37f57c00cb962cf server_cookie=b4015f7a3c440ebe in_seq=0 out_seq=0 2024-03-18T14:59:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.838+0000 7f09d1ffb640 10 client.16386 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:55.923 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.921+0000 7f09d1ffb640 1 -- 172.21.15.76:0/65952364 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558e2fda9f80 con 0x558e2fe5a2b0 2024-03-18T14:59:55.923 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.921+0000 7f09d1ffb640 10 client.16386 handle_command_reply: tid=0 2024-03-18T14:59:55.923 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.921+0000 7f09f2f8b740 1 client.16386 shutdown 2024-03-18T14:59:55.923 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.922+0000 7f09f2f8b740 1 -- 172.21.15.76:0/65952364 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe4a030 msgr2=0x558e2fe4f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:55.923 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.922+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/65952364 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe4a030 0x558e2fe4f540 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f09e00c5410 tx=0x7f09e0003bc0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.923 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.922+0000 7f09f2f8b740 1 -- 172.21.15.76:0/65952364 shutdown_connections 2024-03-18T14:59:55.923 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.922+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/65952364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2fe38a50 0x558e2fe4f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.923 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.922+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/65952364 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x558e2fe5a2b0 0x558e2fec3b50 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.923 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.922+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/65952364 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558e2fe4a030 0x558e2fe4f540 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.922+0000 7f09f2f8b740 1 --2- 172.21.15.76:0/65952364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558e2fe38130 0x558e2fe52030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.922+0000 7f09f2f8b740 1 -- 172.21.15.76:0/65952364 >> 172.21.15.76:0/65952364 conn(0x558e2fe32a50 msgr2=0x558e2fe47070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.922+0000 7f09d1ffb640 0 client.16386 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T14:59:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.922+0000 7f09f2f8b740 1 -- 172.21.15.76:0/65952364 shutdown_connections 2024-03-18T14:59:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.922+0000 7f09f2f8b740 1 -- 172.21.15.76:0/65952364 wait complete. 2024-03-18T14:59:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.922+0000 7f09f2f8b740 20 client.16386 trim_cache size 0 max 16384 2024-03-18T14:59:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T14:59:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T14:59:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T14:59:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T14:59:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.923+0000 7f09f03ab640 1 -- 172.21.15.76:0/3672854412 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f09d40a2a90 msgr2=0x7f09d40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.923+0000 7f09f03ab640 1 --2- 172.21.15.76:0/3672854412 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f09d40a2a90 0x7f09d40a4f50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f09e810ea40 tx=0x7f09d8005f90 comp rx=0 tx=0).stop 2024-03-18T14:59:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.923+0000 7f09f03ab640 1 -- 172.21.15.76:0/3672854412 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e81077b0 msgr2=0x7f09e819eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:59:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.923+0000 7f09f03ab640 1 --2- 172.21.15.76:0/3672854412 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e81077b0 0x7f09e819eac0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f09e0037040 tx=0x7f09e0032cb0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.923+0000 7f09f03ab640 1 -- 172.21.15.76:0/3672854412 shutdown_connections 2024-03-18T14:59:55.926 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.923+0000 7f09f03ab640 1 --2- 172.21.15.76:0/3672854412 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f09d40a2a90 0x7f09d40a4f50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.926 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.923+0000 7f09f03ab640 1 --2- 172.21.15.76:0/3672854412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09e806a980 0x7f09e819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.926 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.923+0000 7f09f03ab640 1 --2- 172.21.15.76:0/3672854412 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e81077b0 0x7f09e819eac0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.926 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.923+0000 7f09f03ab640 1 --2- 172.21.15.76:0/3672854412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09e8069fe0 0x7f09e819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:59:55.926 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.923+0000 7f09f03ab640 1 -- 172.21.15.76:0/3672854412 >> 172.21.15.76:0/3672854412 conn(0x7f09e80774b0 msgr2=0x7f09e80ff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:59:55.926 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.923+0000 7f09f03ab640 1 -- 172.21.15.76:0/3672854412 shutdown_connections 2024-03-18T14:59:55.926 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T14:59:55.923+0000 7f09f03ab640 1 -- 172.21.15.76:0/3672854412 wait complete. 2024-03-18T14:59:56.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:55 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:59:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:55 smithi076 ceph-mon[32205]: pgmap v208: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 212 op/s 2024-03-18T14:59:56.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:55 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:59:56.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:55 smithi174 ceph-mon[36787]: pgmap v208: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 212 op/s 2024-03-18T14:59:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:55 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T14:59:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:55 smithi077 ceph-mon[35865]: pgmap v208: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 212 op/s 2024-03-18T14:59:56.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 643225 9.55 MB/sec execute 237 sec latency 44.527 ms 2024-03-18T14:59:57.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 644941 9.54 MB/sec execute 238 sec latency 23.052 ms 2024-03-18T14:59:57.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:57 smithi077 ceph-mon[35865]: pgmap v209: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 212 op/s 2024-03-18T14:59:58.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:57 smithi076 ceph-mon[32205]: pgmap v209: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 212 op/s 2024-03-18T14:59:58.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:57 smithi174 ceph-mon[36787]: pgmap v209: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 212 op/s 2024-03-18T14:59:58.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 646721 9.55 MB/sec execute 239 sec latency 37.268 ms 2024-03-18T14:59:59.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 648054 9.54 MB/sec execute 240 sec latency 21.654 ms 2024-03-18T14:59:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 14:59:59 smithi077 ceph-mon[35865]: pgmap v210: 97 pgs: 97 active+clean; 197 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.1 MiB/s wr, 352 op/s 2024-03-18T14:59:59.999 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 14:59:59 smithi076 ceph-mon[32205]: pgmap v210: 97 pgs: 97 active+clean; 197 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.1 MiB/s wr, 352 op/s 2024-03-18T15:00:00.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 14:59:59 smithi174 ceph-mon[36787]: pgmap v210: 97 pgs: 97 active+clean; 197 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.1 MiB/s wr, 352 op/s 2024-03-18T15:00:00.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:00 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a[32201]: 2024-03-18T14:59:59.999+0000 7f4cc589e700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 2 MDSs report damaged metadata 2024-03-18T15:00:00.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:00 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a[32201]: 2024-03-18T14:59:59.999+0000 7f4cc589e700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 2 MDSs report damaged metadata 2024-03-18T15:00:00.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:00 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a[32201]: 2024-03-18T14:59:59.999+0000 7f4cc589e700 -1 log_channel(cluster) log [ERR] : mds.c(mds.2): Metadata damage detected 2024-03-18T15:00:00.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:00 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a[32201]: 2024-03-18T14:59:59.999+0000 7f4cc589e700 -1 log_channel(cluster) log [ERR] : mds.i(mds.1): Metadata damage detected 2024-03-18T15:00:00.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 649981 9.55 MB/sec execute 241 sec latency 28.814 ms 2024-03-18T15:00:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:00 smithi077 ceph-mon[35865]: Health detail: HEALTH_ERR 2 MDSs report damaged metadata 2024-03-18T15:00:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:00 smithi077 ceph-mon[35865]: [ERR] MDS_DAMAGE: 2 MDSs report damaged metadata 2024-03-18T15:00:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:00 smithi077 ceph-mon[35865]: mds.c(mds.2): Metadata damage detected 2024-03-18T15:00:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:00 smithi077 ceph-mon[35865]: mds.i(mds.1): Metadata damage detected 2024-03-18T15:00:01.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:00 smithi076 ceph-mon[32205]: Health detail: HEALTH_ERR 2 MDSs report damaged metadata 2024-03-18T15:00:01.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:00 smithi076 ceph-mon[32205]: [ERR] MDS_DAMAGE: 2 MDSs report damaged metadata 2024-03-18T15:00:01.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:00 smithi076 ceph-mon[32205]: mds.c(mds.2): Metadata damage detected 2024-03-18T15:00:01.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:00 smithi076 ceph-mon[32205]: mds.i(mds.1): Metadata damage detected 2024-03-18T15:00:01.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:00 smithi174 ceph-mon[36787]: Health detail: HEALTH_ERR 2 MDSs report damaged metadata 2024-03-18T15:00:01.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:00 smithi174 ceph-mon[36787]: [ERR] MDS_DAMAGE: 2 MDSs report damaged metadata 2024-03-18T15:00:01.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:00 smithi174 ceph-mon[36787]: mds.c(mds.2): Metadata damage detected 2024-03-18T15:00:01.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:00 smithi174 ceph-mon[36787]: mds.i(mds.1): Metadata damage detected 2024-03-18T15:00:01.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 650817 9.52 MB/sec execute 242 sec latency 84.585 ms 2024-03-18T15:00:02.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:01 smithi076 ceph-mon[32205]: pgmap v211: 97 pgs: 97 active+clean; 197 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 252 op/s 2024-03-18T15:00:02.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:01 smithi174 ceph-mon[36787]: pgmap v211: 97 pgs: 97 active+clean; 197 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 252 op/s 2024-03-18T15:00:02.231 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:01 smithi077 ceph-mon[35865]: pgmap v211: 97 pgs: 97 active+clean; 197 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 252 op/s 2024-03-18T15:00:02.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 653079 9.53 MB/sec execute 243 sec latency 21.699 ms 2024-03-18T15:00:03.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 653661 9.51 MB/sec execute 244 sec latency 63.981 ms 2024-03-18T15:00:04.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:03 smithi076 ceph-mon[32205]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T15:00:04.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:03 smithi076 ceph-mon[32205]: pgmap v213: 97 pgs: 7 active+clean+snaptrim, 73 active+clean, 17 active+clean+snaptrim_wait; 204 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 316 op/s 2024-03-18T15:00:04.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:03 smithi174 ceph-mon[36787]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T15:00:04.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:03 smithi174 ceph-mon[36787]: pgmap v213: 97 pgs: 7 active+clean+snaptrim, 73 active+clean, 17 active+clean+snaptrim_wait; 204 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 316 op/s 2024-03-18T15:00:04.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:03 smithi077 ceph-mon[35865]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T15:00:04.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:03 smithi077 ceph-mon[35865]: pgmap v213: 97 pgs: 7 active+clean+snaptrim, 73 active+clean, 17 active+clean+snaptrim_wait; 204 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 316 op/s 2024-03-18T15:00:04.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 654825 9.50 MB/sec execute 245 sec latency 17.301 ms 2024-03-18T15:00:05.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 657133 9.51 MB/sec execute 246 sec latency 24.346 ms 2024-03-18T15:00:05.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:05 smithi077 ceph-mon[35865]: pgmap v214: 97 pgs: 7 active+clean+snaptrim, 73 active+clean, 17 active+clean+snaptrim_wait; 204 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 316 op/s 2024-03-18T15:00:06.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:05 smithi076 ceph-mon[32205]: pgmap v214: 97 pgs: 7 active+clean+snaptrim, 73 active+clean, 17 active+clean+snaptrim_wait; 204 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 316 op/s 2024-03-18T15:00:06.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:05 smithi174 ceph-mon[36787]: pgmap v214: 97 pgs: 7 active+clean+snaptrim, 73 active+clean, 17 active+clean+snaptrim_wait; 204 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 316 op/s 2024-03-18T15:00:06.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 658408 9.50 MB/sec execute 247 sec latency 19.255 ms 2024-03-18T15:00:07.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 660726 9.51 MB/sec execute 248 sec latency 17.538 ms 2024-03-18T15:00:07.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:07 smithi077 ceph-mon[35865]: pgmap v215: 97 pgs: 7 active+clean+snaptrim, 73 active+clean, 17 active+clean+snaptrim_wait; 204 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 316 op/s 2024-03-18T15:00:08.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:07 smithi076 ceph-mon[32205]: pgmap v215: 97 pgs: 7 active+clean+snaptrim, 73 active+clean, 17 active+clean+snaptrim_wait; 204 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 316 op/s 2024-03-18T15:00:08.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:07 smithi174 ceph-mon[36787]: pgmap v215: 97 pgs: 7 active+clean+snaptrim, 73 active+clean, 17 active+clean+snaptrim_wait; 204 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 316 op/s 2024-03-18T15:00:08.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 661665 9.49 MB/sec execute 249 sec latency 72.454 ms 2024-03-18T15:00:09.020 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.017+0000 7f2a58122640 1 -- 172.21.15.76:0/2957636674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a501077a0 msgr2=0x7f2a50107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.017+0000 7f2a58122640 1 --2- 172.21.15.76:0/2957636674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a501077a0 0x7f2a50107b80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c009930 tx=0x7f2a4c02bc30 comp rx=0 tx=0).stop 2024-03-18T15:00:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.017+0000 7f2a58122640 1 -- 172.21.15.76:0/2957636674 shutdown_connections 2024-03-18T15:00:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.017+0000 7f2a58122640 1 --2- 172.21.15.76:0/2957636674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a50102150 0x7f2a5010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.017+0000 7f2a58122640 1 --2- 172.21.15.76:0/2957636674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a50101790 0x7f2a50101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.017+0000 7f2a58122640 1 --2- 172.21.15.76:0/2957636674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a501077a0 0x7f2a50107b80 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.017+0000 7f2a58122640 1 -- 172.21.15.76:0/2957636674 >> 172.21.15.76:0/2957636674 conn(0x7f2a500fd2a0 msgr2=0x7f2a500ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.018+0000 7f2a58122640 1 -- 172.21.15.76:0/2957636674 shutdown_connections 2024-03-18T15:00:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.018+0000 7f2a58122640 1 -- 172.21.15.76:0/2957636674 wait complete. 2024-03-18T15:00:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.018+0000 7f2a58122640 1 Processor -- start 2024-03-18T15:00:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.018+0000 7f2a58122640 1 -- start start 2024-03-18T15:00:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.018+0000 7f2a58122640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a50101790 0x7f2a50197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.018+0000 7f2a58122640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a50102150 0x7f2a50198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.018+0000 7f2a58122640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a501077a0 0x7f2a5019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.018+0000 7f2a58122640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2a50111b70 con 0x7f2a501077a0 2024-03-18T15:00:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.018+0000 7f2a58122640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2a501119f0 con 0x7f2a50101790 2024-03-18T15:00:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.018+0000 7f2a58122640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2a50111cf0 con 0x7f2a50102150 2024-03-18T15:00:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.018+0000 7f2a566bf640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a501077a0 0x7f2a5019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.018+0000 7f2a55ebe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a50101790 0x7f2a50197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.019+0000 7f2a55ebe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a50101790 0x7f2a50197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:40482/0 (socket says 172.21.15.76:40482) 2024-03-18T15:00:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.019+0000 7f2a566bf640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a501077a0 0x7f2a5019c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55334/0 (socket says 172.21.15.76:55334) 2024-03-18T15:00:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.019+0000 7f2a55ebe640 1 -- 172.21.15.76:0/3660126464 learned_addr learned my addr 172.21.15.76:0/3660126464 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.019+0000 7f2a556bd640 1 --2- 172.21.15.76:0/3660126464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a50102150 0x7f2a50198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.019+0000 7f2a55ebe640 1 -- 172.21.15.76:0/3660126464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a50102150 msgr2=0x7f2a50198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.019+0000 7f2a55ebe640 1 --2- 172.21.15.76:0/3660126464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a50102150 0x7f2a50198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.019+0000 7f2a55ebe640 1 -- 172.21.15.76:0/3660126464 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a501077a0 msgr2=0x7f2a5019c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.019+0000 7f2a55ebe640 1 --2- 172.21.15.76:0/3660126464 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a501077a0 0x7f2a5019c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.019+0000 7f2a55ebe640 1 -- 172.21.15.76:0/3660126464 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5019cfe0 con 0x7f2a50101790 2024-03-18T15:00:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.019+0000 7f2a566bf640 1 --2- 172.21.15.76:0/3660126464 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a501077a0 0x7f2a5019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:00:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.019+0000 7f2a55ebe640 1 --2- 172.21.15.76:0/3660126464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a50101790 0x7f2a50197ff0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c009230 tx=0x7f2a4c02cba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.019+0000 7f2a46ffd640 1 -- 172.21.15.76:0/3660126464 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c041070 con 0x7f2a50101790 2024-03-18T15:00:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.019+0000 7f2a58122640 1 -- 172.21.15.76:0/3660126464 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a5019d270 con 0x7f2a50101790 2024-03-18T15:00:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.019+0000 7f2a46ffd640 1 -- 172.21.15.76:0/3660126464 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a4c004450 con 0x7f2a50101790 2024-03-18T15:00:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.019+0000 7f2a46ffd640 1 -- 172.21.15.76:0/3660126464 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c034440 con 0x7f2a50101790 2024-03-18T15:00:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.019+0000 7f2a58122640 1 -- 172.21.15.76:0/3660126464 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a501a4b60 con 0x7f2a50101790 2024-03-18T15:00:09.031 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.021+0000 7f2a46ffd640 1 -- 172.21.15.76:0/3660126464 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f2a4c044070 con 0x7f2a50101790 2024-03-18T15:00:09.031 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.022+0000 7f2a5adeb740 1 Processor -- start 2024-03-18T15:00:09.031 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.022+0000 7f2a5adeb740 1 -- start start 2024-03-18T15:00:09.031 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.022+0000 7f2a5adeb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f85308c130 0x55f85308c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.022+0000 7f2a5adeb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308ca50 0x55f8530952e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.022+0000 7f2a5adeb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f853095820 0x55f853097c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.022+0000 7f2a46ffd640 1 --2- 172.21.15.76:0/3660126464 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f2a3c0a2a10 0x7f2a3c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.023+0000 7f2a566bf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f85308c130 0x55f85308c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.023+0000 7f2a566bf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f85308c130 0x55f85308c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:40484/0 (socket says 172.21.15.76:40484) 2024-03-18T15:00:09.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.023+0000 7f2a566bf640 1 -- 172.21.15.76:0/3989578123 learned_addr learned my addr 172.21.15.76:0/3989578123 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:09.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.023+0000 7f2a55ebe640 1 --2- 172.21.15.76:0/3989578123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308ca50 0x55f8530952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.023+0000 7f2a556bd640 1 --2- 172.21.15.76:0/3660126464 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f2a3c0a2a10 0x7f2a3c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.023+0000 7f2a5adeb740 1 -- 172.21.15.76:0/3989578123 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f852ff0a40 con 0x55f85308ca50 2024-03-18T15:00:09.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.023+0000 7f2a5adeb740 1 -- 172.21.15.76:0/3989578123 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f852f99920 con 0x55f85308c130 2024-03-18T15:00:09.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.023+0000 7f2a5adeb740 1 -- 172.21.15.76:0/3989578123 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f852fecae0 con 0x55f853095820 2024-03-18T15:00:09.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.023+0000 7f2a46ffd640 1 -- 172.21.15.76:0/3660126464 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8528+0+0 (secure 0 0 0) 0x7f2a4c0f0750 con 0x7f2a50101790 2024-03-18T15:00:09.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.023+0000 7f2a566bf640 1 --2- 172.21.15.76:0/3989578123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f853095820 0x55f853097c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.032 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.023+0000 7f2a566bf640 1 -- 172.21.15.76:0/3989578123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f853095820 msgr2=0x55f853097c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:09.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.023+0000 7f2a566bf640 1 --2- 172.21.15.76:0/3989578123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f853095820 0x55f853097c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.023+0000 7f2a566bf640 1 -- 172.21.15.76:0/3989578123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308ca50 msgr2=0x55f8530952e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:09.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.023+0000 7f2a566bf640 1 --2- 172.21.15.76:0/3989578123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308ca50 0x55f8530952e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.023+0000 7f2a566bf640 1 -- 172.21.15.76:0/3989578123 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f852fb1460 con 0x55f85308c130 2024-03-18T15:00:09.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.023+0000 7f2a556bd640 1 --2- 172.21.15.76:0/3660126464 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f2a3c0a2a10 0x7f2a3c0a4ed0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2a500ff860 tx=0x7f2a40005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:09.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.023+0000 7f2a566bf640 1 --2- 172.21.15.76:0/3989578123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f85308c130 0x55f85308c510 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2a48009fd0 tx=0x7f2a48009e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e1b041ec6ebd0f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:09.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.024+0000 7f2a3bfff640 1 -- 172.21.15.76:0/3989578123 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a480120d0 con 0x55f85308c130 2024-03-18T15:00:09.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.024+0000 7f2a3bfff640 1 -- 172.21.15.76:0/3989578123 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a4800ebe0 con 0x55f85308c130 2024-03-18T15:00:09.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.024+0000 7f2a3bfff640 1 -- 172.21.15.76:0/3989578123 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a48010e40 con 0x55f85308c130 2024-03-18T15:00:09.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.024+0000 7f2a5adeb740 1 -- 172.21.15.76:0/3989578123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f85308c130 msgr2=0x55f85308c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:09.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.024+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/3989578123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f85308c130 0x55f85308c510 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2a48009fd0 tx=0x7f2a48009e10 comp rx=0 tx=0).stop 2024-03-18T15:00:09.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.024+0000 7f2a5adeb740 1 -- 172.21.15.76:0/3989578123 shutdown_connections 2024-03-18T15:00:09.033 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.024+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/3989578123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f853095820 0x55f853097c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.024+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/3989578123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308ca50 0x55f8530952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.024+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/3989578123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f85308c130 0x55f85308c510 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.024+0000 7f2a5adeb740 1 -- 172.21.15.76:0/3989578123 >> 172.21.15.76:0/3989578123 conn(0x55f853086a50 msgr2=0x55f853086e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:09.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.024+0000 7f2a5adeb740 1 -- 172.21.15.76:0/3989578123 shutdown_connections 2024-03-18T15:00:09.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.024+0000 7f2a5adeb740 1 -- 172.21.15.76:0/3989578123 wait complete. 2024-03-18T15:00:09.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.025+0000 7f2a5adeb740 1 -- start start 2024-03-18T15:00:09.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.025+0000 7f2a5adeb740 1 Processor -- start 2024-03-18T15:00:09.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.025+0000 7f2a5adeb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308c130 0x55f853092700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.025+0000 7f2a5adeb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f85308ca50 0x55f853092c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.025+0000 7f2a5adeb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f853095820 0x55f853093180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.025+0000 7f2a5adeb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f8530983f0 con 0x55f85308c130 2024-03-18T15:00:09.034 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.025+0000 7f2a5adeb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f852facd70 con 0x55f853095820 2024-03-18T15:00:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.025+0000 7f2a5adeb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f853098570 con 0x55f85308ca50 2024-03-18T15:00:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.025+0000 7f2a5adeb740 10 client.0 _ll_register_callbacks cb 0x55f852d04a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:00:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.025+0000 7f2a55ebe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308c130 0x55f853092700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.025+0000 7f2a55ebe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308c130 0x55f853092700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55356/0 (socket says 172.21.15.76:55356) 2024-03-18T15:00:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.025+0000 7f2a556bd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f85308ca50 0x55f853092c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.025+0000 7f2a55ebe640 1 -- 172.21.15.76:0/2113998022 learned_addr learned my addr 172.21.15.76:0/2113998022 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.025+0000 7f2a566bf640 1 --2- 172.21.15.76:0/2113998022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f853095820 0x55f853093180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.026+0000 7f2a55ebe640 1 -- 172.21.15.76:0/2113998022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f85308ca50 msgr2=0x55f853092c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.026+0000 7f2a55ebe640 1 --2- 172.21.15.76:0/2113998022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f85308ca50 0x55f853092c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.026+0000 7f2a55ebe640 1 -- 172.21.15.76:0/2113998022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f853095820 msgr2=0x55f853093180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.026+0000 7f2a55ebe640 1 --2- 172.21.15.76:0/2113998022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f853095820 0x55f853093180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.026+0000 7f2a55ebe640 1 -- 172.21.15.76:0/2113998022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f852d05210 con 0x55f85308c130 2024-03-18T15:00:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.026+0000 7f2a55ebe640 1 --2- 172.21.15.76:0/2113998022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308c130 0x55f853092700 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c0b9900 tx=0x7f2a4c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=832e1974825c4b47 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:09.035 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.026+0000 7f2a397fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:00:09.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.026+0000 7f2a397fa640 1 -- 172.21.15.76:0/2113998022 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c0bc890 con 0x55f85308c130 2024-03-18T15:00:09.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.026+0000 7f2a397fa640 1 -- 172.21.15.76:0/2113998022 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8528+0+0 (secure 0 0 0) 0x7f2a4c040170 con 0x55f85308c130 2024-03-18T15:00:09.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.026+0000 7f2a5adeb740 1 -- 172.21.15.76:0/2113998022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a4800ebe0 con 0x55f85308c130 2024-03-18T15:00:09.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.026+0000 7f2a397fa640 1 -- 172.21.15.76:0/2113998022 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f2a4c0bb4f0 con 0x55f85308c130 2024-03-18T15:00:09.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.026+0000 7f2a5adeb740 10 client.16398 fetch_fsmap learned FSMap version 41 2024-03-18T15:00:09.036 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.026+0000 7f2a5adeb740 1 -- 172.21.15.76:0/2113998022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f852dd1620 con 0x55f85308c130 2024-03-18T15:00:09.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.026+0000 7f2a397fa640 1 -- 172.21.15.76:0/2113998022 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a4800ebe0 con 0x55f85308c130 2024-03-18T15:00:09.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.026+0000 7f2a397fa640 1 -- 172.21.15.76:0/2113998022 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f2a4c0ee0e0 con 0x55f85308c130 2024-03-18T15:00:09.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.027+0000 7f2a5adeb740 10 client.16398 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:00:09.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.027+0000 7f2a5adeb740 10 client.16398 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:00:09.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.027+0000 7f2a5adeb740 20 client.16398 populate_metadata read hostname 'smithi076' 2024-03-18T15:00:09.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.027+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/2113998022 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f8530a5870 0x55f8530a7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.027+0000 7f2a5adeb740 4 client.16398 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:00:09.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.027+0000 7f2a5adeb740 1 -- 172.21.15.76:0/2113998022 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f852ffdf80 con 0x55f8530a5870 2024-03-18T15:00:09.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.027+0000 7f2a556bd640 1 --2- 172.21.15.76:0/2113998022 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f8530a5870 0x55f8530a7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.027+0000 7f2a397fa640 10 client.16398 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:00:09.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.027+0000 7f2a556bd640 1 --2- 172.21.15.76:0/2113998022 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f8530a5870 0x55f8530a7cc0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a009e5035330c39 server_cookie=7cc3c4e03e57929e in_seq=0 out_seq=0 2024-03-18T15:00:09.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.028+0000 7f2a397fa640 1 -- 172.21.15.76:0/2113998022 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f852ffdf80 con 0x55f8530a5870 2024-03-18T15:00:09.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.028+0000 7f2a397fa640 10 client.16398 handle_command_reply: tid=0 2024-03-18T15:00:09.037 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.028+0000 7f2a5adeb740 1 client.16398 shutdown 2024-03-18T15:00:09.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.028+0000 7f2a5adeb740 1 -- 172.21.15.76:0/2113998022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308c130 msgr2=0x55f853092700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:09.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.028+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/2113998022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308c130 0x55f853092700 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c0b9900 tx=0x7f2a4c02bc30 comp rx=0 tx=0).stop 2024-03-18T15:00:09.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.028+0000 7f2a5adeb740 1 -- 172.21.15.76:0/2113998022 shutdown_connections 2024-03-18T15:00:09.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.028+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/2113998022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f85308ca50 0x55f853092c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.028+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/2113998022 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f8530a5870 0x55f8530a7cc0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.028+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/2113998022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f853095820 0x55f853093180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.028+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/2113998022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308c130 0x55f853092700 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.028+0000 7f2a397fa640 0 client.16398 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:00:09.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.028+0000 7f2a5adeb740 1 -- 172.21.15.76:0/2113998022 >> 172.21.15.76:0/2113998022 conn(0x55f853086a50 msgr2=0x55f8530887a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:09.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.029+0000 7f2a5adeb740 1 -- 172.21.15.76:0/2113998022 shutdown_connections 2024-03-18T15:00:09.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.029+0000 7f2a5adeb740 1 -- 172.21.15.76:0/2113998022 wait complete. 2024-03-18T15:00:09.038 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.029+0000 7f2a5adeb740 20 client.16398 trim_cache size 0 max 16384 2024-03-18T15:00:09.050 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.044+0000 7f2a5adeb740 1 Processor -- start 2024-03-18T15:00:09.050 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.044+0000 7f2a5adeb740 1 -- start start 2024-03-18T15:00:09.050 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.044+0000 7f2a5adeb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308c130 0x55f853094e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.044+0000 7f2a5adeb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f85308ca50 0x55f8530a1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.044+0000 7f2a5adeb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f853095820 0x55f8530ad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.044+0000 7f2a5adeb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2a501afd90 con 0x55f85308c130 2024-03-18T15:00:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.044+0000 7f2a5adeb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2a501ad660 con 0x55f853095820 2024-03-18T15:00:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.044+0000 7f2a5adeb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2a501ab4b0 con 0x55f85308ca50 2024-03-18T15:00:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.045+0000 7f2a55ebe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308c130 0x55f853094e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.045+0000 7f2a55ebe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308c130 0x55f853094e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55370/0 (socket says 172.21.15.76:55370) 2024-03-18T15:00:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.045+0000 7f2a55ebe640 1 -- 172.21.15.76:0/4245402771 learned_addr learned my addr 172.21.15.76:0/4245402771 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.045+0000 7f2a566bf640 1 --2- 172.21.15.76:0/4245402771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f853095820 0x55f8530ad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.045+0000 7f2a556bd640 1 --2- 172.21.15.76:0/4245402771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f85308ca50 0x55f8530a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.045+0000 7f2a55ebe640 1 -- 172.21.15.76:0/4245402771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f85308ca50 msgr2=0x55f8530a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.045+0000 7f2a55ebe640 1 --2- 172.21.15.76:0/4245402771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f85308ca50 0x55f8530a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.045+0000 7f2a55ebe640 1 -- 172.21.15.76:0/4245402771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f853095820 msgr2=0x55f8530ad5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.045+0000 7f2a55ebe640 1 --2- 172.21.15.76:0/4245402771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f853095820 0x55f8530ad5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.045+0000 7f2a55ebe640 1 -- 172.21.15.76:0/4245402771 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f85308c880 con 0x55f85308c130 2024-03-18T15:00:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.045+0000 7f2a566bf640 1 --2- 172.21.15.76:0/4245402771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f853095820 0x55f8530ad5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:00:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.045+0000 7f2a556bd640 1 --2- 172.21.15.76:0/4245402771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f85308ca50 0x55f8530a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:00:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.045+0000 7f2a55ebe640 1 --2- 172.21.15.76:0/4245402771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308c130 0x55f853094e30 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c0b9390 tx=0x7f2a4c0bb880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=533d31b0601f0a80 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.045+0000 7f2a3b7fe640 1 -- 172.21.15.76:0/4245402771 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c0bb6e0 con 0x55f85308c130 2024-03-18T15:00:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.045+0000 7f2a3b7fe640 1 -- 172.21.15.76:0/4245402771 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a4c004c80 con 0x55f85308c130 2024-03-18T15:00:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.045+0000 7f2a3b7fe640 1 -- 172.21.15.76:0/4245402771 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c0c0fd0 con 0x55f85308c130 2024-03-18T15:00:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.046+0000 7f2a5adeb740 1 -- 172.21.15.76:0/4245402771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308c130 msgr2=0x55f853094e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:09.053 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.046+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/4245402771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308c130 0x55f853094e30 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c0b9390 tx=0x7f2a4c0bb880 comp rx=0 tx=0).stop 2024-03-18T15:00:09.053 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.046+0000 7f2a5adeb740 1 -- 172.21.15.76:0/4245402771 shutdown_connections 2024-03-18T15:00:09.053 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.046+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/4245402771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f85308ca50 0x55f8530a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.053 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.046+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/4245402771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f853095820 0x55f8530ad5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.053 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.046+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/4245402771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308c130 0x55f853094e30 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.053 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.046+0000 7f2a5adeb740 1 -- 172.21.15.76:0/4245402771 >> 172.21.15.76:0/4245402771 conn(0x55f853086a50 msgr2=0x55f853085bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:09.053 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.046+0000 7f2a5adeb740 1 -- 172.21.15.76:0/4245402771 shutdown_connections 2024-03-18T15:00:09.053 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.046+0000 7f2a5adeb740 1 -- 172.21.15.76:0/4245402771 wait complete. 2024-03-18T15:00:09.053 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.046+0000 7f2a5adeb740 1 -- start start 2024-03-18T15:00:09.053 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.047+0000 7f2a5adeb740 1 Processor -- start 2024-03-18T15:00:09.053 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.047+0000 7f2a5adeb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f85308c130 0x55f85308f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.053 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.047+0000 7f2a5adeb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308ca50 0x55f85308f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.047+0000 7f2a5adeb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f853095820 0x55f85308fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.047+0000 7f2a5adeb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2a501abb20 con 0x55f85308ca50 2024-03-18T15:00:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.047+0000 7f2a5adeb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2a501acf90 con 0x55f853095820 2024-03-18T15:00:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.047+0000 7f2a5adeb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2a501ac1f0 con 0x55f85308c130 2024-03-18T15:00:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.047+0000 7f2a5adeb740 10 client.0 _ll_register_callbacks cb 0x7f2a4c005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:00:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.047+0000 7f2a556bd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308ca50 0x55f85308f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.047+0000 7f2a556bd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308ca50 0x55f85308f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55384/0 (socket says 172.21.15.76:55384) 2024-03-18T15:00:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.047+0000 7f2a556bd640 1 -- 172.21.15.76:0/4029901519 learned_addr learned my addr 172.21.15.76:0/4029901519 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.047+0000 7f2a566bf640 1 --2- 172.21.15.76:0/4029901519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f853095820 0x55f85308fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.047+0000 7f2a55ebe640 1 --2- 172.21.15.76:0/4029901519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f85308c130 0x55f85308f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.048+0000 7f2a556bd640 1 -- 172.21.15.76:0/4029901519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f85308c130 msgr2=0x55f85308f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.048+0000 7f2a556bd640 1 --2- 172.21.15.76:0/4029901519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f85308c130 0x55f85308f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.048+0000 7f2a556bd640 1 -- 172.21.15.76:0/4029901519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f853095820 msgr2=0x55f85308fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.048+0000 7f2a556bd640 1 --2- 172.21.15.76:0/4029901519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f853095820 0x55f85308fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.048+0000 7f2a556bd640 1 -- 172.21.15.76:0/4029901519 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f85310ad00 con 0x55f85308ca50 2024-03-18T15:00:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.048+0000 7f2a556bd640 1 --2- 172.21.15.76:0/4029901519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308ca50 0x55f85308f8a0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x55f853091f60 tx=0x7f2a40004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f92ecf84fc12c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.048+0000 7f2a397fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:00:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.048+0000 7f2a397fa640 1 -- 172.21.15.76:0/4029901519 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4001fb00 con 0x55f85308ca50 2024-03-18T15:00:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.048+0000 7f2a397fa640 1 -- 172.21.15.76:0/4029901519 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8528+0+0 (secure 0 0 0) 0x7f2a4001fca0 con 0x55f85308ca50 2024-03-18T15:00:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.048+0000 7f2a5adeb740 1 -- 172.21.15.76:0/4029901519 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a4c004c80 con 0x55f85308ca50 2024-03-18T15:00:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.048+0000 7f2a397fa640 1 -- 172.21.15.76:0/4029901519 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f2a4001fea0 con 0x55f85308ca50 2024-03-18T15:00:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.048+0000 7f2a5adeb740 10 client.16404 fetch_fsmap learned FSMap version 41 2024-03-18T15:00:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.048+0000 7f2a5adeb740 1 -- 172.21.15.76:0/4029901519 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f85310c060 con 0x55f85308ca50 2024-03-18T15:00:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.048+0000 7f2a397fa640 1 -- 172.21.15.76:0/4029901519 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a4c004c80 con 0x55f85308ca50 2024-03-18T15:00:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.049+0000 7f2a397fa640 1 -- 172.21.15.76:0/4029901519 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f2a40030070 con 0x55f85308ca50 2024-03-18T15:00:09.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.049+0000 7f2a5adeb740 10 client.16404 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:00:09.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.049+0000 7f2a5adeb740 10 client.16404 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:00:09.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.049+0000 7f2a5adeb740 20 client.16404 populate_metadata read hostname 'smithi076' 2024-03-18T15:00:09.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.049+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/4029901519 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f85310c840 0x55f85310ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:09.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.049+0000 7f2a5adeb740 4 client.16404 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:00:09.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.049+0000 7f2a5adeb740 1 -- 172.21.15.76:0/4029901519 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f852ff82d0 con 0x55f85310c840 2024-03-18T15:00:09.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.049+0000 7f2a55ebe640 1 --2- 172.21.15.76:0/4029901519 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f85310c840 0x55f85310ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:09.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.049+0000 7f2a397fa640 10 client.16404 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:00:09.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.049+0000 7f2a55ebe640 1 --2- 172.21.15.76:0/4029901519 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f85310c840 0x55f85310ec60 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d1d2d4e589ea3b5 server_cookie=86c5446f670d2c84 in_seq=0 out_seq=0 2024-03-18T15:00:09.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.112+0000 7f2a397fa640 1 -- 172.21.15.76:0/4029901519 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f852ff82d0 con 0x55f85310c840 2024-03-18T15:00:09.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.112+0000 7f2a397fa640 10 client.16404 handle_command_reply: tid=0 2024-03-18T15:00:09.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.113+0000 7f2a5adeb740 1 client.16404 shutdown 2024-03-18T15:00:09.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.113+0000 7f2a5adeb740 1 -- 172.21.15.76:0/4029901519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308ca50 msgr2=0x55f85308f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:09.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.113+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/4029901519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308ca50 0x55f85308f8a0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x55f853091f60 tx=0x7f2a40004730 comp rx=0 tx=0).stop 2024-03-18T15:00:09.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.113+0000 7f2a5adeb740 1 -- 172.21.15.76:0/4029901519 shutdown_connections 2024-03-18T15:00:09.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.113+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/4029901519 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f85310c840 0x55f85310ec60 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.113+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/4029901519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f853095820 0x55f85308fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.113+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/4029901519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f85308ca50 0x55f85308f8a0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.113+0000 7f2a397fa640 0 client.16404 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:00:09.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.113+0000 7f2a5adeb740 1 --2- 172.21.15.76:0/4029901519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f85308c130 0x55f85308f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.113+0000 7f2a5adeb740 1 -- 172.21.15.76:0/4029901519 >> 172.21.15.76:0/4029901519 conn(0x55f853086a50 msgr2=0x55f8530926c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:09.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.113+0000 7f2a5adeb740 1 -- 172.21.15.76:0/4029901519 shutdown_connections 2024-03-18T15:00:09.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.113+0000 7f2a5adeb740 1 -- 172.21.15.76:0/4029901519 wait complete. 2024-03-18T15:00:09.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.113+0000 7f2a5adeb740 20 client.16404 trim_cache size 0 max 16384 2024-03-18T15:00:09.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:00:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:00:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:00:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:00:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.114+0000 7f2a58122640 1 -- 172.21.15.76:0/3660126464 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f2a3c0a2a10 msgr2=0x7f2a3c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.114+0000 7f2a58122640 1 --2- 172.21.15.76:0/3660126464 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f2a3c0a2a10 0x7f2a3c0a4ed0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2a500ff860 tx=0x7f2a40005f90 comp rx=0 tx=0).stop 2024-03-18T15:00:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.114+0000 7f2a58122640 1 -- 172.21.15.76:0/3660126464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a50101790 msgr2=0x7f2a50197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.114+0000 7f2a58122640 1 --2- 172.21.15.76:0/3660126464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a50101790 0x7f2a50197ff0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c009230 tx=0x7f2a4c02cba0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.114+0000 7f2a58122640 1 -- 172.21.15.76:0/3660126464 shutdown_connections 2024-03-18T15:00:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.114+0000 7f2a58122640 1 --2- 172.21.15.76:0/3660126464 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f2a3c0a2a10 0x7f2a3c0a4ed0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.114+0000 7f2a58122640 1 --2- 172.21.15.76:0/3660126464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a50102150 0x7f2a50198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.114+0000 7f2a58122640 1 --2- 172.21.15.76:0/3660126464 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a501077a0 0x7f2a5019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.114+0000 7f2a58122640 1 --2- 172.21.15.76:0/3660126464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a50101790 0x7f2a50197ff0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.114+0000 7f2a58122640 1 -- 172.21.15.76:0/3660126464 >> 172.21.15.76:0/3660126464 conn(0x7f2a500fd2a0 msgr2=0x7f2a500fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.114+0000 7f2a58122640 1 -- 172.21.15.76:0/3660126464 shutdown_connections 2024-03-18T15:00:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:09.114+0000 7f2a58122640 1 -- 172.21.15.76:0/3660126464 wait complete. 2024-03-18T15:00:09.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 664039 9.51 MB/sec execute 250 sec latency 40.767 ms 2024-03-18T15:00:09.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:09 smithi077 ceph-mon[35865]: pgmap v216: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-03-18T15:00:10.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:09 smithi076 ceph-mon[32205]: pgmap v216: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-03-18T15:00:10.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:09 smithi174 ceph-mon[36787]: pgmap v216: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-03-18T15:00:10.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 664986 9.48 MB/sec execute 251 sec latency 15.872 ms 2024-03-18T15:00:11.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 667600 9.51 MB/sec execute 252 sec latency 18.799 ms 2024-03-18T15:00:11.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:11 smithi077 ceph-mon[35865]: pgmap v217: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-03-18T15:00:12.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:11 smithi076 ceph-mon[32205]: pgmap v217: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-03-18T15:00:12.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:11 smithi174 ceph-mon[36787]: pgmap v217: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-03-18T15:00:12.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 668525 9.48 MB/sec execute 253 sec latency 50.514 ms 2024-03-18T15:00:13.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 670643 9.49 MB/sec execute 254 sec latency 31.687 ms 2024-03-18T15:00:14.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:14 smithi076 ceph-mon[32205]: osdmap e97: 12 total, 12 up, 12 in 2024-03-18T15:00:14.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:14 smithi076 ceph-mon[32205]: pgmap v219: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 279 op/s 2024-03-18T15:00:14.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:14 smithi174 ceph-mon[36787]: osdmap e97: 12 total, 12 up, 12 in 2024-03-18T15:00:14.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:14 smithi174 ceph-mon[36787]: pgmap v219: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 279 op/s 2024-03-18T15:00:14.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:14 smithi077 ceph-mon[35865]: osdmap e97: 12 total, 12 up, 12 in 2024-03-18T15:00:14.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:14 smithi077 ceph-mon[35865]: pgmap v219: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 279 op/s 2024-03-18T15:00:14.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 671902 9.48 MB/sec execute 255 sec latency 20.195 ms 2024-03-18T15:00:15.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 674239 9.49 MB/sec execute 256 sec latency 25.347 ms 2024-03-18T15:00:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:15 smithi077 ceph-mon[35865]: pgmap v220: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 279 op/s 2024-03-18T15:00:16.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:15 smithi076 ceph-mon[32205]: pgmap v220: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 279 op/s 2024-03-18T15:00:16.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:15 smithi174 ceph-mon[36787]: pgmap v220: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 279 op/s 2024-03-18T15:00:16.748 INFO:tasks.workunit.client.0.smithi076.stdout: 1 675556 9.48 MB/sec execute 257 sec latency 13.109 ms 2024-03-18T15:00:17.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 677788 9.49 MB/sec execute 258 sec latency 23.562 ms 2024-03-18T15:00:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:17 smithi077 ceph-mon[35865]: pgmap v221: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 279 op/s 2024-03-18T15:00:18.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:17 smithi076 ceph-mon[32205]: pgmap v221: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 279 op/s 2024-03-18T15:00:18.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:17 smithi174 ceph-mon[36787]: pgmap v221: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 279 op/s 2024-03-18T15:00:18.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 678971 9.48 MB/sec execute 259 sec latency 55.536 ms 2024-03-18T15:00:19.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 680807 9.47 MB/sec execute 260 sec latency 26.833 ms 2024-03-18T15:00:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:19 smithi077 ceph-mon[35865]: pgmap v222: 97 pgs: 97 active+clean; 205 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 268 op/s 2024-03-18T15:00:20.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:19 smithi076 ceph-mon[32205]: pgmap v222: 97 pgs: 97 active+clean; 205 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 268 op/s 2024-03-18T15:00:20.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:19 smithi174 ceph-mon[36787]: pgmap v222: 97 pgs: 97 active+clean; 205 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 268 op/s 2024-03-18T15:00:20.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 682452 9.48 MB/sec execute 261 sec latency 19.819 ms 2024-03-18T15:00:21.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 684735 9.49 MB/sec execute 262 sec latency 21.224 ms 2024-03-18T15:00:22.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:21 smithi076 ceph-mon[32205]: pgmap v223: 97 pgs: 97 active+clean; 205 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 268 op/s 2024-03-18T15:00:22.206 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.203+0000 7fe2689ab640 1 -- 172.21.15.76:0/1250667183 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe264101790 msgr2=0x7fe264101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:22.206 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.203+0000 7fe2689ab640 1 --2- 172.21.15.76:0/1250667183 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe264101790 0x7fe264101c10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fe258009930 tx=0x7fe25802bc30 comp rx=0 tx=0).stop 2024-03-18T15:00:22.206 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.203+0000 7fe2689ab640 1 -- 172.21.15.76:0/1250667183 shutdown_connections 2024-03-18T15:00:22.207 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.203+0000 7fe2689ab640 1 --2- 172.21.15.76:0/1250667183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe264102150 0x7fe26410e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.207 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.203+0000 7fe2689ab640 1 --2- 172.21.15.76:0/1250667183 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe264101790 0x7fe264101c10 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.207 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.203+0000 7fe2689ab640 1 --2- 172.21.15.76:0/1250667183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2641077a0 0x7fe264107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.207 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.203+0000 7fe2689ab640 1 -- 172.21.15.76:0/1250667183 >> 172.21.15.76:0/1250667183 conn(0x7fe2640fd2a0 msgr2=0x7fe2640ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:22.207 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.203+0000 7fe2689ab640 1 -- 172.21.15.76:0/1250667183 shutdown_connections 2024-03-18T15:00:22.207 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.203+0000 7fe2689ab640 1 -- 172.21.15.76:0/1250667183 wait complete. 2024-03-18T15:00:22.207 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.204+0000 7fe2689ab640 1 Processor -- start 2024-03-18T15:00:22.207 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.204+0000 7fe2689ab640 1 -- start start 2024-03-18T15:00:22.207 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.204+0000 7fe2689ab640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe264101790 0x7fe264197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.207 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.204+0000 7fe2689ab640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe264102150 0x7fe264198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.207 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.204+0000 7fe2689ab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2641077a0 0x7fe26419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.207 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.204+0000 7fe2689ab640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe264111b70 con 0x7fe264101790 2024-03-18T15:00:22.207 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.204+0000 7fe2689ab640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe2641119f0 con 0x7fe2641077a0 2024-03-18T15:00:22.208 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.204+0000 7fe2689ab640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe264111cf0 con 0x7fe264102150 2024-03-18T15:00:22.208 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.204+0000 7fe26259c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe264101790 0x7fe264197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.208 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.204+0000 7fe26259c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe264101790 0x7fe264197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50000/0 (socket says 172.21.15.76:50000) 2024-03-18T15:00:22.208 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.204+0000 7fe26259c640 1 -- 172.21.15.76:0/410158803 learned_addr learned my addr 172.21.15.76:0/410158803 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:22.208 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.205+0000 7fe262d9d640 1 --2- 172.21.15.76:0/410158803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2641077a0 0x7fe26419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.208 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.205+0000 7fe261d9b640 1 --2- 172.21.15.76:0/410158803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe264102150 0x7fe264198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.208 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.205+0000 7fe26259c640 1 -- 172.21.15.76:0/410158803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe264102150 msgr2=0x7fe264198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:22.208 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.205+0000 7fe26259c640 1 --2- 172.21.15.76:0/410158803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe264102150 0x7fe264198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.208 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.205+0000 7fe26259c640 1 -- 172.21.15.76:0/410158803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2641077a0 msgr2=0x7fe26419c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:22.208 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.205+0000 7fe26259c640 1 --2- 172.21.15.76:0/410158803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2641077a0 0x7fe26419c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.208 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.205+0000 7fe26259c640 1 -- 172.21.15.76:0/410158803 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26419cfe0 con 0x7fe264101790 2024-03-18T15:00:22.208 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.205+0000 7fe262d9d640 1 --2- 172.21.15.76:0/410158803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2641077a0 0x7fe26419c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:00:22.208 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.205+0000 7fe26259c640 1 --2- 172.21.15.76:0/410158803 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe264101790 0x7fe264197ff0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fe25400e9f0 tx=0x7fe25400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:22.208 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.205+0000 7fe2537fe640 1 -- 172.21.15.76:0/410158803 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25400cd80 con 0x7fe264101790 2024-03-18T15:00:22.208 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.205+0000 7fe2689ab640 1 -- 172.21.15.76:0/410158803 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe26419d2d0 con 0x7fe264101790 2024-03-18T15:00:22.209 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.205+0000 7fe2537fe640 1 -- 172.21.15.76:0/410158803 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe254010910 con 0x7fe264101790 2024-03-18T15:00:22.209 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.205+0000 7fe2689ab640 1 -- 172.21.15.76:0/410158803 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2641a4b60 con 0x7fe264101790 2024-03-18T15:00:22.209 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.205+0000 7fe2537fe640 1 -- 172.21.15.76:0/410158803 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe254018910 con 0x7fe264101790 2024-03-18T15:00:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.207+0000 7fe2537fe640 1 -- 172.21.15.76:0/410158803 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fe254010ac0 con 0x7fe264101790 2024-03-18T15:00:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.208+0000 7fe2537fe640 1 --2- 172.21.15.76:0/410158803 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe24c0a2a10 0x7fe24c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.208+0000 7fe261d9b640 1 --2- 172.21.15.76:0/410158803 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe24c0a2a10 0x7fe24c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.208+0000 7fe2537fe640 1 -- 172.21.15.76:0/410158803 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7fe2540cf0e0 con 0x7fe264101790 2024-03-18T15:00:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.208+0000 7fe26b58b740 1 Processor -- start 2024-03-18T15:00:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.208+0000 7fe26b58b740 1 -- start start 2024-03-18T15:00:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.208+0000 7fe261d9b640 1 --2- 172.21.15.76:0/410158803 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe24c0a2a10 0x7fe24c0a4ed0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe258005ab0 tx=0x7fe258005fd0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe26b58b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16130 0x55f4b0a16510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe26b58b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a16a50 0x55f4b0a1f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe26b58b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a1f820 0x55f4b0a21c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe26b58b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f4b097aa40 con 0x55f4b0a1f820 2024-03-18T15:00:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe26b58b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f4b0923920 con 0x55f4b0a16a50 2024-03-18T15:00:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe26b58b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f4b0976ae0 con 0x55f4b0a16130 2024-03-18T15:00:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe261d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a16a50 0x55f4b0a1f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe261d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a16a50 0x55f4b0a1f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:50722/0 (socket says 172.21.15.76:50722) 2024-03-18T15:00:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe26259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16130 0x55f4b0a16510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe261d9b640 1 -- 172.21.15.76:0/3280203391 learned_addr learned my addr 172.21.15.76:0/3280203391 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe262d9d640 1 --2- 172.21.15.76:0/3280203391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a1f820 0x55f4b0a21c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe261d9b640 1 -- 172.21.15.76:0/3280203391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16130 msgr2=0x55f4b0a16510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe261d9b640 1 --2- 172.21.15.76:0/3280203391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16130 0x55f4b0a16510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe261d9b640 1 -- 172.21.15.76:0/3280203391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a1f820 msgr2=0x55f4b0a21c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe261d9b640 1 --2- 172.21.15.76:0/3280203391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a1f820 0x55f4b0a21c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe261d9b640 1 -- 172.21.15.76:0/3280203391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4b093b460 con 0x55f4b0a16a50 2024-03-18T15:00:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe261d9b640 1 --2- 172.21.15.76:0/3280203391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a16a50 0x55f4b0a1f2e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe258035000 tx=0x7fe25803f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7354bba6b286302c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe250ff9640 1 -- 172.21.15.76:0/3280203391 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe258004180 con 0x55f4b0a16a50 2024-03-18T15:00:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe250ff9640 1 -- 172.21.15.76:0/3280203391 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe258004ba0 con 0x55f4b0a16a50 2024-03-18T15:00:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.209+0000 7fe250ff9640 1 -- 172.21.15.76:0/3280203391 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2580035f0 con 0x55f4b0a16a50 2024-03-18T15:00:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.210+0000 7fe26b58b740 1 -- 172.21.15.76:0/3280203391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a16a50 msgr2=0x55f4b0a1f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.210+0000 7fe26b58b740 1 --2- 172.21.15.76:0/3280203391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a16a50 0x55f4b0a1f2e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe258035000 tx=0x7fe25803f000 comp rx=0 tx=0).stop 2024-03-18T15:00:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.210+0000 7fe26b58b740 1 -- 172.21.15.76:0/3280203391 shutdown_connections 2024-03-18T15:00:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.210+0000 7fe26b58b740 1 --2- 172.21.15.76:0/3280203391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a1f820 0x55f4b0a21c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.210+0000 7fe26b58b740 1 --2- 172.21.15.76:0/3280203391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a16a50 0x55f4b0a1f2e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.210+0000 7fe26b58b740 1 --2- 172.21.15.76:0/3280203391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16130 0x55f4b0a16510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.210+0000 7fe26b58b740 1 -- 172.21.15.76:0/3280203391 >> 172.21.15.76:0/3280203391 conn(0x55f4b0a10a50 msgr2=0x55f4b0a10e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.210+0000 7fe26b58b740 1 -- 172.21.15.76:0/3280203391 shutdown_connections 2024-03-18T15:00:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.210+0000 7fe26b58b740 1 -- 172.21.15.76:0/3280203391 wait complete. 2024-03-18T15:00:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.210+0000 7fe26b58b740 1 -- start start 2024-03-18T15:00:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe26b58b740 1 Processor -- start 2024-03-18T15:00:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe26b58b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a16130 0x55f4b0a1c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe26b58b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16a50 0x55f4b0a1cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe26b58b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a1f820 0x55f4b0a1d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe26b58b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f4b0a223f0 con 0x55f4b0a16130 2024-03-18T15:00:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe26b58b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f4b0936d70 con 0x55f4b0a1f820 2024-03-18T15:00:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe26b58b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f4b0a22570 con 0x55f4b0a16a50 2024-03-18T15:00:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe26b58b740 10 client.0 _ll_register_callbacks cb 0x55f4b068ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:00:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe26259c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a16130 0x55f4b0a1c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe262d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a1f820 0x55f4b0a1d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe261d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16a50 0x55f4b0a1cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe26259c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a16130 0x55f4b0a1c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50014/0 (socket says 172.21.15.76:50014) 2024-03-18T15:00:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe26259c640 1 -- 172.21.15.76:0/1311221114 learned_addr learned my addr 172.21.15.76:0/1311221114 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe26259c640 1 -- 172.21.15.76:0/1311221114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16a50 msgr2=0x55f4b0a1cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe26259c640 1 --2- 172.21.15.76:0/1311221114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16a50 0x55f4b0a1cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe26259c640 1 -- 172.21.15.76:0/1311221114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a1f820 msgr2=0x55f4b0a1d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe26259c640 1 --2- 172.21.15.76:0/1311221114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a1f820 0x55f4b0a1d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe26259c640 1 -- 172.21.15.76:0/1311221114 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4b068f210 con 0x55f4b0a16130 2024-03-18T15:00:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe26259c640 1 --2- 172.21.15.76:0/1311221114 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a16130 0x55f4b0a1c800 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe25409b510 tx=0x7fe25409b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4125ae9d9d405b73 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe249ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:00:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.211+0000 7fe249ffb640 1 -- 172.21.15.76:0/1311221114 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2540afb40 con 0x55f4b0a16130 2024-03-18T15:00:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.212+0000 7fe249ffb640 1 -- 172.21.15.76:0/1311221114 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7fe25409ac60 con 0x55f4b0a16130 2024-03-18T15:00:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.212+0000 7fe26b58b740 1 -- 172.21.15.76:0/1311221114 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe258004ba0 con 0x55f4b0a16130 2024-03-18T15:00:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.212+0000 7fe249ffb640 1 -- 172.21.15.76:0/1311221114 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fe25409ae60 con 0x55f4b0a16130 2024-03-18T15:00:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.212+0000 7fe26b58b740 10 client.16416 fetch_fsmap learned FSMap version 41 2024-03-18T15:00:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.212+0000 7fe26b58b740 1 -- 172.21.15.76:0/1311221114 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f4b075b620 con 0x55f4b0a16130 2024-03-18T15:00:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.212+0000 7fe249ffb640 1 -- 172.21.15.76:0/1311221114 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe258004ba0 con 0x55f4b0a16130 2024-03-18T15:00:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.212+0000 7fe249ffb640 1 -- 172.21.15.76:0/1311221114 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7fe2540afce0 con 0x55f4b0a16130 2024-03-18T15:00:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.212+0000 7fe26b58b740 10 client.16416 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:00:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.212+0000 7fe26b58b740 10 client.16416 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:00:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.212+0000 7fe26b58b740 20 client.16416 populate_metadata read hostname 'smithi076' 2024-03-18T15:00:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.212+0000 7fe26b58b740 1 --2- 172.21.15.76:0/1311221114 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f4b0a2f9b0 0x55f4b0a31da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.212+0000 7fe26b58b740 4 client.16416 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:00:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.212+0000 7fe26b58b740 1 -- 172.21.15.76:0/1311221114 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4b0987f80 con 0x55f4b0a2f9b0 2024-03-18T15:00:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.212+0000 7fe261d9b640 1 --2- 172.21.15.76:0/1311221114 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f4b0a2f9b0 0x55f4b0a31da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.213+0000 7fe249ffb640 10 client.16416 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:00:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.213+0000 7fe261d9b640 1 --2- 172.21.15.76:0/1311221114 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f4b0a2f9b0 0x55f4b0a31da0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd373c26f5c32e25 server_cookie=3b3ea0d5ac68bfee in_seq=0 out_seq=0 2024-03-18T15:00:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.213+0000 7fe249ffb640 1 -- 172.21.15.76:0/1311221114 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f4b0987f80 con 0x55f4b0a2f9b0 2024-03-18T15:00:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.213+0000 7fe249ffb640 10 client.16416 handle_command_reply: tid=0 2024-03-18T15:00:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.214+0000 7fe26b58b740 1 client.16416 shutdown 2024-03-18T15:00:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.214+0000 7fe26b58b740 1 -- 172.21.15.76:0/1311221114 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a16130 msgr2=0x55f4b0a1c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.214+0000 7fe26b58b740 1 --2- 172.21.15.76:0/1311221114 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a16130 0x55f4b0a1c800 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe25409b510 tx=0x7fe25409b950 comp rx=0 tx=0).stop 2024-03-18T15:00:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.214+0000 7fe26b58b740 1 -- 172.21.15.76:0/1311221114 shutdown_connections 2024-03-18T15:00:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.214+0000 7fe26b58b740 1 --2- 172.21.15.76:0/1311221114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16a50 0x55f4b0a1cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.214+0000 7fe26b58b740 1 --2- 172.21.15.76:0/1311221114 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f4b0a2f9b0 0x55f4b0a31da0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.214+0000 7fe26b58b740 1 --2- 172.21.15.76:0/1311221114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a1f820 0x55f4b0a1d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.214+0000 7fe26b58b740 1 --2- 172.21.15.76:0/1311221114 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a16130 0x55f4b0a1c800 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.214+0000 7fe249ffb640 0 client.16416 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:00:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.214+0000 7fe26b58b740 1 -- 172.21.15.76:0/1311221114 >> 172.21.15.76:0/1311221114 conn(0x55f4b0a10a50 msgr2=0x55f4b0a127a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.214+0000 7fe26b58b740 1 -- 172.21.15.76:0/1311221114 shutdown_connections 2024-03-18T15:00:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.214+0000 7fe26b58b740 1 -- 172.21.15.76:0/1311221114 wait complete. 2024-03-18T15:00:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.214+0000 7fe26b58b740 20 client.16416 trim_cache size 0 max 16384 2024-03-18T15:00:22.222 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:21 smithi174 ceph-mon[36787]: pgmap v223: 97 pgs: 97 active+clean; 205 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 268 op/s 2024-03-18T15:00:22.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:21 smithi077 ceph-mon[35865]: pgmap v223: 97 pgs: 97 active+clean; 205 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 268 op/s 2024-03-18T15:00:22.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe26b58b740 1 Processor -- start 2024-03-18T15:00:22.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe26b58b740 1 -- start start 2024-03-18T15:00:22.233 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe26b58b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16130 0x55f4b0a2b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe26b58b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a16a50 0x55f4b0a375d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe26b58b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a1f820 0x55f4b0a277e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe26b58b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe2641afcb0 con 0x55f4b0a16a50 2024-03-18T15:00:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe26b58b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe2641ad580 con 0x55f4b0a1f820 2024-03-18T15:00:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe26b58b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe2641ab3f0 con 0x55f4b0a16130 2024-03-18T15:00:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe261d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a16a50 0x55f4b0a375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe261d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a16a50 0x55f4b0a375d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50018/0 (socket says 172.21.15.76:50018) 2024-03-18T15:00:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe262d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a1f820 0x55f4b0a277e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe261d9b640 1 -- 172.21.15.76:0/2246449206 learned_addr learned my addr 172.21.15.76:0/2246449206 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe26259c640 1 --2- 172.21.15.76:0/2246449206 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16130 0x55f4b0a2b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe261d9b640 1 -- 172.21.15.76:0/2246449206 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16130 msgr2=0x55f4b0a2b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe261d9b640 1 --2- 172.21.15.76:0/2246449206 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16130 0x55f4b0a2b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe261d9b640 1 -- 172.21.15.76:0/2246449206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a1f820 msgr2=0x55f4b0a277e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe261d9b640 1 --2- 172.21.15.76:0/2246449206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a1f820 0x55f4b0a277e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe261d9b640 1 -- 172.21.15.76:0/2246449206 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4b0a16880 con 0x55f4b0a16a50 2024-03-18T15:00:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe261d9b640 1 --2- 172.21.15.76:0/2246449206 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a16a50 0x55f4b0a375d0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe25802c170 tx=0x7fe258042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae18c4bb000e2fa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.229+0000 7fe24bfff640 1 -- 172.21.15.76:0/2246449206 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe258005530 con 0x55f4b0a16a50 2024-03-18T15:00:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.230+0000 7fe24bfff640 1 -- 172.21.15.76:0/2246449206 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe25802c620 con 0x55f4b0a16a50 2024-03-18T15:00:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.230+0000 7fe24bfff640 1 -- 172.21.15.76:0/2246449206 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25805a760 con 0x55f4b0a16a50 2024-03-18T15:00:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.230+0000 7fe26b58b740 1 -- 172.21.15.76:0/2246449206 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a16a50 msgr2=0x55f4b0a375d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.230+0000 7fe26b58b740 1 --2- 172.21.15.76:0/2246449206 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a16a50 0x55f4b0a375d0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe25802c170 tx=0x7fe258042000 comp rx=0 tx=0).stop 2024-03-18T15:00:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.230+0000 7fe26b58b740 1 -- 172.21.15.76:0/2246449206 shutdown_connections 2024-03-18T15:00:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.230+0000 7fe26b58b740 1 --2- 172.21.15.76:0/2246449206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a1f820 0x55f4b0a277e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.230+0000 7fe26b58b740 1 --2- 172.21.15.76:0/2246449206 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a16a50 0x55f4b0a375d0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.230+0000 7fe26b58b740 1 --2- 172.21.15.76:0/2246449206 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16130 0x55f4b0a2b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.230+0000 7fe26b58b740 1 -- 172.21.15.76:0/2246449206 >> 172.21.15.76:0/2246449206 conn(0x55f4b0a10a50 msgr2=0x55f4b0a0f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.230+0000 7fe26b58b740 1 -- 172.21.15.76:0/2246449206 shutdown_connections 2024-03-18T15:00:22.235 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.230+0000 7fe26b58b740 1 -- 172.21.15.76:0/2246449206 wait complete. 2024-03-18T15:00:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.230+0000 7fe26b58b740 1 -- start start 2024-03-18T15:00:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe26b58b740 1 Processor -- start 2024-03-18T15:00:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe26b58b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a16130 0x55f4b0a277f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe26b58b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16a50 0x55f4b0a247c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe26b58b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a1f820 0x55f4b0a24d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe26b58b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe2641aba40 con 0x55f4b0a1f820 2024-03-18T15:00:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe26b58b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe2641aceb0 con 0x55f4b0a16130 2024-03-18T15:00:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe26b58b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe2641ac110 con 0x55f4b0a16a50 2024-03-18T15:00:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe26b58b740 10 client.0 _ll_register_callbacks cb 0x7fe25409b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:00:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe262d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a1f820 0x55f4b0a24d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe26259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a16130 0x55f4b0a277f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe262d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a1f820 0x55f4b0a24d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50022/0 (socket says 172.21.15.76:50022) 2024-03-18T15:00:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe262d9d640 1 -- 172.21.15.76:0/1204254364 learned_addr learned my addr 172.21.15.76:0/1204254364 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe26259c640 1 --2- 172.21.15.76:0/1204254364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a16130 0x55f4b0a277f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:50754/0 (socket says 172.21.15.76:50754) 2024-03-18T15:00:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe261d9b640 1 --2- 172.21.15.76:0/1204254364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16a50 0x55f4b0a247c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe262d9d640 1 -- 172.21.15.76:0/1204254364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16a50 msgr2=0x55f4b0a247c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe262d9d640 1 --2- 172.21.15.76:0/1204254364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16a50 0x55f4b0a247c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe262d9d640 1 -- 172.21.15.76:0/1204254364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a16130 msgr2=0x55f4b0a277f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe262d9d640 1 --2- 172.21.15.76:0/1204254364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a16130 0x55f4b0a277f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe262d9d640 1 -- 172.21.15.76:0/1204254364 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4b0a25240 con 0x55f4b0a1f820 2024-03-18T15:00:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe262d9d640 1 --2- 172.21.15.76:0/1204254364 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a1f820 0x55f4b0a24d00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fe25c005c00 tx=0x7fe25c0128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0dc3ba052678b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.231+0000 7fe249ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:00:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.232+0000 7fe249ffb640 1 -- 172.21.15.76:0/1204254364 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c004d20 con 0x55f4b0a1f820 2024-03-18T15:00:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.232+0000 7fe249ffb640 1 -- 172.21.15.76:0/1204254364 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7fe25c003970 con 0x55f4b0a1f820 2024-03-18T15:00:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.232+0000 7fe26b58b740 1 -- 172.21.15.76:0/1204254364 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f4b09822d0 con 0x55f4b0a1f820 2024-03-18T15:00:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.232+0000 7fe249ffb640 1 -- 172.21.15.76:0/1204254364 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fe25c003b70 con 0x55f4b0a1f820 2024-03-18T15:00:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.232+0000 7fe26b58b740 10 client.16422 fetch_fsmap learned FSMap version 41 2024-03-18T15:00:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.232+0000 7fe26b58b740 1 -- 172.21.15.76:0/1204254364 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f4b0a96860 con 0x55f4b0a1f820 2024-03-18T15:00:22.238 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.232+0000 7fe249ffb640 1 -- 172.21.15.76:0/1204254364 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f4b09822d0 con 0x55f4b0a1f820 2024-03-18T15:00:22.238 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.232+0000 7fe249ffb640 1 -- 172.21.15.76:0/1204254364 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7fe25c027640 con 0x55f4b0a1f820 2024-03-18T15:00:22.238 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.232+0000 7fe26b58b740 10 client.16422 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:00:22.238 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.232+0000 7fe26b58b740 10 client.16422 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:00:22.238 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.232+0000 7fe26b58b740 20 client.16422 populate_metadata read hostname 'smithi076' 2024-03-18T15:00:22.238 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.232+0000 7fe26b58b740 1 --2- 172.21.15.76:0/1204254364 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f4b0a97030 0x55f4b0a99450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:22.238 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.232+0000 7fe26b58b740 4 client.16422 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:00:22.238 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.232+0000 7fe26b58b740 1 -- 172.21.15.76:0/1204254364 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f4b0900770 con 0x55f4b0a97030 2024-03-18T15:00:22.238 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.232+0000 7fe26259c640 1 --2- 172.21.15.76:0/1204254364 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f4b0a97030 0x55f4b0a99450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:22.238 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.233+0000 7fe249ffb640 10 client.16422 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:00:22.238 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.233+0000 7fe26259c640 1 --2- 172.21.15.76:0/1204254364 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f4b0a97030 0x55f4b0a99450 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa939321f2f6c0fe server_cookie=929bc66697902f0e in_seq=0 out_seq=0 2024-03-18T15:00:22.306 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.305+0000 7fe249ffb640 1 -- 172.21.15.76:0/1204254364 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f4b0900770 con 0x55f4b0a97030 2024-03-18T15:00:22.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.305+0000 7fe249ffb640 10 client.16422 handle_command_reply: tid=0 2024-03-18T15:00:22.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.305+0000 7fe26b58b740 1 client.16422 shutdown 2024-03-18T15:00:22.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.306+0000 7fe26b58b740 1 -- 172.21.15.76:0/1204254364 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a1f820 msgr2=0x55f4b0a24d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:22.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.306+0000 7fe26b58b740 1 --2- 172.21.15.76:0/1204254364 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a1f820 0x55f4b0a24d00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fe25c005c00 tx=0x7fe25c0128e0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.309+0000 7fe26b58b740 1 -- 172.21.15.76:0/1204254364 shutdown_connections 2024-03-18T15:00:22.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.309+0000 7fe26b58b740 1 --2- 172.21.15.76:0/1204254364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f4b0a16a50 0x55f4b0a247c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.309+0000 7fe26b58b740 1 --2- 172.21.15.76:0/1204254364 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f4b0a97030 0x55f4b0a99450 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.309+0000 7fe26b58b740 1 --2- 172.21.15.76:0/1204254364 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4b0a1f820 0x55f4b0a24d00 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.309+0000 7fe26b58b740 1 --2- 172.21.15.76:0/1204254364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4b0a16130 0x55f4b0a277f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.309+0000 7fe26b58b740 1 -- 172.21.15.76:0/1204254364 >> 172.21.15.76:0/1204254364 conn(0x55f4b0a10a50 msgr2=0x55f4b0a1d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:22.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.309+0000 7fe249ffb640 0 client.16422 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:00:22.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.311+0000 7fe26b58b740 1 -- 172.21.15.76:0/1204254364 shutdown_connections 2024-03-18T15:00:22.311 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.311+0000 7fe26b58b740 1 -- 172.21.15.76:0/1204254364 wait complete. 2024-03-18T15:00:22.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.312+0000 7fe26b58b740 20 client.16422 trim_cache size 0 max 16384 2024-03-18T15:00:22.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:00:22.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:00:22.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:00:22.313 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:00:22.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.315+0000 7fe2689ab640 1 -- 172.21.15.76:0/410158803 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe24c0a2a10 msgr2=0x7fe24c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:22.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.315+0000 7fe2689ab640 1 --2- 172.21.15.76:0/410158803 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe24c0a2a10 0x7fe24c0a4ed0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe258005ab0 tx=0x7fe258005fd0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.315+0000 7fe2689ab640 1 -- 172.21.15.76:0/410158803 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe264101790 msgr2=0x7fe264197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:22.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.315+0000 7fe2689ab640 1 --2- 172.21.15.76:0/410158803 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe264101790 0x7fe264197ff0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fe25400e9f0 tx=0x7fe25400eeb0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.316+0000 7fe2689ab640 1 -- 172.21.15.76:0/410158803 shutdown_connections 2024-03-18T15:00:22.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.316+0000 7fe2689ab640 1 --2- 172.21.15.76:0/410158803 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe24c0a2a10 0x7fe24c0a4ed0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.316+0000 7fe2689ab640 1 --2- 172.21.15.76:0/410158803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe264102150 0x7fe264198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.316+0000 7fe2689ab640 1 --2- 172.21.15.76:0/410158803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2641077a0 0x7fe26419c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.316+0000 7fe2689ab640 1 --2- 172.21.15.76:0/410158803 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe264101790 0x7fe264197ff0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:22.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.316+0000 7fe2689ab640 1 -- 172.21.15.76:0/410158803 >> 172.21.15.76:0/410158803 conn(0x7fe2640fd2a0 msgr2=0x7fe2640fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:22.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.316+0000 7fe2689ab640 1 -- 172.21.15.76:0/410158803 shutdown_connections 2024-03-18T15:00:22.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:22.316+0000 7fe2689ab640 1 -- 172.21.15.76:0/410158803 wait complete. 2024-03-18T15:00:22.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 685995 9.48 MB/sec execute 263 sec latency 36.236 ms 2024-03-18T15:00:23.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 688278 9.49 MB/sec execute 264 sec latency 16.134 ms 2024-03-18T15:00:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:23 smithi077 ceph-mon[35865]: pgmap v224: 97 pgs: 97 active+clean; 200 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 427 op/s 2024-03-18T15:00:24.018 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:23 smithi174 ceph-mon[36787]: pgmap v224: 97 pgs: 97 active+clean; 200 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 427 op/s 2024-03-18T15:00:24.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:23 smithi076 ceph-mon[32205]: pgmap v224: 97 pgs: 97 active+clean; 200 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 427 op/s 2024-03-18T15:00:24.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 689550 9.48 MB/sec execute 265 sec latency 37.177 ms 2024-03-18T15:00:25.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 691699 9.49 MB/sec execute 266 sec latency 38.926 ms 2024-03-18T15:00:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:25 smithi077 ceph-mon[35865]: pgmap v225: 97 pgs: 97 active+clean; 200 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 239 op/s 2024-03-18T15:00:26.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:25 smithi076 ceph-mon[32205]: pgmap v225: 97 pgs: 97 active+clean; 200 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 239 op/s 2024-03-18T15:00:26.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:25 smithi174 ceph-mon[36787]: pgmap v225: 97 pgs: 97 active+clean; 200 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 239 op/s 2024-03-18T15:00:26.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 693214 9.48 MB/sec execute 267 sec latency 19.528 ms 2024-03-18T15:00:27.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 695423 9.50 MB/sec execute 268 sec latency 35.090 ms 2024-03-18T15:00:27.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:27 smithi077 ceph-mon[35865]: pgmap v226: 97 pgs: 97 active+clean; 200 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 239 op/s 2024-03-18T15:00:28.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:27 smithi076 ceph-mon[32205]: pgmap v226: 97 pgs: 97 active+clean; 200 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 239 op/s 2024-03-18T15:00:28.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:27 smithi174 ceph-mon[36787]: pgmap v226: 97 pgs: 97 active+clean; 200 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 239 op/s 2024-03-18T15:00:28.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 696830 9.49 MB/sec execute 269 sec latency 16.642 ms 2024-03-18T15:00:29.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:29 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:00:29.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:29 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:00:29.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:29 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:00:29.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 698475 9.48 MB/sec execute 270 sec latency 40.203 ms 2024-03-18T15:00:30.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:30 smithi076 ceph-mon[32205]: pgmap v227: 97 pgs: 97 active+clean; 207 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 343 op/s 2024-03-18T15:00:30.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:30 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:00:30.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:30 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:00:30.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:30 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:00:30.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:30 smithi174 ceph-mon[36787]: pgmap v227: 97 pgs: 97 active+clean; 207 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 343 op/s 2024-03-18T15:00:30.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:30 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:00:30.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:30 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:00:30.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:30 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:00:30.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:30 smithi077 ceph-mon[35865]: pgmap v227: 97 pgs: 97 active+clean; 207 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 343 op/s 2024-03-18T15:00:30.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:30 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:00:30.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:30 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:00:30.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:30 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:00:30.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 700027 9.48 MB/sec execute 271 sec latency 31.973 ms 2024-03-18T15:00:31.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 701930 9.48 MB/sec execute 272 sec latency 23.042 ms 2024-03-18T15:00:31.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:31 smithi077 ceph-mon[35865]: pgmap v228: 97 pgs: 97 active+clean; 207 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 250 op/s 2024-03-18T15:00:32.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:31 smithi076 ceph-mon[32205]: pgmap v228: 97 pgs: 97 active+clean; 207 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 250 op/s 2024-03-18T15:00:32.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:31 smithi174 ceph-mon[36787]: pgmap v228: 97 pgs: 97 active+clean; 207 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 250 op/s 2024-03-18T15:00:32.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 703589 9.49 MB/sec execute 273 sec latency 11.395 ms 2024-03-18T15:00:33.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 705560 9.48 MB/sec execute 274 sec latency 21.225 ms 2024-03-18T15:00:33.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:33 smithi077 ceph-mon[35865]: pgmap v229: 97 pgs: 97 active+clean; 213 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 350 op/s 2024-03-18T15:00:34.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:33 smithi076 ceph-mon[32205]: pgmap v229: 97 pgs: 97 active+clean; 213 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 350 op/s 2024-03-18T15:00:34.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:33 smithi174 ceph-mon[36787]: pgmap v229: 97 pgs: 97 active+clean; 213 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 350 op/s 2024-03-18T15:00:34.749 INFO:tasks.workunit.client.0.smithi076.stdout: 1 707205 9.49 MB/sec execute 275 sec latency 25.138 ms 2024-03-18T15:00:35.415 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.413+0000 7f93ff27e640 1 -- 172.21.15.76:0/1826916095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93f81078b0 msgr2=0x7f93f8107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.413+0000 7f93ff27e640 1 --2- 172.21.15.76:0/1826916095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93f81078b0 0x7f93f8107c90 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009930 tx=0x7f93f402bc30 comp rx=0 tx=0).stop 2024-03-18T15:00:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.413+0000 7f93ff27e640 1 -- 172.21.15.76:0/1826916095 shutdown_connections 2024-03-18T15:00:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.413+0000 7f93ff27e640 1 --2- 172.21.15.76:0/1826916095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93f8068e60 0x7f93f80692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.413+0000 7f93ff27e640 1 --2- 172.21.15.76:0/1826916095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93f8069800 0x7f93f810e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.413+0000 7f93ff27e640 1 --2- 172.21.15.76:0/1826916095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93f81078b0 0x7f93f8107c90 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.413+0000 7f93ff27e640 1 -- 172.21.15.76:0/1826916095 >> 172.21.15.76:0/1826916095 conn(0x7f93f8077530 msgr2=0x7f93f8077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.413+0000 7f93ff27e640 1 -- 172.21.15.76:0/1826916095 shutdown_connections 2024-03-18T15:00:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.413+0000 7f93ff27e640 1 -- 172.21.15.76:0/1826916095 wait complete. 2024-03-18T15:00:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.413+0000 7f93ff27e640 1 Processor -- start 2024-03-18T15:00:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.413+0000 7f93ff27e640 1 -- start start 2024-03-18T15:00:35.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.413+0000 7f93ff27e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93f8068e60 0x7f93f8198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93ff27e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93f8069800 0x7f93f8198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93ff27e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93f81078b0 0x7f93f819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93ff27e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f93f8111ba0 con 0x7f93f81078b0 2024-03-18T15:00:35.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93ff27e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f93f8111a20 con 0x7f93f8068e60 2024-03-18T15:00:35.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93ff27e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f93f8111d20 con 0x7f93f8069800 2024-03-18T15:00:35.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93fd81b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93f81078b0 0x7f93f819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93fd01a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93f8068e60 0x7f93f8198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93fd81b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93f81078b0 0x7f93f819c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33660/0 (socket says 172.21.15.76:33660) 2024-03-18T15:00:35.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93fd81b640 1 -- 172.21.15.76:0/2687977591 learned_addr learned my addr 172.21.15.76:0/2687977591 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:35.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93fc819640 1 --2- 172.21.15.76:0/2687977591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93f8069800 0x7f93f8198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93fd81b640 1 -- 172.21.15.76:0/2687977591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93f8069800 msgr2=0x7f93f8198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:35.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93fd81b640 1 --2- 172.21.15.76:0/2687977591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93f8069800 0x7f93f8198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.417 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93fd81b640 1 -- 172.21.15.76:0/2687977591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93f8068e60 msgr2=0x7f93f8198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:35.418 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93fd81b640 1 --2- 172.21.15.76:0/2687977591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93f8068e60 0x7f93f8198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.418 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93fd81b640 1 -- 172.21.15.76:0/2687977591 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f819d030 con 0x7f93f81078b0 2024-03-18T15:00:35.418 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93fd01a640 1 --2- 172.21.15.76:0/2687977591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93f8068e60 0x7f93f8198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:00:35.418 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93fd81b640 1 --2- 172.21.15.76:0/2687977591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93f81078b0 0x7f93f819c900 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f93f000ea60 tx=0x7f93f000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:35.418 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93ee7fc640 1 -- 172.21.15.76:0/2687977591 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f000ce40 con 0x7f93f81078b0 2024-03-18T15:00:35.418 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93ee7fc640 1 -- 172.21.15.76:0/2687977591 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93f0005020 con 0x7f93f81078b0 2024-03-18T15:00:35.418 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93ff27e640 1 -- 172.21.15.76:0/2687977591 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93f819d320 con 0x7f93f81078b0 2024-03-18T15:00:35.418 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93ee7fc640 1 -- 172.21.15.76:0/2687977591 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f0010690 con 0x7f93f81078b0 2024-03-18T15:00:35.418 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.414+0000 7f93ff27e640 1 -- 172.21.15.76:0/2687977591 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93f81a4bb0 con 0x7f93f81078b0 2024-03-18T15:00:35.424 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.416+0000 7f93ee7fc640 1 -- 172.21.15.76:0/2687977591 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f93f0005520 con 0x7f93f81078b0 2024-03-18T15:00:35.424 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.417+0000 7f9401e62740 1 Processor -- start 2024-03-18T15:00:35.425 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.417+0000 7f9401e62740 1 -- start start 2024-03-18T15:00:35.425 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.417+0000 7f93ee7fc640 1 --2- 172.21.15.76:0/2687977591 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f93e40a2a10 0x7f93e40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.425 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.417+0000 7f93ee7fc640 1 -- 172.21.15.76:0/2687977591 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7f93f00cf280 con 0x7f93f81078b0 2024-03-18T15:00:35.425 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.417+0000 7f9401e62740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc8f8130 0x555fcc8f8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.425 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.417+0000 7f93fc819640 1 --2- 172.21.15.76:0/2687977591 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f93e40a2a10 0x7f93e40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.425 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.417+0000 7f9401e62740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8a50 0x555fcc9056f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.425 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.417+0000 7f9401e62740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc905c30 0x555fcc908020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.425 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.417+0000 7f9401e62740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x555fcc85ca40 con 0x555fcc905c30 2024-03-18T15:00:35.425 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.417+0000 7f9401e62740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555fcc805920 con 0x555fcc8f8a50 2024-03-18T15:00:35.425 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.417+0000 7f9401e62740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555fcc858ae0 con 0x555fcc8f8130 2024-03-18T15:00:35.425 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.417+0000 7f93fd01a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc8f8130 0x555fcc8f8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.425 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.417+0000 7f93fd81b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc905c30 0x555fcc908020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.425 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.417+0000 7f93fd01a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc8f8130 0x555fcc8f8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:39508/0 (socket says 172.21.15.76:39508) 2024-03-18T15:00:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.417+0000 7f93fd81b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc905c30 0x555fcc908020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33668/0 (socket says 172.21.15.76:33668) 2024-03-18T15:00:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.417+0000 7f93fd01a640 1 -- 172.21.15.76:0/3352047054 learned_addr learned my addr 172.21.15.76:0/3352047054 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.418+0000 7f93fc819640 1 --2- 172.21.15.76:0/3352047054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8a50 0x555fcc9056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.418+0000 7f93fd81b640 1 -- 172.21.15.76:0/3352047054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc8f8130 msgr2=0x555fcc8f8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.418+0000 7f93fd81b640 1 --2- 172.21.15.76:0/3352047054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc8f8130 0x555fcc8f8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.418+0000 7f93fd81b640 1 -- 172.21.15.76:0/3352047054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8a50 msgr2=0x555fcc9056f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.418+0000 7f93fd81b640 1 --2- 172.21.15.76:0/3352047054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8a50 0x555fcc9056f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.418+0000 7f93fd81b640 1 -- 172.21.15.76:0/3352047054 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fcc81d460 con 0x555fcc905c30 2024-03-18T15:00:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.418+0000 7f93fd01a640 1 --2- 172.21.15.76:0/3352047054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc8f8130 0x555fcc8f8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:00:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.418+0000 7f93fc819640 1 --2- 172.21.15.76:0/2687977591 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f93e40a2a10 0x7f93e40a4ed0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f93f8077aa0 tx=0x7f93e8011920 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.418+0000 7f93fd81b640 1 --2- 172.21.15.76:0/3352047054 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc905c30 0x555fcc908020 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f93f00cc9f0 tx=0x7f93f009b320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbbf8c3c6a24e079 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.418+0000 7f93e37fe640 1 -- 172.21.15.76:0/3352047054 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f009fa50 con 0x555fcc905c30 2024-03-18T15:00:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.418+0000 7f93e37fe640 1 -- 172.21.15.76:0/3352047054 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93f00a40a0 con 0x555fcc905c30 2024-03-18T15:00:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.418+0000 7f93e37fe640 1 -- 172.21.15.76:0/3352047054 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f00860e0 con 0x555fcc905c30 2024-03-18T15:00:35.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.419+0000 7f9401e62740 1 -- 172.21.15.76:0/3352047054 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc905c30 msgr2=0x555fcc908020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:35.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.419+0000 7f9401e62740 1 --2- 172.21.15.76:0/3352047054 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc905c30 0x555fcc908020 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f93f00cc9f0 tx=0x7f93f009b320 comp rx=0 tx=0).stop 2024-03-18T15:00:35.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.419+0000 7f9401e62740 1 -- 172.21.15.76:0/3352047054 shutdown_connections 2024-03-18T15:00:35.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.419+0000 7f9401e62740 1 --2- 172.21.15.76:0/3352047054 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc905c30 0x555fcc908020 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.419+0000 7f9401e62740 1 --2- 172.21.15.76:0/3352047054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8a50 0x555fcc9056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.419+0000 7f9401e62740 1 --2- 172.21.15.76:0/3352047054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc8f8130 0x555fcc8f8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.419+0000 7f9401e62740 1 -- 172.21.15.76:0/3352047054 >> 172.21.15.76:0/3352047054 conn(0x555fcc8f2a50 msgr2=0x555fcc8f2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:35.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.419+0000 7f9401e62740 1 -- 172.21.15.76:0/3352047054 shutdown_connections 2024-03-18T15:00:35.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.419+0000 7f9401e62740 1 -- 172.21.15.76:0/3352047054 wait complete. 2024-03-18T15:00:35.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.419+0000 7f9401e62740 1 -- start start 2024-03-18T15:00:35.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.419+0000 7f9401e62740 1 Processor -- start 2024-03-18T15:00:35.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f9401e62740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8130 0x555fcc902d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f9401e62740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc8f8a50 0x555fcc903250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f9401e62740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc905c30 0x555fcc903790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.427 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f9401e62740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x555fcc908860 con 0x555fcc8f8a50 2024-03-18T15:00:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f9401e62740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555fcc818d70 con 0x555fcc8f8130 2024-03-18T15:00:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f9401e62740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555fcc9089e0 con 0x555fcc905c30 2024-03-18T15:00:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f9401e62740 10 client.0 _ll_register_callbacks cb 0x555fcc570a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:00:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f93fc819640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc8f8a50 0x555fcc903250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f93fd01a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8130 0x555fcc902d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f93fc819640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc8f8a50 0x555fcc903250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33670/0 (socket says 172.21.15.76:33670) 2024-03-18T15:00:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f93fd01a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8130 0x555fcc902d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:59972/0 (socket says 172.21.15.76:59972) 2024-03-18T15:00:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f93fc819640 1 -- 172.21.15.76:0/2064472002 learned_addr learned my addr 172.21.15.76:0/2064472002 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f93fd81b640 1 --2- 172.21.15.76:0/2064472002 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc905c30 0x555fcc903790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f93fd01a640 1 -- 172.21.15.76:0/2064472002 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc905c30 msgr2=0x555fcc903790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f93fd01a640 1 --2- 172.21.15.76:0/2064472002 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc905c30 0x555fcc903790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f93fd01a640 1 -- 172.21.15.76:0/2064472002 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc8f8a50 msgr2=0x555fcc903250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f93fd01a640 1 --2- 172.21.15.76:0/2064472002 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc8f8a50 0x555fcc903250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f93fd01a640 1 -- 172.21.15.76:0/2064472002 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fcc571210 con 0x555fcc8f8130 2024-03-18T15:00:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f93fc819640 1 --2- 172.21.15.76:0/2064472002 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc8f8a50 0x555fcc903250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:00:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f93fd01a640 1 --2- 172.21.15.76:0/2064472002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8130 0x555fcc902d10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f93f4006010 tx=0x7f93f4004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b900b5c4d75132b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f93e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T15:00:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f93e0ff9640 1 -- 172.21.15.76:0/2064472002 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f4004430 con 0x555fcc8f8130 2024-03-18T15:00:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f9401e62740 1 -- 172.21.15.76:0/2064472002 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93f00a40a0 con 0x555fcc8f8130 2024-03-18T15:00:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.420+0000 7f93e0ff9640 1 -- 172.21.15.76:0/2064472002 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7f93f402c8b0 con 0x555fcc8f8130 2024-03-18T15:00:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.421+0000 7f93e0ff9640 1 -- 172.21.15.76:0/2064472002 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f93f402cab0 con 0x555fcc8f8130 2024-03-18T15:00:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.421+0000 7f9401e62740 10 client.26297 fetch_fsmap learned FSMap version 41 2024-03-18T15:00:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.421+0000 7f9401e62740 1 -- 172.21.15.76:0/2064472002 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x555fcc63d620 con 0x555fcc8f8130 2024-03-18T15:00:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.421+0000 7f93e0ff9640 1 -- 172.21.15.76:0/2064472002 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93f00a40a0 con 0x555fcc8f8130 2024-03-18T15:00:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.421+0000 7f93e0ff9640 1 -- 172.21.15.76:0/2064472002 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f93f404f070 con 0x555fcc8f8130 2024-03-18T15:00:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.421+0000 7f9401e62740 10 client.26297 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:00:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.421+0000 7f9401e62740 10 client.26297 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:00:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.421+0000 7f9401e62740 20 client.26297 populate_metadata read hostname 'smithi076' 2024-03-18T15:00:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.421+0000 7f9401e62740 1 --2- 172.21.15.76:0/2064472002 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x555fcc915f10 0x555fcc918300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.421+0000 7f9401e62740 4 client.26297 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:00:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.421+0000 7f9401e62740 1 -- 172.21.15.76:0/2064472002 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555fcc869f80 con 0x555fcc915f10 2024-03-18T15:00:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.421+0000 7f93fc819640 1 --2- 172.21.15.76:0/2064472002 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x555fcc915f10 0x555fcc918300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.422+0000 7f93fc819640 1 --2- 172.21.15.76:0/2064472002 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x555fcc915f10 0x555fcc918300 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8444e42b7cae8a4b server_cookie=b3e79d5486787da7 in_seq=0 out_seq=0 2024-03-18T15:00:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.422+0000 7f93e0ff9640 10 client.26297 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:00:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.422+0000 7f93e0ff9640 1 -- 172.21.15.76:0/2064472002 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555fcc869f80 con 0x555fcc915f10 2024-03-18T15:00:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.422+0000 7f93e0ff9640 10 client.26297 handle_command_reply: tid=0 2024-03-18T15:00:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.422+0000 7f9401e62740 1 client.26297 shutdown 2024-03-18T15:00:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.423+0000 7f9401e62740 1 -- 172.21.15.76:0/2064472002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8130 msgr2=0x555fcc902d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.423+0000 7f9401e62740 1 --2- 172.21.15.76:0/2064472002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8130 0x555fcc902d10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f93f4006010 tx=0x7f93f4004040 comp rx=0 tx=0).stop 2024-03-18T15:00:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.423+0000 7f9401e62740 1 -- 172.21.15.76:0/2064472002 shutdown_connections 2024-03-18T15:00:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.423+0000 7f9401e62740 1 --2- 172.21.15.76:0/2064472002 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc905c30 0x555fcc903790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.423+0000 7f9401e62740 1 --2- 172.21.15.76:0/2064472002 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x555fcc915f10 0x555fcc918300 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.423+0000 7f9401e62740 1 --2- 172.21.15.76:0/2064472002 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc8f8a50 0x555fcc903250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.423+0000 7f9401e62740 1 --2- 172.21.15.76:0/2064472002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8130 0x555fcc902d10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.423+0000 7f93e0ff9640 0 client.26297 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:00:35.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.423+0000 7f9401e62740 1 -- 172.21.15.76:0/2064472002 >> 172.21.15.76:0/2064472002 conn(0x555fcc8f2a50 msgr2=0x555fcc8f47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:35.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.423+0000 7f9401e62740 1 -- 172.21.15.76:0/2064472002 shutdown_connections 2024-03-18T15:00:35.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.423+0000 7f9401e62740 1 -- 172.21.15.76:0/2064472002 wait complete. 2024-03-18T15:00:35.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.423+0000 7f9401e62740 20 client.26297 trim_cache size 0 max 16384 2024-03-18T15:00:35.444 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f9401e62740 1 Processor -- start 2024-03-18T15:00:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f9401e62740 1 -- start start 2024-03-18T15:00:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f9401e62740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc8f8130 0x555fcc911600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f9401e62740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8a50 0x555fcc91dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f9401e62740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc905c30 0x555fcc90dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f9401e62740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f93f81ada80 con 0x555fcc905c30 2024-03-18T15:00:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f9401e62740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f93f81ab350 con 0x555fcc8f8a50 2024-03-18T15:00:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f9401e62740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f93f81a91c0 con 0x555fcc8f8130 2024-03-18T15:00:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f93fd81b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc905c30 0x555fcc90dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f93fc819640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8a50 0x555fcc91dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f93fd81b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc905c30 0x555fcc90dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33686/0 (socket says 172.21.15.76:33686) 2024-03-18T15:00:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f93fd81b640 1 -- 172.21.15.76:0/4175992467 learned_addr learned my addr 172.21.15.76:0/4175992467 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f93fd01a640 1 --2- 172.21.15.76:0/4175992467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc8f8130 0x555fcc911600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f93fd81b640 1 -- 172.21.15.76:0/4175992467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc8f8130 msgr2=0x555fcc911600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:35.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f93fd81b640 1 --2- 172.21.15.76:0/4175992467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc8f8130 0x555fcc911600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f93fd81b640 1 -- 172.21.15.76:0/4175992467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8a50 msgr2=0x555fcc91dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:35.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f93fd81b640 1 --2- 172.21.15.76:0/4175992467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8a50 0x555fcc91dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f93fd81b640 1 -- 172.21.15.76:0/4175992467 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fcc908390 con 0x555fcc905c30 2024-03-18T15:00:35.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f93fc819640 1 --2- 172.21.15.76:0/4175992467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8a50 0x555fcc91dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:00:35.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f93fd81b640 1 --2- 172.21.15.76:0/4175992467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc905c30 0x555fcc90dc20 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f93f00a3b70 tx=0x7f93f0005920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7606bb7592a1c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:35.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.439+0000 7f93e2ffd640 1 -- 172.21.15.76:0/4175992467 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f0099c00 con 0x555fcc905c30 2024-03-18T15:00:35.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.440+0000 7f93e2ffd640 1 -- 172.21.15.76:0/4175992467 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93f00051d0 con 0x555fcc905c30 2024-03-18T15:00:35.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.440+0000 7f93e2ffd640 1 -- 172.21.15.76:0/4175992467 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f00a3c10 con 0x555fcc905c30 2024-03-18T15:00:35.446 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.440+0000 7f9401e62740 1 -- 172.21.15.76:0/4175992467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc905c30 msgr2=0x555fcc90dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:35.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.440+0000 7f9401e62740 1 --2- 172.21.15.76:0/4175992467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc905c30 0x555fcc90dc20 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f93f00a3b70 tx=0x7f93f0005920 comp rx=0 tx=0).stop 2024-03-18T15:00:35.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.440+0000 7f9401e62740 1 -- 172.21.15.76:0/4175992467 shutdown_connections 2024-03-18T15:00:35.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.440+0000 7f9401e62740 1 --2- 172.21.15.76:0/4175992467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc905c30 0x555fcc90dc20 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.440+0000 7f9401e62740 1 --2- 172.21.15.76:0/4175992467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8a50 0x555fcc91dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.440+0000 7f9401e62740 1 --2- 172.21.15.76:0/4175992467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc8f8130 0x555fcc911600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.440+0000 7f9401e62740 1 -- 172.21.15.76:0/4175992467 >> 172.21.15.76:0/4175992467 conn(0x555fcc8f2a50 msgr2=0x555fcc8f1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:35.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.440+0000 7f9401e62740 1 -- 172.21.15.76:0/4175992467 shutdown_connections 2024-03-18T15:00:35.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.440+0000 7f9401e62740 1 -- 172.21.15.76:0/4175992467 wait complete. 2024-03-18T15:00:35.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.441+0000 7f9401e62740 1 -- start start 2024-03-18T15:00:35.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.441+0000 7f9401e62740 1 Processor -- start 2024-03-18T15:00:35.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.441+0000 7f9401e62740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc8f8130 0x555fcc90dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.441+0000 7f9401e62740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8a50 0x555fcc90a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.441+0000 7f9401e62740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc905c30 0x555fcc90a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.441+0000 7f9401e62740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f93f81a9810 con 0x555fcc8f8130 2024-03-18T15:00:35.447 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.441+0000 7f9401e62740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f93f81aac80 con 0x555fcc8f8a50 2024-03-18T15:00:35.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.441+0000 7f9401e62740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f93f81a9ee0 con 0x555fcc905c30 2024-03-18T15:00:35.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.441+0000 7f9401e62740 10 client.0 _ll_register_callbacks cb 0x7f93f4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:00:35.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.441+0000 7f93fd01a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc8f8130 0x555fcc90dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.441+0000 7f93fd01a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc8f8130 0x555fcc90dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33698/0 (socket says 172.21.15.76:33698) 2024-03-18T15:00:35.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.441+0000 7f93fd01a640 1 -- 172.21.15.76:0/2624932414 learned_addr learned my addr 172.21.15.76:0/2624932414 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:35.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.441+0000 7f93fc819640 1 --2- 172.21.15.76:0/2624932414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8a50 0x555fcc90a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.441+0000 7f93fd81b640 1 --2- 172.21.15.76:0/2624932414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc905c30 0x555fcc90a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.442+0000 7f93fd01a640 1 -- 172.21.15.76:0/2624932414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc905c30 msgr2=0x555fcc90a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:35.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.442+0000 7f93fd01a640 1 --2- 172.21.15.76:0/2624932414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc905c30 0x555fcc90a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.448 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.442+0000 7f93fd01a640 1 -- 172.21.15.76:0/2624932414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8a50 msgr2=0x555fcc90a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:35.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.442+0000 7f93fd01a640 1 --2- 172.21.15.76:0/2624932414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8a50 0x555fcc90a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.442+0000 7f93fd01a640 1 -- 172.21.15.76:0/2624932414 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fcc90aa90 con 0x555fcc8f8130 2024-03-18T15:00:35.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.442+0000 7f93fc819640 1 --2- 172.21.15.76:0/2624932414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8a50 0x555fcc90a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:00:35.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.442+0000 7f93fd01a640 1 --2- 172.21.15.76:0/2624932414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc8f8130 0x555fcc90dbf0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f93f4006010 tx=0x7f93f402ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f48dc54b3d045d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:35.449 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.442+0000 7f93e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:00:35.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.442+0000 7f93e0ff9640 1 -- 172.21.15.76:0/2624932414 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f4055040 con 0x555fcc8f8130 2024-03-18T15:00:35.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.442+0000 7f93e0ff9640 1 -- 172.21.15.76:0/2624932414 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7f93f40035f0 con 0x555fcc8f8130 2024-03-18T15:00:35.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.442+0000 7f9401e62740 1 -- 172.21.15.76:0/2624932414 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93f00051d0 con 0x555fcc8f8130 2024-03-18T15:00:35.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.442+0000 7f93e0ff9640 1 -- 172.21.15.76:0/2624932414 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f93f4009520 con 0x555fcc8f8130 2024-03-18T15:00:35.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.442+0000 7f9401e62740 10 client.16440 fetch_fsmap learned FSMap version 41 2024-03-18T15:00:35.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.442+0000 7f9401e62740 1 -- 172.21.15.76:0/2624932414 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x555fcc97cf40 con 0x555fcc8f8130 2024-03-18T15:00:35.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.442+0000 7f93e0ff9640 1 -- 172.21.15.76:0/2624932414 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93f00051d0 con 0x555fcc8f8130 2024-03-18T15:00:35.450 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.442+0000 7f93e0ff9640 1 -- 172.21.15.76:0/2624932414 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f93f4060030 con 0x555fcc8f8130 2024-03-18T15:00:35.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.443+0000 7f9401e62740 10 client.16440 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:00:35.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.443+0000 7f9401e62740 10 client.16440 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:00:35.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.443+0000 7f9401e62740 20 client.16440 populate_metadata read hostname 'smithi076' 2024-03-18T15:00:35.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.443+0000 7f9401e62740 1 --2- 172.21.15.76:0/2624932414 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x555fcc97d740 0x555fcc97fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:35.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.443+0000 7f9401e62740 4 client.16440 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:00:35.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.443+0000 7f9401e62740 1 -- 172.21.15.76:0/2624932414 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555fcc8642d0 con 0x555fcc97d740 2024-03-18T15:00:35.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.443+0000 7f93fc819640 1 --2- 172.21.15.76:0/2624932414 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x555fcc97d740 0x555fcc97fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:35.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.443+0000 7f93fc819640 1 --2- 172.21.15.76:0/2624932414 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x555fcc97d740 0x555fcc97fb60 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c062ee36bc821515 server_cookie=72e504d967b07178 in_seq=0 out_seq=0 2024-03-18T15:00:35.452 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.443+0000 7f93e0ff9640 10 client.16440 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:00:35.519 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.517+0000 7f93e0ff9640 1 -- 172.21.15.76:0/2624932414 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555fcc8642d0 con 0x555fcc97d740 2024-03-18T15:00:35.519 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.517+0000 7f93e0ff9640 10 client.16440 handle_command_reply: tid=0 2024-03-18T15:00:35.519 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.517+0000 7f9401e62740 1 client.16440 shutdown 2024-03-18T15:00:35.519 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.518+0000 7f9401e62740 1 -- 172.21.15.76:0/2624932414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc8f8130 msgr2=0x555fcc90dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:35.519 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.518+0000 7f9401e62740 1 --2- 172.21.15.76:0/2624932414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc8f8130 0x555fcc90dbf0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f93f4006010 tx=0x7f93f402ce80 comp rx=0 tx=0).stop 2024-03-18T15:00:35.520 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.518+0000 7f9401e62740 1 -- 172.21.15.76:0/2624932414 shutdown_connections 2024-03-18T15:00:35.520 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.518+0000 7f9401e62740 1 --2- 172.21.15.76:0/2624932414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555fcc905c30 0x555fcc90a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.520 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.518+0000 7f9401e62740 1 --2- 172.21.15.76:0/2624932414 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x555fcc97d740 0x555fcc97fb60 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.520 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.518+0000 7f9401e62740 1 --2- 172.21.15.76:0/2624932414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fcc8f8a50 0x555fcc90a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.520 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.518+0000 7f93e0ff9640 0 client.16440 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:00:35.520 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.518+0000 7f9401e62740 1 --2- 172.21.15.76:0/2624932414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555fcc8f8130 0x555fcc90dbf0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.520 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.518+0000 7f9401e62740 1 -- 172.21.15.76:0/2624932414 >> 172.21.15.76:0/2624932414 conn(0x555fcc8f2a50 msgr2=0x555fcc8f1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:35.520 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.518+0000 7f9401e62740 1 -- 172.21.15.76:0/2624932414 shutdown_connections 2024-03-18T15:00:35.520 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.518+0000 7f9401e62740 1 -- 172.21.15.76:0/2624932414 wait complete. 2024-03-18T15:00:35.520 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.518+0000 7f9401e62740 20 client.16440 trim_cache size 0 max 16384 2024-03-18T15:00:35.520 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:00:35.521 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:00:35.521 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:00:35.521 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:00:35.521 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.519+0000 7f93ff27e640 1 -- 172.21.15.76:0/2687977591 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f93e40a2a10 msgr2=0x7f93e40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:35.522 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.519+0000 7f93ff27e640 1 --2- 172.21.15.76:0/2687977591 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f93e40a2a10 0x7f93e40a4ed0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f93f8077aa0 tx=0x7f93e8011920 comp rx=0 tx=0).stop 2024-03-18T15:00:35.522 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.519+0000 7f93ff27e640 1 -- 172.21.15.76:0/2687977591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93f81078b0 msgr2=0x7f93f819c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:35.522 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.519+0000 7f93ff27e640 1 --2- 172.21.15.76:0/2687977591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93f81078b0 0x7f93f819c900 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f93f000ea60 tx=0x7f93f000ef20 comp rx=0 tx=0).stop 2024-03-18T15:00:35.522 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.519+0000 7f93ff27e640 1 -- 172.21.15.76:0/2687977591 shutdown_connections 2024-03-18T15:00:35.522 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.519+0000 7f93ff27e640 1 --2- 172.21.15.76:0/2687977591 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f93e40a2a10 0x7f93e40a4ed0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.522 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.519+0000 7f93ff27e640 1 --2- 172.21.15.76:0/2687977591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93f8069800 0x7f93f8198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.522 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.519+0000 7f93ff27e640 1 --2- 172.21.15.76:0/2687977591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93f81078b0 0x7f93f819c900 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.522 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.519+0000 7f93ff27e640 1 --2- 172.21.15.76:0/2687977591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93f8068e60 0x7f93f8198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:35.522 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.519+0000 7f93ff27e640 1 -- 172.21.15.76:0/2687977591 >> 172.21.15.76:0/2687977591 conn(0x7f93f8077530 msgr2=0x7f93f80fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:35.522 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.519+0000 7f93ff27e640 1 -- 172.21.15.76:0/2687977591 shutdown_connections 2024-03-18T15:00:35.522 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:35.519+0000 7f93ff27e640 1 -- 172.21.15.76:0/2687977591 wait complete. 2024-03-18T15:00:35.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 708969 9.48 MB/sec execute 276 sec latency 60.467 ms 2024-03-18T15:00:35.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:35 smithi077 ceph-mon[35865]: pgmap v230: 97 pgs: 97 active+clean; 213 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-18T15:00:36.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:35 smithi076 ceph-mon[32205]: pgmap v230: 97 pgs: 97 active+clean; 213 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-18T15:00:36.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:35 smithi174 ceph-mon[36787]: pgmap v230: 97 pgs: 97 active+clean; 213 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-18T15:00:36.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 710691 9.49 MB/sec execute 277 sec latency 14.192 ms 2024-03-18T15:00:37.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 712593 9.48 MB/sec execute 278 sec latency 21.856 ms 2024-03-18T15:00:38.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:37 smithi076 ceph-mon[32205]: pgmap v231: 97 pgs: 97 active+clean; 213 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-18T15:00:38.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:37 smithi174 ceph-mon[36787]: pgmap v231: 97 pgs: 97 active+clean; 213 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-18T15:00:38.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:37 smithi077 ceph-mon[35865]: pgmap v231: 97 pgs: 97 active+clean; 213 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-18T15:00:38.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 714305 9.49 MB/sec execute 279 sec latency 11.916 ms 2024-03-18T15:00:39.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 716101 9.48 MB/sec execute 280 sec latency 32.114 ms 2024-03-18T15:00:40.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:39 smithi076 ceph-mon[32205]: pgmap v232: 97 pgs: 97 active+clean; 209 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 335 op/s 2024-03-18T15:00:40.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:39 smithi174 ceph-mon[36787]: pgmap v232: 97 pgs: 97 active+clean; 209 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 335 op/s 2024-03-18T15:00:40.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:39 smithi077 ceph-mon[35865]: pgmap v232: 97 pgs: 97 active+clean; 209 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 335 op/s 2024-03-18T15:00:40.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 717768 9.49 MB/sec execute 281 sec latency 22.253 ms 2024-03-18T15:00:41.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 719730 9.48 MB/sec execute 282 sec latency 19.280 ms 2024-03-18T15:00:42.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:41 smithi076 ceph-mon[32205]: pgmap v233: 97 pgs: 97 active+clean; 209 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 231 op/s 2024-03-18T15:00:42.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:41 smithi174 ceph-mon[36787]: pgmap v233: 97 pgs: 97 active+clean; 209 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 231 op/s 2024-03-18T15:00:42.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:41 smithi077 ceph-mon[35865]: pgmap v233: 97 pgs: 97 active+clean; 209 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 231 op/s 2024-03-18T15:00:42.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 721450 9.49 MB/sec execute 283 sec latency 16.723 ms 2024-03-18T15:00:43.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 723637 9.49 MB/sec execute 284 sec latency 20.321 ms 2024-03-18T15:00:44.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:43 smithi076 ceph-mon[32205]: pgmap v234: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 334 op/s 2024-03-18T15:00:44.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:43 smithi174 ceph-mon[36787]: pgmap v234: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 334 op/s 2024-03-18T15:00:44.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:43 smithi077 ceph-mon[35865]: pgmap v234: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 334 op/s 2024-03-18T15:00:44.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 725082 9.49 MB/sec execute 285 sec latency 52.022 ms 2024-03-18T15:00:45.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 727333 9.50 MB/sec execute 286 sec latency 29.248 ms 2024-03-18T15:00:46.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:45 smithi076 ceph-mon[32205]: pgmap v235: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 234 op/s 2024-03-18T15:00:46.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:45 smithi174 ceph-mon[36787]: pgmap v235: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 234 op/s 2024-03-18T15:00:46.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:45 smithi077 ceph-mon[35865]: pgmap v235: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 234 op/s 2024-03-18T15:00:46.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 728727 9.49 MB/sec execute 287 sec latency 18.153 ms 2024-03-18T15:00:47.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 731133 9.50 MB/sec execute 288 sec latency 21.537 ms 2024-03-18T15:00:48.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:47 smithi076 ceph-mon[32205]: pgmap v236: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 234 op/s 2024-03-18T15:00:48.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:47 smithi174 ceph-mon[36787]: pgmap v236: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 234 op/s 2024-03-18T15:00:48.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:47 smithi077 ceph-mon[35865]: pgmap v236: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 234 op/s 2024-03-18T15:00:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.609+0000 7fb9d0353640 1 -- 172.21.15.76:0/803674718 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9c81006d0 msgr2=0x7fb9c8100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.609+0000 7fb9d0353640 1 --2- 172.21.15.76:0/803674718 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9c81006d0 0x7fb9c8100b50 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8009930 tx=0x7fb9b802bc30 comp rx=0 tx=0).stop 2024-03-18T15:00:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.609+0000 7fb9d0353640 1 -- 172.21.15.76:0/803674718 shutdown_connections 2024-03-18T15:00:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.609+0000 7fb9d0353640 1 --2- 172.21.15.76:0/803674718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9c8101090 0x7fb9c810d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.609+0000 7fb9d0353640 1 --2- 172.21.15.76:0/803674718 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9c81006d0 0x7fb9c8100b50 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.609+0000 7fb9d0353640 1 --2- 172.21.15.76:0/803674718 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9c81066e0 0x7fb9c8106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.609+0000 7fb9d0353640 1 -- 172.21.15.76:0/803674718 >> 172.21.15.76:0/803674718 conn(0x7fb9c80778f0 msgr2=0x7fb9c8077d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.609+0000 7fb9d0353640 1 -- 172.21.15.76:0/803674718 shutdown_connections 2024-03-18T15:00:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.609+0000 7fb9d0353640 1 -- 172.21.15.76:0/803674718 wait complete. 2024-03-18T15:00:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.610+0000 7fb9d0353640 1 Processor -- start 2024-03-18T15:00:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.610+0000 7fb9d0353640 1 -- start start 2024-03-18T15:00:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.610+0000 7fb9d0353640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9c81006d0 0x7fb9c8198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.610+0000 7fb9d0353640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9c8101090 0x7fb9c8198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.610+0000 7fb9d0353640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9c81066e0 0x7fb9c819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.610+0000 7fb9d0353640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c8110af0 con 0x7fb9c8101090 2024-03-18T15:00:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.610+0000 7fb9d0353640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c8110970 con 0x7fb9c81006d0 2024-03-18T15:00:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.610+0000 7fb9d0353640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c8110c70 con 0x7fb9c81066e0 2024-03-18T15:00:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.610+0000 7fb9cd8ee640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9c8101090 0x7fb9c8198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.610+0000 7fb9cd8ee640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9c8101090 0x7fb9c8198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60886/0 (socket says 172.21.15.76:60886) 2024-03-18T15:00:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.610+0000 7fb9cd8ee640 1 -- 172.21.15.76:0/3700504798 learned_addr learned my addr 172.21.15.76:0/3700504798 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.610+0000 7fb9ce0ef640 1 --2- 172.21.15.76:0/3700504798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9c81006d0 0x7fb9c8198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.610+0000 7fb9ce8f0640 1 --2- 172.21.15.76:0/3700504798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9c81066e0 0x7fb9c819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.610+0000 7fb9cd8ee640 1 -- 172.21.15.76:0/3700504798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9c81066e0 msgr2=0x7fb9c819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.610+0000 7fb9cd8ee640 1 --2- 172.21.15.76:0/3700504798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9c81066e0 0x7fb9c819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.611+0000 7fb9cd8ee640 1 -- 172.21.15.76:0/3700504798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9c81006d0 msgr2=0x7fb9c8198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.611+0000 7fb9cd8ee640 1 --2- 172.21.15.76:0/3700504798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9c81006d0 0x7fb9c8198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.611+0000 7fb9cd8ee640 1 -- 172.21.15.76:0/3700504798 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c819d000 con 0x7fb9c8101090 2024-03-18T15:00:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.611+0000 7fb9cd8ee640 1 --2- 172.21.15.76:0/3700504798 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9c8101090 0x7fb9c8198550 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8009230 tx=0x7fb9b802cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.611+0000 7fb9bf7fe640 1 -- 172.21.15.76:0/3700504798 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8041070 con 0x7fb9c8101090 2024-03-18T15:00:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.611+0000 7fb9d0353640 1 -- 172.21.15.76:0/3700504798 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9c819d290 con 0x7fb9c8101090 2024-03-18T15:00:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.611+0000 7fb9bf7fe640 1 -- 172.21.15.76:0/3700504798 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9b8004450 con 0x7fb9c8101090 2024-03-18T15:00:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.611+0000 7fb9bf7fe640 1 -- 172.21.15.76:0/3700504798 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8003e50 con 0x7fb9c8101090 2024-03-18T15:00:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.611+0000 7fb9d0353640 1 -- 172.21.15.76:0/3700504798 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9c81a4b80 con 0x7fb9c8101090 2024-03-18T15:00:48.621 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.613+0000 7fb9bf7fe640 1 -- 172.21.15.76:0/3700504798 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fb9b8044070 con 0x7fb9c8101090 2024-03-18T15:00:48.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.614+0000 7fb9bf7fe640 1 --2- 172.21.15.76:0/3700504798 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb9b40a2bb0 0x7fb9b40a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.614+0000 7fb9d3019740 1 Processor -- start 2024-03-18T15:00:48.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.614+0000 7fb9bf7fe640 1 -- 172.21.15.76:0/3700504798 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7fb9b80f05f0 con 0x7fb9c8101090 2024-03-18T15:00:48.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.614+0000 7fb9ce0ef640 1 --2- 172.21.15.76:0/3700504798 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb9b40a2bb0 0x7fb9b40a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.614+0000 7fb9d3019740 1 -- start start 2024-03-18T15:00:48.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9d3019740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 0x55fe368ae510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9d3019740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 0x55fe368b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9d3019740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368b7820 0x55fe368b9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9d3019740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55fe36812a40 con 0x55fe368aea50 2024-03-18T15:00:48.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9d3019740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fe367bb920 con 0x55fe368b7820 2024-03-18T15:00:48.622 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9d3019740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fe3680eae0 con 0x55fe368ae130 2024-03-18T15:00:48.623 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9ce0ef640 1 --2- 172.21.15.76:0/3700504798 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb9b40a2bb0 0x7fb9b40a5070 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4009b50 tx=0x7fb9c4005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:48.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9cd8ee640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 0x55fe368b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9ce0ef640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 0x55fe368ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9ce8f0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368b7820 0x55fe368b9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9cd8ee640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 0x55fe368b72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60902/0 (socket says 172.21.15.76:60902) 2024-03-18T15:00:48.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9ce0ef640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 0x55fe368ae510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:34178/0 (socket says 172.21.15.76:34178) 2024-03-18T15:00:48.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9cd8ee640 1 -- 172.21.15.76:0/3465745 learned_addr learned my addr 172.21.15.76:0/3465745 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:48.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9cd8ee640 1 -- 172.21.15.76:0/3465745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 msgr2=0x55fe368ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:48.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9cd8ee640 1 --2- 172.21.15.76:0/3465745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 0x55fe368ae510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9cd8ee640 1 -- 172.21.15.76:0/3465745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368b7820 msgr2=0x55fe368b9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:48.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9cd8ee640 1 --2- 172.21.15.76:0/3465745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368b7820 0x55fe368b9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9cd8ee640 1 -- 172.21.15.76:0/3465745 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe367d3460 con 0x55fe368aea50 2024-03-18T15:00:48.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9ce8f0640 1 --2- 172.21.15.76:0/3465745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368b7820 0x55fe368b9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:00:48.624 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9cd8ee640 1 --2- 172.21.15.76:0/3465745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 0x55fe368b72e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb9b80bcb30 tx=0x7fb9b8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5a3e8320173180 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9bcbfe640 1 -- 172.21.15.76:0/3465745 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b804b040 con 0x55fe368aea50 2024-03-18T15:00:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9bcbfe640 1 -- 172.21.15.76:0/3465745 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9b802cd20 con 0x55fe368aea50 2024-03-18T15:00:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.615+0000 7fb9bcbfe640 1 -- 172.21.15.76:0/3465745 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b80b94a0 con 0x55fe368aea50 2024-03-18T15:00:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.616+0000 7fb9d3019740 1 -- 172.21.15.76:0/3465745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 msgr2=0x55fe368b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.616+0000 7fb9d3019740 1 --2- 172.21.15.76:0/3465745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 0x55fe368b72e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb9b80bcb30 tx=0x7fb9b8009260 comp rx=0 tx=0).stop 2024-03-18T15:00:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.616+0000 7fb9d3019740 1 -- 172.21.15.76:0/3465745 shutdown_connections 2024-03-18T15:00:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.616+0000 7fb9d3019740 1 --2- 172.21.15.76:0/3465745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368b7820 0x55fe368b9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.616+0000 7fb9d3019740 1 --2- 172.21.15.76:0/3465745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 0x55fe368b72e0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.616+0000 7fb9d3019740 1 --2- 172.21.15.76:0/3465745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 0x55fe368ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.616+0000 7fb9d3019740 1 -- 172.21.15.76:0/3465745 >> 172.21.15.76:0/3465745 conn(0x55fe368a8a50 msgr2=0x55fe368a8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.616+0000 7fb9d3019740 1 -- 172.21.15.76:0/3465745 shutdown_connections 2024-03-18T15:00:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.617+0000 7fb9d3019740 1 -- 172.21.15.76:0/3465745 wait complete. 2024-03-18T15:00:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.617+0000 7fb9d3019740 1 -- start start 2024-03-18T15:00:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.617+0000 7fb9d3019740 1 Processor -- start 2024-03-18T15:00:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.617+0000 7fb9d3019740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 0x55fe368b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.617+0000 7fb9d3019740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368aea50 0x55fe368b4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.617+0000 7fb9d3019740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368b7820 0x55fe368b5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.617+0000 7fb9d3019740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55fe368ba210 con 0x55fe368b7820 2024-03-18T15:00:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.617+0000 7fb9d3019740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fe367ced70 con 0x55fe368aea50 2024-03-18T15:00:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.617+0000 7fb9d3019740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fe368ba390 con 0x55fe368ae130 2024-03-18T15:00:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.617+0000 7fb9d3019740 10 client.0 _ll_register_callbacks cb 0x55fe36526a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:00:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.617+0000 7fb9ce8f0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368b7820 0x55fe368b5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.617+0000 7fb9ce8f0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368b7820 0x55fe368b5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60910/0 (socket says 172.21.15.76:60910) 2024-03-18T15:00:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.617+0000 7fb9ce8f0640 1 -- 172.21.15.76:0/3889774941 learned_addr learned my addr 172.21.15.76:0/3889774941 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.617+0000 7fb9cd8ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368aea50 0x55fe368b4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.617+0000 7fb9ce0ef640 1 --2- 172.21.15.76:0/3889774941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 0x55fe368b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.618+0000 7fb9cd8ee640 1 -- 172.21.15.76:0/3889774941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 msgr2=0x55fe368b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.618+0000 7fb9cd8ee640 1 --2- 172.21.15.76:0/3889774941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 0x55fe368b48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.618+0000 7fb9cd8ee640 1 -- 172.21.15.76:0/3889774941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368b7820 msgr2=0x55fe368b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.618+0000 7fb9cd8ee640 1 --2- 172.21.15.76:0/3889774941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368b7820 0x55fe368b5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.618+0000 7fb9cd8ee640 1 -- 172.21.15.76:0/3889774941 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe36527210 con 0x55fe368aea50 2024-03-18T15:00:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.618+0000 7fb9ce8f0640 1 --2- 172.21.15.76:0/3889774941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368b7820 0x55fe368b5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:00:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.618+0000 7fb9cd8ee640 1 --2- 172.21.15.76:0/3889774941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368aea50 0x55fe368b4e00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb9b80bcff0 tx=0x7fb9b80b96d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fac2356d5d7e4d81 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.618+0000 7fb9b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T15:00:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.618+0000 7fb9b1ffb640 1 -- 172.21.15.76:0/3889774941 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b80d9030 con 0x55fe368aea50 2024-03-18T15:00:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.618+0000 7fb9d3019740 1 -- 172.21.15.76:0/3889774941 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9b802cd20 con 0x55fe368aea50 2024-03-18T15:00:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.618+0000 7fb9b1ffb640 1 -- 172.21.15.76:0/3889774941 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7fb9b80bb130 con 0x55fe368aea50 2024-03-18T15:00:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.618+0000 7fb9b1ffb640 1 -- 172.21.15.76:0/3889774941 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fb9b804b050 con 0x55fe368aea50 2024-03-18T15:00:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.618+0000 7fb9d3019740 10 client.26315 fetch_fsmap learned FSMap version 41 2024-03-18T15:00:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.618+0000 7fb9d3019740 1 -- 172.21.15.76:0/3889774941 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55fe365f3620 con 0x55fe368aea50 2024-03-18T15:00:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.619+0000 7fb9b1ffb640 1 -- 172.21.15.76:0/3889774941 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9b802cd20 con 0x55fe368aea50 2024-03-18T15:00:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.619+0000 7fb9b1ffb640 1 -- 172.21.15.76:0/3889774941 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7fb9b80a2030 con 0x55fe368aea50 2024-03-18T15:00:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.619+0000 7fb9d3019740 10 client.26315 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:00:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.619+0000 7fb9d3019740 10 client.26315 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:00:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.619+0000 7fb9d3019740 20 client.26315 populate_metadata read hostname 'smithi076' 2024-03-18T15:00:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.619+0000 7fb9d3019740 1 --2- 172.21.15.76:0/3889774941 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55fe368cb9d0 0x55fe368cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.619+0000 7fb9d3019740 4 client.26315 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:00:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.619+0000 7fb9d3019740 1 -- 172.21.15.76:0/3889774941 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe3681ff80 con 0x55fe368cb9d0 2024-03-18T15:00:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.619+0000 7fb9ce0ef640 1 --2- 172.21.15.76:0/3889774941 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55fe368cb9d0 0x55fe368cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.619+0000 7fb9b1ffb640 10 client.26315 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:00:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.619+0000 7fb9ce0ef640 1 --2- 172.21.15.76:0/3889774941 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55fe368cb9d0 0x55fe368cddc0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd6f5a49bd66c6e5 server_cookie=d37990ae5630875a in_seq=0 out_seq=0 2024-03-18T15:00:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.620+0000 7fb9b1ffb640 1 -- 172.21.15.76:0/3889774941 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fe3681ff80 con 0x55fe368cb9d0 2024-03-18T15:00:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.620+0000 7fb9b1ffb640 10 client.26315 handle_command_reply: tid=0 2024-03-18T15:00:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.620+0000 7fb9d3019740 1 client.26315 shutdown 2024-03-18T15:00:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.620+0000 7fb9d3019740 1 -- 172.21.15.76:0/3889774941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368aea50 msgr2=0x55fe368b4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.620+0000 7fb9d3019740 1 --2- 172.21.15.76:0/3889774941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368aea50 0x55fe368b4e00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb9b80bcff0 tx=0x7fb9b80b96d0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.620+0000 7fb9d3019740 1 -- 172.21.15.76:0/3889774941 shutdown_connections 2024-03-18T15:00:48.629 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.620+0000 7fb9d3019740 1 --2- 172.21.15.76:0/3889774941 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55fe368cb9d0 0x55fe368cddc0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.629 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.620+0000 7fb9d3019740 1 --2- 172.21.15.76:0/3889774941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368b7820 0x55fe368b5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.629 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.620+0000 7fb9b1ffb640 0 client.26315 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:00:48.629 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.620+0000 7fb9d3019740 1 --2- 172.21.15.76:0/3889774941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368aea50 0x55fe368b4e00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.629 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.620+0000 7fb9d3019740 1 --2- 172.21.15.76:0/3889774941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 0x55fe368b48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.629 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.620+0000 7fb9d3019740 1 -- 172.21.15.76:0/3889774941 >> 172.21.15.76:0/3889774941 conn(0x55fe368a8a50 msgr2=0x55fe368aa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:48.629 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.620+0000 7fb9d3019740 1 -- 172.21.15.76:0/3889774941 shutdown_connections 2024-03-18T15:00:48.629 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.620+0000 7fb9d3019740 1 -- 172.21.15.76:0/3889774941 wait complete. 2024-03-18T15:00:48.629 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.620+0000 7fb9d3019740 20 client.26315 trim_cache size 0 max 16384 2024-03-18T15:00:48.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.636+0000 7fb9d3019740 1 Processor -- start 2024-03-18T15:00:48.638 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.636+0000 7fb9d3019740 1 -- start start 2024-03-18T15:00:48.641 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.636+0000 7fb9d3019740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368ae130 0x55fe368c2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.641 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.636+0000 7fb9d3019740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 0x55fe368a7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.641 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.636+0000 7fb9d3019740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368b7820 0x55fe368b5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.641 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.636+0000 7fb9d3019740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c81ad9e0 con 0x55fe368aea50 2024-03-18T15:00:48.641 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.636+0000 7fb9d3019740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c81ab2b0 con 0x55fe368ae130 2024-03-18T15:00:48.641 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.636+0000 7fb9d3019740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c81a9140 con 0x55fe368b7820 2024-03-18T15:00:48.641 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.636+0000 7fb9cd8ee640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 0x55fe368a7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.642 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.637+0000 7fb9ce0ef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368ae130 0x55fe368c2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.642 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.637+0000 7fb9ce0ef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368ae130 0x55fe368c2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:49544/0 (socket says 172.21.15.76:49544) 2024-03-18T15:00:48.642 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.637+0000 7fb9cd8ee640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 0x55fe368a7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60922/0 (socket says 172.21.15.76:60922) 2024-03-18T15:00:48.642 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.637+0000 7fb9ce0ef640 1 -- 172.21.15.76:0/449931384 learned_addr learned my addr 172.21.15.76:0/449931384 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:48.642 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.637+0000 7fb9ce8f0640 1 --2- 172.21.15.76:0/449931384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368b7820 0x55fe368b5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.642 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.637+0000 7fb9ce0ef640 1 -- 172.21.15.76:0/449931384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368b7820 msgr2=0x55fe368b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:48.642 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.637+0000 7fb9ce0ef640 1 --2- 172.21.15.76:0/449931384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368b7820 0x55fe368b5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.642 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.637+0000 7fb9ce0ef640 1 -- 172.21.15.76:0/449931384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 msgr2=0x55fe368a7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:48.642 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.637+0000 7fb9ce0ef640 1 --2- 172.21.15.76:0/449931384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 0x55fe368a7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.642 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.637+0000 7fb9ce0ef640 1 -- 172.21.15.76:0/449931384 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe368a2fc0 con 0x55fe368ae130 2024-03-18T15:00:48.642 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.637+0000 7fb9cd8ee640 1 --2- 172.21.15.76:0/449931384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 0x55fe368a7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:00:48.642 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.637+0000 7fb9ce0ef640 1 --2- 172.21.15.76:0/449931384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368ae130 0x55fe368c2f20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb9c40118d0 tx=0x7fb9c4014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e13379ac7027e8da server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:48.642 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.637+0000 7fb9b3fff640 1 -- 172.21.15.76:0/449931384 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c402ab90 con 0x55fe368ae130 2024-03-18T15:00:48.642 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.637+0000 7fb9b3fff640 1 -- 172.21.15.76:0/449931384 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9c402ad30 con 0x55fe368ae130 2024-03-18T15:00:48.642 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.637+0000 7fb9b3fff640 1 -- 172.21.15.76:0/449931384 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4004a00 con 0x55fe368ae130 2024-03-18T15:00:48.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.638+0000 7fb9d3019740 1 -- 172.21.15.76:0/449931384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368ae130 msgr2=0x55fe368c2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:48.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.638+0000 7fb9d3019740 1 --2- 172.21.15.76:0/449931384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368ae130 0x55fe368c2f20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb9c40118d0 tx=0x7fb9c4014000 comp rx=0 tx=0).stop 2024-03-18T15:00:48.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.638+0000 7fb9d3019740 1 -- 172.21.15.76:0/449931384 shutdown_connections 2024-03-18T15:00:48.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.638+0000 7fb9d3019740 1 --2- 172.21.15.76:0/449931384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368b7820 0x55fe368b5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.638+0000 7fb9d3019740 1 --2- 172.21.15.76:0/449931384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 0x55fe368a7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.638+0000 7fb9d3019740 1 --2- 172.21.15.76:0/449931384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368ae130 0x55fe368c2f20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.638+0000 7fb9d3019740 1 -- 172.21.15.76:0/449931384 >> 172.21.15.76:0/449931384 conn(0x55fe368a8a50 msgr2=0x55fe368a7320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:48.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.638+0000 7fb9d3019740 1 -- 172.21.15.76:0/449931384 shutdown_connections 2024-03-18T15:00:48.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.638+0000 7fb9d3019740 1 -- 172.21.15.76:0/449931384 wait complete. 2024-03-18T15:00:48.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.638+0000 7fb9d3019740 1 -- start start 2024-03-18T15:00:48.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.638+0000 7fb9d3019740 1 Processor -- start 2024-03-18T15:00:48.646 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9d3019740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 0x55fe368b1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9d3019740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 0x55fe368b1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9d3019740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368b7820 0x55fe368b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9d3019740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c81a9790 con 0x55fe368aea50 2024-03-18T15:00:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9d3019740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c81aabe0 con 0x55fe368b7820 2024-03-18T15:00:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9d3019740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c81a9e40 con 0x55fe368ae130 2024-03-18T15:00:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9d3019740 10 client.0 _ll_register_callbacks cb 0x7fb9c0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:00:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9ce0ef640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 0x55fe368b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9cd8ee640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 0x55fe368b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9ce0ef640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 0x55fe368b1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:34206/0 (socket says 172.21.15.76:34206) 2024-03-18T15:00:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9ce0ef640 1 -- 172.21.15.76:0/1468468090 learned_addr learned my addr 172.21.15.76:0/1468468090 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:00:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9cd8ee640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 0x55fe368b1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60932/0 (socket says 172.21.15.76:60932) 2024-03-18T15:00:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9ce8f0640 1 --2- 172.21.15.76:0/1468468090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368b7820 0x55fe368b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9ce0ef640 1 -- 172.21.15.76:0/1468468090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368b7820 msgr2=0x55fe368b1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9ce0ef640 1 --2- 172.21.15.76:0/1468468090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368b7820 0x55fe368b1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9ce0ef640 1 -- 172.21.15.76:0/1468468090 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 msgr2=0x55fe368b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:00:48.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9ce0ef640 1 --2- 172.21.15.76:0/1468468090 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 0x55fe368b1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.639+0000 7fb9ce0ef640 1 -- 172.21.15.76:0/1468468090 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe36930b70 con 0x55fe368ae130 2024-03-18T15:00:48.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.640+0000 7fb9ce0ef640 1 --2- 172.21.15.76:0/1468468090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 0x55fe368b1410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb9c40042d0 tx=0x7fb9c40043d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b508b9cdcbb791e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:00:48.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.640+0000 7fb9b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-18T15:00:48.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.640+0000 7fb9b1ffb640 1 -- 172.21.15.76:0/1468468090 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c402ab90 con 0x55fe368ae130 2024-03-18T15:00:48.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.640+0000 7fb9b1ffb640 1 -- 172.21.15.76:0/1468468090 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7fb9c402a640 con 0x55fe368ae130 2024-03-18T15:00:48.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.640+0000 7fb9d3019740 1 -- 172.21.15.76:0/1468468090 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9c402ad30 con 0x55fe368ae130 2024-03-18T15:00:48.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.640+0000 7fb9b1ffb640 1 -- 172.21.15.76:0/1468468090 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fb9c4041050 con 0x55fe368ae130 2024-03-18T15:00:48.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.640+0000 7fb9d3019740 10 client.26308 fetch_fsmap learned FSMap version 41 2024-03-18T15:00:48.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.640+0000 7fb9d3019740 1 -- 172.21.15.76:0/1468468090 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55fe36931da0 con 0x55fe368ae130 2024-03-18T15:00:48.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.640+0000 7fb9b1ffb640 1 -- 172.21.15.76:0/1468468090 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9c402ad30 con 0x55fe368ae130 2024-03-18T15:00:48.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.640+0000 7fb9b1ffb640 1 -- 172.21.15.76:0/1468468090 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7fb9c401e4b0 con 0x55fe368ae130 2024-03-18T15:00:48.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.640+0000 7fb9d3019740 10 client.26308 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:00:48.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.640+0000 7fb9d3019740 10 client.26308 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:00:48.648 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.640+0000 7fb9d3019740 20 client.26308 populate_metadata read hostname 'smithi076' 2024-03-18T15:00:48.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.640+0000 7fb9d3019740 1 --2- 172.21.15.76:0/1468468090 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55fe368cb9d0 0x55fe369344f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:00:48.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.640+0000 7fb9d3019740 4 client.26308 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:00:48.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.640+0000 7fb9d3019740 1 -- 172.21.15.76:0/1468468090 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fe3681a2d0 con 0x55fe368cb9d0 2024-03-18T15:00:48.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.641+0000 7fb9cd8ee640 1 --2- 172.21.15.76:0/1468468090 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55fe368cb9d0 0x55fe369344f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:00:48.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.641+0000 7fb9cd8ee640 1 --2- 172.21.15.76:0/1468468090 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55fe368cb9d0 0x55fe369344f0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e18a10ced89edef server_cookie=cf5f6407c1c625de in_seq=0 out_seq=0 2024-03-18T15:00:48.649 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.641+0000 7fb9b1ffb640 10 client.26308 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:00:48.717 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.716+0000 7fb9b1ffb640 1 -- 172.21.15.76:0/1468468090 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fe3681a2d0 con 0x55fe368cb9d0 2024-03-18T15:00:48.717 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.716+0000 7fb9b1ffb640 10 client.26308 handle_command_reply: tid=0 2024-03-18T15:00:48.718 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.717+0000 7fb9d3019740 1 client.26308 shutdown 2024-03-18T15:00:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.720+0000 7fb9d3019740 1 -- 172.21.15.76:0/1468468090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 msgr2=0x55fe368b1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.720+0000 7fb9d3019740 1 --2- 172.21.15.76:0/1468468090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 0x55fe368b1410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb9c40042d0 tx=0x7fb9c40043d0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.721+0000 7fb9d3019740 1 -- 172.21.15.76:0/1468468090 shutdown_connections 2024-03-18T15:00:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.721+0000 7fb9d3019740 1 --2- 172.21.15.76:0/1468468090 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55fe368cb9d0 0x55fe369344f0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.721+0000 7fb9d3019740 1 --2- 172.21.15.76:0/1468468090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fe368b7820 0x55fe368b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.721+0000 7fb9d3019740 1 --2- 172.21.15.76:0/1468468090 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe368aea50 0x55fe368b1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.721+0000 7fb9d3019740 1 --2- 172.21.15.76:0/1468468090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe368ae130 0x55fe368b1410 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.721+0000 7fb9d3019740 1 -- 172.21.15.76:0/1468468090 >> 172.21.15.76:0/1468468090 conn(0x55fe368a8a50 msgr2=0x55fe368a7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:48.723 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.722+0000 7fb9b1ffb640 0 client.26308 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:00:48.724 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.723+0000 7fb9d3019740 1 -- 172.21.15.76:0/1468468090 shutdown_connections 2024-03-18T15:00:48.724 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.724+0000 7fb9d3019740 1 -- 172.21.15.76:0/1468468090 wait complete. 2024-03-18T15:00:48.724 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.724+0000 7fb9d3019740 20 client.26308 trim_cache size 0 max 16384 2024-03-18T15:00:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:00:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:00:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:00:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:00:48.726 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.725+0000 7fb9d0353640 1 -- 172.21.15.76:0/3700504798 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb9b40a2bb0 msgr2=0x7fb9b40a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:48.726 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.725+0000 7fb9d0353640 1 --2- 172.21.15.76:0/3700504798 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb9b40a2bb0 0x7fb9b40a5070 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4009b50 tx=0x7fb9c4005f90 comp rx=0 tx=0).stop 2024-03-18T15:00:48.726 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.725+0000 7fb9d0353640 1 -- 172.21.15.76:0/3700504798 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9c8101090 msgr2=0x7fb9c8198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:00:48.726 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.725+0000 7fb9d0353640 1 --2- 172.21.15.76:0/3700504798 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9c8101090 0x7fb9c8198550 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8009230 tx=0x7fb9b802cb30 comp rx=0 tx=0).stop 2024-03-18T15:00:48.726 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.725+0000 7fb9d0353640 1 -- 172.21.15.76:0/3700504798 shutdown_connections 2024-03-18T15:00:48.727 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.726+0000 7fb9d0353640 1 --2- 172.21.15.76:0/3700504798 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb9b40a2bb0 0x7fb9b40a5070 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.727 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.726+0000 7fb9d0353640 1 --2- 172.21.15.76:0/3700504798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9c81066e0 0x7fb9c819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.727 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.726+0000 7fb9d0353640 1 --2- 172.21.15.76:0/3700504798 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9c8101090 0x7fb9c8198550 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.727 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.726+0000 7fb9d0353640 1 --2- 172.21.15.76:0/3700504798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9c81006d0 0x7fb9c8198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:00:48.727 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.726+0000 7fb9d0353640 1 -- 172.21.15.76:0/3700504798 >> 172.21.15.76:0/3700504798 conn(0x7fb9c80778f0 msgr2=0x7fb9c8069c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:00:48.727 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.726+0000 7fb9d0353640 1 -- 172.21.15.76:0/3700504798 shutdown_connections 2024-03-18T15:00:48.727 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:00:48.726+0000 7fb9d0353640 1 -- 172.21.15.76:0/3700504798 wait complete. 2024-03-18T15:00:48.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 732237 9.49 MB/sec execute 289 sec latency 52.834 ms 2024-03-18T15:00:49.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 734388 9.50 MB/sec execute 290 sec latency 61.075 ms 2024-03-18T15:00:50.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:49 smithi076 ceph-mon[32205]: pgmap v237: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 337 op/s 2024-03-18T15:00:50.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:49 smithi174 ceph-mon[36787]: pgmap v237: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 337 op/s 2024-03-18T15:00:50.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:49 smithi077 ceph-mon[35865]: pgmap v237: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 337 op/s 2024-03-18T15:00:50.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 735704 9.49 MB/sec execute 291 sec latency 20.444 ms 2024-03-18T15:00:51.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 737811 9.50 MB/sec execute 292 sec latency 25.245 ms 2024-03-18T15:00:52.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:51 smithi076 ceph-mon[32205]: pgmap v238: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 207 op/s 2024-03-18T15:00:52.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:51 smithi174 ceph-mon[36787]: pgmap v238: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 207 op/s 2024-03-18T15:00:52.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:51 smithi077 ceph-mon[35865]: pgmap v238: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 207 op/s 2024-03-18T15:00:52.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 739253 9.49 MB/sec execute 293 sec latency 46.857 ms 2024-03-18T15:00:53.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 741572 9.50 MB/sec execute 294 sec latency 15.786 ms 2024-03-18T15:00:54.018 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:53 smithi174 ceph-mon[36787]: pgmap v239: 97 pgs: 97 active+clean; 217 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 355 op/s 2024-03-18T15:00:54.028 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:53 smithi077 ceph-mon[35865]: pgmap v239: 97 pgs: 97 active+clean; 217 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 355 op/s 2024-03-18T15:00:54.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:53 smithi076 ceph-mon[32205]: pgmap v239: 97 pgs: 97 active+clean; 217 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 355 op/s 2024-03-18T15:00:54.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 742795 9.49 MB/sec execute 295 sec latency 52.373 ms 2024-03-18T15:00:55.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 745006 9.50 MB/sec execute 296 sec latency 97.253 ms 2024-03-18T15:00:56.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:55 smithi076 ceph-mon[32205]: pgmap v240: 97 pgs: 97 active+clean; 217 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 251 op/s 2024-03-18T15:00:56.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:55 smithi174 ceph-mon[36787]: pgmap v240: 97 pgs: 97 active+clean; 217 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 251 op/s 2024-03-18T15:00:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:55 smithi077 ceph-mon[35865]: pgmap v240: 97 pgs: 97 active+clean; 217 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 251 op/s 2024-03-18T15:00:56.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 746353 9.49 MB/sec execute 297 sec latency 19.953 ms 2024-03-18T15:00:57.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 748395 9.50 MB/sec execute 298 sec latency 20.008 ms 2024-03-18T15:00:58.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:57 smithi076 ceph-mon[32205]: pgmap v241: 97 pgs: 97 active+clean; 217 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 251 op/s 2024-03-18T15:00:58.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:57 smithi174 ceph-mon[36787]: pgmap v241: 97 pgs: 97 active+clean; 217 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 251 op/s 2024-03-18T15:00:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:57 smithi077 ceph-mon[35865]: pgmap v241: 97 pgs: 97 active+clean; 217 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 251 op/s 2024-03-18T15:00:58.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 749821 9.49 MB/sec execute 299 sec latency 17.115 ms 2024-03-18T15:00:59.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 751683 9.49 MB/sec execute 300 sec latency 32.748 ms 2024-03-18T15:01:00.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:00:59 smithi076 ceph-mon[32205]: pgmap v242: 97 pgs: 97 active+clean; 223 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.2 MiB/s wr, 353 op/s 2024-03-18T15:01:00.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:00:59 smithi174 ceph-mon[36787]: pgmap v242: 97 pgs: 97 active+clean; 223 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.2 MiB/s wr, 353 op/s 2024-03-18T15:01:00.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:00:59 smithi077 ceph-mon[35865]: pgmap v242: 97 pgs: 97 active+clean; 223 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.2 MiB/s wr, 353 op/s 2024-03-18T15:01:00.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 753019 9.49 MB/sec execute 301 sec latency 33.201 ms 2024-03-18T15:01:01.750 INFO:tasks.workunit.client.0.smithi076.stdout: 1 754287 9.48 MB/sec execute 302 sec latency 21.709 ms 2024-03-18T15:01:01.825 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.824+0000 7f42c72dc640 1 -- 172.21.15.76:0/1293953890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42c0100930 msgr2=0x7f42c0100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:01.825 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.824+0000 7f42c72dc640 1 --2- 172.21.15.76:0/1293953890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42c0100930 0x7f42c0100db0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009930 tx=0x7f42b002bc30 comp rx=0 tx=0).stop 2024-03-18T15:01:01.825 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.824+0000 7f42c72dc640 1 -- 172.21.15.76:0/1293953890 shutdown_connections 2024-03-18T15:01:01.825 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.824+0000 7f42c72dc640 1 --2- 172.21.15.76:0/1293953890 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42c01012f0 0x7f42c010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.825 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.824+0000 7f42c72dc640 1 --2- 172.21.15.76:0/1293953890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42c0100930 0x7f42c0100db0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.825 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.824+0000 7f42c72dc640 1 --2- 172.21.15.76:0/1293953890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42c0105680 0x7f42c0105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.825 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.824+0000 7f42c72dc640 1 -- 172.21.15.76:0/1293953890 >> 172.21.15.76:0/1293953890 conn(0x7f42c0077530 msgr2=0x7f42c0077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:01.825 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.824+0000 7f42c72dc640 1 -- 172.21.15.76:0/1293953890 shutdown_connections 2024-03-18T15:01:01.825 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.824+0000 7f42c72dc640 1 -- 172.21.15.76:0/1293953890 wait complete. 2024-03-18T15:01:01.826 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.824+0000 7f42c72dc640 1 Processor -- start 2024-03-18T15:01:01.826 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c72dc640 1 -- start start 2024-03-18T15:01:01.827 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c72dc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42c0100930 0x7f42c0198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.827 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c72dc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42c01012f0 0x7f42c0198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c72dc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42c0105680 0x7f42c019c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c72dc640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f42c0110ba0 con 0x7f42c01012f0 2024-03-18T15:01:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c72dc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f42c0110a20 con 0x7f42c0105680 2024-03-18T15:01:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c72dc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f42c0110d20 con 0x7f42c0100930 2024-03-18T15:01:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c4877640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42c01012f0 0x7f42c0198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c5879640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42c0105680 0x7f42c019c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c5078640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42c0100930 0x7f42c0198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c4877640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42c01012f0 0x7f42c0198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49068/0 (socket says 172.21.15.76:49068) 2024-03-18T15:01:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c5879640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42c0105680 0x7f42c019c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:60996/0 (socket says 172.21.15.76:60996) 2024-03-18T15:01:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c4877640 1 -- 172.21.15.76:0/292266942 learned_addr learned my addr 172.21.15.76:0/292266942 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c4877640 1 -- 172.21.15.76:0/292266942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42c0100930 msgr2=0x7f42c0198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c4877640 1 --2- 172.21.15.76:0/292266942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42c0100930 0x7f42c0198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c4877640 1 -- 172.21.15.76:0/292266942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42c0105680 msgr2=0x7f42c019c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c4877640 1 --2- 172.21.15.76:0/292266942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42c0105680 0x7f42c019c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c4877640 1 -- 172.21.15.76:0/292266942 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c019d100 con 0x7f42c01012f0 2024-03-18T15:01:01.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c5078640 1 --2- 172.21.15.76:0/292266942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42c0100930 0x7f42c0198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:01:01.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c5879640 1 --2- 172.21.15.76:0/292266942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42c0105680 0x7f42c019c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:01:01.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.825+0000 7f42c4877640 1 --2- 172.21.15.76:0/292266942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42c01012f0 0x7f42c0198650 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009230 tx=0x7f42b002cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:01.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.826+0000 7f42b67fc640 1 -- 172.21.15.76:0/292266942 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0041070 con 0x7f42c01012f0 2024-03-18T15:01:01.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.826+0000 7f42c72dc640 1 -- 172.21.15.76:0/292266942 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42c019d390 con 0x7f42c01012f0 2024-03-18T15:01:01.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.826+0000 7f42b67fc640 1 -- 172.21.15.76:0/292266942 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42b0004450 con 0x7f42c01012f0 2024-03-18T15:01:01.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.826+0000 7f42b67fc640 1 -- 172.21.15.76:0/292266942 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0003e50 con 0x7f42c01012f0 2024-03-18T15:01:01.829 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.826+0000 7f42c72dc640 1 -- 172.21.15.76:0/292266942 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42c01a4c80 con 0x7f42c01012f0 2024-03-18T15:01:01.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42b67fc640 1 -- 172.21.15.76:0/292266942 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f42b0044070 con 0x7f42c01012f0 2024-03-18T15:01:01.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c9fa3740 1 Processor -- start 2024-03-18T15:01:01.835 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c9fa3740 1 -- start start 2024-03-18T15:01:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c9fa3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 0x563e70022510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c9fa3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 0x563e7002b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c9fa3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 0x563e7002dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c9fa3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563e6ff86a40 con 0x563e70022a50 2024-03-18T15:01:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c9fa3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563e6ff2f920 con 0x563e70022130 2024-03-18T15:01:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c9fa3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563e6ff82ae0 con 0x563e7002b820 2024-03-18T15:01:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c5078640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 0x563e7002b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c5078640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 0x563e7002b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49076/0 (socket says 172.21.15.76:49076) 2024-03-18T15:01:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c5078640 1 -- 172.21.15.76:0/658046636 learned_addr learned my addr 172.21.15.76:0/658046636 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c5879640 1 --2- 172.21.15.76:0/658046636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 0x563e70022510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c4877640 1 --2- 172.21.15.76:0/658046636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 0x563e7002dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c5078640 1 -- 172.21.15.76:0/658046636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 msgr2=0x563e7002dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c5078640 1 --2- 172.21.15.76:0/658046636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 0x563e7002dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c5078640 1 -- 172.21.15.76:0/658046636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 msgr2=0x563e70022510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:01.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c5078640 1 --2- 172.21.15.76:0/658046636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 0x563e70022510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c5078640 1 -- 172.21.15.76:0/658046636 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e6ff47460 con 0x563e70022a50 2024-03-18T15:01:01.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42b67fc640 1 --2- 172.21.15.76:0/292266942 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f42ac0a2bb0 0x7f42ac0a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c4877640 1 --2- 172.21.15.76:0/658046636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 0x563e7002dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:01:01.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.829+0000 7f42c5879640 1 --2- 172.21.15.76:0/658046636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 0x563e70022510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:01:01.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.830+0000 7f42c4877640 1 --2- 172.21.15.76:0/292266942 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f42ac0a2bb0 0x7f42ac0a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.830+0000 7f42c5078640 1 --2- 172.21.15.76:0/658046636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 0x563e7002b2e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0114f0 tx=0x7f42bc0119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779d90afb497a5d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:01.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.830+0000 7f42b67fc640 1 -- 172.21.15.76:0/292266942 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7f42b00f0620 con 0x7f42c01012f0 2024-03-18T15:01:01.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.830+0000 7f42ab7fe640 1 -- 172.21.15.76:0/658046636 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc01e070 con 0x563e70022a50 2024-03-18T15:01:01.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.830+0000 7f42ab7fe640 1 -- 172.21.15.76:0/658046636 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42bc00faa0 con 0x563e70022a50 2024-03-18T15:01:01.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.830+0000 7f42ab7fe640 1 -- 172.21.15.76:0/658046636 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc019b50 con 0x563e70022a50 2024-03-18T15:01:01.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.830+0000 7f42c9fa3740 1 -- 172.21.15.76:0/658046636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 msgr2=0x563e7002b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:01.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.830+0000 7f42c4877640 1 --2- 172.21.15.76:0/292266942 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f42ac0a2bb0 0x7f42ac0a5070 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f42b00edf50 tx=0x7f42b0000bc0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:01.837 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.830+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/658046636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 0x563e7002b2e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0114f0 tx=0x7f42bc0119b0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.831+0000 7f42c9fa3740 1 -- 172.21.15.76:0/658046636 shutdown_connections 2024-03-18T15:01:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.831+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/658046636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 0x563e7002dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.831+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/658046636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 0x563e7002b2e0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.831+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/658046636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 0x563e70022510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.831+0000 7f42c9fa3740 1 -- 172.21.15.76:0/658046636 >> 172.21.15.76:0/658046636 conn(0x563e7001ca50 msgr2=0x563e7001ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.831+0000 7f42c9fa3740 1 -- 172.21.15.76:0/658046636 shutdown_connections 2024-03-18T15:01:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.831+0000 7f42c9fa3740 1 -- 172.21.15.76:0/658046636 wait complete. 2024-03-18T15:01:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.831+0000 7f42c9fa3740 1 -- start start 2024-03-18T15:01:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.831+0000 7f42c9fa3740 1 Processor -- start 2024-03-18T15:01:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.831+0000 7f42c9fa3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 0x563e70028840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.831+0000 7f42c9fa3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 0x563e70028d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.831+0000 7f42c9fa3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 0x563e700292c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.831+0000 7f42c9fa3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563e7002e450 con 0x563e70022a50 2024-03-18T15:01:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.831+0000 7f42c9fa3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563e6ff42d70 con 0x563e70022130 2024-03-18T15:01:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.831+0000 7f42c9fa3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563e7002e5d0 con 0x563e7002b820 2024-03-18T15:01:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42c9fa3740 10 client.0 _ll_register_callbacks cb 0x563e6fc9aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:01:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42c5078640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 0x563e70028d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42c5879640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 0x563e70028840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42c5078640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 0x563e70028d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49092/0 (socket says 172.21.15.76:49092) 2024-03-18T15:01:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42c5078640 1 -- 172.21.15.76:0/1794573157 learned_addr learned my addr 172.21.15.76:0/1794573157 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42c5879640 1 --2- 172.21.15.76:0/1794573157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 0x563e700292c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42c5879640 1 -- 172.21.15.76:0/1794573157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 msgr2=0x563e700292c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42c5879640 1 --2- 172.21.15.76:0/1794573157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 0x563e700292c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42c5879640 1 -- 172.21.15.76:0/1794573157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 msgr2=0x563e70028d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42c5879640 1 --2- 172.21.15.76:0/1794573157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 0x563e70028d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42c5879640 1 -- 172.21.15.76:0/1794573157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e6fc9b210 con 0x563e70022130 2024-03-18T15:01:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42c5879640 1 --2- 172.21.15.76:0/1794573157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 0x563e70028840 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f42b8010d90 tx=0x7f42b8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=251c93e6d9f25d7e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T15:01:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42a8ff9640 1 -- 172.21.15.76:0/1794573157 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b8019040 con 0x563e70022130 2024-03-18T15:01:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42a8ff9640 1 -- 172.21.15.76:0/1794573157 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7f42b800ec40 con 0x563e70022130 2024-03-18T15:01:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42c9fa3740 1 -- 172.21.15.76:0/1794573157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42bc00faa0 con 0x563e70022130 2024-03-18T15:01:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42a8ff9640 1 -- 172.21.15.76:0/1794573157 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f42b800fe60 con 0x563e70022130 2024-03-18T15:01:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42c9fa3740 10 client.26333 fetch_fsmap learned FSMap version 41 2024-03-18T15:01:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.832+0000 7f42c9fa3740 1 -- 172.21.15.76:0/1794573157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x563e6fd67620 con 0x563e70022130 2024-03-18T15:01:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.833+0000 7f42a8ff9640 1 -- 172.21.15.76:0/1794573157 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42bc00faa0 con 0x563e70022130 2024-03-18T15:01:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.833+0000 7f42a8ff9640 1 -- 172.21.15.76:0/1794573157 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f42b8027050 con 0x563e70022130 2024-03-18T15:01:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.833+0000 7f42c9fa3740 10 client.26333 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:01:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.833+0000 7f42c9fa3740 10 client.26333 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:01:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.833+0000 7f42c9fa3740 20 client.26333 populate_metadata read hostname 'smithi076' 2024-03-18T15:01:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.833+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/1794573157 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x563e7003b980 0x563e7003dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.833+0000 7f42c9fa3740 4 client.26333 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:01:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.833+0000 7f42c9fa3740 1 -- 172.21.15.76:0/1794573157 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e6ff93f80 con 0x563e7003b980 2024-03-18T15:01:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.833+0000 7f42c5078640 1 --2- 172.21.15.76:0/1794573157 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x563e7003b980 0x563e7003dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.833+0000 7f42c5078640 1 --2- 172.21.15.76:0/1794573157 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x563e7003b980 0x563e7003dd90 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c4f1219052061b1 server_cookie=b0b4d2069bc0cbd3 in_seq=0 out_seq=0 2024-03-18T15:01:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.833+0000 7f42a8ff9640 10 client.26333 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.834+0000 7f42a8ff9640 1 -- 172.21.15.76:0/1794573157 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563e6ff93f80 con 0x563e7003b980 2024-03-18T15:01:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.834+0000 7f42a8ff9640 10 client.26333 handle_command_reply: tid=0 2024-03-18T15:01:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.834+0000 7f42c9fa3740 1 client.26333 shutdown 2024-03-18T15:01:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.834+0000 7f42c9fa3740 1 -- 172.21.15.76:0/1794573157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 msgr2=0x563e70028840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.834+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/1794573157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 0x563e70028840 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f42b8010d90 tx=0x7f42b8004a40 comp rx=0 tx=0).stop 2024-03-18T15:01:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.834+0000 7f42c9fa3740 1 -- 172.21.15.76:0/1794573157 shutdown_connections 2024-03-18T15:01:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.835+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/1794573157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 0x563e700292c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.835+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/1794573157 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x563e7003b980 0x563e7003dd90 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.835+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/1794573157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 0x563e70028d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.835+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/1794573157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 0x563e70028840 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.835+0000 7f42a8ff9640 0 client.26333 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:01.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.835+0000 7f42c9fa3740 1 -- 172.21.15.76:0/1794573157 >> 172.21.15.76:0/1794573157 conn(0x563e7001ca50 msgr2=0x563e7001e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:01.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.835+0000 7f42c9fa3740 1 -- 172.21.15.76:0/1794573157 shutdown_connections 2024-03-18T15:01:01.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.835+0000 7f42c9fa3740 1 -- 172.21.15.76:0/1794573157 wait complete. 2024-03-18T15:01:01.842 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.835+0000 7f42c9fa3740 20 client.26333 trim_cache size 0 max 16384 2024-03-18T15:01:01.852 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c9fa3740 1 Processor -- start 2024-03-18T15:01:01.852 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c9fa3740 1 -- start start 2024-03-18T15:01:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c9fa3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022130 0x563e7002af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c9fa3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022a50 0x563e700371f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c9fa3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 0x563e700436a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c9fa3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f42c01afd30 con 0x563e70022130 2024-03-18T15:01:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c9fa3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f42c01ad600 con 0x563e70022a50 2024-03-18T15:01:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c9fa3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f42c01ab470 con 0x563e7002b820 2024-03-18T15:01:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c5879640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022130 0x563e7002af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c5078640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022a50 0x563e700371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c5879640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022130 0x563e7002af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49108/0 (socket says 172.21.15.76:49108) 2024-03-18T15:01:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c5879640 1 -- 172.21.15.76:0/1973684451 learned_addr learned my addr 172.21.15.76:0/1973684451 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c5078640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022a50 0x563e700371f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:32794/0 (socket says 172.21.15.76:32794) 2024-03-18T15:01:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c5879640 1 --2- 172.21.15.76:0/1973684451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 0x563e700436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.853 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c5078640 1 -- 172.21.15.76:0/1973684451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 msgr2=0x563e700436a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c5078640 1 --2- 172.21.15.76:0/1973684451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 0x563e700436a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c5078640 1 -- 172.21.15.76:0/1973684451 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022130 msgr2=0x563e7002af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c5078640 1 --2- 172.21.15.76:0/1973684451 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022130 0x563e7002af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c5078640 1 -- 172.21.15.76:0/1973684451 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e7002df80 con 0x563e70022a50 2024-03-18T15:01:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.851+0000 7f42c5879640 1 --2- 172.21.15.76:0/1973684451 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022130 0x563e7002af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:01:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.852+0000 7f42c5078640 1 --2- 172.21.15.76:0/1973684451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022a50 0x563e700371f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f42bc009700 tx=0x7f42bc0197b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cf8316bc10aa7b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.852+0000 7f42aaffd640 1 -- 172.21.15.76:0/1973684451 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc01e040 con 0x563e70022a50 2024-03-18T15:01:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.852+0000 7f42aaffd640 1 -- 172.21.15.76:0/1973684451 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42bc00faa0 con 0x563e70022a50 2024-03-18T15:01:01.854 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.852+0000 7f42aaffd640 1 -- 172.21.15.76:0/1973684451 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc005330 con 0x563e70022a50 2024-03-18T15:01:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.853+0000 7f42c9fa3740 1 -- 172.21.15.76:0/1973684451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022a50 msgr2=0x563e700371f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.853+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/1973684451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022a50 0x563e700371f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f42bc009700 tx=0x7f42bc0197b0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.853+0000 7f42c9fa3740 1 -- 172.21.15.76:0/1973684451 shutdown_connections 2024-03-18T15:01:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.853+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/1973684451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 0x563e700436a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.853+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/1973684451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022a50 0x563e700371f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.853+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/1973684451 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022130 0x563e7002af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.853+0000 7f42c9fa3740 1 -- 172.21.15.76:0/1973684451 >> 172.21.15.76:0/1973684451 conn(0x563e7001ca50 msgr2=0x563e7001b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.853+0000 7f42c9fa3740 1 -- 172.21.15.76:0/1973684451 shutdown_connections 2024-03-18T15:01:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.853+0000 7f42c9fa3740 1 -- 172.21.15.76:0/1973684451 wait complete. 2024-03-18T15:01:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.853+0000 7f42c9fa3740 1 -- start start 2024-03-18T15:01:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.853+0000 7f42c9fa3740 1 Processor -- start 2024-03-18T15:01:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.854+0000 7f42c9fa3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 0x563e70025c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.854+0000 7f42c9fa3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 0x563e70026170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.854+0000 7f42c9fa3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 0x563e7009aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.854+0000 7f42c9fa3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f42c01abac0 con 0x563e70022a50 2024-03-18T15:01:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.854+0000 7f42c9fa3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f42c01acf30 con 0x563e70022130 2024-03-18T15:01:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.854+0000 7f42c9fa3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f42c01ac190 con 0x563e7002b820 2024-03-18T15:01:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.854+0000 7f42c9fa3740 10 client.0 _ll_register_callbacks cb 0x7f42b80106d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:01:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.854+0000 7f42c5879640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 0x563e70025c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.854+0000 7f42c5078640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 0x563e70026170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.854+0000 7f42c5879640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 0x563e70025c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:32796/0 (socket says 172.21.15.76:32796) 2024-03-18T15:01:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.854+0000 7f42c5078640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 0x563e70026170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49122/0 (socket says 172.21.15.76:49122) 2024-03-18T15:01:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.854+0000 7f42c5879640 1 -- 172.21.15.76:0/2571662621 learned_addr learned my addr 172.21.15.76:0/2571662621 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.854+0000 7f42c5879640 1 --2- 172.21.15.76:0/2571662621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 0x563e7009aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.855+0000 7f42c5879640 1 -- 172.21.15.76:0/2571662621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 msgr2=0x563e7009aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.855+0000 7f42c5879640 1 --2- 172.21.15.76:0/2571662621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 0x563e7009aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.855+0000 7f42c5879640 1 -- 172.21.15.76:0/2571662621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 msgr2=0x563e70026170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.855+0000 7f42c5879640 1 --2- 172.21.15.76:0/2571662621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 0x563e70026170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.855+0000 7f42c5879640 1 -- 172.21.15.76:0/2571662621 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e7009b290 con 0x563e70022130 2024-03-18T15:01:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.855+0000 7f42c5078640 1 --2- 172.21.15.76:0/2571662621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 0x563e70026170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:01:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.855+0000 7f42c5879640 1 --2- 172.21.15.76:0/2571662621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 0x563e70025c30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f42b8005d70 tx=0x7f42b8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32b423dc566cea8c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.855+0000 7f42a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T15:01:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.855+0000 7f42a8ff9640 1 -- 172.21.15.76:0/2571662621 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b8003970 con 0x563e70022130 2024-03-18T15:01:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.855+0000 7f42c9fa3740 1 -- 172.21.15.76:0/2571662621 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42bc00faa0 con 0x563e70022130 2024-03-18T15:01:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.855+0000 7f42a8ff9640 1 -- 172.21.15.76:0/2571662621 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7f42b8003b10 con 0x563e70022130 2024-03-18T15:01:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.855+0000 7f42a8ff9640 1 -- 172.21.15.76:0/2571662621 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f42b8003d10 con 0x563e70022130 2024-03-18T15:01:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.855+0000 7f42c9fa3740 10 client.26339 fetch_fsmap learned FSMap version 41 2024-03-18T15:01:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.855+0000 7f42c9fa3740 1 -- 172.21.15.76:0/2571662621 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x563e700a2620 con 0x563e70022130 2024-03-18T15:01:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.855+0000 7f42a8ff9640 1 -- 172.21.15.76:0/2571662621 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42bc00faa0 con 0x563e70022130 2024-03-18T15:01:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.856+0000 7f42a8ff9640 1 -- 172.21.15.76:0/2571662621 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f42b8027040 con 0x563e70022130 2024-03-18T15:01:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.856+0000 7f42c9fa3740 10 client.26339 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:01:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.856+0000 7f42c9fa3740 10 client.26339 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:01:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.856+0000 7f42c9fa3740 20 client.26339 populate_metadata read hostname 'smithi076' 2024-03-18T15:01:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.856+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/2571662621 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x563e700a2cc0 0x563e700a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.856+0000 7f42c9fa3740 4 client.26339 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:01:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.856+0000 7f42c9fa3740 1 -- 172.21.15.76:0/2571662621 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563e6ff8e2d0 con 0x563e700a2cc0 2024-03-18T15:01:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.856+0000 7f42c5078640 1 --2- 172.21.15.76:0/2571662621 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x563e700a2cc0 0x563e700a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.856+0000 7f42a8ff9640 10 client.26339 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.856+0000 7f42c5078640 1 --2- 172.21.15.76:0/2571662621 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x563e700a2cc0 0x563e700a50e0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4bfb144d7262c5 server_cookie=a16bc78d50478886 in_seq=0 out_seq=0 2024-03-18T15:01:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.929+0000 7f42a8ff9640 1 -- 172.21.15.76:0/2571662621 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563e6ff8e2d0 con 0x563e700a2cc0 2024-03-18T15:01:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.929+0000 7f42a8ff9640 10 client.26339 handle_command_reply: tid=0 2024-03-18T15:01:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.929+0000 7f42c9fa3740 1 client.26339 shutdown 2024-03-18T15:01:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.930+0000 7f42c9fa3740 1 -- 172.21.15.76:0/2571662621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 msgr2=0x563e70025c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.930+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/2571662621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 0x563e70025c30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f42b8005d70 tx=0x7f42b8005fd0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.930+0000 7f42c9fa3740 1 -- 172.21.15.76:0/2571662621 shutdown_connections 2024-03-18T15:01:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.930+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/2571662621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e7002b820 0x563e7009aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.930+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/2571662621 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x563e700a2cc0 0x563e700a50e0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.930+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/2571662621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e70022a50 0x563e70026170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.930+0000 7f42c9fa3740 1 --2- 172.21.15.76:0/2571662621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e70022130 0x563e70025c30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.930+0000 7f42c9fa3740 1 -- 172.21.15.76:0/2571662621 >> 172.21.15.76:0/2571662621 conn(0x563e7001ca50 msgr2=0x563e70029090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.930+0000 7f42a8ff9640 0 client.26339 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.930+0000 7f42c9fa3740 1 -- 172.21.15.76:0/2571662621 shutdown_connections 2024-03-18T15:01:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.930+0000 7f42c9fa3740 1 -- 172.21.15.76:0/2571662621 wait complete. 2024-03-18T15:01:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.930+0000 7f42c9fa3740 20 client.26339 trim_cache size 0 max 16384 2024-03-18T15:01:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:01:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:01:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:01:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:01:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.931+0000 7f42c72dc640 1 -- 172.21.15.76:0/292266942 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f42ac0a2bb0 msgr2=0x7f42ac0a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.931+0000 7f42c72dc640 1 --2- 172.21.15.76:0/292266942 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f42ac0a2bb0 0x7f42ac0a5070 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f42b00edf50 tx=0x7f42b0000bc0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.931+0000 7f42c72dc640 1 -- 172.21.15.76:0/292266942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42c01012f0 msgr2=0x7f42c0198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.931+0000 7f42c72dc640 1 --2- 172.21.15.76:0/292266942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42c01012f0 0x7f42c0198650 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009230 tx=0x7f42b002cb30 comp rx=0 tx=0).stop 2024-03-18T15:01:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.932+0000 7f42c72dc640 1 -- 172.21.15.76:0/292266942 shutdown_connections 2024-03-18T15:01:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.932+0000 7f42c72dc640 1 --2- 172.21.15.76:0/292266942 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f42ac0a2bb0 0x7f42ac0a5070 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.932+0000 7f42c72dc640 1 --2- 172.21.15.76:0/292266942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42c0105680 0x7f42c019c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.932+0000 7f42c72dc640 1 --2- 172.21.15.76:0/292266942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42c01012f0 0x7f42c0198650 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.932+0000 7f42c72dc640 1 --2- 172.21.15.76:0/292266942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42c0100930 0x7f42c0198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.932+0000 7f42c72dc640 1 -- 172.21.15.76:0/292266942 >> 172.21.15.76:0/292266942 conn(0x7f42c0077530 msgr2=0x7f42c00696f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.932+0000 7f42c72dc640 1 -- 172.21.15.76:0/292266942 shutdown_connections 2024-03-18T15:01:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:01.932+0000 7f42c72dc640 1 -- 172.21.15.76:0/292266942 wait complete. 2024-03-18T15:01:02.067 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:01 smithi077 ceph-mon[35865]: pgmap v243: 97 pgs: 97 active+clean; 223 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 249 op/s 2024-03-18T15:01:02.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:01 smithi076 ceph-mon[32205]: pgmap v243: 97 pgs: 97 active+clean; 223 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 249 op/s 2024-03-18T15:01:02.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:01 smithi174 ceph-mon[36787]: pgmap v243: 97 pgs: 97 active+clean; 223 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 249 op/s 2024-03-18T15:01:02.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 756546 9.49 MB/sec execute 303 sec latency 59.060 ms 2024-03-18T15:01:03.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 756767 9.46 MB/sec execute 304 sec latency 44.008 ms 2024-03-18T15:01:04.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:03 smithi076 ceph-mon[32205]: osdmap e98: 12 total, 12 up, 12 in 2024-03-18T15:01:04.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:03 smithi076 ceph-mon[32205]: pgmap v245: 97 pgs: 11 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 81 active+clean; 214 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 310 op/s 2024-03-18T15:01:04.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:03 smithi174 ceph-mon[36787]: osdmap e98: 12 total, 12 up, 12 in 2024-03-18T15:01:04.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:03 smithi174 ceph-mon[36787]: pgmap v245: 97 pgs: 11 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 81 active+clean; 214 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 310 op/s 2024-03-18T15:01:04.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:03 smithi077 ceph-mon[35865]: osdmap e98: 12 total, 12 up, 12 in 2024-03-18T15:01:04.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:03 smithi077 ceph-mon[35865]: pgmap v245: 97 pgs: 11 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 81 active+clean; 214 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 310 op/s 2024-03-18T15:01:04.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 758620 9.45 MB/sec execute 305 sec latency 20.507 ms 2024-03-18T15:01:05.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 760246 9.46 MB/sec execute 306 sec latency 30.210 ms 2024-03-18T15:01:06.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:05 smithi076 ceph-mon[32205]: pgmap v246: 97 pgs: 11 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 81 active+clean; 214 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 310 op/s 2024-03-18T15:01:06.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:05 smithi174 ceph-mon[36787]: pgmap v246: 97 pgs: 11 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 81 active+clean; 214 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 310 op/s 2024-03-18T15:01:06.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:05 smithi077 ceph-mon[35865]: pgmap v246: 97 pgs: 11 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 81 active+clean; 214 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 310 op/s 2024-03-18T15:01:06.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 761758 9.45 MB/sec execute 307 sec latency 18.961 ms 2024-03-18T15:01:07.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 763791 9.46 MB/sec execute 308 sec latency 19.890 ms 2024-03-18T15:01:08.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:07 smithi076 ceph-mon[32205]: pgmap v247: 97 pgs: 11 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 81 active+clean; 214 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 310 op/s 2024-03-18T15:01:08.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:07 smithi174 ceph-mon[36787]: pgmap v247: 97 pgs: 11 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 81 active+clean; 214 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 310 op/s 2024-03-18T15:01:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:07 smithi077 ceph-mon[35865]: pgmap v247: 97 pgs: 11 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 81 active+clean; 214 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 310 op/s 2024-03-18T15:01:08.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 765235 9.45 MB/sec execute 309 sec latency 20.752 ms 2024-03-18T15:01:09.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 767336 9.46 MB/sec execute 310 sec latency 16.846 ms 2024-03-18T15:01:10.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:09 smithi076 ceph-mon[32205]: pgmap v248: 97 pgs: 97 active+clean; 220 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 307 op/s 2024-03-18T15:01:10.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:09 smithi174 ceph-mon[36787]: pgmap v248: 97 pgs: 97 active+clean; 220 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 307 op/s 2024-03-18T15:01:10.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:09 smithi077 ceph-mon[35865]: pgmap v248: 97 pgs: 97 active+clean; 220 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 307 op/s 2024-03-18T15:01:10.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 768625 9.45 MB/sec execute 311 sec latency 65.287 ms 2024-03-18T15:01:11.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 770870 9.46 MB/sec execute 312 sec latency 30.737 ms 2024-03-18T15:01:12.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:11 smithi076 ceph-mon[32205]: pgmap v249: 97 pgs: 97 active+clean; 220 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 307 op/s 2024-03-18T15:01:12.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:11 smithi174 ceph-mon[36787]: pgmap v249: 97 pgs: 97 active+clean; 220 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 307 op/s 2024-03-18T15:01:12.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:11 smithi077 ceph-mon[35865]: pgmap v249: 97 pgs: 97 active+clean; 220 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 307 op/s 2024-03-18T15:01:12.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 772455 9.45 MB/sec execute 313 sec latency 20.739 ms 2024-03-18T15:01:13.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 774467 9.46 MB/sec execute 314 sec latency 27.980 ms 2024-03-18T15:01:14.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:14 smithi076 ceph-mon[32205]: osdmap e99: 12 total, 12 up, 12 in 2024-03-18T15:01:14.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:14 smithi076 ceph-mon[32205]: pgmap v251: 97 pgs: 97 active+clean; 230 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-03-18T15:01:14.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:14 smithi174 ceph-mon[36787]: osdmap e99: 12 total, 12 up, 12 in 2024-03-18T15:01:14.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:14 smithi174 ceph-mon[36787]: pgmap v251: 97 pgs: 97 active+clean; 230 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-03-18T15:01:14.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:14 smithi077 ceph-mon[35865]: osdmap e99: 12 total, 12 up, 12 in 2024-03-18T15:01:14.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:14 smithi077 ceph-mon[35865]: pgmap v251: 97 pgs: 97 active+clean; 230 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-03-18T15:01:14.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 775910 9.45 MB/sec execute 315 sec latency 76.195 ms 2024-03-18T15:01:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.044+0000 7f12f23f3640 1 -- 172.21.15.76:0/1304208851 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12ec101780 msgr2=0x7f12ec101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.044+0000 7f12f23f3640 1 --2- 172.21.15.76:0/1304208851 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12ec101780 0x7f12ec101c00 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f12e0009930 tx=0x7f12e002bc30 comp rx=0 tx=0).stop 2024-03-18T15:01:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.044+0000 7f12f23f3640 1 -- 172.21.15.76:0/1304208851 shutdown_connections 2024-03-18T15:01:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.044+0000 7f12f23f3640 1 --2- 172.21.15.76:0/1304208851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12ec102140 0x7f12ec10e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.044+0000 7f12f23f3640 1 --2- 172.21.15.76:0/1304208851 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12ec101780 0x7f12ec101c00 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.044+0000 7f12f23f3640 1 --2- 172.21.15.76:0/1304208851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ec107790 0x7f12ec107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.044+0000 7f12f23f3640 1 -- 172.21.15.76:0/1304208851 >> 172.21.15.76:0/1304208851 conn(0x7f12ec0fd270 msgr2=0x7f12ec0ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.044+0000 7f12f23f3640 1 -- 172.21.15.76:0/1304208851 shutdown_connections 2024-03-18T15:01:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.044+0000 7f12f23f3640 1 -- 172.21.15.76:0/1304208851 wait complete. 2024-03-18T15:01:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.045+0000 7f12f23f3640 1 Processor -- start 2024-03-18T15:01:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.045+0000 7f12f23f3640 1 -- start start 2024-03-18T15:01:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.045+0000 7f12f23f3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12ec101780 0x7f12ec197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.045+0000 7f12f23f3640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12ec102140 0x7f12ec1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.045+0000 7f12f23f3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ec107790 0x7f12ec19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.045+0000 7f12f23f3640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec111ba0 con 0x7f12ec102140 2024-03-18T15:01:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.045+0000 7f12f23f3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec111a20 con 0x7f12ec107790 2024-03-18T15:01:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.045+0000 7f12f23f3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec111d20 con 0x7f12ec101780 2024-03-18T15:01:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.045+0000 7f12eb7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12ec102140 0x7f12ec1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.046+0000 7f12eb7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12ec102140 0x7f12ec1984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52938/0 (socket says 172.21.15.76:52938) 2024-03-18T15:01:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.046+0000 7f12eb7fe640 1 -- 172.21.15.76:0/529569254 learned_addr learned my addr 172.21.15.76:0/529569254 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.046+0000 7f12f0990640 1 --2- 172.21.15.76:0/529569254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ec107790 0x7f12ec19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.046+0000 7f12ebfff640 1 --2- 172.21.15.76:0/529569254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12ec101780 0x7f12ec197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.046+0000 7f12f0990640 1 -- 172.21.15.76:0/529569254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12ec101780 msgr2=0x7f12ec197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.046+0000 7f12f0990640 1 --2- 172.21.15.76:0/529569254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12ec101780 0x7f12ec197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.046+0000 7f12f0990640 1 -- 172.21.15.76:0/529569254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12ec102140 msgr2=0x7f12ec1984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.046+0000 7f12f0990640 1 --2- 172.21.15.76:0/529569254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12ec102140 0x7f12ec1984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.046+0000 7f12f0990640 1 -- 172.21.15.76:0/529569254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12ec19cf90 con 0x7f12ec107790 2024-03-18T15:01:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.046+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/529569254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12ec102140 0x7f12ec1984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:01:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.046+0000 7f12f0990640 1 --2- 172.21.15.76:0/529569254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ec107790 0x7f12ec19c860 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f12e4009fd0 tx=0x7f12e400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.046+0000 7f12e97fa640 1 -- 172.21.15.76:0/529569254 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e400ce20 con 0x7f12ec107790 2024-03-18T15:01:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.046+0000 7f12f23f3640 1 -- 172.21.15.76:0/529569254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12ec19d280 con 0x7f12ec107790 2024-03-18T15:01:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.046+0000 7f12e97fa640 1 -- 172.21.15.76:0/529569254 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12e4005020 con 0x7f12ec107790 2024-03-18T15:01:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.046+0000 7f12e97fa640 1 -- 172.21.15.76:0/529569254 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e4010720 con 0x7f12ec107790 2024-03-18T15:01:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.047+0000 7f12f23f3640 1 -- 172.21.15.76:0/529569254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12ec1a4b10 con 0x7f12ec107790 2024-03-18T15:01:15.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.049+0000 7f12f4fdb740 1 Processor -- start 2024-03-18T15:01:15.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.049+0000 7f12e97fa640 1 -- 172.21.15.76:0/529569254 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f12e4005540 con 0x7f12ec107790 2024-03-18T15:01:15.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.049+0000 7f12f4fdb740 1 -- start start 2024-03-18T15:01:15.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.049+0000 7f12f4fdb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6ec130 0x5577af6ec510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12f4fdb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6eca50 0x5577af6f52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12f4fdb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6f5820 0x5577af6f7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12f4fdb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5577af650a40 con 0x5577af6ec130 2024-03-18T15:01:15.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12f4fdb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5577af5f9920 con 0x5577af6f5820 2024-03-18T15:01:15.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12f4fdb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5577af64cae0 con 0x5577af6eca50 2024-03-18T15:01:15.055 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12eb7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6ec130 0x5577af6ec510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12eb7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6ec130 0x5577af6ec510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52950/0 (socket says 172.21.15.76:52950) 2024-03-18T15:01:15.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12eb7fe640 1 -- 172.21.15.76:0/2045210314 learned_addr learned my addr 172.21.15.76:0/2045210314 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:15.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12ebfff640 1 --2- 172.21.15.76:0/2045210314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6eca50 0x5577af6f52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/2045210314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6f5820 0x5577af6f7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12e97fa640 1 --2- 172.21.15.76:0/529569254 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f12d40a2a10 0x7f12d40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12eb7fe640 1 -- 172.21.15.76:0/2045210314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6eca50 msgr2=0x5577af6f52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:15.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/2045210314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6eca50 0x5577af6f52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12eb7fe640 1 -- 172.21.15.76:0/2045210314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6f5820 msgr2=0x5577af6f7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:15.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/2045210314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6f5820 0x5577af6f7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12eb7fe640 1 -- 172.21.15.76:0/2045210314 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577af611460 con 0x5577af6ec130 2024-03-18T15:01:15.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12f0990640 1 --2- 172.21.15.76:0/529569254 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f12d40a2a10 0x7f12d40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12e97fa640 1 -- 172.21.15.76:0/529569254 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f12e40cf480 con 0x7f12ec107790 2024-03-18T15:01:15.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/2045210314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6ec130 0x5577af6ec510 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f12e0005ec0 tx=0x7f12e0005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdef650e07ecc3c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:15.056 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12d27fc640 1 -- 172.21.15.76:0/2045210314 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e002ca80 con 0x5577af6ec130 2024-03-18T15:01:15.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.050+0000 7f12d27fc640 1 -- 172.21.15.76:0/2045210314 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12e002cc20 con 0x5577af6ec130 2024-03-18T15:01:15.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.051+0000 7f12d27fc640 1 -- 172.21.15.76:0/2045210314 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e0005560 con 0x5577af6ec130 2024-03-18T15:01:15.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.051+0000 7f12f4fdb740 1 -- 172.21.15.76:0/2045210314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6ec130 msgr2=0x5577af6ec510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:15.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.051+0000 7f12f0990640 1 --2- 172.21.15.76:0/529569254 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f12d40a2a10 0x7f12d40a4ed0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f12e409ba90 tx=0x7f12e406d730 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:15.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.051+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/2045210314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6ec130 0x5577af6ec510 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f12e0005ec0 tx=0x7f12e0005b50 comp rx=0 tx=0).stop 2024-03-18T15:01:15.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.051+0000 7f12f4fdb740 1 -- 172.21.15.76:0/2045210314 shutdown_connections 2024-03-18T15:01:15.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.051+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/2045210314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6eca50 0x5577af6f52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.051+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/2045210314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6f5820 0x5577af6f7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.051+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/2045210314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6ec130 0x5577af6ec510 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.051+0000 7f12f4fdb740 1 -- 172.21.15.76:0/2045210314 >> 172.21.15.76:0/2045210314 conn(0x5577af6e6a50 msgr2=0x5577af6e6e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:15.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.051+0000 7f12f4fdb740 1 -- 172.21.15.76:0/2045210314 shutdown_connections 2024-03-18T15:01:15.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.051+0000 7f12f4fdb740 1 -- 172.21.15.76:0/2045210314 wait complete. 2024-03-18T15:01:15.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12f4fdb740 1 -- start start 2024-03-18T15:01:15.057 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12f4fdb740 1 Processor -- start 2024-03-18T15:01:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12f4fdb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6ec130 0x5577af6f2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12f4fdb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6eca50 0x5577af6f2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12f4fdb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 0x5577af6f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12f4fdb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5577af6f83f0 con 0x5577af6f5820 2024-03-18T15:01:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12f4fdb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5577af60cd70 con 0x5577af6ec130 2024-03-18T15:01:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12f4fdb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5577af6f8570 con 0x5577af6eca50 2024-03-18T15:01:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12f4fdb740 10 client.0 _ll_register_callbacks cb 0x5577af364a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:01:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12eb7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 0x5577af6f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12ebfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6eca50 0x5577af6f2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12eb7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 0x5577af6f32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52952/0 (socket says 172.21.15.76:52952) 2024-03-18T15:01:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12eb7fe640 1 -- 172.21.15.76:0/4150001944 learned_addr learned my addr 172.21.15.76:0/4150001944 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/4150001944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6ec130 0x5577af6f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12eb7fe640 1 -- 172.21.15.76:0/4150001944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6eca50 msgr2=0x5577af6f2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/4150001944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6eca50 0x5577af6f2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12eb7fe640 1 -- 172.21.15.76:0/4150001944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6ec130 msgr2=0x5577af6f2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/4150001944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6ec130 0x5577af6f2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.052+0000 7f12eb7fe640 1 -- 172.21.15.76:0/4150001944 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577af365210 con 0x5577af6f5820 2024-03-18T15:01:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.053+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/4150001944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 0x5577af6f32d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f12e002c820 tx=0x7f12e0005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78230962a7f02318 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.053+0000 7f12cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:01:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.053+0000 7f12cb7fe640 1 -- 172.21.15.76:0/4150001944 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e0004230 con 0x5577af6f5820 2024-03-18T15:01:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.053+0000 7f12cb7fe640 1 -- 172.21.15.76:0/4150001944 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f12e00043d0 con 0x5577af6f5820 2024-03-18T15:01:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.053+0000 7f12f4fdb740 1 -- 172.21.15.76:0/4150001944 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12e002cc20 con 0x5577af6f5820 2024-03-18T15:01:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.053+0000 7f12cb7fe640 1 -- 172.21.15.76:0/4150001944 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f12e00035f0 con 0x5577af6f5820 2024-03-18T15:01:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.053+0000 7f12f4fdb740 10 client.16488 fetch_fsmap learned FSMap version 41 2024-03-18T15:01:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.053+0000 7f12f4fdb740 1 -- 172.21.15.76:0/4150001944 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5577af431620 con 0x5577af6f5820 2024-03-18T15:01:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.053+0000 7f12cb7fe640 1 -- 172.21.15.76:0/4150001944 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12e002cc20 con 0x5577af6f5820 2024-03-18T15:01:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.053+0000 7f12cb7fe640 1 -- 172.21.15.76:0/4150001944 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f12e00407c0 con 0x5577af6f5820 2024-03-18T15:01:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.053+0000 7f12f4fdb740 10 client.16488 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:01:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.053+0000 7f12f4fdb740 10 client.16488 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:01:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.053+0000 7f12f4fdb740 20 client.16488 populate_metadata read hostname 'smithi076' 2024-03-18T15:01:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.053+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/4150001944 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5577af7057d0 0x5577af707bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.053+0000 7f12f4fdb740 4 client.16488 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:01:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.053+0000 7f12f4fdb740 1 -- 172.21.15.76:0/4150001944 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577af65df80 con 0x5577af7057d0 2024-03-18T15:01:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.054+0000 7f12ebfff640 1 --2- 172.21.15.76:0/4150001944 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5577af7057d0 0x5577af707bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.054+0000 7f12cb7fe640 10 client.16488 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.054+0000 7f12ebfff640 1 --2- 172.21.15.76:0/4150001944 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5577af7057d0 0x5577af707bc0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de83577d07bff783 server_cookie=81110da408eb2d57 in_seq=0 out_seq=0 2024-03-18T15:01:15.061 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.055+0000 7f12cb7fe640 1 -- 172.21.15.76:0/4150001944 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5577af65df80 con 0x5577af7057d0 2024-03-18T15:01:15.061 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.055+0000 7f12cb7fe640 10 client.16488 handle_command_reply: tid=0 2024-03-18T15:01:15.061 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.055+0000 7f12f4fdb740 1 client.16488 shutdown 2024-03-18T15:01:15.061 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.055+0000 7f12f4fdb740 1 -- 172.21.15.76:0/4150001944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 msgr2=0x5577af6f32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:15.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.055+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/4150001944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 0x5577af6f32d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f12e002c820 tx=0x7f12e0005280 comp rx=0 tx=0).stop 2024-03-18T15:01:15.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.055+0000 7f12f4fdb740 1 -- 172.21.15.76:0/4150001944 shutdown_connections 2024-03-18T15:01:15.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.055+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/4150001944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6eca50 0x5577af6f2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.055+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/4150001944 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5577af7057d0 0x5577af707bc0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.055+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/4150001944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 0x5577af6f32d0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.055+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/4150001944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6ec130 0x5577af6f2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.055+0000 7f12cb7fe640 0 client.16488 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:15.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.055+0000 7f12f4fdb740 1 -- 172.21.15.76:0/4150001944 >> 172.21.15.76:0/4150001944 conn(0x5577af6e6a50 msgr2=0x5577af6e87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:15.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.056+0000 7f12f4fdb740 1 -- 172.21.15.76:0/4150001944 shutdown_connections 2024-03-18T15:01:15.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.056+0000 7f12f4fdb740 1 -- 172.21.15.76:0/4150001944 wait complete. 2024-03-18T15:01:15.062 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.056+0000 7f12f4fdb740 20 client.16488 trim_cache size 0 max 16384 2024-03-18T15:01:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.071+0000 7f12f4fdb740 1 Processor -- start 2024-03-18T15:01:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.071+0000 7f12f4fdb740 1 -- start start 2024-03-18T15:01:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.071+0000 7f12f4fdb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6ec130 0x5577af6f4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.071+0000 7f12f4fdb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6eca50 0x5577af6e5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.071+0000 7f12f4fdb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 0x5577af70abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.071+0000 7f12f4fdb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec1afbd0 con 0x5577af6f5820 2024-03-18T15:01:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.071+0000 7f12f4fdb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec1ad4a0 con 0x5577af6eca50 2024-03-18T15:01:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.071+0000 7f12f4fdb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec1ab310 con 0x5577af6ec130 2024-03-18T15:01:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.071+0000 7f12eb7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 0x5577af70abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.071+0000 7f12eb7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 0x5577af70abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52954/0 (socket says 172.21.15.76:52954) 2024-03-18T15:01:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.071+0000 7f12eb7fe640 1 -- 172.21.15.76:0/2016208788 learned_addr learned my addr 172.21.15.76:0/2016208788 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.071+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/2016208788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6ec130 0x5577af6f4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.071+0000 7f12ebfff640 1 --2- 172.21.15.76:0/2016208788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6eca50 0x5577af6e5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.072+0000 7f12eb7fe640 1 -- 172.21.15.76:0/2016208788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6ec130 msgr2=0x5577af6f4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.072+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/2016208788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6ec130 0x5577af6f4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.072+0000 7f12eb7fe640 1 -- 172.21.15.76:0/2016208788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6eca50 msgr2=0x5577af6e5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.072+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/2016208788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6eca50 0x5577af6e5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.072+0000 7f12eb7fe640 1 -- 172.21.15.76:0/2016208788 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577af6e0fc0 con 0x5577af6f5820 2024-03-18T15:01:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.072+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/2016208788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 0x5577af70abe0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f12e0040be0 tx=0x7f12e0005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35d9613f4d9a5564 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.072+0000 7f12d1ffb640 1 -- 172.21.15.76:0/2016208788 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e0004af0 con 0x5577af6f5820 2024-03-18T15:01:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.072+0000 7f12d1ffb640 1 -- 172.21.15.76:0/2016208788 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12e0009520 con 0x5577af6f5820 2024-03-18T15:01:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.072+0000 7f12d1ffb640 1 -- 172.21.15.76:0/2016208788 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e00429c0 con 0x5577af6f5820 2024-03-18T15:01:15.078 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.073+0000 7f12f4fdb740 1 -- 172.21.15.76:0/2016208788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 msgr2=0x5577af70abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:15.078 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.073+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/2016208788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 0x5577af70abe0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f12e0040be0 tx=0x7f12e0005b50 comp rx=0 tx=0).stop 2024-03-18T15:01:15.078 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.073+0000 7f12f4fdb740 1 -- 172.21.15.76:0/2016208788 shutdown_connections 2024-03-18T15:01:15.078 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.073+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/2016208788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 0x5577af70abe0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.078 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.073+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/2016208788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6eca50 0x5577af6e5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.078 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.073+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/2016208788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6ec130 0x5577af6f4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.078 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.073+0000 7f12f4fdb740 1 -- 172.21.15.76:0/2016208788 >> 172.21.15.76:0/2016208788 conn(0x5577af6e6a50 msgr2=0x5577af6e5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:15.078 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.073+0000 7f12f4fdb740 1 -- 172.21.15.76:0/2016208788 shutdown_connections 2024-03-18T15:01:15.078 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.073+0000 7f12f4fdb740 1 -- 172.21.15.76:0/2016208788 wait complete. 2024-03-18T15:01:15.078 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.073+0000 7f12f4fdb740 1 -- start start 2024-03-18T15:01:15.078 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.073+0000 7f12f4fdb740 1 Processor -- start 2024-03-18T15:01:15.078 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12f4fdb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6ec130 0x5577af708c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.079 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12f4fdb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6eca50 0x5577af709190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.079 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12f4fdb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 0x5577af76a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.079 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12f4fdb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec1ab960 con 0x5577af6f5820 2024-03-18T15:01:15.079 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12f4fdb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec1acdd0 con 0x5577af6eca50 2024-03-18T15:01:15.079 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12f4fdb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec1ac030 con 0x5577af6ec130 2024-03-18T15:01:15.079 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12f4fdb740 10 client.0 _ll_register_callbacks cb 0x7f12e0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:01:15.079 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12ebfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6eca50 0x5577af709190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.079 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12ebfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6eca50 0x5577af709190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:46912/0 (socket says 172.21.15.76:46912) 2024-03-18T15:01:15.079 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12ebfff640 1 -- 172.21.15.76:0/624322757 learned_addr learned my addr 172.21.15.76:0/624322757 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:15.079 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/624322757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 0x5577af76a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.080 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/624322757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6ec130 0x5577af708c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.080 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12ebfff640 1 -- 172.21.15.76:0/624322757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6ec130 msgr2=0x5577af708c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:15.080 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12ebfff640 1 --2- 172.21.15.76:0/624322757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6ec130 0x5577af708c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.080 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12ebfff640 1 -- 172.21.15.76:0/624322757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 msgr2=0x5577af76a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:15.080 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12ebfff640 1 --2- 172.21.15.76:0/624322757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 0x5577af76a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.080 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12ebfff640 1 -- 172.21.15.76:0/624322757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577af76acb0 con 0x5577af6eca50 2024-03-18T15:01:15.080 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/624322757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6ec130 0x5577af708c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:01:15.080 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12ebfff640 1 --2- 172.21.15.76:0/624322757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6eca50 0x5577af709190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x5577af6f20b0 tx=0x7f12dc004410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98775af0260bc2f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:15.080 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T15:01:15.080 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.074+0000 7f12cb7fe640 1 -- 172.21.15.76:0/624322757 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12dc003cc0 con 0x5577af6eca50 2024-03-18T15:01:15.080 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.075+0000 7f12cb7fe640 1 -- 172.21.15.76:0/624322757 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f12dc005020 con 0x5577af6eca50 2024-03-18T15:01:15.080 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.075+0000 7f12f4fdb740 1 -- 172.21.15.76:0/624322757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12e0009520 con 0x5577af6eca50 2024-03-18T15:01:15.080 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.075+0000 7f12cb7fe640 1 -- 172.21.15.76:0/624322757 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f12dc003e60 con 0x5577af6eca50 2024-03-18T15:01:15.080 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.075+0000 7f12f4fdb740 10 client.26357 fetch_fsmap learned FSMap version 41 2024-03-18T15:01:15.080 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.075+0000 7f12f4fdb740 1 -- 172.21.15.76:0/624322757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5577af76c050 con 0x5577af6eca50 2024-03-18T15:01:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.075+0000 7f12cb7fe640 1 -- 172.21.15.76:0/624322757 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12e0009520 con 0x5577af6eca50 2024-03-18T15:01:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.075+0000 7f12cb7fe640 1 -- 172.21.15.76:0/624322757 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f12dc02f020 con 0x5577af6eca50 2024-03-18T15:01:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.075+0000 7f12f4fdb740 10 client.26357 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:01:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.075+0000 7f12f4fdb740 10 client.26357 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:01:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.075+0000 7f12f4fdb740 20 client.26357 populate_metadata read hostname 'smithi076' 2024-03-18T15:01:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.075+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/624322757 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5577af76c730 0x5577af76eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.075+0000 7f12f4fdb740 4 client.26357 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:01:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.075+0000 7f12f4fdb740 1 -- 172.21.15.76:0/624322757 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5577af6582d0 con 0x5577af76c730 2024-03-18T15:01:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.075+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/624322757 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5577af76c730 0x5577af76eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.076+0000 7f12cb7fe640 10 client.26357 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.076+0000 7f12eb7fe640 1 --2- 172.21.15.76:0/624322757 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5577af76c730 0x5577af76eb50 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ff256e29e845d93 server_cookie=50e39be25a91508b in_seq=0 out_seq=0 2024-03-18T15:01:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.158+0000 7f12cb7fe640 1 -- 172.21.15.76:0/624322757 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5577af6582d0 con 0x5577af76c730 2024-03-18T15:01:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.158+0000 7f12cb7fe640 10 client.26357 handle_command_reply: tid=0 2024-03-18T15:01:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.158+0000 7f12f4fdb740 1 client.26357 shutdown 2024-03-18T15:01:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.160+0000 7f12f4fdb740 1 -- 172.21.15.76:0/624322757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6eca50 msgr2=0x5577af709190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.160+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/624322757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6eca50 0x5577af709190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x5577af6f20b0 tx=0x7f12dc004410 comp rx=0 tx=0).stop 2024-03-18T15:01:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.160+0000 7f12f4fdb740 1 -- 172.21.15.76:0/624322757 shutdown_connections 2024-03-18T15:01:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.160+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/624322757 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5577af76c730 0x5577af76eb50 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.160+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/624322757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5577af6f5820 0x5577af76a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.160+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/624322757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577af6eca50 0x5577af709190 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.160+0000 7f12f4fdb740 1 --2- 172.21.15.76:0/624322757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5577af6ec130 0x5577af708c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.160+0000 7f12f4fdb740 1 -- 172.21.15.76:0/624322757 >> 172.21.15.76:0/624322757 conn(0x5577af6e6a50 msgr2=0x5577af6f0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.160+0000 7f12cb7fe640 0 client.26357 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.161+0000 7f12f4fdb740 1 -- 172.21.15.76:0/624322757 shutdown_connections 2024-03-18T15:01:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.162+0000 7f12f4fdb740 1 -- 172.21.15.76:0/624322757 wait complete. 2024-03-18T15:01:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.162+0000 7f12f4fdb740 20 client.26357 trim_cache size 0 max 16384 2024-03-18T15:01:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:01:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:01:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:01:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:01:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.163+0000 7f12f23f3640 1 -- 172.21.15.76:0/529569254 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f12d40a2a10 msgr2=0x7f12d40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.163+0000 7f12f23f3640 1 --2- 172.21.15.76:0/529569254 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f12d40a2a10 0x7f12d40a4ed0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f12e409ba90 tx=0x7f12e406d730 comp rx=0 tx=0).stop 2024-03-18T15:01:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.163+0000 7f12f23f3640 1 -- 172.21.15.76:0/529569254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ec107790 msgr2=0x7f12ec19c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.163+0000 7f12f23f3640 1 --2- 172.21.15.76:0/529569254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ec107790 0x7f12ec19c860 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f12e4009fd0 tx=0x7f12e400ef10 comp rx=0 tx=0).stop 2024-03-18T15:01:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.163+0000 7f12f23f3640 1 -- 172.21.15.76:0/529569254 shutdown_connections 2024-03-18T15:01:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.163+0000 7f12f23f3640 1 --2- 172.21.15.76:0/529569254 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f12d40a2a10 0x7f12d40a4ed0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.163+0000 7f12f23f3640 1 --2- 172.21.15.76:0/529569254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ec107790 0x7f12ec19c860 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.163+0000 7f12f23f3640 1 --2- 172.21.15.76:0/529569254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12ec102140 0x7f12ec1984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.163+0000 7f12f23f3640 1 --2- 172.21.15.76:0/529569254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12ec101780 0x7f12ec197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.163+0000 7f12f23f3640 1 -- 172.21.15.76:0/529569254 >> 172.21.15.76:0/529569254 conn(0x7f12ec0fd270 msgr2=0x7f12ec0fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.163+0000 7f12f23f3640 1 -- 172.21.15.76:0/529569254 shutdown_connections 2024-03-18T15:01:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:15.163+0000 7f12f23f3640 1 -- 172.21.15.76:0/529569254 wait complete. 2024-03-18T15:01:15.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 777894 9.46 MB/sec execute 316 sec latency 54.266 ms 2024-03-18T15:01:16.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:15 smithi076 ceph-mon[32205]: pgmap v252: 97 pgs: 97 active+clean; 230 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-03-18T15:01:16.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:15 smithi174 ceph-mon[36787]: pgmap v252: 97 pgs: 97 active+clean; 230 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-03-18T15:01:16.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:15 smithi077 ceph-mon[35865]: pgmap v252: 97 pgs: 97 active+clean; 230 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-03-18T15:01:16.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 779249 9.45 MB/sec execute 317 sec latency 26.180 ms 2024-03-18T15:01:17.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 781494 9.47 MB/sec execute 318 sec latency 17.044 ms 2024-03-18T15:01:18.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:17 smithi076 ceph-mon[32205]: pgmap v253: 97 pgs: 97 active+clean; 230 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-03-18T15:01:18.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:17 smithi174 ceph-mon[36787]: pgmap v253: 97 pgs: 97 active+clean; 230 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-03-18T15:01:18.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:17 smithi077 ceph-mon[35865]: pgmap v253: 97 pgs: 97 active+clean; 230 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-03-18T15:01:18.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 782965 9.45 MB/sec execute 319 sec latency 20.421 ms 2024-03-18T15:01:19.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 785090 9.47 MB/sec execute 320 sec latency 21.320 ms 2024-03-18T15:01:20.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:19 smithi076 ceph-mon[32205]: pgmap v254: 97 pgs: 97 active+clean; 225 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 292 op/s 2024-03-18T15:01:20.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:19 smithi174 ceph-mon[36787]: pgmap v254: 97 pgs: 97 active+clean; 225 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 292 op/s 2024-03-18T15:01:20.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:19 smithi077 ceph-mon[35865]: pgmap v254: 97 pgs: 97 active+clean; 225 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 292 op/s 2024-03-18T15:01:20.751 INFO:tasks.workunit.client.0.smithi076.stdout: 1 786597 9.45 MB/sec execute 321 sec latency 32.239 ms 2024-03-18T15:01:21.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 788633 9.47 MB/sec execute 322 sec latency 26.223 ms 2024-03-18T15:01:22.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:21 smithi076 ceph-mon[32205]: pgmap v255: 97 pgs: 97 active+clean; 225 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 292 op/s 2024-03-18T15:01:22.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:21 smithi174 ceph-mon[36787]: pgmap v255: 97 pgs: 97 active+clean; 225 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 292 op/s 2024-03-18T15:01:22.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:21 smithi077 ceph-mon[35865]: pgmap v255: 97 pgs: 97 active+clean; 225 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 292 op/s 2024-03-18T15:01:22.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 790614 9.46 MB/sec execute 323 sec latency 19.222 ms 2024-03-18T15:01:23.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 792215 9.47 MB/sec execute 324 sec latency 17.796 ms 2024-03-18T15:01:24.018 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:23 smithi174 ceph-mon[36787]: pgmap v256: 97 pgs: 97 active+clean; 232 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.3 MiB/s wr, 406 op/s 2024-03-18T15:01:24.028 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:23 smithi077 ceph-mon[35865]: pgmap v256: 97 pgs: 97 active+clean; 232 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.3 MiB/s wr, 406 op/s 2024-03-18T15:01:24.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:23 smithi076 ceph-mon[32205]: pgmap v256: 97 pgs: 97 active+clean; 232 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.3 MiB/s wr, 406 op/s 2024-03-18T15:01:24.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 793859 9.46 MB/sec execute 325 sec latency 35.035 ms 2024-03-18T15:01:25.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 795628 9.47 MB/sec execute 326 sec latency 62.479 ms 2024-03-18T15:01:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:25 smithi077 ceph-mon[35865]: pgmap v257: 97 pgs: 97 active+clean; 232 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 247 op/s 2024-03-18T15:01:26.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:25 smithi076 ceph-mon[32205]: pgmap v257: 97 pgs: 97 active+clean; 232 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 247 op/s 2024-03-18T15:01:26.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:25 smithi174 ceph-mon[36787]: pgmap v257: 97 pgs: 97 active+clean; 232 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 247 op/s 2024-03-18T15:01:26.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 796840 9.45 MB/sec execute 327 sec latency 26.164 ms 2024-03-18T15:01:27.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 799125 9.47 MB/sec execute 328 sec latency 17.244 ms 2024-03-18T15:01:28.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:27 smithi076 ceph-mon[32205]: pgmap v258: 97 pgs: 97 active+clean; 232 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 247 op/s 2024-03-18T15:01:28.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:27 smithi174 ceph-mon[36787]: pgmap v258: 97 pgs: 97 active+clean; 232 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 247 op/s 2024-03-18T15:01:28.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:27 smithi077 ceph-mon[35865]: pgmap v258: 97 pgs: 97 active+clean; 232 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 247 op/s 2024-03-18T15:01:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.259+0000 7fa94d6e8640 1 -- 172.21.15.76:0/1693317268 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa948103370 msgr2=0x7fa94810b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.259+0000 7fa94d6e8640 1 --2- 172.21.15.76:0/1693317268 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa948103370 0x7fa94810b850 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa93c009900 tx=0x7fa93c02bcd0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.259+0000 7fa94d6e8640 1 -- 172.21.15.76:0/1693317268 shutdown_connections 2024-03-18T15:01:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.259+0000 7fa94d6e8640 1 --2- 172.21.15.76:0/1693317268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa94810be20 0x7fa94810e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.259+0000 7fa94d6e8640 1 --2- 172.21.15.76:0/1693317268 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa948103370 0x7fa94810b850 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.259+0000 7fa94d6e8640 1 --2- 172.21.15.76:0/1693317268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa948102a50 0x7fa948102e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.259+0000 7fa94d6e8640 1 -- 172.21.15.76:0/1693317268 >> 172.21.15.76:0/1693317268 conn(0x7fa948077800 msgr2=0x7fa948077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.259+0000 7fa94d6e8640 1 -- 172.21.15.76:0/1693317268 shutdown_connections 2024-03-18T15:01:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.259+0000 7fa94d6e8640 1 -- 172.21.15.76:0/1693317268 wait complete. 2024-03-18T15:01:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa94d6e8640 1 Processor -- start 2024-03-18T15:01:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa94d6e8640 1 -- start start 2024-03-18T15:01:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa94d6e8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa948102a50 0x7fa9481980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa94d6e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa948103370 0x7fa9481985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa94d6e8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa94810be20 0x7fa94819c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa94d6e8640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa948111c00 con 0x7fa948102a50 2024-03-18T15:01:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa94d6e8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa948111a80 con 0x7fa948103370 2024-03-18T15:01:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa94d6e8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa948111d80 con 0x7fa94810be20 2024-03-18T15:01:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa946ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa948102a50 0x7fa9481980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa946ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa948102a50 0x7fa9481980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47498/0 (socket says 172.21.15.76:47498) 2024-03-18T15:01:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa946ffd640 1 -- 172.21.15.76:0/2812130284 learned_addr learned my addr 172.21.15.76:0/2812130284 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa9467fc640 1 --2- 172.21.15.76:0/2812130284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa948103370 0x7fa9481985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa9477fe640 1 --2- 172.21.15.76:0/2812130284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa94810be20 0x7fa94819c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa946ffd640 1 -- 172.21.15.76:0/2812130284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa94810be20 msgr2=0x7fa94819c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa946ffd640 1 --2- 172.21.15.76:0/2812130284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa94810be20 0x7fa94819c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa946ffd640 1 -- 172.21.15.76:0/2812130284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa948103370 msgr2=0x7fa9481985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa946ffd640 1 --2- 172.21.15.76:0/2812130284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa948103370 0x7fa9481985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa946ffd640 1 -- 172.21.15.76:0/2812130284 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94819d040 con 0x7fa948102a50 2024-03-18T15:01:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.260+0000 7fa9467fc640 1 --2- 172.21.15.76:0/2812130284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa948103370 0x7fa9481985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:01:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.261+0000 7fa946ffd640 1 --2- 172.21.15.76:0/2812130284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa948102a50 0x7fa9481980a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa93800e9f0 tx=0x7fa93800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.261+0000 7fa92ffff640 1 -- 172.21.15.76:0/2812130284 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93800cd80 con 0x7fa948102a50 2024-03-18T15:01:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.261+0000 7fa92ffff640 1 -- 172.21.15.76:0/2812130284 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa938005500 con 0x7fa948102a50 2024-03-18T15:01:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.261+0000 7fa94d6e8640 1 -- 172.21.15.76:0/2812130284 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94819d330 con 0x7fa948102a50 2024-03-18T15:01:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.261+0000 7fa92ffff640 1 -- 172.21.15.76:0/2812130284 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa938010690 con 0x7fa948102a50 2024-03-18T15:01:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.261+0000 7fa94d6e8640 1 -- 172.21.15.76:0/2812130284 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9481a4bc0 con 0x7fa948102a50 2024-03-18T15:01:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.263+0000 7fa92ffff640 1 -- 172.21.15.76:0/2812130284 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fa9380046c0 con 0x7fa948102a50 2024-03-18T15:01:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.263+0000 7fa9503ac740 1 Processor -- start 2024-03-18T15:01:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.263+0000 7fa9503ac740 1 -- start start 2024-03-18T15:01:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.263+0000 7fa9503ac740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57130 0x561e0bc57510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa92ffff640 1 --2- 172.21.15.76:0/2812130284 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa9300a2a90 0x7fa9300a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa9503ac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc57a50 0x561e0bc602e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa9503ac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc60820 0x561e0bc62c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa9503ac740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561e0bbbba40 con 0x561e0bc57130 2024-03-18T15:01:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa9503ac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561e0bb64920 con 0x561e0bc60820 2024-03-18T15:01:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa9503ac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561e0bbb7ae0 con 0x561e0bc57a50 2024-03-18T15:01:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa9477fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57130 0x561e0bc57510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa9477fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57130 0x561e0bc57510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47508/0 (socket says 172.21.15.76:47508) 2024-03-18T15:01:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa946ffd640 1 --2- 172.21.15.76:0/2812130284 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa9300a2a90 0x7fa9300a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa9477fe640 1 -- 172.21.15.76:0/2136523225 learned_addr learned my addr 172.21.15.76:0/2136523225 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa9467fc640 1 --2- 172.21.15.76:0/2136523225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc57a50 0x561e0bc602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa9477fe640 1 --2- 172.21.15.76:0/2136523225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc60820 0x561e0bc62c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa92ffff640 1 -- 172.21.15.76:0/2812130284 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7fa9380cf1b0 con 0x7fa948102a50 2024-03-18T15:01:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa9477fe640 1 -- 172.21.15.76:0/2136523225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc57a50 msgr2=0x561e0bc602e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa9477fe640 1 --2- 172.21.15.76:0/2136523225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc57a50 0x561e0bc602e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa9477fe640 1 -- 172.21.15.76:0/2136523225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc60820 msgr2=0x561e0bc62c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa9477fe640 1 --2- 172.21.15.76:0/2136523225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc60820 0x561e0bc62c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa9477fe640 1 -- 172.21.15.76:0/2136523225 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e0bb7c460 con 0x561e0bc57130 2024-03-18T15:01:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa9467fc640 1 --2- 172.21.15.76:0/2136523225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc57a50 0x561e0bc602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:01:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa946ffd640 1 --2- 172.21.15.76:0/2812130284 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa9300a2a90 0x7fa9300a4f50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa93800e9c0 tx=0x7fa938086040 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa9477fe640 1 --2- 172.21.15.76:0/2136523225 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57130 0x561e0bc57510 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa940013a90 tx=0x7fa940009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246fbeb4b6e42659 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa92d7fa640 1 -- 172.21.15.76:0/2136523225 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940013170 con 0x561e0bc57130 2024-03-18T15:01:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa92d7fa640 1 -- 172.21.15.76:0/2136523225 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa94000f940 con 0x561e0bc57130 2024-03-18T15:01:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.264+0000 7fa92d7fa640 1 -- 172.21.15.76:0/2136523225 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9400052f0 con 0x561e0bc57130 2024-03-18T15:01:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.265+0000 7fa9503ac740 1 -- 172.21.15.76:0/2136523225 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57130 msgr2=0x561e0bc57510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.265+0000 7fa9503ac740 1 --2- 172.21.15.76:0/2136523225 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57130 0x561e0bc57510 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa940013a90 tx=0x7fa940009500 comp rx=0 tx=0).stop 2024-03-18T15:01:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.265+0000 7fa9503ac740 1 -- 172.21.15.76:0/2136523225 shutdown_connections 2024-03-18T15:01:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.265+0000 7fa9503ac740 1 --2- 172.21.15.76:0/2136523225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc57a50 0x561e0bc602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.265+0000 7fa9503ac740 1 --2- 172.21.15.76:0/2136523225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc60820 0x561e0bc62c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.265+0000 7fa9503ac740 1 --2- 172.21.15.76:0/2136523225 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57130 0x561e0bc57510 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.265+0000 7fa9503ac740 1 -- 172.21.15.76:0/2136523225 >> 172.21.15.76:0/2136523225 conn(0x561e0bc51a50 msgr2=0x561e0bc51e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.265+0000 7fa9503ac740 1 -- 172.21.15.76:0/2136523225 shutdown_connections 2024-03-18T15:01:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.265+0000 7fa9503ac740 1 -- 172.21.15.76:0/2136523225 wait complete. 2024-03-18T15:01:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9503ac740 1 -- start start 2024-03-18T15:01:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9503ac740 1 Processor -- start 2024-03-18T15:01:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9503ac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc57130 0x561e0bc5d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9503ac740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57a50 0x561e0bc5dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9503ac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc60820 0x561e0bc5e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9503ac740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561e0bc633f0 con 0x561e0bc57a50 2024-03-18T15:01:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9503ac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561e0bb77d70 con 0x561e0bc60820 2024-03-18T15:01:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9503ac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561e0bc63570 con 0x561e0bc57130 2024-03-18T15:01:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9503ac740 10 client.0 _ll_register_callbacks cb 0x561e0b8cfa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:01:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9467fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57a50 0x561e0bc5dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9477fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc57130 0x561e0bc5d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9477fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc57130 0x561e0bc5d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:46438/0 (socket says 172.21.15.76:46438) 2024-03-18T15:01:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9477fe640 1 -- 172.21.15.76:0/2698714800 learned_addr learned my addr 172.21.15.76:0/2698714800 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9477fe640 1 --2- 172.21.15.76:0/2698714800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc60820 0x561e0bc5e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9467fc640 1 -- 172.21.15.76:0/2698714800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc57130 msgr2=0x561e0bc5d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9467fc640 1 --2- 172.21.15.76:0/2698714800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc57130 0x561e0bc5d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9467fc640 1 -- 172.21.15.76:0/2698714800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc60820 msgr2=0x561e0bc5e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9467fc640 1 --2- 172.21.15.76:0/2698714800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc60820 0x561e0bc5e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9467fc640 1 -- 172.21.15.76:0/2698714800 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e0b8d0210 con 0x561e0bc57a50 2024-03-18T15:01:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.266+0000 7fa9477fe640 1 --2- 172.21.15.76:0/2698714800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc57130 0x561e0bc5d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:01:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.267+0000 7fa9467fc640 1 --2- 172.21.15.76:0/2698714800 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57a50 0x561e0bc5dcd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa93c02c210 tx=0x7fa93c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbcf0fb43ee7bc1b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.267+0000 7fa92aff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:01:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.267+0000 7fa92aff5640 1 -- 172.21.15.76:0/2698714800 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c043070 con 0x561e0bc57a50 2024-03-18T15:01:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.267+0000 7fa92aff5640 1 -- 172.21.15.76:0/2698714800 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7fa93c02c920 con 0x561e0bc57a50 2024-03-18T15:01:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.267+0000 7fa9503ac740 1 -- 172.21.15.76:0/2698714800 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa94000f940 con 0x561e0bc57a50 2024-03-18T15:01:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.267+0000 7fa92aff5640 1 -- 172.21.15.76:0/2698714800 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fa93c02cb60 con 0x561e0bc57a50 2024-03-18T15:01:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.267+0000 7fa9503ac740 10 client.16506 fetch_fsmap learned FSMap version 41 2024-03-18T15:01:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.267+0000 7fa9503ac740 1 -- 172.21.15.76:0/2698714800 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x561e0b99c620 con 0x561e0bc57a50 2024-03-18T15:01:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.267+0000 7fa92aff5640 1 -- 172.21.15.76:0/2698714800 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa94000f940 con 0x561e0bc57a50 2024-03-18T15:01:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.267+0000 7fa92aff5640 1 -- 172.21.15.76:0/2698714800 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7fa93c04e020 con 0x561e0bc57a50 2024-03-18T15:01:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.267+0000 7fa9503ac740 10 client.16506 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:01:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.267+0000 7fa9503ac740 10 client.16506 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:01:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.267+0000 7fa9503ac740 20 client.16506 populate_metadata read hostname 'smithi076' 2024-03-18T15:01:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.267+0000 7fa9503ac740 1 --2- 172.21.15.76:0/2698714800 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561e0bc70860 0x561e0bc72c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.267+0000 7fa9503ac740 4 client.16506 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:01:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.267+0000 7fa9503ac740 1 -- 172.21.15.76:0/2698714800 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e0bbc8f80 con 0x561e0bc70860 2024-03-18T15:01:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.268+0000 7fa9477fe640 1 --2- 172.21.15.76:0/2698714800 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561e0bc70860 0x561e0bc72c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.268+0000 7fa9477fe640 1 --2- 172.21.15.76:0/2698714800 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561e0bc70860 0x561e0bc72c50 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aaa73564e4aada6f server_cookie=9b22f23b3a10e260 in_seq=0 out_seq=0 2024-03-18T15:01:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.268+0000 7fa92aff5640 10 client.16506 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.269+0000 7fa92aff5640 1 -- 172.21.15.76:0/2698714800 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561e0bbc8f80 con 0x561e0bc70860 2024-03-18T15:01:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.269+0000 7fa92aff5640 10 client.16506 handle_command_reply: tid=0 2024-03-18T15:01:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.269+0000 7fa9503ac740 1 client.16506 shutdown 2024-03-18T15:01:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.269+0000 7fa9503ac740 1 -- 172.21.15.76:0/2698714800 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57a50 msgr2=0x561e0bc5dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.269+0000 7fa9503ac740 1 --2- 172.21.15.76:0/2698714800 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57a50 0x561e0bc5dcd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa93c02c210 tx=0x7fa93c004040 comp rx=0 tx=0).stop 2024-03-18T15:01:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.269+0000 7fa9503ac740 1 -- 172.21.15.76:0/2698714800 shutdown_connections 2024-03-18T15:01:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.269+0000 7fa9503ac740 1 --2- 172.21.15.76:0/2698714800 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561e0bc70860 0x561e0bc72c50 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.269+0000 7fa9503ac740 1 --2- 172.21.15.76:0/2698714800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc60820 0x561e0bc5e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.269+0000 7fa9503ac740 1 --2- 172.21.15.76:0/2698714800 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57a50 0x561e0bc5dcd0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.269+0000 7fa92aff5640 0 client.16506 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.269+0000 7fa9503ac740 1 --2- 172.21.15.76:0/2698714800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc57130 0x561e0bc5d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.269+0000 7fa9503ac740 1 -- 172.21.15.76:0/2698714800 >> 172.21.15.76:0/2698714800 conn(0x561e0bc51a50 msgr2=0x561e0bc537a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.269+0000 7fa9503ac740 1 -- 172.21.15.76:0/2698714800 shutdown_connections 2024-03-18T15:01:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.269+0000 7fa9503ac740 1 -- 172.21.15.76:0/2698714800 wait complete. 2024-03-18T15:01:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.269+0000 7fa9503ac740 20 client.16506 trim_cache size 0 max 16384 2024-03-18T15:01:28.288 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.287+0000 7fa9503ac740 1 Processor -- start 2024-03-18T15:01:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.287+0000 7fa9503ac740 1 -- start start 2024-03-18T15:01:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.287+0000 7fa9503ac740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57130 0x561e0bc5fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.287+0000 7fa9503ac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc57a50 0x561e0bc6c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.287+0000 7fa9503ac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc60820 0x561e0bc78540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.287+0000 7fa9503ac740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481afcd0 con 0x561e0bc57130 2024-03-18T15:01:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.287+0000 7fa9503ac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481ad5a0 con 0x561e0bc57a50 2024-03-18T15:01:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.287+0000 7fa9503ac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481ab410 con 0x561e0bc60820 2024-03-18T15:01:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.287+0000 7fa9477fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57130 0x561e0bc5fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.287+0000 7fa9467fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc57a50 0x561e0bc6c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.287+0000 7fa9477fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57130 0x561e0bc5fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47512/0 (socket says 172.21.15.76:47512) 2024-03-18T15:01:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.287+0000 7fa9477fe640 1 -- 172.21.15.76:0/1406271325 learned_addr learned my addr 172.21.15.76:0/1406271325 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.287+0000 7fa9477fe640 1 --2- 172.21.15.76:0/1406271325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc60820 0x561e0bc78540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.288+0000 7fa9467fc640 1 -- 172.21.15.76:0/1406271325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc60820 msgr2=0x561e0bc78540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.288+0000 7fa9467fc640 1 --2- 172.21.15.76:0/1406271325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc60820 0x561e0bc78540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.288+0000 7fa9467fc640 1 -- 172.21.15.76:0/1406271325 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57130 msgr2=0x561e0bc5fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.288+0000 7fa9467fc640 1 --2- 172.21.15.76:0/1406271325 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57130 0x561e0bc5fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.288+0000 7fa9467fc640 1 -- 172.21.15.76:0/1406271325 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e0bc732b0 con 0x561e0bc57a50 2024-03-18T15:01:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.288+0000 7fa9467fc640 1 --2- 172.21.15.76:0/1406271325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc57a50 0x561e0bc6c190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa93c005ab0 tx=0x7fa93c009230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7aa774d825345624 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.288+0000 7fa92cff9640 1 -- 172.21.15.76:0/1406271325 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c043040 con 0x561e0bc57a50 2024-03-18T15:01:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.288+0000 7fa92cff9640 1 -- 172.21.15.76:0/1406271325 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa93c055020 con 0x561e0bc57a50 2024-03-18T15:01:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.288+0000 7fa92cff9640 1 -- 172.21.15.76:0/1406271325 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c02c9b0 con 0x561e0bc57a50 2024-03-18T15:01:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.289+0000 7fa9503ac740 1 -- 172.21.15.76:0/1406271325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc57a50 msgr2=0x561e0bc6c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.289+0000 7fa9503ac740 1 --2- 172.21.15.76:0/1406271325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc57a50 0x561e0bc6c190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa93c005ab0 tx=0x7fa93c009230 comp rx=0 tx=0).stop 2024-03-18T15:01:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.289+0000 7fa9503ac740 1 -- 172.21.15.76:0/1406271325 shutdown_connections 2024-03-18T15:01:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.289+0000 7fa9503ac740 1 --2- 172.21.15.76:0/1406271325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc60820 0x561e0bc78540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.289+0000 7fa9503ac740 1 --2- 172.21.15.76:0/1406271325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc57a50 0x561e0bc6c190 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.289+0000 7fa9503ac740 1 --2- 172.21.15.76:0/1406271325 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57130 0x561e0bc5fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.289+0000 7fa9503ac740 1 -- 172.21.15.76:0/1406271325 >> 172.21.15.76:0/1406271325 conn(0x561e0bc51a50 msgr2=0x561e0bc5d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.289+0000 7fa9503ac740 1 -- 172.21.15.76:0/1406271325 shutdown_connections 2024-03-18T15:01:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.289+0000 7fa9503ac740 1 -- 172.21.15.76:0/1406271325 wait complete. 2024-03-18T15:01:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.289+0000 7fa9503ac740 1 -- start start 2024-03-18T15:01:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.289+0000 7fa9503ac740 1 Processor -- start 2024-03-18T15:01:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9503ac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc57130 0x561e0bc5a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9503ac740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57a50 0x561e0bc5aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9503ac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc60820 0x561e0bc5afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9503ac740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481aba60 con 0x561e0bc57a50 2024-03-18T15:01:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9503ac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481aced0 con 0x561e0bc57130 2024-03-18T15:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9503ac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481ac130 con 0x561e0bc60820 2024-03-18T15:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9503ac740 10 client.0 _ll_register_callbacks cb 0x7fa93c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9467fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57a50 0x561e0bc5aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc57130 0x561e0bc5a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9467fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57a50 0x561e0bc5aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47528/0 (socket says 172.21.15.76:47528) 2024-03-18T15:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9467fc640 1 -- 172.21.15.76:0/2858352341 learned_addr learned my addr 172.21.15.76:0/2858352341 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9477fe640 1 --2- 172.21.15.76:0/2858352341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc60820 0x561e0bc5afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9467fc640 1 -- 172.21.15.76:0/2858352341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc60820 msgr2=0x561e0bc5afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9467fc640 1 --2- 172.21.15.76:0/2858352341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc60820 0x561e0bc5afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9467fc640 1 -- 172.21.15.76:0/2858352341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc57130 msgr2=0x561e0bc5a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9467fc640 1 --2- 172.21.15.76:0/2858352341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc57130 0x561e0bc5a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9467fc640 1 -- 172.21.15.76:0/2858352341 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e0bc51880 con 0x561e0bc57a50 2024-03-18T15:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9477fe640 1 --2- 172.21.15.76:0/2858352341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc57130 0x561e0bc5a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa9467fc640 1 --2- 172.21.15.76:0/2858352341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57a50 0x561e0bc5aa70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fa93c006010 tx=0x7fa93c0040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3852fc24bf048ec5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.290+0000 7fa92aff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.291+0000 7fa92aff5640 1 -- 172.21.15.76:0/2858352341 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c043040 con 0x561e0bc57a50 2024-03-18T15:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.291+0000 7fa92aff5640 1 -- 172.21.15.76:0/2858352341 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7fa93c0050f0 con 0x561e0bc57a50 2024-03-18T15:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.291+0000 7fa9503ac740 1 -- 172.21.15.76:0/2858352341 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa93c055020 con 0x561e0bc57a50 2024-03-18T15:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.291+0000 7fa92aff5640 1 -- 172.21.15.76:0/2858352341 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fa93c0036d0 con 0x561e0bc57a50 2024-03-18T15:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.291+0000 7fa9503ac740 10 client.16512 fetch_fsmap learned FSMap version 41 2024-03-18T15:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.291+0000 7fa9503ac740 1 -- 172.21.15.76:0/2858352341 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x561e0bcd6e50 con 0x561e0bc57a50 2024-03-18T15:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.291+0000 7fa92aff5640 1 -- 172.21.15.76:0/2858352341 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa93c055020 con 0x561e0bc57a50 2024-03-18T15:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.291+0000 7fa92aff5640 1 -- 172.21.15.76:0/2858352341 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7fa93c061020 con 0x561e0bc57a50 2024-03-18T15:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.291+0000 7fa9503ac740 10 client.16512 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.291+0000 7fa9503ac740 10 client.16512 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.291+0000 7fa9503ac740 20 client.16512 populate_metadata read hostname 'smithi076' 2024-03-18T15:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.291+0000 7fa9503ac740 1 --2- 172.21.15.76:0/2858352341 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561e0bcd7510 0x561e0bcd9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.291+0000 7fa9503ac740 4 client.16512 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.291+0000 7fa9503ac740 1 -- 172.21.15.76:0/2858352341 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561e0bbc32d0 con 0x561e0bcd7510 2024-03-18T15:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.291+0000 7fa9477fe640 1 --2- 172.21.15.76:0/2858352341 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561e0bcd7510 0x561e0bcd9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.292+0000 7fa92aff5640 10 client.16512 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.292+0000 7fa9477fe640 1 --2- 172.21.15.76:0/2858352341 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561e0bcd7510 0x561e0bcd9930 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24805598b8e67052 server_cookie=ef9fdf731392382c in_seq=0 out_seq=0 2024-03-18T15:01:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.374+0000 7fa92aff5640 1 -- 172.21.15.76:0/2858352341 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561e0bbc32d0 con 0x561e0bcd7510 2024-03-18T15:01:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.374+0000 7fa92aff5640 10 client.16512 handle_command_reply: tid=0 2024-03-18T15:01:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.374+0000 7fa9503ac740 1 client.16512 shutdown 2024-03-18T15:01:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.375+0000 7fa9503ac740 1 -- 172.21.15.76:0/2858352341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57a50 msgr2=0x561e0bc5aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.375+0000 7fa9503ac740 1 --2- 172.21.15.76:0/2858352341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57a50 0x561e0bc5aa70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fa93c006010 tx=0x7fa93c0040b0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.375+0000 7fa9503ac740 1 -- 172.21.15.76:0/2858352341 shutdown_connections 2024-03-18T15:01:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.375+0000 7fa9503ac740 1 --2- 172.21.15.76:0/2858352341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e0bc60820 0x561e0bc5afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.375+0000 7fa9503ac740 1 --2- 172.21.15.76:0/2858352341 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x561e0bcd7510 0x561e0bcd9930 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.375+0000 7fa9503ac740 1 --2- 172.21.15.76:0/2858352341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561e0bc57a50 0x561e0bc5aa70 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.375+0000 7fa9503ac740 1 --2- 172.21.15.76:0/2858352341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e0bc57130 0x561e0bc5a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.375+0000 7fa92aff5640 0 client.16512 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.375+0000 7fa9503ac740 1 -- 172.21.15.76:0/2858352341 >> 172.21.15.76:0/2858352341 conn(0x561e0bc51a50 msgr2=0x561e0bc53660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.375+0000 7fa9503ac740 1 -- 172.21.15.76:0/2858352341 shutdown_connections 2024-03-18T15:01:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.375+0000 7fa9503ac740 1 -- 172.21.15.76:0/2858352341 wait complete. 2024-03-18T15:01:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.375+0000 7fa9503ac740 20 client.16512 trim_cache size 0 max 16384 2024-03-18T15:01:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:01:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:01:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:01:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:01:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.375+0000 7fa94d6e8640 1 -- 172.21.15.76:0/2812130284 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa9300a2a90 msgr2=0x7fa9300a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.375+0000 7fa94d6e8640 1 --2- 172.21.15.76:0/2812130284 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa9300a2a90 0x7fa9300a4f50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa93800e9c0 tx=0x7fa938086040 comp rx=0 tx=0).stop 2024-03-18T15:01:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.376+0000 7fa94d6e8640 1 -- 172.21.15.76:0/2812130284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa948102a50 msgr2=0x7fa9481980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.376+0000 7fa94d6e8640 1 --2- 172.21.15.76:0/2812130284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa948102a50 0x7fa9481980a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa93800e9f0 tx=0x7fa93800edf0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.376+0000 7fa94d6e8640 1 -- 172.21.15.76:0/2812130284 shutdown_connections 2024-03-18T15:01:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.376+0000 7fa94d6e8640 1 --2- 172.21.15.76:0/2812130284 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa9300a2a90 0x7fa9300a4f50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.376+0000 7fa94d6e8640 1 --2- 172.21.15.76:0/2812130284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa94810be20 0x7fa94819c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.376+0000 7fa94d6e8640 1 --2- 172.21.15.76:0/2812130284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa948103370 0x7fa9481985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.376+0000 7fa94d6e8640 1 --2- 172.21.15.76:0/2812130284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa948102a50 0x7fa9481980a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.376+0000 7fa94d6e8640 1 -- 172.21.15.76:0/2812130284 >> 172.21.15.76:0/2812130284 conn(0x7fa948077800 msgr2=0x7fa9480fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.376+0000 7fa94d6e8640 1 -- 172.21.15.76:0/2812130284 shutdown_connections 2024-03-18T15:01:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:28.376+0000 7fa94d6e8640 1 -- 172.21.15.76:0/2812130284 wait complete. 2024-03-18T15:01:28.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 800358 9.46 MB/sec execute 329 sec latency 41.170 ms 2024-03-18T15:01:29.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 802653 9.47 MB/sec execute 330 sec latency 14.326 ms 2024-03-18T15:01:30.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:29 smithi076 ceph-mon[32205]: pgmap v259: 97 pgs: 97 active+clean; 227 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 383 op/s 2024-03-18T15:01:30.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:29 smithi174 ceph-mon[36787]: pgmap v259: 97 pgs: 97 active+clean; 227 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 383 op/s 2024-03-18T15:01:30.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:29 smithi077 ceph-mon[35865]: pgmap v259: 97 pgs: 97 active+clean; 227 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 383 op/s 2024-03-18T15:01:30.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 803566 9.45 MB/sec execute 331 sec latency 69.869 ms 2024-03-18T15:01:31.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:30 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:01:31.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:30 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:01:31.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:30 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:01:31.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 805779 9.46 MB/sec execute 332 sec latency 64.792 ms 2024-03-18T15:01:32.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:31 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:01:32.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:31 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:01:32.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:31 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:01:32.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:31 smithi076 ceph-mon[32205]: pgmap v260: 97 pgs: 97 active+clean; 227 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 244 op/s 2024-03-18T15:01:32.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:31 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:01:32.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:31 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:01:32.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:31 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:01:32.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:31 smithi174 ceph-mon[36787]: pgmap v260: 97 pgs: 97 active+clean; 227 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 244 op/s 2024-03-18T15:01:32.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:31 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:01:32.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:31 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:01:32.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:31 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:01:32.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:31 smithi077 ceph-mon[35865]: pgmap v260: 97 pgs: 97 active+clean; 227 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 244 op/s 2024-03-18T15:01:32.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 806933 9.44 MB/sec execute 333 sec latency 25.320 ms 2024-03-18T15:01:33.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 809496 9.47 MB/sec execute 334 sec latency 23.223 ms 2024-03-18T15:01:34.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:33 smithi076 ceph-mon[32205]: pgmap v261: 97 pgs: 97 active+clean; 233 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 341 op/s 2024-03-18T15:01:34.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:33 smithi174 ceph-mon[36787]: pgmap v261: 97 pgs: 97 active+clean; 233 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 341 op/s 2024-03-18T15:01:34.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:33 smithi077 ceph-mon[35865]: pgmap v261: 97 pgs: 97 active+clean; 233 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 341 op/s 2024-03-18T15:01:34.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 810627 9.45 MB/sec execute 335 sec latency 16.596 ms 2024-03-18T15:01:35.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 813073 9.47 MB/sec execute 336 sec latency 36.324 ms 2024-03-18T15:01:35.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:35 smithi077 ceph-mon[35865]: pgmap v262: 97 pgs: 97 active+clean; 233 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 233 op/s 2024-03-18T15:01:36.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:35 smithi076 ceph-mon[32205]: pgmap v262: 97 pgs: 97 active+clean; 233 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 233 op/s 2024-03-18T15:01:36.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:35 smithi174 ceph-mon[36787]: pgmap v262: 97 pgs: 97 active+clean; 233 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 233 op/s 2024-03-18T15:01:36.753 INFO:tasks.workunit.client.0.smithi076.stdout: 1 813988 9.45 MB/sec execute 337 sec latency 25.424 ms 2024-03-18T15:01:37.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 816597 9.47 MB/sec execute 338 sec latency 18.405 ms 2024-03-18T15:01:38.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:37 smithi076 ceph-mon[32205]: pgmap v263: 97 pgs: 97 active+clean; 233 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 233 op/s 2024-03-18T15:01:38.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:37 smithi174 ceph-mon[36787]: pgmap v263: 97 pgs: 97 active+clean; 233 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 233 op/s 2024-03-18T15:01:38.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:37 smithi077 ceph-mon[35865]: pgmap v263: 97 pgs: 97 active+clean; 233 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 233 op/s 2024-03-18T15:01:38.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 817748 9.45 MB/sec execute 339 sec latency 19.935 ms 2024-03-18T15:01:39.752 INFO:tasks.workunit.client.0.smithi076.stdout: 1 820275 9.47 MB/sec execute 340 sec latency 18.276 ms 2024-03-18T15:01:40.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:39 smithi076 ceph-mon[32205]: pgmap v264: 97 pgs: 97 active+clean; 240 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 331 op/s 2024-03-18T15:01:40.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:39 smithi174 ceph-mon[36787]: pgmap v264: 97 pgs: 97 active+clean; 240 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 331 op/s 2024-03-18T15:01:40.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:39 smithi077 ceph-mon[35865]: pgmap v264: 97 pgs: 97 active+clean; 240 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 331 op/s 2024-03-18T15:01:40.753 INFO:tasks.workunit.client.0.smithi076.stdout: 1 821520 9.46 MB/sec execute 341 sec latency 20.865 ms 2024-03-18T15:01:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.472+0000 7f10ff47e640 1 -- 172.21.15.76:0/3697106352 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f8105680 msgr2=0x7f10f8105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:41.475 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.472+0000 7f10ff47e640 1 --2- 172.21.15.76:0/3697106352 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f8105680 0x7f10f8105a60 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009900 tx=0x7f10f002bc30 comp rx=0 tx=0).stop 2024-03-18T15:01:41.475 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.472+0000 7f10ff47e640 1 -- 172.21.15.76:0/3697106352 shutdown_connections 2024-03-18T15:01:41.475 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.472+0000 7f10ff47e640 1 --2- 172.21.15.76:0/3697106352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10f81012f0 0x7f10f810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.475 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.472+0000 7f10ff47e640 1 --2- 172.21.15.76:0/3697106352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10f8100930 0x7f10f8100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.475 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.472+0000 7f10ff47e640 1 --2- 172.21.15.76:0/3697106352 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f8105680 0x7f10f8105a60 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.475 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.472+0000 7f10ff47e640 1 -- 172.21.15.76:0/3697106352 >> 172.21.15.76:0/3697106352 conn(0x7f10f8077530 msgr2=0x7f10f8077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:41.475 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.472+0000 7f10ff47e640 1 -- 172.21.15.76:0/3697106352 shutdown_connections 2024-03-18T15:01:41.475 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.472+0000 7f10ff47e640 1 -- 172.21.15.76:0/3697106352 wait complete. 2024-03-18T15:01:41.475 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.472+0000 7f10ff47e640 1 Processor -- start 2024-03-18T15:01:41.475 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.472+0000 7f10ff47e640 1 -- start start 2024-03-18T15:01:41.475 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10ff47e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10f8100930 0x7f10f810e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.475 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10ff47e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f81012f0 0x7f10f810e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.476 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10ff47e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10f8105680 0x7f10f810ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.476 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10ff47e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f10f8111cc0 con 0x7f10f81012f0 2024-03-18T15:01:41.476 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10ff47e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f10f8111b40 con 0x7f10f8105680 2024-03-18T15:01:41.476 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10ff47e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f10f8111e40 con 0x7f10f8100930 2024-03-18T15:01:41.476 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10fca19640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f81012f0 0x7f10f810e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.476 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10fd21a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10f8100930 0x7f10f810e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.476 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10fca19640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f81012f0 0x7f10f810e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41210/0 (socket says 172.21.15.76:41210) 2024-03-18T15:01:41.476 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10fda1b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10f8105680 0x7f10f810ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.476 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10fca19640 1 -- 172.21.15.76:0/6499741 learned_addr learned my addr 172.21.15.76:0/6499741 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:41.476 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10fca19640 1 -- 172.21.15.76:0/6499741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10f8100930 msgr2=0x7f10f810e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:41.476 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10fca19640 1 --2- 172.21.15.76:0/6499741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10f8100930 0x7f10f810e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.476 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10fca19640 1 -- 172.21.15.76:0/6499741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10f8105680 msgr2=0x7f10f810ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:41.476 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10fca19640 1 --2- 172.21.15.76:0/6499741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10f8105680 0x7f10f810ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.476 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10fca19640 1 -- 172.21.15.76:0/6499741 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f81a6540 con 0x7f10f81012f0 2024-03-18T15:01:41.477 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10fd21a640 1 --2- 172.21.15.76:0/6499741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10f8100930 0x7f10f810e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:01:41.477 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10fca19640 1 --2- 172.21.15.76:0/6499741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f81012f0 0x7f10f810e590 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f10ec00e9f0 tx=0x7f10ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:41.477 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10f67fc640 1 -- 172.21.15.76:0/6499741 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec00cd80 con 0x7f10f81012f0 2024-03-18T15:01:41.477 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10f67fc640 1 -- 172.21.15.76:0/6499741 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10ec010910 con 0x7f10f81012f0 2024-03-18T15:01:41.477 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.473+0000 7f10ff47e640 1 -- 172.21.15.76:0/6499741 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f81a6830 con 0x7f10f81012f0 2024-03-18T15:01:41.477 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.474+0000 7f10f67fc640 1 -- 172.21.15.76:0/6499741 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0188f0 con 0x7f10f81012f0 2024-03-18T15:01:41.477 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.474+0000 7f10ff47e640 1 -- 172.21.15.76:0/6499741 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f81a6d80 con 0x7f10f81012f0 2024-03-18T15:01:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.476+0000 7f10f67fc640 1 -- 172.21.15.76:0/6499741 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f10ec010ac0 con 0x7f10f81012f0 2024-03-18T15:01:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.476+0000 7f1102065740 1 Processor -- start 2024-03-18T15:01:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.476+0000 7f1102065740 1 -- start start 2024-03-18T15:01:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.476+0000 7f10f67fc640 1 --2- 172.21.15.76:0/6499741 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f10e00a2a10 0x7f10e00a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.476+0000 7f1102065740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057cd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f1102065740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 0x5630057d62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f1102065740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057d6820 0x5630057d8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f1102065740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563005731a40 con 0x5630057cda50 2024-03-18T15:01:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f1102065740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5630056da920 con 0x5630057cd130 2024-03-18T15:01:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f1102065740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56300572dae0 con 0x5630057d6820 2024-03-18T15:01:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10fda1b640 1 --2- 172.21.15.76:0/6499741 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f10e00a2a10 0x7f10e00a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10f67fc640 1 -- 172.21.15.76:0/6499741 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f10ec0cf200 con 0x7f10f81012f0 2024-03-18T15:01:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10fd21a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10fca19640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 0x5630057d62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10fd21a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057cd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:50224/0 (socket says 172.21.15.76:50224) 2024-03-18T15:01:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10fca19640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 0x5630057d62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41216/0 (socket says 172.21.15.76:41216) 2024-03-18T15:01:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10fd21a640 1 -- 172.21.15.76:0/281280136 learned_addr learned my addr 172.21.15.76:0/281280136 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10fda1b640 1 --2- 172.21.15.76:0/281280136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057d6820 0x5630057d8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10fd21a640 1 -- 172.21.15.76:0/281280136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057d6820 msgr2=0x5630057d8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10fd21a640 1 --2- 172.21.15.76:0/281280136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057d6820 0x5630057d8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10fd21a640 1 -- 172.21.15.76:0/281280136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 msgr2=0x5630057d62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10fd21a640 1 --2- 172.21.15.76:0/281280136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 0x5630057d62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10fd21a640 1 -- 172.21.15.76:0/281280136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630056f2460 con 0x5630057cd130 2024-03-18T15:01:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10fca19640 1 --2- 172.21.15.76:0/281280136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 0x5630057d62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:01:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10fda1b640 1 --2- 172.21.15.76:0/6499741 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f10e00a2a10 0x7f10e00a4ed0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f10e8009fd0 tx=0x7f10e8011b40 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10fd21a640 1 --2- 172.21.15.76:0/281280136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057cd510 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009900 tx=0x7f10f0005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e439b60b80fa8dd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10df7fe640 1 -- 172.21.15.76:0/281280136 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0042070 con 0x5630057cd130 2024-03-18T15:01:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10df7fe640 1 -- 172.21.15.76:0/281280136 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10f0004a70 con 0x5630057cd130 2024-03-18T15:01:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.477+0000 7f10df7fe640 1 -- 172.21.15.76:0/281280136 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0005210 con 0x5630057cd130 2024-03-18T15:01:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.478+0000 7f1102065740 1 -- 172.21.15.76:0/281280136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 msgr2=0x5630057cd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.478+0000 7f1102065740 1 --2- 172.21.15.76:0/281280136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057cd510 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009900 tx=0x7f10f0005e80 comp rx=0 tx=0).stop 2024-03-18T15:01:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.478+0000 7f1102065740 1 -- 172.21.15.76:0/281280136 shutdown_connections 2024-03-18T15:01:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.478+0000 7f1102065740 1 --2- 172.21.15.76:0/281280136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057d6820 0x5630057d8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.478+0000 7f1102065740 1 --2- 172.21.15.76:0/281280136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 0x5630057d62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.478+0000 7f1102065740 1 --2- 172.21.15.76:0/281280136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057cd510 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.478+0000 7f1102065740 1 -- 172.21.15.76:0/281280136 >> 172.21.15.76:0/281280136 conn(0x5630057c7a50 msgr2=0x5630057c7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.478+0000 7f1102065740 1 -- 172.21.15.76:0/281280136 shutdown_connections 2024-03-18T15:01:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.478+0000 7f1102065740 1 -- 172.21.15.76:0/281280136 wait complete. 2024-03-18T15:01:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.479+0000 7f1102065740 1 -- start start 2024-03-18T15:01:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.479+0000 7f1102065740 1 Processor -- start 2024-03-18T15:01:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.479+0000 7f1102065740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057d3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.479+0000 7f1102065740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 0x5630057d3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.479+0000 7f1102065740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057d6820 0x5630057d42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.479+0000 7f1102065740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5630057d93f0 con 0x5630057cda50 2024-03-18T15:01:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.479+0000 7f1102065740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5630056edd70 con 0x5630057cd130 2024-03-18T15:01:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.479+0000 7f1102065740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5630057d9570 con 0x5630057d6820 2024-03-18T15:01:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.479+0000 7f1102065740 10 client.0 _ll_register_callbacks cb 0x563005445a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:01:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.479+0000 7f10fca19640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 0x5630057d3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.479+0000 7f10fd21a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057d3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.479+0000 7f10fd21a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057d3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:50228/0 (socket says 172.21.15.76:50228) 2024-03-18T15:01:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.479+0000 7f10fd21a640 1 -- 172.21.15.76:0/712541651 learned_addr learned my addr 172.21.15.76:0/712541651 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.479+0000 7f10fca19640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 0x5630057d3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41224/0 (socket says 172.21.15.76:41224) 2024-03-18T15:01:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.479+0000 7f10fda1b640 1 --2- 172.21.15.76:0/712541651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057d6820 0x5630057d42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.479+0000 7f10fca19640 1 -- 172.21.15.76:0/712541651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057d6820 msgr2=0x5630057d42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.479+0000 7f10fca19640 1 --2- 172.21.15.76:0/712541651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057d6820 0x5630057d42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f10fca19640 1 -- 172.21.15.76:0/712541651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 msgr2=0x5630057d3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f10fca19640 1 --2- 172.21.15.76:0/712541651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057d3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f10fca19640 1 -- 172.21.15.76:0/712541651 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563005446210 con 0x5630057cda50 2024-03-18T15:01:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f10fd21a640 1 --2- 172.21.15.76:0/712541651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057d3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:01:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f10fca19640 1 --2- 172.21.15.76:0/712541651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 0x5630057d3db0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f10ec0cc940 tx=0x7f10ec097d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c36ab56617c9229 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f10dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:01:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f10dcff9640 1 -- 172.21.15.76:0/712541651 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0a3d90 con 0x5630057cda50 2024-03-18T15:01:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f1102065740 1 -- 172.21.15.76:0/712541651 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10f0004a70 con 0x5630057cda50 2024-03-18T15:01:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f10dcff9640 1 -- 172.21.15.76:0/712541651 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f10ec09cbb0 con 0x5630057cda50 2024-03-18T15:01:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f10dcff9640 1 -- 172.21.15.76:0/712541651 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f10ec09f9d0 con 0x5630057cda50 2024-03-18T15:01:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f1102065740 10 client.16524 fetch_fsmap learned FSMap version 41 2024-03-18T15:01:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f1102065740 1 -- 172.21.15.76:0/712541651 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x563005512620 con 0x5630057cda50 2024-03-18T15:01:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f10dcff9640 1 -- 172.21.15.76:0/712541651 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10f0004a70 con 0x5630057cda50 2024-03-18T15:01:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f10dcff9640 1 -- 172.21.15.76:0/712541651 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f10ec097680 con 0x5630057cda50 2024-03-18T15:01:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f1102065740 10 client.16524 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:01:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f1102065740 10 client.16524 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:01:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f1102065740 20 client.16524 populate_metadata read hostname 'smithi076' 2024-03-18T15:01:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f1102065740 1 --2- 172.21.15.76:0/712541651 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5630057e6970 0x5630057e8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.480+0000 7f1102065740 4 client.16524 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:01:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.481+0000 7f1102065740 1 -- 172.21.15.76:0/712541651 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56300573ef80 con 0x5630057e6970 2024-03-18T15:01:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.481+0000 7f10fd21a640 1 --2- 172.21.15.76:0/712541651 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5630057e6970 0x5630057e8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.481+0000 7f10dcff9640 10 client.16524 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.481+0000 7f10fd21a640 1 --2- 172.21.15.76:0/712541651 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5630057e6970 0x5630057e8da0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c092ada8b946c23a server_cookie=2cdca4e56029cc28 in_seq=0 out_seq=0 2024-03-18T15:01:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.482+0000 7f10dcff9640 1 -- 172.21.15.76:0/712541651 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56300573ef80 con 0x5630057e6970 2024-03-18T15:01:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.482+0000 7f10dcff9640 10 client.16524 handle_command_reply: tid=0 2024-03-18T15:01:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.482+0000 7f1102065740 1 client.16524 shutdown 2024-03-18T15:01:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.482+0000 7f1102065740 1 -- 172.21.15.76:0/712541651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 msgr2=0x5630057d3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.482+0000 7f1102065740 1 --2- 172.21.15.76:0/712541651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 0x5630057d3db0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f10ec0cc940 tx=0x7f10ec097d00 comp rx=0 tx=0).stop 2024-03-18T15:01:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.482+0000 7f1102065740 1 -- 172.21.15.76:0/712541651 shutdown_connections 2024-03-18T15:01:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.482+0000 7f1102065740 1 --2- 172.21.15.76:0/712541651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057d6820 0x5630057d42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.482+0000 7f1102065740 1 --2- 172.21.15.76:0/712541651 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5630057e6970 0x5630057e8da0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.483+0000 7f1102065740 1 --2- 172.21.15.76:0/712541651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 0x5630057d3db0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.483+0000 7f10dcff9640 0 client.16524 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.483+0000 7f1102065740 1 --2- 172.21.15.76:0/712541651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057d3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.483+0000 7f1102065740 1 -- 172.21.15.76:0/712541651 >> 172.21.15.76:0/712541651 conn(0x5630057c7a50 msgr2=0x5630057c97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.483+0000 7f1102065740 1 -- 172.21.15.76:0/712541651 shutdown_connections 2024-03-18T15:01:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.483+0000 7f1102065740 1 -- 172.21.15.76:0/712541651 wait complete. 2024-03-18T15:01:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.483+0000 7f1102065740 20 client.16524 trim_cache size 0 max 16384 2024-03-18T15:01:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.498+0000 7f1102065740 1 Processor -- start 2024-03-18T15:01:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.498+0000 7f1102065740 1 -- start start 2024-03-18T15:01:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.498+0000 7f1102065740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057cd130 0x5630057d5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.498+0000 7f1102065740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cda50 0x5630057e2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.498+0000 7f1102065740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057d6820 0x5630057ee680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.498+0000 7f1102065740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81b1db0 con 0x5630057d6820 2024-03-18T15:01:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.498+0000 7f1102065740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81af680 con 0x5630057cda50 2024-03-18T15:01:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.498+0000 7f1102065740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81ad4f0 con 0x5630057cd130 2024-03-18T15:01:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.498+0000 7f10fda1b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057d6820 0x5630057ee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.498+0000 7f10fca19640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cda50 0x5630057e2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.498+0000 7f10fd21a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057cd130 0x5630057d5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.498+0000 7f10fca19640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cda50 0x5630057e2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:50234/0 (socket says 172.21.15.76:50234) 2024-03-18T15:01:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.498+0000 7f10fda1b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057d6820 0x5630057ee680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41240/0 (socket says 172.21.15.76:41240) 2024-03-18T15:01:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.498+0000 7f10fca19640 1 -- 172.21.15.76:0/1056559089 learned_addr learned my addr 172.21.15.76:0/1056559089 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.499+0000 7f10fca19640 1 -- 172.21.15.76:0/1056559089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057cd130 msgr2=0x5630057d5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.499+0000 7f10fca19640 1 --2- 172.21.15.76:0/1056559089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057cd130 0x5630057d5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.501 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.499+0000 7f10fca19640 1 -- 172.21.15.76:0/1056559089 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057d6820 msgr2=0x5630057ee680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:41.501 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.499+0000 7f10fca19640 1 --2- 172.21.15.76:0/1056559089 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057d6820 0x5630057ee680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.501 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.499+0000 7f10fca19640 1 -- 172.21.15.76:0/1056559089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630057c7580 con 0x5630057cda50 2024-03-18T15:01:41.501 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.499+0000 7f10fca19640 1 --2- 172.21.15.76:0/1056559089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cda50 0x5630057e2190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f10ec09b8c0 tx=0x7f10ec09bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3e843dd5be0bc42 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:41.501 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.499+0000 7f10deffd640 1 -- 172.21.15.76:0/1056559089 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec09b290 con 0x5630057cda50 2024-03-18T15:01:41.501 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.499+0000 7f10deffd640 1 -- 172.21.15.76:0/1056559089 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10ec0a4020 con 0x5630057cda50 2024-03-18T15:01:41.501 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.499+0000 7f10deffd640 1 -- 172.21.15.76:0/1056559089 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec09afe0 con 0x5630057cda50 2024-03-18T15:01:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.500+0000 7f1102065740 1 -- 172.21.15.76:0/1056559089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cda50 msgr2=0x5630057e2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.500+0000 7f1102065740 1 --2- 172.21.15.76:0/1056559089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cda50 0x5630057e2190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f10ec09b8c0 tx=0x7f10ec09bab0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.500+0000 7f1102065740 1 -- 172.21.15.76:0/1056559089 shutdown_connections 2024-03-18T15:01:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.500+0000 7f1102065740 1 --2- 172.21.15.76:0/1056559089 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057d6820 0x5630057ee680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.500+0000 7f1102065740 1 --2- 172.21.15.76:0/1056559089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cda50 0x5630057e2190 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.500+0000 7f1102065740 1 --2- 172.21.15.76:0/1056559089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057cd130 0x5630057d5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.500+0000 7f1102065740 1 -- 172.21.15.76:0/1056559089 >> 172.21.15.76:0/1056559089 conn(0x5630057c7a50 msgr2=0x5630057c9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.500+0000 7f1102065740 1 -- 172.21.15.76:0/1056559089 shutdown_connections 2024-03-18T15:01:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.500+0000 7f1102065740 1 -- 172.21.15.76:0/1056559089 wait complete. 2024-03-18T15:01:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.500+0000 7f1102065740 1 -- start start 2024-03-18T15:01:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.500+0000 7f1102065740 1 Processor -- start 2024-03-18T15:01:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.501+0000 7f1102065740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057da740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.501+0000 7f1102065740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 0x5630057dac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.501+0000 7f1102065740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057d6820 0x56300584bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.501+0000 7f10fd21a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057da740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.501+0000 7f1102065740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81adb40 con 0x5630057cda50 2024-03-18T15:01:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.501+0000 7f1102065740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81aefb0 con 0x5630057cd130 2024-03-18T15:01:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.501+0000 7f10fd21a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057da740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:50240/0 (socket says 172.21.15.76:50240) 2024-03-18T15:01:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.501+0000 7f10fd21a640 1 -- 172.21.15.76:0/3425493739 learned_addr learned my addr 172.21.15.76:0/3425493739 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.501+0000 7f1102065740 1 -- 172.21.15.76:0/3425493739 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81ae210 con 0x5630057d6820 2024-03-18T15:01:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.501+0000 7f1102065740 10 client.0 _ll_register_callbacks cb 0x7f10ec09b790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:01:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.501+0000 7f10fca19640 1 --2- 172.21.15.76:0/3425493739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 0x5630057dac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.501+0000 7f10fda1b640 1 --2- 172.21.15.76:0/3425493739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057d6820 0x56300584bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.501+0000 7f10fd21a640 1 -- 172.21.15.76:0/3425493739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057d6820 msgr2=0x56300584bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.501+0000 7f10fd21a640 1 --2- 172.21.15.76:0/3425493739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057d6820 0x56300584bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.501+0000 7f10fd21a640 1 -- 172.21.15.76:0/3425493739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 msgr2=0x5630057dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.501+0000 7f10fd21a640 1 --2- 172.21.15.76:0/3425493739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 0x5630057dac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.502+0000 7f10fd21a640 1 -- 172.21.15.76:0/3425493739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56300584c1f0 con 0x5630057cd130 2024-03-18T15:01:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.502+0000 7f10fda1b640 1 --2- 172.21.15.76:0/3425493739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057d6820 0x56300584bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:01:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.502+0000 7f10fca19640 1 --2- 172.21.15.76:0/3425493739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 0x5630057dac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:01:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.502+0000 7f10fd21a640 1 --2- 172.21.15.76:0/3425493739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057da740 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009900 tx=0x7f10f002c1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbc73924b9d74180 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.502+0000 7f10dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T15:01:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.502+0000 7f10dcff9640 1 -- 172.21.15.76:0/3425493739 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0042040 con 0x5630057cd130 2024-03-18T15:01:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.502+0000 7f10dcff9640 1 -- 172.21.15.76:0/3425493739 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f10f004fca0 con 0x5630057cd130 2024-03-18T15:01:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.502+0000 7f1102065740 1 -- 172.21.15.76:0/3425493739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10ec0a4020 con 0x5630057cd130 2024-03-18T15:01:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.502+0000 7f10dcff9640 1 -- 172.21.15.76:0/3425493739 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f10f004f3b0 con 0x5630057cd130 2024-03-18T15:01:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.502+0000 7f1102065740 10 client.26393 fetch_fsmap learned FSMap version 41 2024-03-18T15:01:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.502+0000 7f1102065740 1 -- 172.21.15.76:0/3425493739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x56300584d400 con 0x5630057cd130 2024-03-18T15:01:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.502+0000 7f10dcff9640 1 -- 172.21.15.76:0/3425493739 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10ec0a4020 con 0x5630057cd130 2024-03-18T15:01:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.502+0000 7f10dcff9640 1 -- 172.21.15.76:0/3425493739 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f10f0004fb0 con 0x5630057cd130 2024-03-18T15:01:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.502+0000 7f1102065740 10 client.26393 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:01:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.502+0000 7f1102065740 10 client.26393 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:01:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.503+0000 7f1102065740 20 client.26393 populate_metadata read hostname 'smithi076' 2024-03-18T15:01:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.503+0000 7f1102065740 1 --2- 172.21.15.76:0/3425493739 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5630057e6970 0x56300584f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.503+0000 7f1102065740 4 client.26393 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:01:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.503+0000 7f1102065740 1 -- 172.21.15.76:0/3425493739 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5630057392d0 con 0x5630057e6970 2024-03-18T15:01:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.503+0000 7f10fca19640 1 --2- 172.21.15.76:0/3425493739 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5630057e6970 0x56300584f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.503+0000 7f10dcff9640 10 client.26393 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.503+0000 7f10fca19640 1 --2- 172.21.15.76:0/3425493739 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5630057e6970 0x56300584f700 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fff080618d3b1aff server_cookie=63f1a6ec8c46f4a0 in_seq=0 out_seq=0 2024-03-18T15:01:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.582+0000 7f10dcff9640 1 -- 172.21.15.76:0/3425493739 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5630057392d0 con 0x5630057e6970 2024-03-18T15:01:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.582+0000 7f10dcff9640 10 client.26393 handle_command_reply: tid=0 2024-03-18T15:01:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.582+0000 7f1102065740 1 client.26393 shutdown 2024-03-18T15:01:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.582+0000 7f1102065740 1 -- 172.21.15.76:0/3425493739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 msgr2=0x5630057da740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.582+0000 7f1102065740 1 --2- 172.21.15.76:0/3425493739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057da740 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009900 tx=0x7f10f002c1f0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.582+0000 7f1102065740 1 -- 172.21.15.76:0/3425493739 shutdown_connections 2024-03-18T15:01:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.582+0000 7f1102065740 1 --2- 172.21.15.76:0/3425493739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5630057d6820 0x56300584bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.582+0000 7f1102065740 1 --2- 172.21.15.76:0/3425493739 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5630057e6970 0x56300584f700 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.582+0000 7f1102065740 1 --2- 172.21.15.76:0/3425493739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5630057cda50 0x5630057dac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.582+0000 7f10dcff9640 0 client.26393 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.582+0000 7f1102065740 1 --2- 172.21.15.76:0/3425493739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630057cd130 0x5630057da740 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.582+0000 7f1102065740 1 -- 172.21.15.76:0/3425493739 >> 172.21.15.76:0/3425493739 conn(0x5630057c7a50 msgr2=0x5630057c8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.582+0000 7f1102065740 1 -- 172.21.15.76:0/3425493739 shutdown_connections 2024-03-18T15:01:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.582+0000 7f1102065740 1 -- 172.21.15.76:0/3425493739 wait complete. 2024-03-18T15:01:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.582+0000 7f1102065740 20 client.26393 trim_cache size 0 max 16384 2024-03-18T15:01:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:01:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:01:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:01:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:01:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.583+0000 7f10ff47e640 1 -- 172.21.15.76:0/6499741 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f10e00a2a10 msgr2=0x7f10e00a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.583+0000 7f10ff47e640 1 --2- 172.21.15.76:0/6499741 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f10e00a2a10 0x7f10e00a4ed0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f10e8009fd0 tx=0x7f10e8011b40 comp rx=0 tx=0).stop 2024-03-18T15:01:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.583+0000 7f10ff47e640 1 -- 172.21.15.76:0/6499741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f81012f0 msgr2=0x7f10f810e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.583+0000 7f10ff47e640 1 --2- 172.21.15.76:0/6499741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f81012f0 0x7f10f810e590 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f10ec00e9f0 tx=0x7f10ec00eeb0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.583+0000 7f10ff47e640 1 -- 172.21.15.76:0/6499741 shutdown_connections 2024-03-18T15:01:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.583+0000 7f10ff47e640 1 --2- 172.21.15.76:0/6499741 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f10e00a2a10 0x7f10e00a4ed0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.583+0000 7f10ff47e640 1 --2- 172.21.15.76:0/6499741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10f8105680 0x7f10f810ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.583+0000 7f10ff47e640 1 --2- 172.21.15.76:0/6499741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f10f81012f0 0x7f10f810e590 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.583+0000 7f10ff47e640 1 --2- 172.21.15.76:0/6499741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10f8100930 0x7f10f810e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.583+0000 7f10ff47e640 1 -- 172.21.15.76:0/6499741 >> 172.21.15.76:0/6499741 conn(0x7f10f8077530 msgr2=0x7f10f80fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.583+0000 7f10ff47e640 1 -- 172.21.15.76:0/6499741 shutdown_connections 2024-03-18T15:01:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:41.583+0000 7f10ff47e640 1 -- 172.21.15.76:0/6499741 wait complete. 2024-03-18T15:01:41.753 INFO:tasks.workunit.client.0.smithi076.stdout: 1 823512 9.46 MB/sec execute 342 sec latency 37.153 ms 2024-03-18T15:01:42.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:41 smithi076 ceph-mon[32205]: pgmap v265: 97 pgs: 97 active+clean; 240 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 194 op/s 2024-03-18T15:01:42.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:41 smithi174 ceph-mon[36787]: pgmap v265: 97 pgs: 97 active+clean; 240 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 194 op/s 2024-03-18T15:01:42.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:41 smithi077 ceph-mon[35865]: pgmap v265: 97 pgs: 97 active+clean; 240 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 194 op/s 2024-03-18T15:01:42.753 INFO:tasks.workunit.client.0.smithi076.stdout: 1 824607 9.45 MB/sec execute 343 sec latency 36.108 ms 2024-03-18T15:01:43.753 INFO:tasks.workunit.client.0.smithi076.stdout: 1 826924 9.46 MB/sec execute 344 sec latency 21.428 ms 2024-03-18T15:01:44.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:43 smithi076 ceph-mon[32205]: pgmap v266: 97 pgs: 97 active+clean; 239 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 330 op/s 2024-03-18T15:01:44.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:43 smithi174 ceph-mon[36787]: pgmap v266: 97 pgs: 97 active+clean; 239 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 330 op/s 2024-03-18T15:01:44.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:43 smithi077 ceph-mon[35865]: pgmap v266: 97 pgs: 97 active+clean; 239 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 330 op/s 2024-03-18T15:01:44.753 INFO:tasks.workunit.client.0.smithi076.stdout: 1 828141 9.45 MB/sec execute 345 sec latency 17.132 ms 2024-03-18T15:01:45.753 INFO:tasks.workunit.client.0.smithi076.stdout: 1 830237 9.45 MB/sec execute 346 sec latency 28.479 ms 2024-03-18T15:01:46.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:45 smithi076 ceph-mon[32205]: pgmap v267: 97 pgs: 97 active+clean; 239 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 233 op/s 2024-03-18T15:01:46.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:45 smithi174 ceph-mon[36787]: pgmap v267: 97 pgs: 97 active+clean; 239 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 233 op/s 2024-03-18T15:01:46.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:45 smithi077 ceph-mon[35865]: pgmap v267: 97 pgs: 97 active+clean; 239 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 233 op/s 2024-03-18T15:01:46.753 INFO:tasks.workunit.client.0.smithi076.stdout: 1 831395 9.45 MB/sec execute 347 sec latency 58.225 ms 2024-03-18T15:01:47.753 INFO:tasks.workunit.client.0.smithi076.stdout: 1 833365 9.44 MB/sec execute 348 sec latency 21.393 ms 2024-03-18T15:01:48.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:47 smithi076 ceph-mon[32205]: pgmap v268: 97 pgs: 97 active+clean; 239 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 233 op/s 2024-03-18T15:01:48.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:47 smithi174 ceph-mon[36787]: pgmap v268: 97 pgs: 97 active+clean; 239 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 233 op/s 2024-03-18T15:01:48.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:47 smithi077 ceph-mon[35865]: pgmap v268: 97 pgs: 97 active+clean; 239 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 233 op/s 2024-03-18T15:01:48.753 INFO:tasks.workunit.client.0.smithi076.stdout: 1 834911 9.45 MB/sec execute 349 sec latency 12.972 ms 2024-03-18T15:01:49.753 INFO:tasks.workunit.client.0.smithi076.stdout: 1 836953 9.44 MB/sec execute 350 sec latency 22.209 ms 2024-03-18T15:01:50.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:49 smithi076 ceph-mon[32205]: pgmap v269: 97 pgs: 97 active+clean; 245 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 335 op/s 2024-03-18T15:01:50.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:49 smithi174 ceph-mon[36787]: pgmap v269: 97 pgs: 97 active+clean; 245 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 335 op/s 2024-03-18T15:01:50.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:49 smithi077 ceph-mon[35865]: pgmap v269: 97 pgs: 97 active+clean; 245 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 335 op/s 2024-03-18T15:01:50.753 INFO:tasks.workunit.client.0.smithi076.stdout: 1 838500 9.45 MB/sec execute 351 sec latency 14.288 ms 2024-03-18T15:01:51.753 INFO:tasks.workunit.client.0.smithi076.stdout: 1 839999 9.44 MB/sec execute 352 sec latency 114.402 ms 2024-03-18T15:01:52.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:51 smithi076 ceph-mon[32205]: pgmap v270: 97 pgs: 97 active+clean; 245 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 236 op/s 2024-03-18T15:01:52.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:51 smithi174 ceph-mon[36787]: pgmap v270: 97 pgs: 97 active+clean; 245 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 236 op/s 2024-03-18T15:01:52.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:51 smithi077 ceph-mon[35865]: pgmap v270: 97 pgs: 97 active+clean; 245 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 236 op/s 2024-03-18T15:01:52.753 INFO:tasks.workunit.client.0.smithi076.stdout: 1 841836 9.45 MB/sec execute 353 sec latency 41.274 ms 2024-03-18T15:01:53.753 INFO:tasks.workunit.client.0.smithi076.stdout: 1 843152 9.44 MB/sec execute 354 sec latency 27.459 ms 2024-03-18T15:01:54.017 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:53 smithi174 ceph-mon[36787]: pgmap v271: 97 pgs: 97 active+clean; 252 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 336 op/s 2024-03-18T15:01:54.028 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:53 smithi077 ceph-mon[35865]: pgmap v271: 97 pgs: 97 active+clean; 252 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 336 op/s 2024-03-18T15:01:54.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:53 smithi076 ceph-mon[32205]: pgmap v271: 97 pgs: 97 active+clean; 252 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 336 op/s 2024-03-18T15:01:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.674+0000 7f778353b640 1 -- 172.21.15.76:0/274971835 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c1012f0 msgr2=0x7f777c10d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.674+0000 7f778353b640 1 --2- 172.21.15.76:0/274971835 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c1012f0 0x7f777c10d510 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7774009ca0 tx=0x7f777402c0f0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.674+0000 7f778353b640 1 -- 172.21.15.76:0/274971835 shutdown_connections 2024-03-18T15:01:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.674+0000 7f778353b640 1 --2- 172.21.15.76:0/274971835 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c1012f0 0x7f777c10d510 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.674+0000 7f778353b640 1 --2- 172.21.15.76:0/274971835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f777c100930 0x7f777c100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.674+0000 7f778353b640 1 --2- 172.21.15.76:0/274971835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f777c105680 0x7f777c105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.674+0000 7f778353b640 1 -- 172.21.15.76:0/274971835 >> 172.21.15.76:0/274971835 conn(0x7f777c077530 msgr2=0x7f777c077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.674+0000 7f778353b640 1 -- 172.21.15.76:0/274971835 shutdown_connections 2024-03-18T15:01:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.674+0000 7f778353b640 1 -- 172.21.15.76:0/274971835 wait complete. 2024-03-18T15:01:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f778353b640 1 Processor -- start 2024-03-18T15:01:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f778353b640 1 -- start start 2024-03-18T15:01:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f778353b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c100930 0x7f777c198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f778353b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f777c1012f0 0x7f777c199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f778353b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f777c105680 0x7f777c1a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f778353b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f777c110ba0 con 0x7f777c100930 2024-03-18T15:01:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f778353b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f777c110a20 con 0x7f777c105680 2024-03-18T15:01:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f778353b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f777c110d20 con 0x7f777c1012f0 2024-03-18T15:01:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f77812d7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c100930 0x7f777c198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f77812d7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c100930 0x7f777c198ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39198/0 (socket says 172.21.15.76:39198) 2024-03-18T15:01:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f77812d7640 1 -- 172.21.15.76:0/2532498732 learned_addr learned my addr 172.21.15.76:0/2532498732 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f7780ad6640 1 --2- 172.21.15.76:0/2532498732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f777c1012f0 0x7f777c199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f7781ad8640 1 --2- 172.21.15.76:0/2532498732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f777c105680 0x7f777c1a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f77812d7640 1 -- 172.21.15.76:0/2532498732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f777c1012f0 msgr2=0x7f777c199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f77812d7640 1 --2- 172.21.15.76:0/2532498732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f777c1012f0 0x7f777c199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f77812d7640 1 -- 172.21.15.76:0/2532498732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f777c105680 msgr2=0x7f777c1a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f77812d7640 1 --2- 172.21.15.76:0/2532498732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f777c105680 0x7f777c1a3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.675+0000 7f77812d7640 1 -- 172.21.15.76:0/2532498732 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777c1a44a0 con 0x7f777c100930 2024-03-18T15:01:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.676+0000 7f77812d7640 1 --2- 172.21.15.76:0/2532498732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c100930 0x7f777c198ee0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f777800e9f0 tx=0x7f777800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.676+0000 7f77727fc640 1 -- 172.21.15.76:0/2532498732 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777800cd80 con 0x7f777c100930 2024-03-18T15:01:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.676+0000 7f77727fc640 1 -- 172.21.15.76:0/2532498732 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7778005500 con 0x7f777c100930 2024-03-18T15:01:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.676+0000 7f778353b640 1 -- 172.21.15.76:0/2532498732 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f777c1a4670 con 0x7f777c100930 2024-03-18T15:01:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.676+0000 7f77727fc640 1 -- 172.21.15.76:0/2532498732 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7778010690 con 0x7f777c100930 2024-03-18T15:01:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.676+0000 7f778353b640 1 -- 172.21.15.76:0/2532498732 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f777c1a4b90 con 0x7f777c100930 2024-03-18T15:01:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.678+0000 7f77727fc640 1 -- 172.21.15.76:0/2532498732 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f77780046c0 con 0x7f777c100930 2024-03-18T15:01:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.679+0000 7f77861ff740 1 Processor -- start 2024-03-18T15:01:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.679+0000 7f77861ff740 1 -- start start 2024-03-18T15:01:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.679+0000 7f77727fc640 1 --2- 172.21.15.76:0/2532498732 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f77680a2a90 0x7f77680a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.679+0000 7f77861ff740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c505130 0x55ce0c505510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.679+0000 7f77861ff740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505a50 0x55ce0c50e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.679+0000 7f77861ff740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c50e820 0x55ce0c510c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.679+0000 7f77861ff740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ce0c469a40 con 0x55ce0c50e820 2024-03-18T15:01:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.679+0000 7f77861ff740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ce0c412920 con 0x55ce0c505a50 2024-03-18T15:01:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.679+0000 7f77861ff740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ce0c465ae0 con 0x55ce0c505130 2024-03-18T15:01:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.679+0000 7f77812d7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c505130 0x55ce0c505510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.679+0000 7f7780ad6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505a50 0x55ce0c50e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.679+0000 7f7780ad6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505a50 0x55ce0c50e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:34780/0 (socket says 172.21.15.76:34780) 2024-03-18T15:01:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.679+0000 7f7780ad6640 1 -- 172.21.15.76:0/3025517964 learned_addr learned my addr 172.21.15.76:0/3025517964 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.679+0000 7f7781ad8640 1 --2- 172.21.15.76:0/3025517964 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c50e820 0x55ce0c510c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.679+0000 7f77727fc640 1 -- 172.21.15.76:0/2532498732 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f77780cf120 con 0x7f777c100930 2024-03-18T15:01:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.680+0000 7f7780ad6640 1 -- 172.21.15.76:0/3025517964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c505130 msgr2=0x55ce0c505510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.680+0000 7f7780ad6640 1 --2- 172.21.15.76:0/3025517964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c505130 0x55ce0c505510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.680+0000 7f7780ad6640 1 -- 172.21.15.76:0/3025517964 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c50e820 msgr2=0x55ce0c510c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.680+0000 7f7780ad6640 1 --2- 172.21.15.76:0/3025517964 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c50e820 0x55ce0c510c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.680+0000 7f7781ad8640 1 --2- 172.21.15.76:0/2532498732 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f77680a2a90 0x7f77680a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.680+0000 7f7780ad6640 1 -- 172.21.15.76:0/3025517964 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce0c42a460 con 0x55ce0c505a50 2024-03-18T15:01:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.680+0000 7f77812d7640 1 --2- 172.21.15.76:0/3025517964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c505130 0x55ce0c505510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:01:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.680+0000 7f7780ad6640 1 --2- 172.21.15.76:0/3025517964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505a50 0x55ce0c50e2e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f776c009c90 tx=0x7f776c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71553314dbc63f91 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.680+0000 7f7781ad8640 1 --2- 172.21.15.76:0/2532498732 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f77680a2a90 0x7f77680a4f50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7774004af0 tx=0x7f7774005c50 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.680+0000 7f77677fe640 1 -- 172.21.15.76:0/3025517964 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c01e070 con 0x55ce0c505a50 2024-03-18T15:01:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.680+0000 7f77677fe640 1 -- 172.21.15.76:0/3025517964 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f776c019b10 con 0x55ce0c505a50 2024-03-18T15:01:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.680+0000 7f77677fe640 1 -- 172.21.15.76:0/3025517964 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c00ea70 con 0x55ce0c505a50 2024-03-18T15:01:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.681+0000 7f77861ff740 1 -- 172.21.15.76:0/3025517964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505a50 msgr2=0x55ce0c50e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.681+0000 7f77861ff740 1 --2- 172.21.15.76:0/3025517964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505a50 0x55ce0c50e2e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f776c009c90 tx=0x7f776c011810 comp rx=0 tx=0).stop 2024-03-18T15:01:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.681+0000 7f77861ff740 1 -- 172.21.15.76:0/3025517964 shutdown_connections 2024-03-18T15:01:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.681+0000 7f77861ff740 1 --2- 172.21.15.76:0/3025517964 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c50e820 0x55ce0c510c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.681+0000 7f77861ff740 1 --2- 172.21.15.76:0/3025517964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505a50 0x55ce0c50e2e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.681+0000 7f77861ff740 1 --2- 172.21.15.76:0/3025517964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c505130 0x55ce0c505510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.681+0000 7f77861ff740 1 -- 172.21.15.76:0/3025517964 >> 172.21.15.76:0/3025517964 conn(0x55ce0c4ffa50 msgr2=0x55ce0c4ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.681+0000 7f77861ff740 1 -- 172.21.15.76:0/3025517964 shutdown_connections 2024-03-18T15:01:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.681+0000 7f77861ff740 1 -- 172.21.15.76:0/3025517964 wait complete. 2024-03-18T15:01:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.681+0000 7f77861ff740 1 -- start start 2024-03-18T15:01:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.682+0000 7f77861ff740 1 Processor -- start 2024-03-18T15:01:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.682+0000 7f77861ff740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505130 0x55ce0c50b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.682+0000 7f77861ff740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 0x55ce0c50bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.682+0000 7f77861ff740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c50e820 0x55ce0c50c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.682+0000 7f77861ff740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ce0c511450 con 0x55ce0c505a50 2024-03-18T15:01:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.682+0000 7f77861ff740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ce0c425d70 con 0x55ce0c505130 2024-03-18T15:01:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.682+0000 7f77861ff740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ce0c5115d0 con 0x55ce0c50e820 2024-03-18T15:01:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.682+0000 7f77861ff740 10 client.0 _ll_register_callbacks cb 0x55ce0c17da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:01:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.682+0000 7f77812d7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505130 0x55ce0c50b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.682+0000 7f77812d7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505130 0x55ce0c50b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:34786/0 (socket says 172.21.15.76:34786) 2024-03-18T15:01:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.682+0000 7f7780ad6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 0x55ce0c50bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.682+0000 7f77812d7640 1 -- 172.21.15.76:0/60052243 learned_addr learned my addr 172.21.15.76:0/60052243 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.682+0000 7f7781ad8640 1 --2- 172.21.15.76:0/60052243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c50e820 0x55ce0c50c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f7780ad6640 1 -- 172.21.15.76:0/60052243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c50e820 msgr2=0x55ce0c50c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f7780ad6640 1 --2- 172.21.15.76:0/60052243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c50e820 0x55ce0c50c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f7780ad6640 1 -- 172.21.15.76:0/60052243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505130 msgr2=0x55ce0c50b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f7780ad6640 1 --2- 172.21.15.76:0/60052243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505130 0x55ce0c50b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f7780ad6640 1 -- 172.21.15.76:0/60052243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce0c17e210 con 0x55ce0c505a50 2024-03-18T15:01:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f7780ad6640 1 --2- 172.21.15.76:0/60052243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 0x55ce0c50bd90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f776c009c90 tx=0x7f776c019a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2836713b7be809e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f7764ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:01:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f7764ff9640 1 -- 172.21.15.76:0/60052243 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c01e050 con 0x55ce0c505a50 2024-03-18T15:01:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f7764ff9640 1 -- 172.21.15.76:0/60052243 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f776c005450 con 0x55ce0c505a50 2024-03-18T15:01:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f77861ff740 1 -- 172.21.15.76:0/60052243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f776c019b10 con 0x55ce0c505a50 2024-03-18T15:01:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f7764ff9640 1 -- 172.21.15.76:0/60052243 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f776c005650 con 0x55ce0c505a50 2024-03-18T15:01:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f77861ff740 10 client.16542 fetch_fsmap learned FSMap version 41 2024-03-18T15:01:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f77861ff740 1 -- 172.21.15.76:0/60052243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55ce0c24a620 con 0x55ce0c505a50 2024-03-18T15:01:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f7764ff9640 1 -- 172.21.15.76:0/60052243 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f776c019b10 con 0x55ce0c505a50 2024-03-18T15:01:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f7764ff9640 1 -- 172.21.15.76:0/60052243 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f776c00ec60 con 0x55ce0c505a50 2024-03-18T15:01:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f77861ff740 10 client.16542 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:01:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f77861ff740 10 client.16542 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:01:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f77861ff740 20 client.16542 populate_metadata read hostname 'smithi076' 2024-03-18T15:01:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f77861ff740 1 --2- 172.21.15.76:0/60052243 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ce0c522ca0 0x55ce0c525090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.683+0000 7f77861ff740 4 client.16542 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:01:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.684+0000 7f77861ff740 1 -- 172.21.15.76:0/60052243 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce0c476f80 con 0x55ce0c522ca0 2024-03-18T15:01:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.684+0000 7f77812d7640 1 --2- 172.21.15.76:0/60052243 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ce0c522ca0 0x55ce0c525090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.684+0000 7f7764ff9640 10 client.16542 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.684+0000 7f77812d7640 1 --2- 172.21.15.76:0/60052243 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ce0c522ca0 0x55ce0c525090 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8c862575441ac77 server_cookie=945cbc38d721f734 in_seq=0 out_seq=0 2024-03-18T15:01:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.685+0000 7f7764ff9640 1 -- 172.21.15.76:0/60052243 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ce0c476f80 con 0x55ce0c522ca0 2024-03-18T15:01:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.685+0000 7f7764ff9640 10 client.16542 handle_command_reply: tid=0 2024-03-18T15:01:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.685+0000 7f77861ff740 1 client.16542 shutdown 2024-03-18T15:01:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.685+0000 7f77861ff740 1 -- 172.21.15.76:0/60052243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 msgr2=0x55ce0c50bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.685+0000 7f77861ff740 1 --2- 172.21.15.76:0/60052243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 0x55ce0c50bd90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f776c009c90 tx=0x7f776c019a10 comp rx=0 tx=0).stop 2024-03-18T15:01:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.685+0000 7f77861ff740 1 -- 172.21.15.76:0/60052243 shutdown_connections 2024-03-18T15:01:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.685+0000 7f77861ff740 1 --2- 172.21.15.76:0/60052243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c50e820 0x55ce0c50c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.685+0000 7f77861ff740 1 --2- 172.21.15.76:0/60052243 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ce0c522ca0 0x55ce0c525090 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.685+0000 7f77861ff740 1 --2- 172.21.15.76:0/60052243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 0x55ce0c50bd90 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.685+0000 7f77861ff740 1 --2- 172.21.15.76:0/60052243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505130 0x55ce0c50b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.685+0000 7f7764ff9640 0 client.16542 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.685+0000 7f77861ff740 1 -- 172.21.15.76:0/60052243 >> 172.21.15.76:0/60052243 conn(0x55ce0c4ffa50 msgr2=0x55ce0c5017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.686+0000 7f77861ff740 1 -- 172.21.15.76:0/60052243 shutdown_connections 2024-03-18T15:01:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.686+0000 7f77861ff740 1 -- 172.21.15.76:0/60052243 wait complete. 2024-03-18T15:01:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.686+0000 7f77861ff740 20 client.16542 trim_cache size 0 max 16384 2024-03-18T15:01:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.702+0000 7f77861ff740 1 Processor -- start 2024-03-18T15:01:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.702+0000 7f77861ff740 1 -- start start 2024-03-18T15:01:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.702+0000 7f77861ff740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c505130 0x55ce0c50df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.702+0000 7f77861ff740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 0x55ce0c51e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.702+0000 7f77861ff740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c50e820 0x55ce0c52a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.702+0000 7f77861ff740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f777c1adab0 con 0x55ce0c505a50 2024-03-18T15:01:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.702+0000 7f77861ff740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f777c1ab380 con 0x55ce0c50e820 2024-03-18T15:01:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.702+0000 7f77861ff740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f777c1a91f0 con 0x55ce0c505130 2024-03-18T15:01:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.702+0000 7f7780ad6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 0x55ce0c51e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.702+0000 7f7780ad6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 0x55ce0c51e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39236/0 (socket says 172.21.15.76:39236) 2024-03-18T15:01:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.702+0000 7f7780ad6640 1 -- 172.21.15.76:0/4003975117 learned_addr learned my addr 172.21.15.76:0/4003975117 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.702+0000 7f7781ad8640 1 --2- 172.21.15.76:0/4003975117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c50e820 0x55ce0c52a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.702+0000 7f77812d7640 1 --2- 172.21.15.76:0/4003975117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c505130 0x55ce0c50df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.703+0000 7f7780ad6640 1 -- 172.21.15.76:0/4003975117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c505130 msgr2=0x55ce0c50df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.703+0000 7f7780ad6640 1 --2- 172.21.15.76:0/4003975117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c505130 0x55ce0c50df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.703+0000 7f7780ad6640 1 -- 172.21.15.76:0/4003975117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c50e820 msgr2=0x55ce0c52a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.703+0000 7f7780ad6640 1 --2- 172.21.15.76:0/4003975117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c50e820 0x55ce0c52a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.703+0000 7f7780ad6640 1 -- 172.21.15.76:0/4003975117 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce0c509da0 con 0x55ce0c505a50 2024-03-18T15:01:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.703+0000 7f7780ad6640 1 --2- 172.21.15.76:0/4003975117 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 0x55ce0c51e640 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f776c012fd0 tx=0x7f776c001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74a2b564f9dcde14 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.703+0000 7f7766ffd640 1 -- 172.21.15.76:0/4003975117 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c01e040 con 0x55ce0c505a50 2024-03-18T15:01:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.703+0000 7f7766ffd640 1 -- 172.21.15.76:0/4003975117 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f776c01f040 con 0x55ce0c505a50 2024-03-18T15:01:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.703+0000 7f7766ffd640 1 -- 172.21.15.76:0/4003975117 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c005250 con 0x55ce0c505a50 2024-03-18T15:01:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.704+0000 7f77861ff740 1 -- 172.21.15.76:0/4003975117 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 msgr2=0x55ce0c51e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.704+0000 7f77861ff740 1 --2- 172.21.15.76:0/4003975117 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 0x55ce0c51e640 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f776c012fd0 tx=0x7f776c001e50 comp rx=0 tx=0).stop 2024-03-18T15:01:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.704+0000 7f77861ff740 1 -- 172.21.15.76:0/4003975117 shutdown_connections 2024-03-18T15:01:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.704+0000 7f77861ff740 1 --2- 172.21.15.76:0/4003975117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c50e820 0x55ce0c52a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.704+0000 7f77861ff740 1 --2- 172.21.15.76:0/4003975117 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 0x55ce0c51e640 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.704+0000 7f77861ff740 1 --2- 172.21.15.76:0/4003975117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c505130 0x55ce0c50df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.704+0000 7f77861ff740 1 -- 172.21.15.76:0/4003975117 >> 172.21.15.76:0/4003975117 conn(0x55ce0c4ffa50 msgr2=0x55ce0c50b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.704+0000 7f77861ff740 1 -- 172.21.15.76:0/4003975117 shutdown_connections 2024-03-18T15:01:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.704+0000 7f77861ff740 1 -- 172.21.15.76:0/4003975117 wait complete. 2024-03-18T15:01:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.704+0000 7f77861ff740 1 -- start start 2024-03-18T15:01:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.704+0000 7f77861ff740 1 Processor -- start 2024-03-18T15:01:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f77861ff740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505130 0x55ce0c5165b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f77861ff740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 0x55ce0c516af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f77861ff740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c50e820 0x55ce0c517030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f77861ff740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f777c1a9840 con 0x55ce0c505a50 2024-03-18T15:01:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f77861ff740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f777c1aacb0 con 0x55ce0c505130 2024-03-18T15:01:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f77861ff740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f777c1a9f10 con 0x55ce0c50e820 2024-03-18T15:01:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f77861ff740 10 client.0 _ll_register_callbacks cb 0x7f776c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:01:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f7780ad6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 0x55ce0c516af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f7780ad6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 0x55ce0c516af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39244/0 (socket says 172.21.15.76:39244) 2024-03-18T15:01:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f7780ad6640 1 -- 172.21.15.76:0/2308254802 learned_addr learned my addr 172.21.15.76:0/2308254802 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:01:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f77812d7640 1 --2- 172.21.15.76:0/2308254802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505130 0x55ce0c5165b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f7781ad8640 1 --2- 172.21.15.76:0/2308254802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c50e820 0x55ce0c517030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f7780ad6640 1 -- 172.21.15.76:0/2308254802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c50e820 msgr2=0x55ce0c517030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f7780ad6640 1 --2- 172.21.15.76:0/2308254802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c50e820 0x55ce0c517030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f7780ad6640 1 -- 172.21.15.76:0/2308254802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505130 msgr2=0x55ce0c5165b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:01:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f7780ad6640 1 --2- 172.21.15.76:0/2308254802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505130 0x55ce0c5165b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f7780ad6640 1 -- 172.21.15.76:0/2308254802 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce0c587e00 con 0x55ce0c505a50 2024-03-18T15:01:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f7780ad6640 1 --2- 172.21.15.76:0/2308254802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 0x55ce0c516af0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f776c0058c0 tx=0x7f776c011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb94264028692fcf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:01:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f7764ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:01:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.705+0000 7f7764ff9640 1 -- 172.21.15.76:0/2308254802 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c01e040 con 0x55ce0c505a50 2024-03-18T15:01:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.706+0000 7f7764ff9640 1 -- 172.21.15.76:0/2308254802 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f776c019a10 con 0x55ce0c505a50 2024-03-18T15:01:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.706+0000 7f77861ff740 1 -- 172.21.15.76:0/2308254802 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f776c01f040 con 0x55ce0c505a50 2024-03-18T15:01:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.706+0000 7f7764ff9640 1 -- 172.21.15.76:0/2308254802 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f776c004900 con 0x55ce0c505a50 2024-03-18T15:01:54.711 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.706+0000 7f77861ff740 10 client.16548 fetch_fsmap learned FSMap version 41 2024-03-18T15:01:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.706+0000 7f77861ff740 1 -- 172.21.15.76:0/2308254802 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55ce0c5890f0 con 0x55ce0c505a50 2024-03-18T15:01:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.706+0000 7f7764ff9640 1 -- 172.21.15.76:0/2308254802 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f776c01f040 con 0x55ce0c505a50 2024-03-18T15:01:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.706+0000 7f7764ff9640 1 -- 172.21.15.76:0/2308254802 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f776c045030 con 0x55ce0c505a50 2024-03-18T15:01:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.706+0000 7f77861ff740 10 client.16548 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:01:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.706+0000 7f77861ff740 10 client.16548 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:01:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.706+0000 7f77861ff740 20 client.16548 populate_metadata read hostname 'smithi076' 2024-03-18T15:01:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.706+0000 7f77861ff740 1 --2- 172.21.15.76:0/2308254802 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ce0c5897d0 0x55ce0c58bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:01:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.706+0000 7f77861ff740 4 client.16548 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:01:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.706+0000 7f77861ff740 1 -- 172.21.15.76:0/2308254802 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ce0c4712d0 con 0x55ce0c5897d0 2024-03-18T15:01:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.706+0000 7f77812d7640 1 --2- 172.21.15.76:0/2308254802 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ce0c5897d0 0x55ce0c58bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:01:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.707+0000 7f7764ff9640 10 client.16548 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.707+0000 7f77812d7640 1 --2- 172.21.15.76:0/2308254802 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ce0c5897d0 0x55ce0c58bbf0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1788bfa2c79827e9 server_cookie=806e275ffdff8f62 in_seq=0 out_seq=0 2024-03-18T15:01:54.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 845285 9.45 MB/sec execute 355 sec latency 39.681 ms 2024-03-18T15:01:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.789+0000 7f7764ff9640 1 -- 172.21.15.76:0/2308254802 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ce0c4712d0 con 0x55ce0c5897d0 2024-03-18T15:01:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.789+0000 7f7764ff9640 10 client.16548 handle_command_reply: tid=0 2024-03-18T15:01:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.789+0000 7f77861ff740 1 client.16548 shutdown 2024-03-18T15:01:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.789+0000 7f77861ff740 1 -- 172.21.15.76:0/2308254802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 msgr2=0x55ce0c516af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.789+0000 7f77861ff740 1 --2- 172.21.15.76:0/2308254802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 0x55ce0c516af0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f776c0058c0 tx=0x7f776c011e30 comp rx=0 tx=0).stop 2024-03-18T15:01:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.789+0000 7f77861ff740 1 -- 172.21.15.76:0/2308254802 shutdown_connections 2024-03-18T15:01:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.789+0000 7f77861ff740 1 --2- 172.21.15.76:0/2308254802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce0c50e820 0x55ce0c517030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.789+0000 7f77861ff740 1 --2- 172.21.15.76:0/2308254802 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55ce0c5897d0 0x55ce0c58bbf0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.789+0000 7f77861ff740 1 --2- 172.21.15.76:0/2308254802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce0c505a50 0x55ce0c516af0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.789+0000 7f77861ff740 1 --2- 172.21.15.76:0/2308254802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce0c505130 0x55ce0c5165b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.789+0000 7f77861ff740 1 -- 172.21.15.76:0/2308254802 >> 172.21.15.76:0/2308254802 conn(0x55ce0c4ffa50 msgr2=0x55ce0c4febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.789+0000 7f7764ff9640 0 client.16548 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:01:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.789+0000 7f77861ff740 1 -- 172.21.15.76:0/2308254802 shutdown_connections 2024-03-18T15:01:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.789+0000 7f77861ff740 1 -- 172.21.15.76:0/2308254802 wait complete. 2024-03-18T15:01:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.789+0000 7f77861ff740 20 client.16548 trim_cache size 0 max 16384 2024-03-18T15:01:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:01:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:01:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:01:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:01:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.790+0000 7f778353b640 1 -- 172.21.15.76:0/2532498732 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f77680a2a90 msgr2=0x7f77680a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.790+0000 7f778353b640 1 --2- 172.21.15.76:0/2532498732 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f77680a2a90 0x7f77680a4f50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7774004af0 tx=0x7f7774005c50 comp rx=0 tx=0).stop 2024-03-18T15:01:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.790+0000 7f778353b640 1 -- 172.21.15.76:0/2532498732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c100930 msgr2=0x7f777c198ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:01:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.790+0000 7f778353b640 1 --2- 172.21.15.76:0/2532498732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c100930 0x7f777c198ee0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f777800e9f0 tx=0x7f777800edf0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.790+0000 7f778353b640 1 -- 172.21.15.76:0/2532498732 shutdown_connections 2024-03-18T15:01:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.790+0000 7f778353b640 1 --2- 172.21.15.76:0/2532498732 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f77680a2a90 0x7f77680a4f50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.790+0000 7f778353b640 1 --2- 172.21.15.76:0/2532498732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f777c1012f0 0x7f777c199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.790+0000 7f778353b640 1 --2- 172.21.15.76:0/2532498732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f777c105680 0x7f777c1a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.790+0000 7f778353b640 1 --2- 172.21.15.76:0/2532498732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c100930 0x7f777c198ee0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:01:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.790+0000 7f778353b640 1 -- 172.21.15.76:0/2532498732 >> 172.21.15.76:0/2532498732 conn(0x7f777c077530 msgr2=0x7f777c0fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:01:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.790+0000 7f778353b640 1 -- 172.21.15.76:0/2532498732 shutdown_connections 2024-03-18T15:01:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:01:54.790+0000 7f778353b640 1 -- 172.21.15.76:0/2532498732 wait complete. 2024-03-18T15:01:55.754 INFO:tasks.workunit.client.0.smithi076.stdout: 1 846550 9.44 MB/sec execute 356 sec latency 42.024 ms 2024-03-18T15:01:56.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:55 smithi076 ceph-mon[32205]: pgmap v272: 97 pgs: 97 active+clean; 252 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 201 op/s 2024-03-18T15:01:56.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:55 smithi174 ceph-mon[36787]: pgmap v272: 97 pgs: 97 active+clean; 252 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 201 op/s 2024-03-18T15:01:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:55 smithi077 ceph-mon[35865]: pgmap v272: 97 pgs: 97 active+clean; 252 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 201 op/s 2024-03-18T15:01:56.754 INFO:tasks.workunit.client.0.smithi076.stdout: 1 848774 9.45 MB/sec execute 357 sec latency 82.029 ms 2024-03-18T15:01:57.754 INFO:tasks.workunit.client.0.smithi076.stdout: 1 850011 9.44 MB/sec execute 358 sec latency 31.713 ms 2024-03-18T15:01:58.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:57 smithi076 ceph-mon[32205]: pgmap v273: 97 pgs: 97 active+clean; 252 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 201 op/s 2024-03-18T15:01:58.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:57 smithi174 ceph-mon[36787]: pgmap v273: 97 pgs: 97 active+clean; 252 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 201 op/s 2024-03-18T15:01:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:57 smithi077 ceph-mon[35865]: pgmap v273: 97 pgs: 97 active+clean; 252 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 201 op/s 2024-03-18T15:01:58.754 INFO:tasks.workunit.client.0.smithi076.stdout: 1 852332 9.45 MB/sec execute 359 sec latency 16.917 ms 2024-03-18T15:01:59.754 INFO:tasks.workunit.client.0.smithi076.stdout: 1 853617 9.44 MB/sec execute 360 sec latency 21.269 ms 2024-03-18T15:02:00.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:01:59 smithi076 ceph-mon[32205]: pgmap v274: 97 pgs: 97 active+clean; 246 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 343 op/s 2024-03-18T15:02:00.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:01:59 smithi174 ceph-mon[36787]: pgmap v274: 97 pgs: 97 active+clean; 246 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 343 op/s 2024-03-18T15:02:00.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:01:59 smithi077 ceph-mon[35865]: pgmap v274: 97 pgs: 97 active+clean; 246 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 343 op/s 2024-03-18T15:02:00.754 INFO:tasks.workunit.client.0.smithi076.stdout: 1 855369 9.44 MB/sec execute 361 sec latency 26.978 ms 2024-03-18T15:02:01.754 INFO:tasks.workunit.client.0.smithi076.stdout: 1 856364 9.43 MB/sec execute 362 sec latency 46.375 ms 2024-03-18T15:02:02.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:01 smithi076 ceph-mon[32205]: pgmap v275: 97 pgs: 97 active+clean; 246 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 242 op/s 2024-03-18T15:02:02.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:01 smithi174 ceph-mon[36787]: pgmap v275: 97 pgs: 97 active+clean; 246 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 242 op/s 2024-03-18T15:02:02.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:01 smithi077 ceph-mon[35865]: pgmap v275: 97 pgs: 97 active+clean; 246 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 242 op/s 2024-03-18T15:02:02.754 INFO:tasks.workunit.client.0.smithi076.stdout: 1 858211 9.42 MB/sec execute 363 sec latency 32.979 ms 2024-03-18T15:02:03.754 INFO:tasks.workunit.client.0.smithi076.stdout: 1 859206 9.42 MB/sec execute 364 sec latency 44.235 ms 2024-03-18T15:02:04.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:03 smithi076 ceph-mon[32205]: osdmap e100: 12 total, 12 up, 12 in 2024-03-18T15:02:04.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:03 smithi076 ceph-mon[32205]: pgmap v277: 97 pgs: 17 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 72 active+clean; 249 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 318 op/s 2024-03-18T15:02:04.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:03 smithi174 ceph-mon[36787]: osdmap e100: 12 total, 12 up, 12 in 2024-03-18T15:02:04.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:03 smithi174 ceph-mon[36787]: pgmap v277: 97 pgs: 17 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 72 active+clean; 249 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 318 op/s 2024-03-18T15:02:04.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:03 smithi077 ceph-mon[35865]: osdmap e100: 12 total, 12 up, 12 in 2024-03-18T15:02:04.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:03 smithi077 ceph-mon[35865]: pgmap v277: 97 pgs: 17 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 72 active+clean; 249 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 318 op/s 2024-03-18T15:02:04.754 INFO:tasks.workunit.client.0.smithi076.stdout: 1 860261 9.40 MB/sec execute 365 sec latency 16.661 ms 2024-03-18T15:02:05.754 INFO:tasks.workunit.client.0.smithi076.stdout: 1 862713 9.42 MB/sec execute 366 sec latency 20.215 ms 2024-03-18T15:02:06.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:05 smithi076 ceph-mon[32205]: pgmap v278: 97 pgs: 17 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 72 active+clean; 249 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 318 op/s 2024-03-18T15:02:06.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:05 smithi174 ceph-mon[36787]: pgmap v278: 97 pgs: 17 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 72 active+clean; 249 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 318 op/s 2024-03-18T15:02:06.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:05 smithi077 ceph-mon[35865]: pgmap v278: 97 pgs: 17 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 72 active+clean; 249 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 318 op/s 2024-03-18T15:02:06.754 INFO:tasks.workunit.client.0.smithi076.stdout: 1 863725 9.40 MB/sec execute 367 sec latency 35.466 ms 2024-03-18T15:02:07.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 866254 9.42 MB/sec execute 368 sec latency 24.179 ms 2024-03-18T15:02:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.887+0000 7f3f1eade640 1 -- 172.21.15.76:0/3761777138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f1810bf60 msgr2=0x7f3f1810e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.887+0000 7f3f1eade640 1 --2- 172.21.15.76:0/3761777138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f1810bf60 0x7f3f1810e3f0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f3f00009900 tx=0x7f3f0002bc30 comp rx=0 tx=0).stop 2024-03-18T15:02:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.887+0000 7f3f1eade640 1 -- 172.21.15.76:0/3761777138 shutdown_connections 2024-03-18T15:02:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.887+0000 7f3f1eade640 1 --2- 172.21.15.76:0/3761777138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f18103400 0x7f3f1810b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.887+0000 7f3f1eade640 1 --2- 172.21.15.76:0/3761777138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f1810bf60 0x7f3f1810e3f0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.887+0000 7f3f1eade640 1 --2- 172.21.15.76:0/3761777138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f18102a50 0x7f3f18102e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.887+0000 7f3f1eade640 1 -- 172.21.15.76:0/3761777138 >> 172.21.15.76:0/3761777138 conn(0x7f3f18077800 msgr2=0x7f3f18077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.887+0000 7f3f1eade640 1 -- 172.21.15.76:0/3761777138 shutdown_connections 2024-03-18T15:02:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.887+0000 7f3f1eade640 1 -- 172.21.15.76:0/3761777138 wait complete. 2024-03-18T15:02:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f1eade640 1 Processor -- start 2024-03-18T15:02:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f1eade640 1 -- start start 2024-03-18T15:02:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f1eade640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f18102a50 0x7f3f181981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f1eade640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f1810bf60 0x7f3f181986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f1eade640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f1819ca60 0x7f3f1819cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f1eade640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3f18111dc0 con 0x7f3f1810bf60 2024-03-18T15:02:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f1eade640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3f18111c40 con 0x7f3f1819ca60 2024-03-18T15:02:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f1eade640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3f18111f40 con 0x7f3f18102a50 2024-03-18T15:02:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f17fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f1810bf60 0x7f3f181986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f17fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f1810bf60 0x7f3f181986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45910/0 (socket says 172.21.15.76:45910) 2024-03-18T15:02:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f1d07b640 1 --2- 172.21.15.76:0/108061688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f1819ca60 0x7f3f1819cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f17fff640 1 -- 172.21.15.76:0/108061688 learned_addr learned my addr 172.21.15.76:0/108061688 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f1c87a640 1 --2- 172.21.15.76:0/108061688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f18102a50 0x7f3f181981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f17fff640 1 -- 172.21.15.76:0/108061688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f18102a50 msgr2=0x7f3f181981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f17fff640 1 --2- 172.21.15.76:0/108061688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f18102a50 0x7f3f181981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f17fff640 1 -- 172.21.15.76:0/108061688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f1819ca60 msgr2=0x7f3f1819cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f17fff640 1 --2- 172.21.15.76:0/108061688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f1819ca60 0x7f3f1819cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f17fff640 1 -- 172.21.15.76:0/108061688 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f1819d550 con 0x7f3f1810bf60 2024-03-18T15:02:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f1d07b640 1 --2- 172.21.15.76:0/108061688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f1819ca60 0x7f3f1819cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:02:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.888+0000 7f3f17fff640 1 --2- 172.21.15.76:0/108061688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f1810bf60 0x7f3f181986e0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800e9c0 tx=0x7f3f0800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.889+0000 7f3f15ffb640 1 -- 172.21.15.76:0/108061688 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0800ccc0 con 0x7f3f1810bf60 2024-03-18T15:02:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.889+0000 7f3f1eade640 1 -- 172.21.15.76:0/108061688 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f1819d840 con 0x7f3f1810bf60 2024-03-18T15:02:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.889+0000 7f3f15ffb640 1 -- 172.21.15.76:0/108061688 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f08005500 con 0x7f3f1810bf60 2024-03-18T15:02:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.889+0000 7f3f15ffb640 1 -- 172.21.15.76:0/108061688 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f08010690 con 0x7f3f1810bf60 2024-03-18T15:02:07.895 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.889+0000 7f3f1eade640 1 -- 172.21.15.76:0/108061688 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f181a5030 con 0x7f3f1810bf60 2024-03-18T15:02:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.891+0000 7f3f15ffb640 1 -- 172.21.15.76:0/108061688 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f3f08004e80 con 0x7f3f1810bf60 2024-03-18T15:02:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.891+0000 7f3f2179b740 1 Processor -- start 2024-03-18T15:02:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.891+0000 7f3f2179b740 1 -- start start 2024-03-18T15:02:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.891+0000 7f3f15ffb640 1 --2- 172.21.15.76:0/108061688 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3f040a2a10 0x7f3f040a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.891+0000 7f3f2179b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062baaf130 0x56062baaf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f2179b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baafa50 0x56062bab82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f2179b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062bab8820 0x56062babac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f2179b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56062ba13a40 con 0x56062baaf130 2024-03-18T15:02:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f2179b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56062b9bc920 con 0x56062baafa50 2024-03-18T15:02:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f2179b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56062ba0fae0 con 0x56062bab8820 2024-03-18T15:02:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f1d07b640 1 --2- 172.21.15.76:0/108061688 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3f040a2a10 0x7f3f040a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f1c87a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062baaf130 0x56062baaf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f1c87a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062baaf130 0x56062baaf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45920/0 (socket says 172.21.15.76:45920) 2024-03-18T15:02:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f17fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baafa50 0x56062bab82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f1c87a640 1 -- 172.21.15.76:0/29568628 learned_addr learned my addr 172.21.15.76:0/29568628 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f15ffb640 1 -- 172.21.15.76:0/108061688 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8528+0+0 (secure 0 0 0) 0x7f3f080cf100 con 0x7f3f1810bf60 2024-03-18T15:02:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f1d07b640 1 --2- 172.21.15.76:0/29568628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062bab8820 0x56062babac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f17fff640 1 -- 172.21.15.76:0/29568628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062bab8820 msgr2=0x56062babac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f17fff640 1 --2- 172.21.15.76:0/29568628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062bab8820 0x56062babac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f17fff640 1 -- 172.21.15.76:0/29568628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062baaf130 msgr2=0x56062baaf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f17fff640 1 --2- 172.21.15.76:0/29568628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062baaf130 0x56062baaf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f17fff640 1 -- 172.21.15.76:0/29568628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56062b9d4460 con 0x56062baafa50 2024-03-18T15:02:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f1c87a640 1 --2- 172.21.15.76:0/29568628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062baaf130 0x56062baaf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:02:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f1d07b640 1 --2- 172.21.15.76:0/108061688 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3f040a2a10 0x7f3f040a4ed0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3f00006010 tx=0x7f3f000095d0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f17fff640 1 --2- 172.21.15.76:0/29568628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baafa50 0x56062bab82e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3f080cc8f0 tx=0x7f3f0809b5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ced28593834dc4a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f0effd640 1 -- 172.21.15.76:0/29568628 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0807f4e0 con 0x56062baafa50 2024-03-18T15:02:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f0effd640 1 -- 172.21.15.76:0/29568628 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f0809bf00 con 0x56062baafa50 2024-03-18T15:02:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.892+0000 7f3f0effd640 1 -- 172.21.15.76:0/29568628 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0800ce60 con 0x56062baafa50 2024-03-18T15:02:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.893+0000 7f3f2179b740 1 -- 172.21.15.76:0/29568628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baafa50 msgr2=0x56062bab82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.893+0000 7f3f2179b740 1 --2- 172.21.15.76:0/29568628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baafa50 0x56062bab82e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3f080cc8f0 tx=0x7f3f0809b5d0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.893+0000 7f3f2179b740 1 -- 172.21.15.76:0/29568628 shutdown_connections 2024-03-18T15:02:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.893+0000 7f3f2179b740 1 --2- 172.21.15.76:0/29568628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062bab8820 0x56062babac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.893+0000 7f3f2179b740 1 --2- 172.21.15.76:0/29568628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baafa50 0x56062bab82e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.893+0000 7f3f2179b740 1 --2- 172.21.15.76:0/29568628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062baaf130 0x56062baaf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.893+0000 7f3f2179b740 1 -- 172.21.15.76:0/29568628 >> 172.21.15.76:0/29568628 conn(0x56062baa9a50 msgr2=0x56062baa9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.893+0000 7f3f2179b740 1 -- 172.21.15.76:0/29568628 shutdown_connections 2024-03-18T15:02:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.893+0000 7f3f2179b740 1 -- 172.21.15.76:0/29568628 wait complete. 2024-03-18T15:02:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.893+0000 7f3f2179b740 1 -- start start 2024-03-18T15:02:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f2179b740 1 Processor -- start 2024-03-18T15:02:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f2179b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062baaf130 0x56062bab5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f2179b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baafa50 0x56062bab5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f2179b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bab6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f2179b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56062babb390 con 0x56062bab8820 2024-03-18T15:02:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f2179b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56062b9cfd70 con 0x56062baafa50 2024-03-18T15:02:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f2179b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56062babb510 con 0x56062baaf130 2024-03-18T15:02:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f2179b740 10 client.0 _ll_register_callbacks cb 0x56062b727a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:02:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f1d07b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bab6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f1d07b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bab6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45928/0 (socket says 172.21.15.76:45928) 2024-03-18T15:02:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f1d07b640 1 -- 172.21.15.76:0/1999736265 learned_addr learned my addr 172.21.15.76:0/1999736265 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f17fff640 1 --2- 172.21.15.76:0/1999736265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baafa50 0x56062bab5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f1c87a640 1 --2- 172.21.15.76:0/1999736265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062baaf130 0x56062bab5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f1d07b640 1 -- 172.21.15.76:0/1999736265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062baaf130 msgr2=0x56062bab5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f1d07b640 1 --2- 172.21.15.76:0/1999736265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062baaf130 0x56062bab5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f1d07b640 1 -- 172.21.15.76:0/1999736265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baafa50 msgr2=0x56062bab5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f1d07b640 1 --2- 172.21.15.76:0/1999736265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baafa50 0x56062bab5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f1d07b640 1 -- 172.21.15.76:0/1999736265 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56062b728210 con 0x56062bab8820 2024-03-18T15:02:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f17fff640 1 --2- 172.21.15.76:0/1999736265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baafa50 0x56062bab5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:02:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3f1d07b640 1 --2- 172.21.15.76:0/1999736265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bab6380 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f3f00009900 tx=0x7f3f0002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54471cfac0b4b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.894+0000 7f3ef7fff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:02:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.895+0000 7f3ef7fff640 1 -- 172.21.15.76:0/1999736265 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0002cc10 con 0x56062bab8820 2024-03-18T15:02:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.895+0000 7f3ef7fff640 1 -- 172.21.15.76:0/1999736265 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8528+0+0 (secure 0 0 0) 0x7f3f00004f80 con 0x56062bab8820 2024-03-18T15:02:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.895+0000 7f3f2179b740 1 -- 172.21.15.76:0/1999736265 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f0809bf00 con 0x56062bab8820 2024-03-18T15:02:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.895+0000 7f3ef7fff640 1 -- 172.21.15.76:0/1999736265 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f3f0005b050 con 0x56062bab8820 2024-03-18T15:02:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.895+0000 7f3f2179b740 10 client.16560 fetch_fsmap learned FSMap version 41 2024-03-18T15:02:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.895+0000 7f3f2179b740 1 -- 172.21.15.76:0/1999736265 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x56062b7f4620 con 0x56062bab8820 2024-03-18T15:02:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.895+0000 7f3ef7fff640 1 -- 172.21.15.76:0/1999736265 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f0809bf00 con 0x56062bab8820 2024-03-18T15:02:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.895+0000 7f3ef7fff640 1 -- 172.21.15.76:0/1999736265 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f3f00003ab0 con 0x56062bab8820 2024-03-18T15:02:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.895+0000 7f3f2179b740 10 client.16560 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:02:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.895+0000 7f3f2179b740 10 client.16560 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:02:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.895+0000 7f3f2179b740 20 client.16560 populate_metadata read hostname 'smithi076' 2024-03-18T15:02:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.895+0000 7f3f2179b740 1 --2- 172.21.15.76:0/1999736265 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56062bac89f0 0x56062bacade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.895+0000 7f3f2179b740 4 client.16560 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:02:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.895+0000 7f3f2179b740 1 -- 172.21.15.76:0/1999736265 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56062ba20f80 con 0x56062bac89f0 2024-03-18T15:02:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.895+0000 7f3f1c87a640 1 --2- 172.21.15.76:0/1999736265 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56062bac89f0 0x56062bacade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.896+0000 7f3ef7fff640 10 client.16560 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:02:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.896+0000 7f3f1c87a640 1 --2- 172.21.15.76:0/1999736265 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56062bac89f0 0x56062bacade0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77f29d1b52883fbb server_cookie=8a5cf5fd90c1aa0 in_seq=0 out_seq=0 2024-03-18T15:02:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.896+0000 7f3ef7fff640 1 -- 172.21.15.76:0/1999736265 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56062ba20f80 con 0x56062bac89f0 2024-03-18T15:02:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.896+0000 7f3ef7fff640 10 client.16560 handle_command_reply: tid=0 2024-03-18T15:02:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.896+0000 7f3f2179b740 1 client.16560 shutdown 2024-03-18T15:02:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.897+0000 7f3f2179b740 1 -- 172.21.15.76:0/1999736265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 msgr2=0x56062bab6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.897+0000 7f3f2179b740 1 --2- 172.21.15.76:0/1999736265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bab6380 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f3f00009900 tx=0x7f3f0002bc30 comp rx=0 tx=0).stop 2024-03-18T15:02:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.897+0000 7f3f2179b740 1 -- 172.21.15.76:0/1999736265 shutdown_connections 2024-03-18T15:02:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.897+0000 7f3f2179b740 1 --2- 172.21.15.76:0/1999736265 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56062bac89f0 0x56062bacade0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.897+0000 7f3f2179b740 1 --2- 172.21.15.76:0/1999736265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bab6380 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.897+0000 7f3f2179b740 1 --2- 172.21.15.76:0/1999736265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baafa50 0x56062bab5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.897+0000 7f3f2179b740 1 --2- 172.21.15.76:0/1999736265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062baaf130 0x56062bab5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.897+0000 7f3f2179b740 1 -- 172.21.15.76:0/1999736265 >> 172.21.15.76:0/1999736265 conn(0x56062baa9a50 msgr2=0x56062baab7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.897+0000 7f3ef7fff640 0 client.16560 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:02:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.897+0000 7f3f2179b740 1 -- 172.21.15.76:0/1999736265 shutdown_connections 2024-03-18T15:02:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.897+0000 7f3f2179b740 1 -- 172.21.15.76:0/1999736265 wait complete. 2024-03-18T15:02:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.897+0000 7f3f2179b740 20 client.16560 trim_cache size 0 max 16384 2024-03-18T15:02:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.912+0000 7f3f2179b740 1 Processor -- start 2024-03-18T15:02:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.912+0000 7f3f2179b740 1 -- start start 2024-03-18T15:02:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.912+0000 7f3f2179b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baaf130 0x56062bab8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.912+0000 7f3f2179b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062baafa50 0x56062bac4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.912+0000 7f3f2179b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bad06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.912+0000 7f3f2179b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3f181ada70 con 0x56062bab8820 2024-03-18T15:02:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.912+0000 7f3f2179b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3f181ab340 con 0x56062baaf130 2024-03-18T15:02:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.912+0000 7f3f2179b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3f181a9190 con 0x56062baafa50 2024-03-18T15:02:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.912+0000 7f3f1d07b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bad06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.912+0000 7f3f1d07b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bad06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45944/0 (socket says 172.21.15.76:45944) 2024-03-18T15:02:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.912+0000 7f3f1d07b640 1 -- 172.21.15.76:0/1287683055 learned_addr learned my addr 172.21.15.76:0/1287683055 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.912+0000 7f3f1c87a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baaf130 0x56062bab8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.912+0000 7f3f17fff640 1 --2- 172.21.15.76:0/1287683055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062baafa50 0x56062bac4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.913+0000 7f3f1d07b640 1 -- 172.21.15.76:0/1287683055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062baafa50 msgr2=0x56062bac4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.913+0000 7f3f1d07b640 1 --2- 172.21.15.76:0/1287683055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062baafa50 0x56062bac4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.913+0000 7f3f1d07b640 1 -- 172.21.15.76:0/1287683055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baaf130 msgr2=0x56062bab8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.913+0000 7f3f1d07b640 1 --2- 172.21.15.76:0/1287683055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baaf130 0x56062bab8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.913+0000 7f3f1d07b640 1 -- 172.21.15.76:0/1287683055 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56062bab48b0 con 0x56062bab8820 2024-03-18T15:02:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.913+0000 7f3f17fff640 1 --2- 172.21.15.76:0/1287683055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062baafa50 0x56062bac4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:02:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.913+0000 7f3f1c87a640 1 --2- 172.21.15.76:0/1287683055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baaf130 0x56062bab8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:02:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.913+0000 7f3f1d07b640 1 --2- 172.21.15.76:0/1287683055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bad06b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3f0002c170 tx=0x7f3f00009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f8e925ad41caa64 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.913+0000 7f3f0e7fc640 1 -- 172.21.15.76:0/1287683055 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f000044d0 con 0x56062bab8820 2024-03-18T15:02:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.913+0000 7f3f0e7fc640 1 -- 172.21.15.76:0/1287683055 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f00052070 con 0x56062bab8820 2024-03-18T15:02:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.913+0000 7f3f0e7fc640 1 -- 172.21.15.76:0/1287683055 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f000052e0 con 0x56062bab8820 2024-03-18T15:02:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.914+0000 7f3f2179b740 1 -- 172.21.15.76:0/1287683055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 msgr2=0x56062bad06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.914+0000 7f3f2179b740 1 --2- 172.21.15.76:0/1287683055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bad06b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3f0002c170 tx=0x7f3f00009780 comp rx=0 tx=0).stop 2024-03-18T15:02:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.914+0000 7f3f2179b740 1 -- 172.21.15.76:0/1287683055 shutdown_connections 2024-03-18T15:02:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.914+0000 7f3f2179b740 1 --2- 172.21.15.76:0/1287683055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062baafa50 0x56062bac4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.914+0000 7f3f2179b740 1 --2- 172.21.15.76:0/1287683055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bad06b0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.914+0000 7f3f2179b740 1 --2- 172.21.15.76:0/1287683055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baaf130 0x56062bab8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.914+0000 7f3f2179b740 1 -- 172.21.15.76:0/1287683055 >> 172.21.15.76:0/1287683055 conn(0x56062baa9a50 msgr2=0x56062baaae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.914+0000 7f3f2179b740 1 -- 172.21.15.76:0/1287683055 shutdown_connections 2024-03-18T15:02:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.914+0000 7f3f2179b740 1 -- 172.21.15.76:0/1287683055 wait complete. 2024-03-18T15:02:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.914+0000 7f3f2179b740 1 -- start start 2024-03-18T15:02:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.914+0000 7f3f2179b740 1 Processor -- start 2024-03-18T15:02:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.914+0000 7f3f2179b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baaf130 0x56062bab2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f2179b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062baafa50 0x56062bab2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f2179b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bab3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f2179b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3f181a9800 con 0x56062bab8820 2024-03-18T15:02:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f2179b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3f181aac70 con 0x56062baaf130 2024-03-18T15:02:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f2179b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3f181a9ed0 con 0x56062baafa50 2024-03-18T15:02:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f2179b740 10 client.0 _ll_register_callbacks cb 0x7f3f08005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:02:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f1d07b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bab3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f1c87a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baaf130 0x56062bab2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f1d07b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bab3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45956/0 (socket says 172.21.15.76:45956) 2024-03-18T15:02:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f17fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062baafa50 0x56062bab2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f1d07b640 1 -- 172.21.15.76:0/792684402 learned_addr learned my addr 172.21.15.76:0/792684402 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f1c87a640 1 -- 172.21.15.76:0/792684402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062baafa50 msgr2=0x56062bab2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f1c87a640 1 --2- 172.21.15.76:0/792684402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062baafa50 0x56062bab2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f1c87a640 1 -- 172.21.15.76:0/792684402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 msgr2=0x56062bab3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f1c87a640 1 --2- 172.21.15.76:0/792684402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bab3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f1c87a640 1 -- 172.21.15.76:0/792684402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56062bb365a0 con 0x56062baaf130 2024-03-18T15:02:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f1d07b640 1 --2- 172.21.15.76:0/792684402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bab3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:02:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f1c87a640 1 --2- 172.21.15.76:0/792684402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baaf130 0x56062bab2590 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3f10000f80 tx=0x7f3f10012b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1eb269fc2c0d489 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3ef7fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T15:02:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3ef7fff640 1 -- 172.21.15.76:0/792684402 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f10004b20 con 0x56062baaf130 2024-03-18T15:02:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3ef7fff640 1 -- 172.21.15.76:0/792684402 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8528+0+0 (secure 0 0 0) 0x7f3f10004cc0 con 0x56062baaf130 2024-03-18T15:02:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f2179b740 1 -- 172.21.15.76:0/792684402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f00052070 con 0x56062baaf130 2024-03-18T15:02:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3ef7fff640 1 -- 172.21.15.76:0/792684402 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f3f10030050 con 0x56062baaf130 2024-03-18T15:02:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.915+0000 7f3f2179b740 10 client.26429 fetch_fsmap learned FSMap version 41 2024-03-18T15:02:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.916+0000 7f3f2179b740 1 -- 172.21.15.76:0/792684402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x56062bb378f0 con 0x56062baaf130 2024-03-18T15:02:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.916+0000 7f3ef7fff640 1 -- 172.21.15.76:0/792684402 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f00052070 con 0x56062baaf130 2024-03-18T15:02:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.916+0000 7f3ef7fff640 1 -- 172.21.15.76:0/792684402 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f3f10005280 con 0x56062baaf130 2024-03-18T15:02:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.916+0000 7f3f2179b740 10 client.26429 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:02:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.916+0000 7f3f2179b740 10 client.26429 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:02:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.916+0000 7f3f2179b740 20 client.26429 populate_metadata read hostname 'smithi076' 2024-03-18T15:02:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.916+0000 7f3f2179b740 1 --2- 172.21.15.76:0/792684402 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56062bb37ff0 0x56062bb3a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.916+0000 7f3f2179b740 4 client.26429 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:02:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.916+0000 7f3f2179b740 1 -- 172.21.15.76:0/792684402 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56062ba1b2d0 con 0x56062bb37ff0 2024-03-18T15:02:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.916+0000 7f3f17fff640 1 --2- 172.21.15.76:0/792684402 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56062bb37ff0 0x56062bb3a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.916+0000 7f3ef7fff640 10 client.26429 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:02:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.916+0000 7f3f17fff640 1 --2- 172.21.15.76:0/792684402 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56062bb37ff0 0x56062bb3a410 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bc67b4d187ab83c server_cookie=400a0ca1a75f9378 in_seq=0 out_seq=0 2024-03-18T15:02:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.990+0000 7f3ef7fff640 1 -- 172.21.15.76:0/792684402 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56062ba1b2d0 con 0x56062bb37ff0 2024-03-18T15:02:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.990+0000 7f3ef7fff640 10 client.26429 handle_command_reply: tid=0 2024-03-18T15:02:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.990+0000 7f3f2179b740 1 client.26429 shutdown 2024-03-18T15:02:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.990+0000 7f3f2179b740 1 -- 172.21.15.76:0/792684402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baaf130 msgr2=0x56062bab2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.990+0000 7f3f2179b740 1 --2- 172.21.15.76:0/792684402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baaf130 0x56062bab2590 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3f10000f80 tx=0x7f3f10012b80 comp rx=0 tx=0).stop 2024-03-18T15:02:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.990+0000 7f3f2179b740 1 -- 172.21.15.76:0/792684402 shutdown_connections 2024-03-18T15:02:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.990+0000 7f3f2179b740 1 --2- 172.21.15.76:0/792684402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56062baafa50 0x56062bab2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.990+0000 7f3f2179b740 1 --2- 172.21.15.76:0/792684402 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56062bb37ff0 0x56062bb3a410 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.990+0000 7f3f2179b740 1 --2- 172.21.15.76:0/792684402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56062bab8820 0x56062bab3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.990+0000 7f3f2179b740 1 --2- 172.21.15.76:0/792684402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56062baaf130 0x56062bab2590 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.990+0000 7f3ef7fff640 0 client.26429 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:02:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.990+0000 7f3f2179b740 1 -- 172.21.15.76:0/792684402 >> 172.21.15.76:0/792684402 conn(0x56062baa9a50 msgr2=0x56062baa8bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.990+0000 7f3f2179b740 1 -- 172.21.15.76:0/792684402 shutdown_connections 2024-03-18T15:02:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.990+0000 7f3f2179b740 1 -- 172.21.15.76:0/792684402 wait complete. 2024-03-18T15:02:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.990+0000 7f3f2179b740 20 client.26429 trim_cache size 0 max 16384 2024-03-18T15:02:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:02:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:02:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:02:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:02:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.991+0000 7f3f1eade640 1 -- 172.21.15.76:0/108061688 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3f040a2a10 msgr2=0x7f3f040a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.991+0000 7f3f1eade640 1 --2- 172.21.15.76:0/108061688 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3f040a2a10 0x7f3f040a4ed0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3f00006010 tx=0x7f3f000095d0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.991+0000 7f3f1eade640 1 -- 172.21.15.76:0/108061688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f1810bf60 msgr2=0x7f3f181986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.991+0000 7f3f1eade640 1 --2- 172.21.15.76:0/108061688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f1810bf60 0x7f3f181986e0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800e9c0 tx=0x7f3f0800ee80 comp rx=0 tx=0).stop 2024-03-18T15:02:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.991+0000 7f3f1eade640 1 -- 172.21.15.76:0/108061688 shutdown_connections 2024-03-18T15:02:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.991+0000 7f3f1eade640 1 --2- 172.21.15.76:0/108061688 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3f040a2a10 0x7f3f040a4ed0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.991+0000 7f3f1eade640 1 --2- 172.21.15.76:0/108061688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f1819ca60 0x7f3f1819cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.991+0000 7f3f1eade640 1 --2- 172.21.15.76:0/108061688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f1810bf60 0x7f3f181986e0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.991+0000 7f3f1eade640 1 --2- 172.21.15.76:0/108061688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f18102a50 0x7f3f181981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.991+0000 7f3f1eade640 1 -- 172.21.15.76:0/108061688 >> 172.21.15.76:0/108061688 conn(0x7f3f18077800 msgr2=0x7f3f180fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.991+0000 7f3f1eade640 1 -- 172.21.15.76:0/108061688 shutdown_connections 2024-03-18T15:02:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:07.991+0000 7f3f1eade640 1 -- 172.21.15.76:0/108061688 wait complete. 2024-03-18T15:02:08.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:07 smithi076 ceph-mon[32205]: pgmap v279: 97 pgs: 17 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 72 active+clean; 249 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 318 op/s 2024-03-18T15:02:08.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:07 smithi174 ceph-mon[36787]: pgmap v279: 97 pgs: 17 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 72 active+clean; 249 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 318 op/s 2024-03-18T15:02:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:07 smithi077 ceph-mon[35865]: pgmap v279: 97 pgs: 17 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 72 active+clean; 249 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 318 op/s 2024-03-18T15:02:08.754 INFO:tasks.workunit.client.0.smithi076.stdout: 1 867176 9.40 MB/sec execute 369 sec latency 35.741 ms 2024-03-18T15:02:09.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 869502 9.41 MB/sec execute 370 sec latency 19.321 ms 2024-03-18T15:02:10.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:09 smithi076 ceph-mon[32205]: pgmap v280: 97 pgs: 97 active+clean; 244 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 308 op/s 2024-03-18T15:02:10.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:09 smithi174 ceph-mon[36787]: pgmap v280: 97 pgs: 97 active+clean; 244 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 308 op/s 2024-03-18T15:02:10.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:09 smithi077 ceph-mon[35865]: pgmap v280: 97 pgs: 97 active+clean; 244 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 308 op/s 2024-03-18T15:02:10.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 870814 9.40 MB/sec execute 371 sec latency 18.144 ms 2024-03-18T15:02:11.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 873031 9.41 MB/sec execute 372 sec latency 83.425 ms 2024-03-18T15:02:12.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:11 smithi076 ceph-mon[32205]: pgmap v281: 97 pgs: 97 active+clean; 244 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 308 op/s 2024-03-18T15:02:12.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:11 smithi174 ceph-mon[36787]: pgmap v281: 97 pgs: 97 active+clean; 244 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 308 op/s 2024-03-18T15:02:12.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:11 smithi077 ceph-mon[35865]: pgmap v281: 97 pgs: 97 active+clean; 244 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 308 op/s 2024-03-18T15:02:12.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 874117 9.40 MB/sec execute 373 sec latency 36.731 ms 2024-03-18T15:02:13.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 876406 9.41 MB/sec execute 374 sec latency 23.849 ms 2024-03-18T15:02:14.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:14 smithi076 ceph-mon[32205]: osdmap e101: 12 total, 12 up, 12 in 2024-03-18T15:02:14.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:14 smithi076 ceph-mon[32205]: pgmap v283: 97 pgs: 97 active+clean; 252 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 285 op/s 2024-03-18T15:02:14.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:14 smithi174 ceph-mon[36787]: osdmap e101: 12 total, 12 up, 12 in 2024-03-18T15:02:14.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:14 smithi174 ceph-mon[36787]: pgmap v283: 97 pgs: 97 active+clean; 252 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 285 op/s 2024-03-18T15:02:14.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:14 smithi077 ceph-mon[35865]: osdmap e101: 12 total, 12 up, 12 in 2024-03-18T15:02:14.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:14 smithi077 ceph-mon[35865]: pgmap v283: 97 pgs: 97 active+clean; 252 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 285 op/s 2024-03-18T15:02:14.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 877718 9.40 MB/sec execute 375 sec latency 23.370 ms 2024-03-18T15:02:15.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 880114 9.41 MB/sec execute 376 sec latency 23.986 ms 2024-03-18T15:02:16.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:15 smithi076 ceph-mon[32205]: pgmap v284: 97 pgs: 97 active+clean; 252 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 285 op/s 2024-03-18T15:02:16.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:15 smithi174 ceph-mon[36787]: pgmap v284: 97 pgs: 97 active+clean; 252 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 285 op/s 2024-03-18T15:02:16.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:15 smithi077 ceph-mon[35865]: pgmap v284: 97 pgs: 97 active+clean; 252 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 285 op/s 2024-03-18T15:02:16.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 881207 9.41 MB/sec execute 377 sec latency 54.206 ms 2024-03-18T15:02:17.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 883391 9.41 MB/sec execute 378 sec latency 25.735 ms 2024-03-18T15:02:18.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:17 smithi076 ceph-mon[32205]: pgmap v285: 97 pgs: 97 active+clean; 252 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 285 op/s 2024-03-18T15:02:18.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:17 smithi174 ceph-mon[36787]: pgmap v285: 97 pgs: 97 active+clean; 252 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 285 op/s 2024-03-18T15:02:18.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:17 smithi077 ceph-mon[35865]: pgmap v285: 97 pgs: 97 active+clean; 252 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 285 op/s 2024-03-18T15:02:18.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 884637 9.41 MB/sec execute 379 sec latency 18.648 ms 2024-03-18T15:02:19.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 887009 9.41 MB/sec execute 380 sec latency 15.238 ms 2024-03-18T15:02:20.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:19 smithi076 ceph-mon[32205]: pgmap v286: 97 pgs: 97 active+clean; 259 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 247 op/s 2024-03-18T15:02:20.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:19 smithi174 ceph-mon[36787]: pgmap v286: 97 pgs: 97 active+clean; 259 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 247 op/s 2024-03-18T15:02:20.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:19 smithi077 ceph-mon[35865]: pgmap v286: 97 pgs: 97 active+clean; 259 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 247 op/s 2024-03-18T15:02:20.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 888345 9.41 MB/sec execute 381 sec latency 25.039 ms 2024-03-18T15:02:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.082+0000 7f9604844640 1 -- 172.21.15.76:0/1643121009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9600100930 msgr2=0x7f9600100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.082+0000 7f9604844640 1 --2- 172.21.15.76:0/1643121009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9600100930 0x7f9600100db0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009890 tx=0x7f95e802bb70 comp rx=0 tx=0).stop 2024-03-18T15:02:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.082+0000 7f9604844640 1 -- 172.21.15.76:0/1643121009 shutdown_connections 2024-03-18T15:02:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.082+0000 7f9604844640 1 --2- 172.21.15.76:0/1643121009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9600100930 0x7f9600100db0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.082+0000 7f9604844640 1 --2- 172.21.15.76:0/1643121009 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f96001012f0 0x7f960010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.082+0000 7f9604844640 1 --2- 172.21.15.76:0/1643121009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9600105680 0x7f9600105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.082+0000 7f9604844640 1 -- 172.21.15.76:0/1643121009 >> 172.21.15.76:0/1643121009 conn(0x7f9600077530 msgr2=0x7f9600077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.082+0000 7f9604844640 1 -- 172.21.15.76:0/1643121009 shutdown_connections 2024-03-18T15:02:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.082+0000 7f9604844640 1 -- 172.21.15.76:0/1643121009 wait complete. 2024-03-18T15:02:21.084 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.082+0000 7f9604844640 1 Processor -- start 2024-03-18T15:02:21.084 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f9604844640 1 -- start start 2024-03-18T15:02:21.085 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f9604844640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9600100930 0x7f96001980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.085 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f9604844640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96001012f0 0x7f9600198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f9604844640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9600105680 0x7f960019c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f9604844640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9600110ba0 con 0x7f9600100930 2024-03-18T15:02:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f9604844640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9600110a20 con 0x7f96001012f0 2024-03-18T15:02:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f9604844640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9600110d20 con 0x7f9600105680 2024-03-18T15:02:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f95fe59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9600100930 0x7f96001980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f95fe59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9600100930 0x7f96001980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47914/0 (socket says 172.21.15.76:47914) 2024-03-18T15:02:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f95fe59c640 1 -- 172.21.15.76:0/261210977 learned_addr learned my addr 172.21.15.76:0/261210977 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f95f5d9b640 1 --2- 172.21.15.76:0/261210977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96001012f0 0x7f9600198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f95fed9d640 1 --2- 172.21.15.76:0/261210977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9600105680 0x7f960019c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f95fe59c640 1 -- 172.21.15.76:0/261210977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9600105680 msgr2=0x7f960019c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f95fe59c640 1 --2- 172.21.15.76:0/261210977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9600105680 0x7f960019c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f95fe59c640 1 -- 172.21.15.76:0/261210977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96001012f0 msgr2=0x7f9600198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f95fe59c640 1 --2- 172.21.15.76:0/261210977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96001012f0 0x7f9600198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f95fe59c640 1 -- 172.21.15.76:0/261210977 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960019d0c0 con 0x7f9600100930 2024-03-18T15:02:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f95f5d9b640 1 --2- 172.21.15.76:0/261210977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96001012f0 0x7f9600198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:02:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.083+0000 7f95fe59c640 1 --2- 172.21.15.76:0/261210977 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9600100930 0x7f96001980d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f95f000ea60 tx=0x7f95f000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.084+0000 7f95f7fff640 1 -- 172.21.15.76:0/261210977 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f000cde0 con 0x7f9600100930 2024-03-18T15:02:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.084+0000 7f95f7fff640 1 -- 172.21.15.76:0/261210977 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95f00035f0 con 0x7f9600100930 2024-03-18T15:02:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.084+0000 7f9604844640 1 -- 172.21.15.76:0/261210977 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f960019d3b0 con 0x7f9600100930 2024-03-18T15:02:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.084+0000 7f95f7fff640 1 -- 172.21.15.76:0/261210977 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f0010670 con 0x7f9600100930 2024-03-18T15:02:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.084+0000 7f9604844640 1 -- 172.21.15.76:0/261210977 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96001a4c40 con 0x7f9600100930 2024-03-18T15:02:21.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.086+0000 7f95f7fff640 1 -- 172.21.15.76:0/261210977 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f95f00037a0 con 0x7f9600100930 2024-03-18T15:02:21.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.086+0000 7f960742b740 1 Processor -- start 2024-03-18T15:02:21.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.086+0000 7f960742b740 1 -- start start 2024-03-18T15:02:21.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.086+0000 7f95f7fff640 1 --2- 172.21.15.76:0/261210977 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f95ec0a2a10 0x7f95ec0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.086+0000 7f960742b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1183bd0 0x5603e1183fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.086+0000 7f960742b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e11844f0 0x5603e11916f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f960742b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1191c30 0x5603e1194020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.094 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f960742b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5603e10e8a40 con 0x5603e1191c30 2024-03-18T15:02:21.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f960742b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5603e1091920 con 0x5603e11844f0 2024-03-18T15:02:21.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f960742b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5603e10e4ae0 con 0x5603e1183bd0 2024-03-18T15:02:21.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95fe59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1183bd0 0x5603e1183fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95fed9d640 1 --2- 172.21.15.76:0/261210977 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f95ec0a2a10 0x7f95ec0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95f5d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e11844f0 0x5603e11916f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95f5d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e11844f0 0x5603e11916f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:57716/0 (socket says 172.21.15.76:57716) 2024-03-18T15:02:21.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95fe59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1183bd0 0x5603e1183fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:55730/0 (socket says 172.21.15.76:55730) 2024-03-18T15:02:21.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95f5d9b640 1 -- 172.21.15.76:0/525154182 learned_addr learned my addr 172.21.15.76:0/525154182 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:21.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95f7fff640 1 -- 172.21.15.76:0/261210977 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7f95f00cf130 con 0x7f9600100930 2024-03-18T15:02:21.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95fed9d640 1 --2- 172.21.15.76:0/525154182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1191c30 0x5603e1194020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95f5d9b640 1 -- 172.21.15.76:0/525154182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1183bd0 msgr2=0x5603e1183fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:21.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95f5d9b640 1 --2- 172.21.15.76:0/525154182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1183bd0 0x5603e1183fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95f5d9b640 1 -- 172.21.15.76:0/525154182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1191c30 msgr2=0x5603e1194020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:21.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95f5d9b640 1 --2- 172.21.15.76:0/525154182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1191c30 0x5603e1194020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95f5d9b640 1 -- 172.21.15.76:0/525154182 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603e10a9460 con 0x5603e11844f0 2024-03-18T15:02:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95fed9d640 1 --2- 172.21.15.76:0/261210977 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f95ec0a2a10 0x7f95ec0a4ed0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f95f8005fd0 tx=0x7f95f8009e20 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95f5d9b640 1 --2- 172.21.15.76:0/525154182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e11844f0 0x5603e11916f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009890 tx=0x7f95e8005890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc96b674e2f20f94 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95f4d99640 1 -- 172.21.15.76:0/525154182 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e8039040 con 0x5603e11844f0 2024-03-18T15:02:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95f4d99640 1 -- 172.21.15.76:0/525154182 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95e8004030 con 0x5603e11844f0 2024-03-18T15:02:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.087+0000 7f95f4d99640 1 -- 172.21.15.76:0/525154182 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e802ca10 con 0x5603e11844f0 2024-03-18T15:02:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.088+0000 7f960742b740 1 -- 172.21.15.76:0/525154182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e11844f0 msgr2=0x5603e11916f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.088+0000 7f960742b740 1 --2- 172.21.15.76:0/525154182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e11844f0 0x5603e11916f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009890 tx=0x7f95e8005890 comp rx=0 tx=0).stop 2024-03-18T15:02:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.088+0000 7f960742b740 1 -- 172.21.15.76:0/525154182 shutdown_connections 2024-03-18T15:02:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.088+0000 7f960742b740 1 --2- 172.21.15.76:0/525154182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1191c30 0x5603e1194020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.088+0000 7f960742b740 1 --2- 172.21.15.76:0/525154182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e11844f0 0x5603e11916f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.088+0000 7f960742b740 1 --2- 172.21.15.76:0/525154182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1183bd0 0x5603e1183fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.088+0000 7f960742b740 1 -- 172.21.15.76:0/525154182 >> 172.21.15.76:0/525154182 conn(0x5603e117ea50 msgr2=0x5603e117ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.088+0000 7f960742b740 1 -- 172.21.15.76:0/525154182 shutdown_connections 2024-03-18T15:02:21.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.088+0000 7f960742b740 1 -- 172.21.15.76:0/525154182 wait complete. 2024-03-18T15:02:21.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.088+0000 7f960742b740 1 -- start start 2024-03-18T15:02:21.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f960742b740 1 Processor -- start 2024-03-18T15:02:21.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f960742b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1183bd0 0x5603e118eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f960742b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e11844f0 0x5603e118f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f960742b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1191c30 0x5603e118f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f960742b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5603e1194800 con 0x5603e1191c30 2024-03-18T15:02:21.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f960742b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5603e10a4d70 con 0x5603e11844f0 2024-03-18T15:02:21.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f960742b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5603e1194980 con 0x5603e1183bd0 2024-03-18T15:02:21.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f960742b740 10 client.0 _ll_register_callbacks cb 0x5603e0dfca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:02:21.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f95fed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1191c30 0x5603e118f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f95fe59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1183bd0 0x5603e118eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f95fe59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1183bd0 0x5603e118eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:55742/0 (socket says 172.21.15.76:55742) 2024-03-18T15:02:21.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f95fed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1191c30 0x5603e118f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47920/0 (socket says 172.21.15.76:47920) 2024-03-18T15:02:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f95fe59c640 1 -- 172.21.15.76:0/2392624294 learned_addr learned my addr 172.21.15.76:0/2392624294 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f95f5d9b640 1 --2- 172.21.15.76:0/2392624294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e11844f0 0x5603e118f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f95fed9d640 1 -- 172.21.15.76:0/2392624294 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1183bd0 msgr2=0x5603e118eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f95fed9d640 1 --2- 172.21.15.76:0/2392624294 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1183bd0 0x5603e118eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f95fed9d640 1 -- 172.21.15.76:0/2392624294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e11844f0 msgr2=0x5603e118f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f95fed9d640 1 --2- 172.21.15.76:0/2392624294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e11844f0 0x5603e118f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f95fed9d640 1 -- 172.21.15.76:0/2392624294 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603e0dfd210 con 0x5603e1191c30 2024-03-18T15:02:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f95fed9d640 1 --2- 172.21.15.76:0/2392624294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1191c30 0x5603e118f610 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f95f800c800 tx=0x7f95f801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d724cc64053dae14 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f95e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:02:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.089+0000 7f95e5ffb640 1 -- 172.21.15.76:0/2392624294 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f801de60 con 0x5603e1191c30 2024-03-18T15:02:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.090+0000 7f95e5ffb640 1 -- 172.21.15.76:0/2392624294 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7f95f801c930 con 0x5603e1191c30 2024-03-18T15:02:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.090+0000 7f960742b740 1 -- 172.21.15.76:0/2392624294 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95e8004030 con 0x5603e1191c30 2024-03-18T15:02:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.090+0000 7f95e5ffb640 1 -- 172.21.15.76:0/2392624294 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f95f801cb30 con 0x5603e1191c30 2024-03-18T15:02:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.090+0000 7f960742b740 10 client.16578 fetch_fsmap learned FSMap version 41 2024-03-18T15:02:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.090+0000 7f960742b740 1 -- 172.21.15.76:0/2392624294 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5603e0ec9620 con 0x5603e1191c30 2024-03-18T15:02:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.090+0000 7f95e5ffb640 1 -- 172.21.15.76:0/2392624294 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95e8004030 con 0x5603e1191c30 2024-03-18T15:02:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.090+0000 7f95e5ffb640 1 -- 172.21.15.76:0/2392624294 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f95f8030020 con 0x5603e1191c30 2024-03-18T15:02:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.090+0000 7f960742b740 10 client.16578 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:02:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.090+0000 7f960742b740 10 client.16578 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:02:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.090+0000 7f960742b740 20 client.16578 populate_metadata read hostname 'smithi076' 2024-03-18T15:02:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.090+0000 7f960742b740 1 --2- 172.21.15.76:0/2392624294 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5603e11a1d80 0x5603e11a4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.090+0000 7f960742b740 4 client.16578 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:02:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.090+0000 7f960742b740 1 -- 172.21.15.76:0/2392624294 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603e10f5f80 con 0x5603e11a1d80 2024-03-18T15:02:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.090+0000 7f95fe59c640 1 --2- 172.21.15.76:0/2392624294 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5603e11a1d80 0x5603e11a4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.091+0000 7f95e5ffb640 10 client.16578 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:02:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.091+0000 7f95fe59c640 1 --2- 172.21.15.76:0/2392624294 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5603e11a1d80 0x5603e11a4190 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ad8e9e9c268c605 server_cookie=92467ee5db52c212 in_seq=0 out_seq=0 2024-03-18T15:02:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.091+0000 7f95e5ffb640 1 -- 172.21.15.76:0/2392624294 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5603e10f5f80 con 0x5603e11a1d80 2024-03-18T15:02:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.091+0000 7f95e5ffb640 10 client.16578 handle_command_reply: tid=0 2024-03-18T15:02:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.091+0000 7f960742b740 1 client.16578 shutdown 2024-03-18T15:02:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.091+0000 7f960742b740 1 -- 172.21.15.76:0/2392624294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1191c30 msgr2=0x5603e118f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.091+0000 7f960742b740 1 --2- 172.21.15.76:0/2392624294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1191c30 0x5603e118f610 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f95f800c800 tx=0x7f95f801d000 comp rx=0 tx=0).stop 2024-03-18T15:02:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.091+0000 7f960742b740 1 -- 172.21.15.76:0/2392624294 shutdown_connections 2024-03-18T15:02:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.091+0000 7f960742b740 1 --2- 172.21.15.76:0/2392624294 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5603e11a1d80 0x5603e11a4190 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.091+0000 7f960742b740 1 --2- 172.21.15.76:0/2392624294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1191c30 0x5603e118f610 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.091+0000 7f95e5ffb640 0 client.16578 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:02:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.091+0000 7f960742b740 1 --2- 172.21.15.76:0/2392624294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e11844f0 0x5603e118f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.091+0000 7f960742b740 1 --2- 172.21.15.76:0/2392624294 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1183bd0 0x5603e118eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.091+0000 7f960742b740 1 -- 172.21.15.76:0/2392624294 >> 172.21.15.76:0/2392624294 conn(0x5603e117ea50 msgr2=0x5603e11807a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.091+0000 7f960742b740 1 -- 172.21.15.76:0/2392624294 shutdown_connections 2024-03-18T15:02:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.091+0000 7f960742b740 1 -- 172.21.15.76:0/2392624294 wait complete. 2024-03-18T15:02:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.092+0000 7f960742b740 20 client.16578 trim_cache size 0 max 16384 2024-03-18T15:02:21.112 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f960742b740 1 Processor -- start 2024-03-18T15:02:21.112 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f960742b740 1 -- start start 2024-03-18T15:02:21.112 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f960742b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e1183bd0 0x5603e11912c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.112 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f960742b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e11844f0 0x5603e119d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.112 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f960742b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1191c30 0x5603e11805b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.112 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f960742b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f96001ada70 con 0x5603e11844f0 2024-03-18T15:02:21.112 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f960742b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f96001ab340 con 0x5603e1183bd0 2024-03-18T15:02:21.112 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f960742b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f96001a91b0 con 0x5603e1191c30 2024-03-18T15:02:21.112 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f95f5d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e11844f0 0x5603e119d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.112 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f95f5d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e11844f0 0x5603e119d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47932/0 (socket says 172.21.15.76:47932) 2024-03-18T15:02:21.112 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f95f5d9b640 1 -- 172.21.15.76:0/628606996 learned_addr learned my addr 172.21.15.76:0/628606996 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:21.112 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f95fe59c640 1 --2- 172.21.15.76:0/628606996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e1183bd0 0x5603e11912c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.113 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f95fed9d640 1 --2- 172.21.15.76:0/628606996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1191c30 0x5603e11805b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.113 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f95f5d9b640 1 -- 172.21.15.76:0/628606996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1191c30 msgr2=0x5603e11805b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:21.113 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f95f5d9b640 1 --2- 172.21.15.76:0/628606996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1191c30 0x5603e11805b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.113 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f95f5d9b640 1 -- 172.21.15.76:0/628606996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e1183bd0 msgr2=0x5603e11912c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:21.113 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f95f5d9b640 1 --2- 172.21.15.76:0/628606996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e1183bd0 0x5603e11912c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.113 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f95f5d9b640 1 -- 172.21.15.76:0/628606996 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603e117e880 con 0x5603e11844f0 2024-03-18T15:02:21.113 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.107+0000 7f95fe59c640 1 --2- 172.21.15.76:0/628606996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e1183bd0 0x5603e11912c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:02:21.113 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.108+0000 7f95f5d9b640 1 --2- 172.21.15.76:0/628606996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e11844f0 0x5603e119d610 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f95e802c620 tx=0x7f95e8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e96b26e9505083 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:21.113 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.108+0000 7f95e7fff640 1 -- 172.21.15.76:0/628606996 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e8039040 con 0x5603e11844f0 2024-03-18T15:02:21.113 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.108+0000 7f95e7fff640 1 -- 172.21.15.76:0/628606996 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95e8004040 con 0x5603e11844f0 2024-03-18T15:02:21.113 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.108+0000 7f95e7fff640 1 -- 172.21.15.76:0/628606996 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e8032640 con 0x5603e11844f0 2024-03-18T15:02:21.113 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.108+0000 7f960742b740 1 -- 172.21.15.76:0/628606996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e11844f0 msgr2=0x5603e119d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:21.113 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.108+0000 7f960742b740 1 --2- 172.21.15.76:0/628606996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e11844f0 0x5603e119d610 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f95e802c620 tx=0x7f95e8000bc0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.108+0000 7f960742b740 1 -- 172.21.15.76:0/628606996 shutdown_connections 2024-03-18T15:02:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f960742b740 1 --2- 172.21.15.76:0/628606996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1191c30 0x5603e11805b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f960742b740 1 --2- 172.21.15.76:0/628606996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e11844f0 0x5603e119d610 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f960742b740 1 --2- 172.21.15.76:0/628606996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e1183bd0 0x5603e11912c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f960742b740 1 -- 172.21.15.76:0/628606996 >> 172.21.15.76:0/628606996 conn(0x5603e117ea50 msgr2=0x5603e117fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f960742b740 1 -- 172.21.15.76:0/628606996 shutdown_connections 2024-03-18T15:02:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f960742b740 1 -- 172.21.15.76:0/628606996 wait complete. 2024-03-18T15:02:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f960742b740 1 -- start start 2024-03-18T15:02:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f960742b740 1 Processor -- start 2024-03-18T15:02:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f960742b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1183bd0 0x5603e1199c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f960742b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e11844f0 0x5603e1196c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f960742b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1191c30 0x5603e1197150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f960742b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f96001a9800 con 0x5603e1183bd0 2024-03-18T15:02:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f960742b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f96001aac70 con 0x5603e11844f0 2024-03-18T15:02:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f960742b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f96001a9ed0 con 0x5603e1191c30 2024-03-18T15:02:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f960742b740 10 client.0 _ll_register_callbacks cb 0x7f95f801d680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:02:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f95fe59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1183bd0 0x5603e1199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f95fe59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1183bd0 0x5603e1199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47938/0 (socket says 172.21.15.76:47938) 2024-03-18T15:02:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f95fe59c640 1 -- 172.21.15.76:0/4179398465 learned_addr learned my addr 172.21.15.76:0/4179398465 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f95fed9d640 1 --2- 172.21.15.76:0/4179398465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1191c30 0x5603e1197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.109+0000 7f95f5d9b640 1 --2- 172.21.15.76:0/4179398465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e11844f0 0x5603e1196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.110+0000 7f95fe59c640 1 -- 172.21.15.76:0/4179398465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1191c30 msgr2=0x5603e1197150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.110+0000 7f95fe59c640 1 --2- 172.21.15.76:0/4179398465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1191c30 0x5603e1197150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.110+0000 7f95fe59c640 1 -- 172.21.15.76:0/4179398465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e11844f0 msgr2=0x5603e1196c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.110+0000 7f95fe59c640 1 --2- 172.21.15.76:0/4179398465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e11844f0 0x5603e1196c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.110+0000 7f95fe59c640 1 -- 172.21.15.76:0/4179398465 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603e1184ba0 con 0x5603e1183bd0 2024-03-18T15:02:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.110+0000 7f95fe59c640 1 --2- 172.21.15.76:0/4179398465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1183bd0 0x5603e1199c40 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f95f0097490 tx=0x7f95f0015f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb48729ab5d169 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.110+0000 7f95e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:02:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.110+0000 7f95e5ffb640 1 -- 172.21.15.76:0/4179398465 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f009beb0 con 0x5603e1183bd0 2024-03-18T15:02:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.110+0000 7f95e5ffb640 1 -- 172.21.15.76:0/4179398465 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7f95f00860b0 con 0x5603e1183bd0 2024-03-18T15:02:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.110+0000 7f960742b740 1 -- 172.21.15.76:0/4179398465 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95e8004040 con 0x5603e1183bd0 2024-03-18T15:02:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.110+0000 7f95e5ffb640 1 -- 172.21.15.76:0/4179398465 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f95f009b7b0 con 0x5603e1183bd0 2024-03-18T15:02:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.110+0000 7f960742b740 10 client.16584 fetch_fsmap learned FSMap version 41 2024-03-18T15:02:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.110+0000 7f960742b740 1 -- 172.21.15.76:0/4179398465 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5603e1208890 con 0x5603e1183bd0 2024-03-18T15:02:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.110+0000 7f95e5ffb640 1 -- 172.21.15.76:0/4179398465 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95e8004040 con 0x5603e1183bd0 2024-03-18T15:02:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.111+0000 7f95e5ffb640 1 -- 172.21.15.76:0/4179398465 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7f95f0097e20 con 0x5603e1183bd0 2024-03-18T15:02:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.111+0000 7f960742b740 10 client.16584 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:02:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.111+0000 7f960742b740 10 client.16584 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:02:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.111+0000 7f960742b740 20 client.16584 populate_metadata read hostname 'smithi076' 2024-03-18T15:02:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.111+0000 7f960742b740 1 --2- 172.21.15.76:0/4179398465 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5603e1208f60 0x5603e120b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.111+0000 7f960742b740 4 client.16584 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:02:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.111+0000 7f960742b740 1 -- 172.21.15.76:0/4179398465 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5603e10f02d0 con 0x5603e1208f60 2024-03-18T15:02:21.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.111+0000 7f95f5d9b640 1 --2- 172.21.15.76:0/4179398465 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5603e1208f60 0x5603e120b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:21.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.111+0000 7f95e5ffb640 10 client.16584 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:02:21.117 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.111+0000 7f95f5d9b640 1 --2- 172.21.15.76:0/4179398465 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5603e1208f60 0x5603e120b380 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54c7d7aec5244326 server_cookie=d277bb671b854926 in_seq=0 out_seq=0 2024-03-18T15:02:21.184 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.183+0000 7f95e5ffb640 1 -- 172.21.15.76:0/4179398465 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5603e10f02d0 con 0x5603e1208f60 2024-03-18T15:02:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.183+0000 7f95e5ffb640 10 client.16584 handle_command_reply: tid=0 2024-03-18T15:02:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.183+0000 7f960742b740 1 client.16584 shutdown 2024-03-18T15:02:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.183+0000 7f960742b740 1 -- 172.21.15.76:0/4179398465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1183bd0 msgr2=0x5603e1199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.183+0000 7f960742b740 1 --2- 172.21.15.76:0/4179398465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1183bd0 0x5603e1199c40 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f95f0097490 tx=0x7f95f0015f80 comp rx=0 tx=0).stop 2024-03-18T15:02:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.183+0000 7f960742b740 1 -- 172.21.15.76:0/4179398465 shutdown_connections 2024-03-18T15:02:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.183+0000 7f960742b740 1 --2- 172.21.15.76:0/4179398465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603e1191c30 0x5603e1197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.183+0000 7f960742b740 1 --2- 172.21.15.76:0/4179398465 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5603e1208f60 0x5603e120b380 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.183+0000 7f960742b740 1 --2- 172.21.15.76:0/4179398465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5603e11844f0 0x5603e1196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.183+0000 7f960742b740 1 --2- 172.21.15.76:0/4179398465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e1183bd0 0x5603e1199c40 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.183+0000 7f95e5ffb640 0 client.16584 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:02:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.183+0000 7f960742b740 1 -- 172.21.15.76:0/4179398465 >> 172.21.15.76:0/4179398465 conn(0x5603e117ea50 msgr2=0x5603e1180730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.183+0000 7f960742b740 1 -- 172.21.15.76:0/4179398465 shutdown_connections 2024-03-18T15:02:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.183+0000 7f960742b740 1 -- 172.21.15.76:0/4179398465 wait complete. 2024-03-18T15:02:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.183+0000 7f960742b740 20 client.16584 trim_cache size 0 max 16384 2024-03-18T15:02:21.186 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:02:21.186 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:02:21.186 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:02:21.186 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:02:21.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.184+0000 7f9604844640 1 -- 172.21.15.76:0/261210977 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f95ec0a2a10 msgr2=0x7f95ec0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:21.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.184+0000 7f9604844640 1 --2- 172.21.15.76:0/261210977 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f95ec0a2a10 0x7f95ec0a4ed0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f95f8005fd0 tx=0x7f95f8009e20 comp rx=0 tx=0).stop 2024-03-18T15:02:21.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.184+0000 7f9604844640 1 -- 172.21.15.76:0/261210977 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9600100930 msgr2=0x7f96001980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:21.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.184+0000 7f9604844640 1 --2- 172.21.15.76:0/261210977 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9600100930 0x7f96001980d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f95f000ea60 tx=0x7f95f000ef20 comp rx=0 tx=0).stop 2024-03-18T15:02:21.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.184+0000 7f9604844640 1 -- 172.21.15.76:0/261210977 shutdown_connections 2024-03-18T15:02:21.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.184+0000 7f9604844640 1 --2- 172.21.15.76:0/261210977 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f95ec0a2a10 0x7f95ec0a4ed0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.184+0000 7f9604844640 1 --2- 172.21.15.76:0/261210977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9600105680 0x7f960019c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.184+0000 7f9604844640 1 --2- 172.21.15.76:0/261210977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96001012f0 0x7f9600198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.184+0000 7f9604844640 1 --2- 172.21.15.76:0/261210977 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9600100930 0x7f96001980d0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:21.187 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.184+0000 7f9604844640 1 -- 172.21.15.76:0/261210977 >> 172.21.15.76:0/261210977 conn(0x7f9600077530 msgr2=0x7f9600068fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:21.188 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.184+0000 7f9604844640 1 -- 172.21.15.76:0/261210977 shutdown_connections 2024-03-18T15:02:21.188 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:21.184+0000 7f9604844640 1 -- 172.21.15.76:0/261210977 wait complete. 2024-03-18T15:02:21.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 890459 9.41 MB/sec execute 382 sec latency 69.404 ms 2024-03-18T15:02:22.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:21 smithi076 ceph-mon[32205]: pgmap v287: 97 pgs: 97 active+clean; 259 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 247 op/s 2024-03-18T15:02:22.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:21 smithi174 ceph-mon[36787]: pgmap v287: 97 pgs: 97 active+clean; 259 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 247 op/s 2024-03-18T15:02:22.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:21 smithi077 ceph-mon[35865]: pgmap v287: 97 pgs: 97 active+clean; 259 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 247 op/s 2024-03-18T15:02:22.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 891711 9.41 MB/sec execute 383 sec latency 25.698 ms 2024-03-18T15:02:23.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 894040 9.42 MB/sec execute 384 sec latency 21.359 ms 2024-03-18T15:02:24.017 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:23 smithi174 ceph-mon[36787]: pgmap v288: 97 pgs: 97 active+clean; 254 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.9 MiB/s wr, 403 op/s 2024-03-18T15:02:24.028 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:23 smithi077 ceph-mon[35865]: pgmap v288: 97 pgs: 97 active+clean; 254 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.9 MiB/s wr, 403 op/s 2024-03-18T15:02:24.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:23 smithi076 ceph-mon[32205]: pgmap v288: 97 pgs: 97 active+clean; 254 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.9 MiB/s wr, 403 op/s 2024-03-18T15:02:24.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 895437 9.41 MB/sec execute 385 sec latency 15.840 ms 2024-03-18T15:02:25.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 897852 9.42 MB/sec execute 386 sec latency 25.269 ms 2024-03-18T15:02:26.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:25 smithi076 ceph-mon[32205]: pgmap v289: 97 pgs: 97 active+clean; 254 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-03-18T15:02:26.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:25 smithi174 ceph-mon[36787]: pgmap v289: 97 pgs: 97 active+clean; 254 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-03-18T15:02:26.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:25 smithi077 ceph-mon[35865]: pgmap v289: 97 pgs: 97 active+clean; 254 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-03-18T15:02:26.755 INFO:tasks.workunit.client.0.smithi076.stdout: 1 898910 9.41 MB/sec execute 387 sec latency 46.396 ms 2024-03-18T15:02:27.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 900959 9.41 MB/sec execute 388 sec latency 54.743 ms 2024-03-18T15:02:28.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:27 smithi076 ceph-mon[32205]: pgmap v290: 97 pgs: 97 active+clean; 254 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-03-18T15:02:28.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:27 smithi174 ceph-mon[36787]: pgmap v290: 97 pgs: 97 active+clean; 254 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-03-18T15:02:28.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:27 smithi077 ceph-mon[35865]: pgmap v290: 97 pgs: 97 active+clean; 254 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-03-18T15:02:28.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 902416 9.41 MB/sec execute 389 sec latency 18.538 ms 2024-03-18T15:02:29.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 904662 9.42 MB/sec execute 390 sec latency 17.125 ms 2024-03-18T15:02:30.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:29 smithi076 ceph-mon[32205]: pgmap v291: 97 pgs: 97 active+clean; 261 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.3 MiB/s wr, 348 op/s 2024-03-18T15:02:30.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:29 smithi174 ceph-mon[36787]: pgmap v291: 97 pgs: 97 active+clean; 261 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.3 MiB/s wr, 348 op/s 2024-03-18T15:02:30.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:29 smithi077 ceph-mon[35865]: pgmap v291: 97 pgs: 97 active+clean; 261 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.3 MiB/s wr, 348 op/s 2024-03-18T15:02:30.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 905957 9.41 MB/sec execute 391 sec latency 22.161 ms 2024-03-18T15:02:31.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:30 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:02:31.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:30 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:02:31.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:30 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:02:31.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 908057 9.41 MB/sec execute 392 sec latency 21.208 ms 2024-03-18T15:02:32.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:31 smithi076 ceph-mon[32205]: pgmap v292: 97 pgs: 97 active+clean; 261 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 246 op/s 2024-03-18T15:02:32.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:31 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:02:32.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:31 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:02:32.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:31 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:02:32.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:31 smithi174 ceph-mon[36787]: pgmap v292: 97 pgs: 97 active+clean; 261 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 246 op/s 2024-03-18T15:02:32.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:31 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:02:32.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:31 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:02:32.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:31 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:02:32.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:31 smithi077 ceph-mon[35865]: pgmap v292: 97 pgs: 97 active+clean; 261 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 246 op/s 2024-03-18T15:02:32.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:31 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:02:32.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:31 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:02:32.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:31 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:02:32.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 909293 9.41 MB/sec execute 393 sec latency 28.572 ms 2024-03-18T15:02:33.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 911039 9.40 MB/sec execute 394 sec latency 22.896 ms 2024-03-18T15:02:34.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:33 smithi076 ceph-mon[32205]: pgmap v293: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 340 op/s 2024-03-18T15:02:34.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:33 smithi174 ceph-mon[36787]: pgmap v293: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 340 op/s 2024-03-18T15:02:34.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:33 smithi077 ceph-mon[35865]: pgmap v293: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 340 op/s 2024-03-18T15:02:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.276+0000 7ffa498e8640 1 -- 172.21.15.76:0/708272535 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa441078b0 msgr2=0x7ffa44107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.276+0000 7ffa498e8640 1 --2- 172.21.15.76:0/708272535 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa441078b0 0x7ffa44107c90 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ffa34009930 tx=0x7ffa3402bc30 comp rx=0 tx=0).stop 2024-03-18T15:02:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.276+0000 7ffa498e8640 1 -- 172.21.15.76:0/708272535 shutdown_connections 2024-03-18T15:02:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.276+0000 7ffa498e8640 1 --2- 172.21.15.76:0/708272535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa44069800 0x7ffa4410e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.276+0000 7ffa498e8640 1 --2- 172.21.15.76:0/708272535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa44068e60 0x7ffa440692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.276+0000 7ffa498e8640 1 --2- 172.21.15.76:0/708272535 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa441078b0 0x7ffa44107c90 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.276+0000 7ffa498e8640 1 -- 172.21.15.76:0/708272535 >> 172.21.15.76:0/708272535 conn(0x7ffa44077530 msgr2=0x7ffa44077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.276+0000 7ffa498e8640 1 -- 172.21.15.76:0/708272535 shutdown_connections 2024-03-18T15:02:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.276+0000 7ffa498e8640 1 -- 172.21.15.76:0/708272535 wait complete. 2024-03-18T15:02:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.276+0000 7ffa498e8640 1 Processor -- start 2024-03-18T15:02:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa498e8640 1 -- start start 2024-03-18T15:02:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa498e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa44068e60 0x7ffa44198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa498e8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa44069800 0x7ffa44198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa498e8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa441078b0 0x7ffa4419c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa498e8640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffa44111ba0 con 0x7ffa441078b0 2024-03-18T15:02:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa498e8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffa44111a20 con 0x7ffa44068e60 2024-03-18T15:02:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa498e8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffa44111d20 con 0x7ffa44069800 2024-03-18T15:02:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa437fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa441078b0 0x7ffa4419c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa42ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa44068e60 0x7ffa44198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa437fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa441078b0 0x7ffa4419c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40258/0 (socket says 172.21.15.76:40258) 2024-03-18T15:02:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa437fe640 1 -- 172.21.15.76:0/1072361651 learned_addr learned my addr 172.21.15.76:0/1072361651 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa427fc640 1 --2- 172.21.15.76:0/1072361651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa44069800 0x7ffa44198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa437fe640 1 -- 172.21.15.76:0/1072361651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa44069800 msgr2=0x7ffa44198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa437fe640 1 --2- 172.21.15.76:0/1072361651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa44069800 0x7ffa44198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa437fe640 1 -- 172.21.15.76:0/1072361651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa44068e60 msgr2=0x7ffa44198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa437fe640 1 --2- 172.21.15.76:0/1072361651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa44068e60 0x7ffa44198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa437fe640 1 -- 172.21.15.76:0/1072361651 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4419cff0 con 0x7ffa441078b0 2024-03-18T15:02:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa437fe640 1 --2- 172.21.15.76:0/1072361651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa441078b0 0x7ffa4419c8c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00ea60 tx=0x7ffa3c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.277+0000 7ffa488e6640 1 -- 172.21.15.76:0/1072361651 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c00ce40 con 0x7ffa441078b0 2024-03-18T15:02:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.278+0000 7ffa488e6640 1 -- 172.21.15.76:0/1072361651 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa3c005020 con 0x7ffa441078b0 2024-03-18T15:02:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.278+0000 7ffa498e8640 1 -- 172.21.15.76:0/1072361651 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa4419d2e0 con 0x7ffa441078b0 2024-03-18T15:02:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.278+0000 7ffa488e6640 1 -- 172.21.15.76:0/1072361651 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c010690 con 0x7ffa441078b0 2024-03-18T15:02:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.278+0000 7ffa498e8640 1 -- 172.21.15.76:0/1072361651 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa441a4b70 con 0x7ffa441078b0 2024-03-18T15:02:34.287 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.280+0000 7ffa488e6640 1 -- 172.21.15.76:0/1072361651 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7ffa3c005520 con 0x7ffa441078b0 2024-03-18T15:02:34.287 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.280+0000 7ffa4c5ad740 1 Processor -- start 2024-03-18T15:02:34.288 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.280+0000 7ffa4c5ad740 1 -- start start 2024-03-18T15:02:34.288 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.280+0000 7ffa4c5ad740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259130 0x55f676259510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.288 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.280+0000 7ffa488e6640 1 --2- 172.21.15.76:0/1072361651 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ffa2c0a2a90 0x7ffa2c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.288 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.280+0000 7ffa4c5ad740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676259a50 0x55f6762622e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.288 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.280+0000 7ffa4c5ad740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 0x55f676264c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.288 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.280+0000 7ffa4c5ad740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f6761bda40 con 0x55f676262820 2024-03-18T15:02:34.288 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.280+0000 7ffa4c5ad740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f676166920 con 0x55f676259a50 2024-03-18T15:02:34.288 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.280+0000 7ffa4c5ad740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f6761b9ae0 con 0x55f676259130 2024-03-18T15:02:34.288 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.280+0000 7ffa42ffd640 1 --2- 172.21.15.76:0/1072361651 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ffa2c0a2a90 0x7ffa2c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.288 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.280+0000 7ffa437fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 0x55f676264c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.288 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.280+0000 7ffa437fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 0x55f676264c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40274/0 (socket says 172.21.15.76:40274) 2024-03-18T15:02:34.288 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.280+0000 7ffa437fe640 1 -- 172.21.15.76:0/3725650231 learned_addr learned my addr 172.21.15.76:0/3725650231 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:34.288 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.280+0000 7ffa427fc640 1 --2- 172.21.15.76:0/3725650231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676259a50 0x55f6762622e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.288 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.281+0000 7ffa488e6640 1 -- 172.21.15.76:0/1072361651 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7ffa3c0cf1b0 con 0x7ffa441078b0 2024-03-18T15:02:34.288 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.281+0000 7ffa427fc640 1 --2- 172.21.15.76:0/3725650231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259130 0x55f676259510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.288 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.281+0000 7ffa437fe640 1 -- 172.21.15.76:0/3725650231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259130 msgr2=0x55f676259510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.281+0000 7ffa437fe640 1 --2- 172.21.15.76:0/3725650231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259130 0x55f676259510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.281+0000 7ffa437fe640 1 -- 172.21.15.76:0/3725650231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676259a50 msgr2=0x55f6762622e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.281+0000 7ffa437fe640 1 --2- 172.21.15.76:0/3725650231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676259a50 0x55f6762622e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.281+0000 7ffa437fe640 1 -- 172.21.15.76:0/3725650231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f67617e460 con 0x55f676262820 2024-03-18T15:02:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.281+0000 7ffa427fc640 1 --2- 172.21.15.76:0/3725650231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259130 0x55f676259510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:02:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.281+0000 7ffa42ffd640 1 --2- 172.21.15.76:0/1072361651 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ffa2c0a2a90 0x7ffa2c0a4f50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffa34005ab0 tx=0x7ffa34000bc0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.281+0000 7ffa437fe640 1 --2- 172.21.15.76:0/3725650231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 0x55f676264c10 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00c760 tx=0x7ffa3c005920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d85d249035ec65 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.281+0000 7ffa29ffb640 1 -- 172.21.15.76:0/3725650231 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c0b48f0 con 0x55f676262820 2024-03-18T15:02:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.281+0000 7ffa29ffb640 1 -- 172.21.15.76:0/3725650231 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa3c09bf30 con 0x55f676262820 2024-03-18T15:02:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.281+0000 7ffa29ffb640 1 -- 172.21.15.76:0/3725650231 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c0ccba0 con 0x55f676262820 2024-03-18T15:02:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.282+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/3725650231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 msgr2=0x55f676264c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.282+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/3725650231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 0x55f676264c10 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00c760 tx=0x7ffa3c005920 comp rx=0 tx=0).stop 2024-03-18T15:02:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.282+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/3725650231 shutdown_connections 2024-03-18T15:02:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.282+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/3725650231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 0x55f676264c10 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.282+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/3725650231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676259a50 0x55f6762622e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.282+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/3725650231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259130 0x55f676259510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.282+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/3725650231 >> 172.21.15.76:0/3725650231 conn(0x55f676253a50 msgr2=0x55f676253e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:34.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.282+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/3725650231 shutdown_connections 2024-03-18T15:02:34.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.282+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/3725650231 wait complete. 2024-03-18T15:02:34.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.282+0000 7ffa4c5ad740 1 -- start start 2024-03-18T15:02:34.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.282+0000 7ffa4c5ad740 1 Processor -- start 2024-03-18T15:02:34.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa4c5ad740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676259130 0x55f67625f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa4c5ad740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259a50 0x55f67625fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa4c5ad740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 0x55f676260340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa4c5ad740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f676265450 con 0x55f676262820 2024-03-18T15:02:34.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa4c5ad740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f676179d70 con 0x55f676259130 2024-03-18T15:02:34.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa4c5ad740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f6762655d0 con 0x55f676259a50 2024-03-18T15:02:34.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa4c5ad740 10 client.0 _ll_register_callbacks cb 0x55f675ed1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:02:34.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa42ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676259130 0x55f67625f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.290 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa437fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 0x55f676260340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa42ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676259130 0x55f67625f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:58714/0 (socket says 172.21.15.76:58714) 2024-03-18T15:02:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa42ffd640 1 -- 172.21.15.76:0/4013854012 learned_addr learned my addr 172.21.15.76:0/4013854012 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa427fc640 1 --2- 172.21.15.76:0/4013854012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259a50 0x55f67625fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa437fe640 1 -- 172.21.15.76:0/4013854012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259a50 msgr2=0x55f67625fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa437fe640 1 --2- 172.21.15.76:0/4013854012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259a50 0x55f67625fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa437fe640 1 -- 172.21.15.76:0/4013854012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676259130 msgr2=0x55f67625f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa437fe640 1 --2- 172.21.15.76:0/4013854012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676259130 0x55f67625f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa437fe640 1 -- 172.21.15.76:0/4013854012 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f675ed2210 con 0x55f676262820 2024-03-18T15:02:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa42ffd640 1 --2- 172.21.15.76:0/4013854012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676259130 0x55f67625f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:02:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa437fe640 1 --2- 172.21.15.76:0/4013854012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 0x55f676260340 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0a4190 tx=0x7ffa3c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a56dd7cb87303a34 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:02:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.283+0000 7ffa22ffd640 1 -- 172.21.15.76:0/4013854012 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c084020 con 0x55f676262820 2024-03-18T15:02:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.284+0000 7ffa22ffd640 1 -- 172.21.15.76:0/4013854012 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7ffa3c097ea0 con 0x55f676262820 2024-03-18T15:02:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.284+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/4013854012 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa3c09bf30 con 0x55f676262820 2024-03-18T15:02:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.284+0000 7ffa22ffd640 1 -- 172.21.15.76:0/4013854012 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7ffa3c09b830 con 0x55f676262820 2024-03-18T15:02:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.284+0000 7ffa4c5ad740 10 client.16596 fetch_fsmap learned FSMap version 41 2024-03-18T15:02:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.284+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/4013854012 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f675f9e620 con 0x55f676262820 2024-03-18T15:02:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.284+0000 7ffa22ffd640 1 -- 172.21.15.76:0/4013854012 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa3c09bf30 con 0x55f676262820 2024-03-18T15:02:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.284+0000 7ffa22ffd640 1 -- 172.21.15.76:0/4013854012 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7ffa3c09bb20 con 0x55f676262820 2024-03-18T15:02:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.284+0000 7ffa4c5ad740 10 client.16596 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:02:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.284+0000 7ffa4c5ad740 10 client.16596 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:02:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.284+0000 7ffa4c5ad740 20 client.16596 populate_metadata read hostname 'smithi076' 2024-03-18T15:02:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.284+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/4013854012 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f676272890 0x55f676274c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.284+0000 7ffa4c5ad740 4 client.16596 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:02:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.284+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/4013854012 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6761caf80 con 0x55f676272890 2024-03-18T15:02:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.284+0000 7ffa42ffd640 1 --2- 172.21.15.76:0/4013854012 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f676272890 0x55f676274c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.285+0000 7ffa22ffd640 10 client.16596 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:02:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.285+0000 7ffa42ffd640 1 --2- 172.21.15.76:0/4013854012 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f676272890 0x55f676274c80 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c74da585d8288603 server_cookie=32a63e80360a89e6 in_seq=0 out_seq=0 2024-03-18T15:02:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.286+0000 7ffa22ffd640 1 -- 172.21.15.76:0/4013854012 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f6761caf80 con 0x55f676272890 2024-03-18T15:02:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.286+0000 7ffa22ffd640 10 client.16596 handle_command_reply: tid=0 2024-03-18T15:02:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.286+0000 7ffa4c5ad740 1 client.16596 shutdown 2024-03-18T15:02:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.286+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/4013854012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 msgr2=0x55f676260340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.286+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/4013854012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 0x55f676260340 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0a4190 tx=0x7ffa3c00c890 comp rx=0 tx=0).stop 2024-03-18T15:02:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.286+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/4013854012 shutdown_connections 2024-03-18T15:02:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.286+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/4013854012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259a50 0x55f67625fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.286+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/4013854012 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f676272890 0x55f676274c80 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.286+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/4013854012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 0x55f676260340 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.286+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/4013854012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676259130 0x55f67625f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.286+0000 7ffa22ffd640 0 client.16596 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:02:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.286+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/4013854012 >> 172.21.15.76:0/4013854012 conn(0x55f676253a50 msgr2=0x55f6762557a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.286+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/4013854012 shutdown_connections 2024-03-18T15:02:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.286+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/4013854012 wait complete. 2024-03-18T15:02:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.286+0000 7ffa4c5ad740 20 client.16596 trim_cache size 0 max 16384 2024-03-18T15:02:34.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.301+0000 7ffa4c5ad740 1 Processor -- start 2024-03-18T15:02:34.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.301+0000 7ffa4c5ad740 1 -- start start 2024-03-18T15:02:34.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.301+0000 7ffa4c5ad740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676259130 0x55f676261ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.301+0000 7ffa4c5ad740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259a50 0x55f67625ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.301+0000 7ffa4c5ad740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 0x55f67625cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.301+0000 7ffa4c5ad740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffa441afcf0 con 0x55f676262820 2024-03-18T15:02:34.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.301+0000 7ffa4c5ad740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffa441ad5c0 con 0x55f676259130 2024-03-18T15:02:34.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.301+0000 7ffa4c5ad740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffa441ab430 con 0x55f676259a50 2024-03-18T15:02:34.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.301+0000 7ffa437fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 0x55f67625cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.301+0000 7ffa437fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 0x55f67625cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40294/0 (socket says 172.21.15.76:40294) 2024-03-18T15:02:34.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.301+0000 7ffa437fe640 1 -- 172.21.15.76:0/3681368523 learned_addr learned my addr 172.21.15.76:0/3681368523 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:34.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.301+0000 7ffa42ffd640 1 --2- 172.21.15.76:0/3681368523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676259130 0x55f676261ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.301+0000 7ffa427fc640 1 --2- 172.21.15.76:0/3681368523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259a50 0x55f67625ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.303 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.302+0000 7ffa437fe640 1 -- 172.21.15.76:0/3681368523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259a50 msgr2=0x55f67625ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:34.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.302+0000 7ffa437fe640 1 --2- 172.21.15.76:0/3681368523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259a50 0x55f67625ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.302+0000 7ffa437fe640 1 -- 172.21.15.76:0/3681368523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676259130 msgr2=0x55f676261ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:34.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.302+0000 7ffa437fe640 1 --2- 172.21.15.76:0/3681368523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676259130 0x55f676261ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.302+0000 7ffa437fe640 1 -- 172.21.15.76:0/3681368523 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f676259880 con 0x55f676262820 2024-03-18T15:02:34.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.302+0000 7ffa437fe640 1 --2- 172.21.15.76:0/3681368523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 0x55f67625cb20 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0049d0 tx=0x7ffa3c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cad303809bd4950 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:34.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.302+0000 7ffa297fa640 1 -- 172.21.15.76:0/3681368523 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c084070 con 0x55f676262820 2024-03-18T15:02:34.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.302+0000 7ffa297fa640 1 -- 172.21.15.76:0/3681368523 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa3c0a3fd0 con 0x55f676262820 2024-03-18T15:02:34.304 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.302+0000 7ffa297fa640 1 -- 172.21.15.76:0/3681368523 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c0afc10 con 0x55f676262820 2024-03-18T15:02:34.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/3681368523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 msgr2=0x55f67625cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:34.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/3681368523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 0x55f67625cb20 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0049d0 tx=0x7ffa3c00c900 comp rx=0 tx=0).stop 2024-03-18T15:02:34.307 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/3681368523 shutdown_connections 2024-03-18T15:02:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/3681368523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259a50 0x55f67625ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/3681368523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676262820 0x55f67625cb20 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/3681368523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676259130 0x55f676261ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/3681368523 >> 172.21.15.76:0/3681368523 conn(0x55f676253a50 msgr2=0x55f67624ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/3681368523 shutdown_connections 2024-03-18T15:02:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/3681368523 wait complete. 2024-03-18T15:02:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 1 -- start start 2024-03-18T15:02:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 1 Processor -- start 2024-03-18T15:02:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676259130 0x55f676277900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259a50 0x55f676277e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676262820 0x55f676274e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffa441aba80 con 0x55f676259130 2024-03-18T15:02:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffa441acef0 con 0x55f676262820 2024-03-18T15:02:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffa441ac150 con 0x55f676259a50 2024-03-18T15:02:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.303+0000 7ffa4c5ad740 10 client.0 _ll_register_callbacks cb 0x7ffa38009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:02:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa42ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676259130 0x55f676277900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa42ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676259130 0x55f676277900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40296/0 (socket says 172.21.15.76:40296) 2024-03-18T15:02:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa42ffd640 1 -- 172.21.15.76:0/1826184525 learned_addr learned my addr 172.21.15.76:0/1826184525 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa437fe640 1 --2- 172.21.15.76:0/1826184525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676262820 0x55f676274e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa427fc640 1 --2- 172.21.15.76:0/1826184525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259a50 0x55f676277e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa42ffd640 1 -- 172.21.15.76:0/1826184525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259a50 msgr2=0x55f676277e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa42ffd640 1 --2- 172.21.15.76:0/1826184525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259a50 0x55f676277e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa42ffd640 1 -- 172.21.15.76:0/1826184525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676262820 msgr2=0x55f676274e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa42ffd640 1 --2- 172.21.15.76:0/1826184525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676262820 0x55f676274e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa42ffd640 1 -- 172.21.15.76:0/1826184525 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f676275540 con 0x55f676259130 2024-03-18T15:02:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa437fe640 1 --2- 172.21.15.76:0/1826184525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676262820 0x55f676274e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:02:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa42ffd640 1 --2- 172.21.15.76:0/1826184525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676259130 0x55f676277900 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7ffa34036040 tx=0x7ffa3402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9112f3b4a4c1a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:02:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa22ffd640 1 -- 172.21.15.76:0/1826184525 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa34055300 con 0x55f676259130 2024-03-18T15:02:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa22ffd640 1 -- 172.21.15.76:0/1826184525 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7ffa340554a0 con 0x55f676259130 2024-03-18T15:02:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/1826184525 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f6761c52d0 con 0x55f676259130 2024-03-18T15:02:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa22ffd640 1 -- 172.21.15.76:0/1826184525 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7ffa340556a0 con 0x55f676259130 2024-03-18T15:02:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa4c5ad740 10 client.16602 fetch_fsmap learned FSMap version 41 2024-03-18T15:02:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/1826184525 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f6762dc800 con 0x55f676259130 2024-03-18T15:02:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.304+0000 7ffa22ffd640 1 -- 172.21.15.76:0/1826184525 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f6761c52d0 con 0x55f676259130 2024-03-18T15:02:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.305+0000 7ffa22ffd640 1 -- 172.21.15.76:0/1826184525 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7ffa34005250 con 0x55f676259130 2024-03-18T15:02:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.305+0000 7ffa4c5ad740 10 client.16602 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:02:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.305+0000 7ffa4c5ad740 10 client.16602 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:02:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.305+0000 7ffa4c5ad740 20 client.16602 populate_metadata read hostname 'smithi076' 2024-03-18T15:02:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.305+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/1826184525 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f6762dcf30 0x55f6762df350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.305+0000 7ffa4c5ad740 4 client.16602 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:02:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.305+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/1826184525 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f676143770 con 0x55f6762dcf30 2024-03-18T15:02:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.305+0000 7ffa427fc640 1 --2- 172.21.15.76:0/1826184525 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f6762dcf30 0x55f6762df350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.305+0000 7ffa427fc640 1 --2- 172.21.15.76:0/1826184525 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f6762dcf30 0x55f6762df350 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf7d6f80aa5461f7 server_cookie=76da1032f15489e6 in_seq=0 out_seq=0 2024-03-18T15:02:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.305+0000 7ffa22ffd640 10 client.16602 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:02:34.386 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.385+0000 7ffa22ffd640 1 -- 172.21.15.76:0/1826184525 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f676143770 con 0x55f6762dcf30 2024-03-18T15:02:34.386 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.385+0000 7ffa22ffd640 10 client.16602 handle_command_reply: tid=0 2024-03-18T15:02:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.385+0000 7ffa4c5ad740 1 client.16602 shutdown 2024-03-18T15:02:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.385+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/1826184525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676259130 msgr2=0x55f676277900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.385+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/1826184525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676259130 0x55f676277900 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7ffa34036040 tx=0x7ffa3402bc30 comp rx=0 tx=0).stop 2024-03-18T15:02:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.385+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/1826184525 shutdown_connections 2024-03-18T15:02:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.385+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/1826184525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f676259a50 0x55f676277e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.385+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/1826184525 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f6762dcf30 0x55f6762df350 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.385+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/1826184525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f676262820 0x55f676274e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.385+0000 7ffa4c5ad740 1 --2- 172.21.15.76:0/1826184525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f676259130 0x55f676277900 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.385+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/1826184525 >> 172.21.15.76:0/1826184525 conn(0x55f676253a50 msgr2=0x55f676252320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.385+0000 7ffa22ffd640 0 client.16602 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:02:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.385+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/1826184525 shutdown_connections 2024-03-18T15:02:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.385+0000 7ffa4c5ad740 1 -- 172.21.15.76:0/1826184525 wait complete. 2024-03-18T15:02:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.385+0000 7ffa4c5ad740 20 client.16602 trim_cache size 0 max 16384 2024-03-18T15:02:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:02:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:02:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:02:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:02:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.386+0000 7ffa498e8640 1 -- 172.21.15.76:0/1072361651 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ffa2c0a2a90 msgr2=0x7ffa2c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.386+0000 7ffa498e8640 1 --2- 172.21.15.76:0/1072361651 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ffa2c0a2a90 0x7ffa2c0a4f50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffa34005ab0 tx=0x7ffa34000bc0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.386+0000 7ffa498e8640 1 -- 172.21.15.76:0/1072361651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa441078b0 msgr2=0x7ffa4419c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.386+0000 7ffa498e8640 1 --2- 172.21.15.76:0/1072361651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa441078b0 0x7ffa4419c8c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00ea60 tx=0x7ffa3c00ef20 comp rx=0 tx=0).stop 2024-03-18T15:02:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.386+0000 7ffa498e8640 1 -- 172.21.15.76:0/1072361651 shutdown_connections 2024-03-18T15:02:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.386+0000 7ffa498e8640 1 --2- 172.21.15.76:0/1072361651 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ffa2c0a2a90 0x7ffa2c0a4f50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.386+0000 7ffa498e8640 1 --2- 172.21.15.76:0/1072361651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa44069800 0x7ffa44198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.386+0000 7ffa498e8640 1 --2- 172.21.15.76:0/1072361651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa441078b0 0x7ffa4419c8c0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.386+0000 7ffa498e8640 1 --2- 172.21.15.76:0/1072361651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa44068e60 0x7ffa44198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.386+0000 7ffa498e8640 1 -- 172.21.15.76:0/1072361651 >> 172.21.15.76:0/1072361651 conn(0x7ffa44077530 msgr2=0x7ffa440fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.386+0000 7ffa498e8640 1 -- 172.21.15.76:0/1072361651 shutdown_connections 2024-03-18T15:02:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:34.386+0000 7ffa498e8640 1 -- 172.21.15.76:0/1072361651 wait complete. 2024-03-18T15:02:34.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 912763 9.41 MB/sec execute 395 sec latency 38.311 ms 2024-03-18T15:02:35.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 914077 9.40 MB/sec execute 396 sec latency 22.268 ms 2024-03-18T15:02:36.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:35 smithi076 ceph-mon[32205]: pgmap v294: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 197 op/s 2024-03-18T15:02:36.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:35 smithi174 ceph-mon[36787]: pgmap v294: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 197 op/s 2024-03-18T15:02:36.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:35 smithi077 ceph-mon[35865]: pgmap v294: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 197 op/s 2024-03-18T15:02:36.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 916218 9.41 MB/sec execute 397 sec latency 21.917 ms 2024-03-18T15:02:37.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 916895 9.41 MB/sec execute 398 sec latency 52.313 ms 2024-03-18T15:02:38.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:37 smithi076 ceph-mon[32205]: pgmap v295: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 197 op/s 2024-03-18T15:02:38.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:37 smithi174 ceph-mon[36787]: pgmap v295: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 197 op/s 2024-03-18T15:02:38.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:37 smithi077 ceph-mon[35865]: pgmap v295: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 197 op/s 2024-03-18T15:02:38.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 918200 9.43 MB/sec execute 399 sec latency 21.125 ms 2024-03-18T15:02:39.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 920468 9.44 MB/sec execute 400 sec latency 19.424 ms 2024-03-18T15:02:40.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:39 smithi076 ceph-mon[32205]: pgmap v296: 97 pgs: 97 active+clean; 262 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 331 op/s 2024-03-18T15:02:40.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:39 smithi174 ceph-mon[36787]: pgmap v296: 97 pgs: 97 active+clean; 262 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 331 op/s 2024-03-18T15:02:40.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:39 smithi077 ceph-mon[35865]: pgmap v296: 97 pgs: 97 active+clean; 262 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 331 op/s 2024-03-18T15:02:40.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 921672 9.43 MB/sec execute 401 sec latency 21.912 ms 2024-03-18T15:02:41.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 923929 9.44 MB/sec execute 402 sec latency 21.104 ms 2024-03-18T15:02:42.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:41 smithi076 ceph-mon[32205]: pgmap v297: 97 pgs: 97 active+clean; 262 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 228 op/s 2024-03-18T15:02:42.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:41 smithi174 ceph-mon[36787]: pgmap v297: 97 pgs: 97 active+clean; 262 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 228 op/s 2024-03-18T15:02:42.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:41 smithi077 ceph-mon[35865]: pgmap v297: 97 pgs: 97 active+clean; 262 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 228 op/s 2024-03-18T15:02:42.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 924703 9.42 MB/sec execute 403 sec latency 67.977 ms 2024-03-18T15:02:43.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 926964 9.43 MB/sec execute 404 sec latency 21.483 ms 2024-03-18T15:02:44.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:43 smithi076 ceph-mon[32205]: pgmap v298: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 330 op/s 2024-03-18T15:02:44.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:43 smithi174 ceph-mon[36787]: pgmap v298: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 330 op/s 2024-03-18T15:02:44.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:43 smithi077 ceph-mon[35865]: pgmap v298: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 330 op/s 2024-03-18T15:02:44.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 928018 9.42 MB/sec execute 405 sec latency 71.472 ms 2024-03-18T15:02:45.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 930109 9.42 MB/sec execute 406 sec latency 19.474 ms 2024-03-18T15:02:46.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:45 smithi076 ceph-mon[32205]: pgmap v299: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 236 op/s 2024-03-18T15:02:46.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:45 smithi174 ceph-mon[36787]: pgmap v299: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 236 op/s 2024-03-18T15:02:46.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:45 smithi077 ceph-mon[35865]: pgmap v299: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 236 op/s 2024-03-18T15:02:46.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 931447 9.42 MB/sec execute 407 sec latency 26.116 ms 2024-03-18T15:02:47.480 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.479+0000 7fec5c9f3640 1 -- 172.21.15.76:0/647187846 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec5806a900 msgr2=0x7fec5810e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:47.480 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.479+0000 7fec5c9f3640 1 --2- 172.21.15.76:0/647187846 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec5806a900 0x7fec5810e680 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fec50009cd0 tx=0x7fec5002c0f0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.479+0000 7fec5c9f3640 1 -- 172.21.15.76:0/647187846 shutdown_connections 2024-03-18T15:02:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.479+0000 7fec5c9f3640 1 --2- 172.21.15.76:0/647187846 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec58069f80 0x7fec5806a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.479+0000 7fec5c9f3640 1 --2- 172.21.15.76:0/647187846 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec5806a900 0x7fec5810e680 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.479+0000 7fec5c9f3640 1 --2- 172.21.15.76:0/647187846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec581077c0 0x7fec58107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.479+0000 7fec5c9f3640 1 -- 172.21.15.76:0/647187846 >> 172.21.15.76:0/647187846 conn(0x7fec580776c0 msgr2=0x7fec58077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.480+0000 7fec5c9f3640 1 -- 172.21.15.76:0/647187846 shutdown_connections 2024-03-18T15:02:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.480+0000 7fec5c9f3640 1 -- 172.21.15.76:0/647187846 wait complete. 2024-03-18T15:02:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.480+0000 7fec5c9f3640 1 Processor -- start 2024-03-18T15:02:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.480+0000 7fec5c9f3640 1 -- start start 2024-03-18T15:02:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.481+0000 7fec5c9f3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec58069f80 0x7fec5819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.481+0000 7fec5c9f3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec5806a900 0x7fec5819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.481+0000 7fec5c9f3640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec581077c0 0x7fec5819ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.481+0000 7fec5c9f3640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fec58111b80 con 0x7fec581077c0 2024-03-18T15:02:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.481+0000 7fec5c9f3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fec58111a00 con 0x7fec5806a900 2024-03-18T15:02:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.481+0000 7fec5c9f3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fec58111d00 con 0x7fec58069f80 2024-03-18T15:02:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.481+0000 7fec56d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec581077c0 0x7fec5819ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.481+0000 7fec56d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec581077c0 0x7fec5819ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34300/0 (socket says 172.21.15.76:34300) 2024-03-18T15:02:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.481+0000 7fec56d9d640 1 -- 172.21.15.76:0/2158752099 learned_addr learned my addr 172.21.15.76:0/2158752099 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.481+0000 7fec5659c640 1 --2- 172.21.15.76:0/2158752099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec58069f80 0x7fec5819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.481+0000 7fec55d9b640 1 --2- 172.21.15.76:0/2158752099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec5806a900 0x7fec5819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.481+0000 7fec55d9b640 1 -- 172.21.15.76:0/2158752099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec58069f80 msgr2=0x7fec5819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.481+0000 7fec55d9b640 1 --2- 172.21.15.76:0/2158752099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec58069f80 0x7fec5819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.481+0000 7fec55d9b640 1 -- 172.21.15.76:0/2158752099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec581077c0 msgr2=0x7fec5819ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.481+0000 7fec55d9b640 1 --2- 172.21.15.76:0/2158752099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec581077c0 0x7fec5819ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.481+0000 7fec55d9b640 1 -- 172.21.15.76:0/2158752099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec5819f1b0 con 0x7fec5806a900 2024-03-18T15:02:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.481+0000 7fec55d9b640 1 --2- 172.21.15.76:0/2158752099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec5806a900 0x7fec5819a700 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00e9f0 tx=0x7fec4c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.482+0000 7fec477fe640 1 -- 172.21.15.76:0/2158752099 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c00ccc0 con 0x7fec5806a900 2024-03-18T15:02:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.482+0000 7fec477fe640 1 -- 172.21.15.76:0/2158752099 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec4c005020 con 0x7fec5806a900 2024-03-18T15:02:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.482+0000 7fec5c9f3640 1 -- 172.21.15.76:0/2158752099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec5819f4a0 con 0x7fec5806a900 2024-03-18T15:02:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.482+0000 7fec477fe640 1 -- 172.21.15.76:0/2158752099 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c0049e0 con 0x7fec5806a900 2024-03-18T15:02:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.482+0000 7fec5c9f3640 1 -- 172.21.15.76:0/2158752099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec581a6d30 con 0x7fec5806a900 2024-03-18T15:02:47.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.485+0000 7fec5f5d9740 1 Processor -- start 2024-03-18T15:02:47.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.485+0000 7fec5f5d9740 1 -- start start 2024-03-18T15:02:47.493 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.485+0000 7fec477fe640 1 -- 172.21.15.76:0/2158752099 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fec4c005560 con 0x7fec5806a900 2024-03-18T15:02:47.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.485+0000 7fec5f5d9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3db130 0x55f93b3db510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.485+0000 7fec5f5d9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3dba50 0x55f93b3e42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.485+0000 7fec5f5d9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3e4820 0x55f93b3e6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.485+0000 7fec5f5d9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f93b33fa40 con 0x55f93b3db130 2024-03-18T15:02:47.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.485+0000 7fec5f5d9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f93b2e8920 con 0x55f93b3dba50 2024-03-18T15:02:47.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.485+0000 7fec5f5d9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f93b33bae0 con 0x55f93b3e4820 2024-03-18T15:02:47.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.485+0000 7fec56d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3db130 0x55f93b3db510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.485+0000 7fec5659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3dba50 0x55f93b3e42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.485+0000 7fec56d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3db130 0x55f93b3db510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34310/0 (socket says 172.21.15.76:34310) 2024-03-18T15:02:47.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.485+0000 7fec56d9d640 1 -- 172.21.15.76:0/2131989469 learned_addr learned my addr 172.21.15.76:0/2131989469 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:47.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.485+0000 7fec55d9b640 1 --2- 172.21.15.76:0/2131989469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3e4820 0x55f93b3e6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.486+0000 7fec56d9d640 1 -- 172.21.15.76:0/2131989469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3e4820 msgr2=0x55f93b3e6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:47.494 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.486+0000 7fec56d9d640 1 --2- 172.21.15.76:0/2131989469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3e4820 0x55f93b3e6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.486+0000 7fec56d9d640 1 -- 172.21.15.76:0/2131989469 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3dba50 msgr2=0x55f93b3e42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:47.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.486+0000 7fec56d9d640 1 --2- 172.21.15.76:0/2131989469 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3dba50 0x55f93b3e42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.486+0000 7fec56d9d640 1 -- 172.21.15.76:0/2131989469 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f93b300460 con 0x55f93b3db130 2024-03-18T15:02:47.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.486+0000 7fec477fe640 1 --2- 172.21.15.76:0/2158752099 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fec400a2a10 0x7fec400a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.486+0000 7fec56d9d640 1 --2- 172.21.15.76:0/2131989469 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3db130 0x55f93b3db510 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fec50037040 tx=0x7fec50005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a7e9034cc61dfaf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:47.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.486+0000 7fec5659c640 1 --2- 172.21.15.76:0/2158752099 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fec400a2a10 0x7fec400a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.486+0000 7fec477fe640 1 -- 172.21.15.76:0/2158752099 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7fec4c0cee10 con 0x7fec5806a900 2024-03-18T15:02:47.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.486+0000 7fec44ff9640 1 -- 172.21.15.76:0/2131989469 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec50043020 con 0x55f93b3db130 2024-03-18T15:02:47.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.486+0000 7fec44ff9640 1 -- 172.21.15.76:0/2131989469 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec5002cca0 con 0x55f93b3db130 2024-03-18T15:02:47.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.486+0000 7fec44ff9640 1 -- 172.21.15.76:0/2131989469 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec50039040 con 0x55f93b3db130 2024-03-18T15:02:47.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.487+0000 7fec5f5d9740 1 -- 172.21.15.76:0/2131989469 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3db130 msgr2=0x55f93b3db510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:47.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.487+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/2131989469 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3db130 0x55f93b3db510 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fec50037040 tx=0x7fec50005f80 comp rx=0 tx=0).stop 2024-03-18T15:02:47.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.487+0000 7fec5659c640 1 --2- 172.21.15.76:0/2158752099 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fec400a2a10 0x7fec400a4ed0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x55f93b3dabe0 tx=0x7fec48005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:47.495 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.487+0000 7fec5f5d9740 1 -- 172.21.15.76:0/2131989469 shutdown_connections 2024-03-18T15:02:47.496 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.487+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/2131989469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3e4820 0x55f93b3e6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.496 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.487+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/2131989469 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3dba50 0x55f93b3e42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.496 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.487+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/2131989469 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3db130 0x55f93b3db510 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.496 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.487+0000 7fec5f5d9740 1 -- 172.21.15.76:0/2131989469 >> 172.21.15.76:0/2131989469 conn(0x55f93b3d5a50 msgr2=0x55f93b3d5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:47.496 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.487+0000 7fec5f5d9740 1 -- 172.21.15.76:0/2131989469 shutdown_connections 2024-03-18T15:02:47.496 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.487+0000 7fec5f5d9740 1 -- 172.21.15.76:0/2131989469 wait complete. 2024-03-18T15:02:47.496 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec5f5d9740 1 -- start start 2024-03-18T15:02:47.496 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec5f5d9740 1 Processor -- start 2024-03-18T15:02:47.496 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec5f5d9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3db130 0x55f93b3e1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.496 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec5f5d9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3dba50 0x55f93b3e1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.496 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec5f5d9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3e4820 0x55f93b3e22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.496 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec5f5d9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f93b3e73f0 con 0x55f93b3e4820 2024-03-18T15:02:47.496 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec5f5d9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f93b2fbd70 con 0x55f93b3db130 2024-03-18T15:02:47.496 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec5f5d9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f93b3e7570 con 0x55f93b3dba50 2024-03-18T15:02:47.496 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec5f5d9740 10 client.0 _ll_register_callbacks cb 0x55f93b053a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:02:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec56d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3e4820 0x55f93b3e22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec56d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3e4820 0x55f93b3e22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34322/0 (socket says 172.21.15.76:34322) 2024-03-18T15:02:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec56d9d640 1 -- 172.21.15.76:0/132474604 learned_addr learned my addr 172.21.15.76:0/132474604 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec5659c640 1 --2- 172.21.15.76:0/132474604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3db130 0x55f93b3e1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec55d9b640 1 --2- 172.21.15.76:0/132474604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3dba50 0x55f93b3e1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec56d9d640 1 -- 172.21.15.76:0/132474604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3dba50 msgr2=0x55f93b3e1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec56d9d640 1 --2- 172.21.15.76:0/132474604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3dba50 0x55f93b3e1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec56d9d640 1 -- 172.21.15.76:0/132474604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3db130 msgr2=0x55f93b3e1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec56d9d640 1 --2- 172.21.15.76:0/132474604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3db130 0x55f93b3e1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.488+0000 7fec56d9d640 1 -- 172.21.15.76:0/132474604 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f93b054210 con 0x55f93b3e4820 2024-03-18T15:02:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.489+0000 7fec56d9d640 1 --2- 172.21.15.76:0/132474604 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3e4820 0x55f93b3e22d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fec50009a60 tx=0x7fec50003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2145c061cf9c261c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.489+0000 7fec3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:02:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.489+0000 7fec3dffb640 1 -- 172.21.15.76:0/132474604 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec50043030 con 0x55f93b3e4820 2024-03-18T15:02:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.489+0000 7fec3dffb640 1 -- 172.21.15.76:0/132474604 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7fec500329a0 con 0x55f93b3e4820 2024-03-18T15:02:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.489+0000 7fec5f5d9740 1 -- 172.21.15.76:0/132474604 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec5002cca0 con 0x55f93b3e4820 2024-03-18T15:02:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.489+0000 7fec3dffb640 1 -- 172.21.15.76:0/132474604 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fec50039050 con 0x55f93b3e4820 2024-03-18T15:02:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.489+0000 7fec5f5d9740 10 client.16614 fetch_fsmap learned FSMap version 41 2024-03-18T15:02:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.489+0000 7fec5f5d9740 1 -- 172.21.15.76:0/132474604 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f93b120620 con 0x55f93b3e4820 2024-03-18T15:02:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.489+0000 7fec3dffb640 1 -- 172.21.15.76:0/132474604 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec5002cca0 con 0x55f93b3e4820 2024-03-18T15:02:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.489+0000 7fec3dffb640 1 -- 172.21.15.76:0/132474604 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7fec50059020 con 0x55f93b3e4820 2024-03-18T15:02:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.489+0000 7fec5f5d9740 10 client.16614 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:02:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.489+0000 7fec5f5d9740 10 client.16614 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:02:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.489+0000 7fec5f5d9740 20 client.16614 populate_metadata read hostname 'smithi076' 2024-03-18T15:02:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.489+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/132474604 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f93b3f4810 0x55f93b3f6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.489+0000 7fec5f5d9740 4 client.16614 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:02:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.489+0000 7fec5f5d9740 1 -- 172.21.15.76:0/132474604 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f93b34cf80 con 0x55f93b3f4810 2024-03-18T15:02:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.489+0000 7fec5659c640 1 --2- 172.21.15.76:0/132474604 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f93b3f4810 0x55f93b3f6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.490+0000 7fec3dffb640 10 client.16614 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:02:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.490+0000 7fec5659c640 1 --2- 172.21.15.76:0/132474604 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f93b3f4810 0x55f93b3f6c00 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9099c9307e9da32 server_cookie=be1fc26135aa2d3 in_seq=0 out_seq=0 2024-03-18T15:02:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.490+0000 7fec3dffb640 1 -- 172.21.15.76:0/132474604 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f93b34cf80 con 0x55f93b3f4810 2024-03-18T15:02:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.490+0000 7fec3dffb640 10 client.16614 handle_command_reply: tid=0 2024-03-18T15:02:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.490+0000 7fec5f5d9740 1 client.16614 shutdown 2024-03-18T15:02:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.491+0000 7fec5f5d9740 1 -- 172.21.15.76:0/132474604 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3e4820 msgr2=0x55f93b3e22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.491+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/132474604 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3e4820 0x55f93b3e22d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fec50009a60 tx=0x7fec50003a80 comp rx=0 tx=0).stop 2024-03-18T15:02:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.491+0000 7fec5f5d9740 1 -- 172.21.15.76:0/132474604 shutdown_connections 2024-03-18T15:02:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.491+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/132474604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3dba50 0x55f93b3e1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.491+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/132474604 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f93b3f4810 0x55f93b3f6c00 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.491+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/132474604 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3e4820 0x55f93b3e22d0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.491+0000 7fec3dffb640 0 client.16614 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:02:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.491+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/132474604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3db130 0x55f93b3e1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.491+0000 7fec5f5d9740 1 -- 172.21.15.76:0/132474604 >> 172.21.15.76:0/132474604 conn(0x55f93b3d5a50 msgr2=0x55f93b3d77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.491+0000 7fec5f5d9740 1 -- 172.21.15.76:0/132474604 shutdown_connections 2024-03-18T15:02:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.491+0000 7fec5f5d9740 1 -- 172.21.15.76:0/132474604 wait complete. 2024-03-18T15:02:47.500 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.491+0000 7fec5f5d9740 20 client.16614 trim_cache size 0 max 16384 2024-03-18T15:02:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.506+0000 7fec5f5d9740 1 Processor -- start 2024-03-18T15:02:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.506+0000 7fec5f5d9740 1 -- start start 2024-03-18T15:02:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec5f5d9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3db130 0x55f93b3e3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec5f5d9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3dba50 0x55f93b3f9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec5f5d9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3e4820 0x55f93b3deaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec5f5d9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fec581b1e90 con 0x55f93b3e4820 2024-03-18T15:02:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec5f5d9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fec581af760 con 0x55f93b3dba50 2024-03-18T15:02:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec5f5d9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fec581ad5d0 con 0x55f93b3db130 2024-03-18T15:02:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec56d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3e4820 0x55f93b3deaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec5659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3db130 0x55f93b3e3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec56d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3e4820 0x55f93b3deaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34330/0 (socket says 172.21.15.76:34330) 2024-03-18T15:02:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec56d9d640 1 -- 172.21.15.76:0/3885349399 learned_addr learned my addr 172.21.15.76:0/3885349399 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec55d9b640 1 --2- 172.21.15.76:0/3885349399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3dba50 0x55f93b3f9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec56d9d640 1 -- 172.21.15.76:0/3885349399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3db130 msgr2=0x55f93b3e3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec56d9d640 1 --2- 172.21.15.76:0/3885349399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3db130 0x55f93b3e3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec56d9d640 1 -- 172.21.15.76:0/3885349399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3dba50 msgr2=0x55f93b3f9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec56d9d640 1 --2- 172.21.15.76:0/3885349399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3dba50 0x55f93b3f9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec56d9d640 1 -- 172.21.15.76:0/3885349399 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f93b3dd450 con 0x55f93b3e4820 2024-03-18T15:02:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec5659c640 1 --2- 172.21.15.76:0/3885349399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3db130 0x55f93b3e3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:02:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec56d9d640 1 --2- 172.21.15.76:0/3885349399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3e4820 0x55f93b3deaf0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fec50037040 tx=0x7fec50005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ea3829dc737949 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec3ffff640 1 -- 172.21.15.76:0/3885349399 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec50043040 con 0x55f93b3e4820 2024-03-18T15:02:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec3ffff640 1 -- 172.21.15.76:0/3885349399 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec500098a0 con 0x55f93b3e4820 2024-03-18T15:02:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.507+0000 7fec3ffff640 1 -- 172.21.15.76:0/3885349399 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5002de60 con 0x55f93b3e4820 2024-03-18T15:02:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.508+0000 7fec5f5d9740 1 -- 172.21.15.76:0/3885349399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3e4820 msgr2=0x55f93b3deaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.508+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/3885349399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3e4820 0x55f93b3deaf0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fec50037040 tx=0x7fec50005d80 comp rx=0 tx=0).stop 2024-03-18T15:02:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.508+0000 7fec5f5d9740 1 -- 172.21.15.76:0/3885349399 shutdown_connections 2024-03-18T15:02:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.508+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/3885349399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3e4820 0x55f93b3deaf0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.508+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/3885349399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3dba50 0x55f93b3f9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.508+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/3885349399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3db130 0x55f93b3e3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.508+0000 7fec5f5d9740 1 -- 172.21.15.76:0/3885349399 >> 172.21.15.76:0/3885349399 conn(0x55f93b3d5a50 msgr2=0x55f93b3d4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.508+0000 7fec5f5d9740 1 -- 172.21.15.76:0/3885349399 shutdown_connections 2024-03-18T15:02:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.508+0000 7fec5f5d9740 1 -- 172.21.15.76:0/3885349399 wait complete. 2024-03-18T15:02:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.508+0000 7fec5f5d9740 1 -- start start 2024-03-18T15:02:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5f5d9740 1 Processor -- start 2024-03-18T15:02:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5f5d9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3db130 0x55f93b3f9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5f5d9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3dba50 0x55f93b3f9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5f5d9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3e4820 0x55f93b3f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5f5d9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fec581adc20 con 0x55f93b3db130 2024-03-18T15:02:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5f5d9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fec581af090 con 0x55f93b3e4820 2024-03-18T15:02:47.515 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5f5d9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fec581ae2f0 con 0x55f93b3dba50 2024-03-18T15:02:47.515 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5f5d9740 10 client.0 _ll_register_callbacks cb 0x7fec480097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:02:47.515 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3db130 0x55f93b3f9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.515 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3db130 0x55f93b3f9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34344/0 (socket says 172.21.15.76:34344) 2024-03-18T15:02:47.515 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5659c640 1 -- 172.21.15.76:0/820314186 learned_addr learned my addr 172.21.15.76:0/820314186 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:02:47.515 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec55d9b640 1 --2- 172.21.15.76:0/820314186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3dba50 0x55f93b3f9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.515 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec56d9d640 1 --2- 172.21.15.76:0/820314186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3e4820 0x55f93b3f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.515 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5659c640 1 -- 172.21.15.76:0/820314186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3dba50 msgr2=0x55f93b3f9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:47.515 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5659c640 1 --2- 172.21.15.76:0/820314186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3dba50 0x55f93b3f9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.515 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5659c640 1 -- 172.21.15.76:0/820314186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3e4820 msgr2=0x55f93b3f7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:02:47.515 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5659c640 1 --2- 172.21.15.76:0/820314186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3e4820 0x55f93b3f7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.515 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5659c640 1 -- 172.21.15.76:0/820314186 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f93b3f83e0 con 0x55f93b3db130 2024-03-18T15:02:47.515 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5659c640 1 --2- 172.21.15.76:0/820314186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3db130 0x55f93b3f9770 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fec480041d0 tx=0x7fec48017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd86580b27fd089 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:02:47.515 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:02:47.515 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec3dffb640 1 -- 172.21.15.76:0/820314186 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec480044c0 con 0x55f93b3db130 2024-03-18T15:02:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec3dffb640 1 -- 172.21.15.76:0/820314186 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7fec4802bb90 con 0x55f93b3db130 2024-03-18T15:02:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.509+0000 7fec5f5d9740 1 -- 172.21.15.76:0/820314186 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f93b3472d0 con 0x55f93b3db130 2024-03-18T15:02:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.510+0000 7fec3dffb640 1 -- 172.21.15.76:0/820314186 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fec4802bd90 con 0x55f93b3db130 2024-03-18T15:02:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.510+0000 7fec5f5d9740 10 client.16620 fetch_fsmap learned FSMap version 41 2024-03-18T15:02:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.510+0000 7fec5f5d9740 1 -- 172.21.15.76:0/820314186 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f93b45b300 con 0x55f93b3db130 2024-03-18T15:02:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.510+0000 7fec3dffb640 1 -- 172.21.15.76:0/820314186 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f93b3472d0 con 0x55f93b3db130 2024-03-18T15:02:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.510+0000 7fec3dffb640 1 -- 172.21.15.76:0/820314186 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8338+0+0 (secure 0 0 0) 0x7fec4803a020 con 0x55f93b3db130 2024-03-18T15:02:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.510+0000 7fec5f5d9740 10 client.16620 fetch_fsmap finished waiting for FSMap version 41 2024-03-18T15:02:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.510+0000 7fec5f5d9740 10 client.16620 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:02:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.510+0000 7fec5f5d9740 20 client.16620 populate_metadata read hostname 'smithi076' 2024-03-18T15:02:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.510+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/820314186 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f93b45ba80 0x55f93b45dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:02:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.510+0000 7fec5f5d9740 4 client.16620 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:02:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.510+0000 7fec5f5d9740 1 -- 172.21.15.76:0/820314186 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f93b2c5770 con 0x55f93b45ba80 2024-03-18T15:02:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.510+0000 7fec55d9b640 1 --2- 172.21.15.76:0/820314186 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f93b45ba80 0x55f93b45dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:02:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.510+0000 7fec3dffb640 10 client.16620 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:02:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.510+0000 7fec55d9b640 1 --2- 172.21.15.76:0/820314186 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f93b45ba80 0x55f93b45dea0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2a737f095f8f10d server_cookie=6078a4de9f3f2f04 in_seq=0 out_seq=0 2024-03-18T15:02:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.579+0000 7fec3dffb640 1 -- 172.21.15.76:0/820314186 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f93b2c5770 con 0x55f93b45ba80 2024-03-18T15:02:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.579+0000 7fec3dffb640 10 client.16620 handle_command_reply: tid=0 2024-03-18T15:02:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.579+0000 7fec5f5d9740 1 client.16620 shutdown 2024-03-18T15:02:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.579+0000 7fec5f5d9740 1 -- 172.21.15.76:0/820314186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3db130 msgr2=0x55f93b3f9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.579+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/820314186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3db130 0x55f93b3f9770 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fec480041d0 tx=0x7fec48017000 comp rx=0 tx=0).stop 2024-03-18T15:02:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5f5d9740 1 -- 172.21.15.76:0/820314186 shutdown_connections 2024-03-18T15:02:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/820314186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f93b3dba50 0x55f93b3f9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/820314186 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55f93b45ba80 0x55f93b45dea0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/820314186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f93b3e4820 0x55f93b3f7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec3dffb640 0 client.16620 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:02:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5f5d9740 1 --2- 172.21.15.76:0/820314186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f93b3db130 0x55f93b3f9770 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5f5d9740 1 -- 172.21.15.76:0/820314186 >> 172.21.15.76:0/820314186 conn(0x55f93b3d5a50 msgr2=0x55f93b3e1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5f5d9740 1 -- 172.21.15.76:0/820314186 shutdown_connections 2024-03-18T15:02:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5f5d9740 1 -- 172.21.15.76:0/820314186 wait complete. 2024-03-18T15:02:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5f5d9740 20 client.16620 trim_cache size 0 max 16384 2024-03-18T15:02:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:02:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:02:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:02:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:02:47.583 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5c9f3640 1 -- 172.21.15.76:0/2158752099 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fec400a2a10 msgr2=0x7fec400a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:47.583 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5c9f3640 1 --2- 172.21.15.76:0/2158752099 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fec400a2a10 0x7fec400a4ed0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x55f93b3dabe0 tx=0x7fec48005f90 comp rx=0 tx=0).stop 2024-03-18T15:02:47.583 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5c9f3640 1 -- 172.21.15.76:0/2158752099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec5806a900 msgr2=0x7fec5819a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:02:47.583 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5c9f3640 1 --2- 172.21.15.76:0/2158752099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec5806a900 0x7fec5819a700 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00e9f0 tx=0x7fec4c00eeb0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.583 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5c9f3640 1 -- 172.21.15.76:0/2158752099 shutdown_connections 2024-03-18T15:02:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5c9f3640 1 --2- 172.21.15.76:0/2158752099 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fec400a2a10 0x7fec400a4ed0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5c9f3640 1 --2- 172.21.15.76:0/2158752099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec581077c0 0x7fec5819ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5c9f3640 1 --2- 172.21.15.76:0/2158752099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec5806a900 0x7fec5819a700 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5c9f3640 1 --2- 172.21.15.76:0/2158752099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec58069f80 0x7fec5819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:02:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5c9f3640 1 -- 172.21.15.76:0/2158752099 >> 172.21.15.76:0/2158752099 conn(0x7fec580776c0 msgr2=0x7fec58101e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:02:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5c9f3640 1 -- 172.21.15.76:0/2158752099 shutdown_connections 2024-03-18T15:02:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:02:47.580+0000 7fec5c9f3640 1 -- 172.21.15.76:0/2158752099 wait complete. 2024-03-18T15:02:47.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 933083 9.41 MB/sec execute 408 sec latency 69.661 ms 2024-03-18T15:02:48.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:47 smithi076 ceph-mon[32205]: pgmap v300: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 236 op/s 2024-03-18T15:02:48.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:47 smithi174 ceph-mon[36787]: pgmap v300: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 236 op/s 2024-03-18T15:02:48.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:47 smithi077 ceph-mon[35865]: pgmap v300: 97 pgs: 97 active+clean; 268 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 236 op/s 2024-03-18T15:02:48.227 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:02:47 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setuser ceph since I am not root 2024-03-18T15:02:48.227 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:02:47 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setgroup ceph since I am not root 2024-03-18T15:02:48.227 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:02:47 smithi077 ceph-mds[65404]: -- 172.21.15.77:0/2453584881 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f6d27a3a40 con 0x55f6d2776c00 2024-03-18T15:02:48.228 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:02:47 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: starting mds.e at 2024-03-18T15:02:48.757 INFO:tasks.workunit.client.0.smithi076.stdout: 1 934877 9.42 MB/sec execute 409 sec latency 16.226 ms 2024-03-18T15:02:49.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:48 smithi076 ceph-mon[32205]: daemon mds.e restarted 2024-03-18T15:02:49.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:48 smithi076 ceph-mon[32205]: osdmap e102: 12 total, 12 up, 12 in 2024-03-18T15:02:49.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:48 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.77:6834/4214206349,v1:172.21.15.77:6835/4214206349] up:boot 2024-03-18T15:02:49.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:48 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T15:02:49.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:48 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:02:49.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:48 smithi174 ceph-mon[36787]: daemon mds.e restarted 2024-03-18T15:02:49.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:48 smithi174 ceph-mon[36787]: osdmap e102: 12 total, 12 up, 12 in 2024-03-18T15:02:49.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:48 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.77:6834/4214206349,v1:172.21.15.77:6835/4214206349] up:boot 2024-03-18T15:02:49.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:48 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T15:02:49.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:48 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:02:49.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:48 smithi077 ceph-mon[35865]: daemon mds.e restarted 2024-03-18T15:02:49.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:48 smithi077 ceph-mon[35865]: osdmap e102: 12 total, 12 up, 12 in 2024-03-18T15:02:49.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:48 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.77:6834/4214206349,v1:172.21.15.77:6835/4214206349] up:boot 2024-03-18T15:02:49.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:48 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T15:02:49.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:48 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:02:49.756 INFO:tasks.workunit.client.0.smithi076.stdout: 1 936907 9.42 MB/sec execute 410 sec latency 19.134 ms 2024-03-18T15:02:50.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:49 smithi076 ceph-mon[32205]: pgmap v302: 97 pgs: 97 active+clean; 262 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 282 op/s 2024-03-18T15:02:50.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:49 smithi174 ceph-mon[36787]: pgmap v302: 97 pgs: 97 active+clean; 262 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 282 op/s 2024-03-18T15:02:50.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:49 smithi077 ceph-mon[35865]: pgmap v302: 97 pgs: 97 active+clean; 262 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 282 op/s 2024-03-18T15:02:50.757 INFO:tasks.workunit.client.0.smithi076.stdout: 1 938562 9.42 MB/sec execute 411 sec latency 12.461 ms 2024-03-18T15:02:51.757 INFO:tasks.workunit.client.0.smithi076.stdout: 1 940193 9.41 MB/sec execute 412 sec latency 82.075 ms 2024-03-18T15:02:52.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:51 smithi076 ceph-mon[32205]: pgmap v303: 97 pgs: 97 active+clean; 262 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 282 op/s 2024-03-18T15:02:52.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:51 smithi174 ceph-mon[36787]: pgmap v303: 97 pgs: 97 active+clean; 262 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 282 op/s 2024-03-18T15:02:52.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:51 smithi077 ceph-mon[35865]: pgmap v303: 97 pgs: 97 active+clean; 262 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 282 op/s 2024-03-18T15:02:52.757 INFO:tasks.workunit.client.0.smithi076.stdout: 1 941931 9.42 MB/sec execute 413 sec latency 49.677 ms 2024-03-18T15:02:53.757 INFO:tasks.workunit.client.0.smithi076.stdout: 1 943267 9.41 MB/sec execute 414 sec latency 31.848 ms 2024-03-18T15:02:54.018 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:53 smithi174 ceph-mon[36787]: pgmap v304: 97 pgs: 97 active+clean; 269 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 281 op/s 2024-03-18T15:02:54.028 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:53 smithi077 ceph-mon[35865]: pgmap v304: 97 pgs: 97 active+clean; 269 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 281 op/s 2024-03-18T15:02:54.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:53 smithi076 ceph-mon[32205]: pgmap v304: 97 pgs: 97 active+clean; 269 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 281 op/s 2024-03-18T15:02:54.757 INFO:tasks.workunit.client.0.smithi076.stdout: 1 945479 9.42 MB/sec execute 415 sec latency 14.803 ms 2024-03-18T15:02:55.757 INFO:tasks.workunit.client.0.smithi076.stdout: 1 946931 9.41 MB/sec execute 416 sec latency 20.259 ms 2024-03-18T15:02:56.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:55 smithi076 ceph-mon[32205]: pgmap v305: 97 pgs: 97 active+clean; 269 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 281 op/s 2024-03-18T15:02:56.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:55 smithi174 ceph-mon[36787]: pgmap v305: 97 pgs: 97 active+clean; 269 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 281 op/s 2024-03-18T15:02:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:55 smithi077 ceph-mon[35865]: pgmap v305: 97 pgs: 97 active+clean; 269 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 281 op/s 2024-03-18T15:02:56.757 INFO:tasks.workunit.client.0.smithi076.stdout: 1 949038 9.42 MB/sec execute 417 sec latency 14.298 ms 2024-03-18T15:02:57.757 INFO:tasks.workunit.client.0.smithi076.stdout: 1 950463 9.42 MB/sec execute 418 sec latency 59.252 ms 2024-03-18T15:02:58.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:57 smithi076 ceph-mon[32205]: pgmap v306: 97 pgs: 97 active+clean; 269 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 281 op/s 2024-03-18T15:02:58.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:57 smithi174 ceph-mon[36787]: pgmap v306: 97 pgs: 97 active+clean; 269 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 281 op/s 2024-03-18T15:02:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:57 smithi077 ceph-mon[35865]: pgmap v306: 97 pgs: 97 active+clean; 269 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 281 op/s 2024-03-18T15:02:58.757 INFO:tasks.workunit.client.0.smithi076.stdout: 1 952523 9.42 MB/sec execute 419 sec latency 32.703 ms 2024-03-18T15:02:59.757 INFO:tasks.workunit.client.0.smithi076.stdout: 1 953900 9.42 MB/sec execute 420 sec latency 20.600 ms 2024-03-18T15:03:00.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:02:59 smithi076 ceph-mon[32205]: pgmap v307: 97 pgs: 97 active+clean; 276 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.2 MiB/s wr, 352 op/s 2024-03-18T15:03:00.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:02:59 smithi174 ceph-mon[36787]: pgmap v307: 97 pgs: 97 active+clean; 276 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.2 MiB/s wr, 352 op/s 2024-03-18T15:03:00.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:02:59 smithi077 ceph-mon[35865]: pgmap v307: 97 pgs: 97 active+clean; 276 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.2 MiB/s wr, 352 op/s 2024-03-18T15:03:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.674+0000 7faccf20c640 1 -- 172.21.15.76:0/4028291285 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7facc8100930 msgr2=0x7facc8100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.674+0000 7faccf20c640 1 --2- 172.21.15.76:0/4028291285 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7facc8100930 0x7facc8100db0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7facb4009900 tx=0x7facb402bc30 comp rx=0 tx=0).stop 2024-03-18T15:03:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.674+0000 7faccf20c640 1 -- 172.21.15.76:0/4028291285 shutdown_connections 2024-03-18T15:03:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.674+0000 7faccf20c640 1 --2- 172.21.15.76:0/4028291285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facc81012f0 0x7facc810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.674+0000 7faccf20c640 1 --2- 172.21.15.76:0/4028291285 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7facc8100930 0x7facc8100db0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.674+0000 7faccf20c640 1 --2- 172.21.15.76:0/4028291285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facc8105680 0x7facc8105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.674+0000 7faccf20c640 1 -- 172.21.15.76:0/4028291285 >> 172.21.15.76:0/4028291285 conn(0x7facc8077530 msgr2=0x7facc8077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.674+0000 7faccf20c640 1 -- 172.21.15.76:0/4028291285 shutdown_connections 2024-03-18T15:03:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.674+0000 7faccf20c640 1 -- 172.21.15.76:0/4028291285 wait complete. 2024-03-18T15:03:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.674+0000 7faccf20c640 1 Processor -- start 2024-03-18T15:03:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.674+0000 7faccf20c640 1 -- start start 2024-03-18T15:03:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.674+0000 7faccf20c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7facc8100930 0x7facc8198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.674+0000 7faccf20c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facc81012f0 0x7facc8198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.674+0000 7faccf20c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facc8105680 0x7facc819c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.674+0000 7faccf20c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7facc8110ba0 con 0x7facc8100930 2024-03-18T15:03:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.674+0000 7faccf20c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7facc8110a20 con 0x7facc8105680 2024-03-18T15:03:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.674+0000 7faccf20c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7facc8110d20 con 0x7facc81012f0 2024-03-18T15:03:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.675+0000 7facccfa8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7facc8100930 0x7facc8198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.675+0000 7facccfa8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7facc8100930 0x7facc8198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51944/0 (socket says 172.21.15.76:51944) 2024-03-18T15:03:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.675+0000 7facccfa8640 1 -- 172.21.15.76:0/4265355382 learned_addr learned my addr 172.21.15.76:0/4265355382 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.675+0000 7faccd7a9640 1 --2- 172.21.15.76:0/4265355382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facc8105680 0x7facc819c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.675+0000 7facc7fff640 1 --2- 172.21.15.76:0/4265355382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facc81012f0 0x7facc8198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.675+0000 7facccfa8640 1 -- 172.21.15.76:0/4265355382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facc81012f0 msgr2=0x7facc8198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.675+0000 7facccfa8640 1 --2- 172.21.15.76:0/4265355382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facc81012f0 0x7facc8198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.675+0000 7facccfa8640 1 -- 172.21.15.76:0/4265355382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facc8105680 msgr2=0x7facc819c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.675+0000 7facccfa8640 1 --2- 172.21.15.76:0/4265355382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facc8105680 0x7facc819c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.675+0000 7facccfa8640 1 -- 172.21.15.76:0/4265355382 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc819d0f0 con 0x7facc8100930 2024-03-18T15:03:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.675+0000 7facccfa8640 1 --2- 172.21.15.76:0/4265355382 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7facc8100930 0x7facc8198100 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7facbc00e9f0 tx=0x7facbc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.675+0000 7facc5ffb640 1 -- 172.21.15.76:0/4265355382 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc00cd80 con 0x7facc8100930 2024-03-18T15:03:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.675+0000 7facc5ffb640 1 -- 172.21.15.76:0/4265355382 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facbc005500 con 0x7facc8100930 2024-03-18T15:03:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.675+0000 7faccf20c640 1 -- 172.21.15.76:0/4265355382 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facc819d3e0 con 0x7facc8100930 2024-03-18T15:03:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.675+0000 7facc5ffb640 1 -- 172.21.15.76:0/4265355382 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc010690 con 0x7facc8100930 2024-03-18T15:03:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.675+0000 7faccf20c640 1 -- 172.21.15.76:0/4265355382 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facc81a4c70 con 0x7facc8100930 2024-03-18T15:03:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7facc5ffb640 1 -- 172.21.15.76:0/4265355382 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7facbc0046c0 con 0x7facc8100930 2024-03-18T15:03:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7facd1df1740 1 Processor -- start 2024-03-18T15:03:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7facc5ffb640 1 --2- 172.21.15.76:0/4265355382 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7facb80a2a10 0x7facb80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7facd1df1740 1 -- start start 2024-03-18T15:03:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7faccd7a9640 1 --2- 172.21.15.76:0/4265355382 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7facb80a2a10 0x7facb80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7facd1df1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38c3130 0x5622d38c3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7facc5ffb640 1 -- 172.21.15.76:0/4265355382 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8440+0+0 (secure 0 0 0) 0x7facbc0cf1a0 con 0x7facc8100930 2024-03-18T15:03:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7facd1df1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38c3a50 0x5622d38cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7facd1df1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38cc820 0x5622d38cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7facd1df1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5622d3827a40 con 0x5622d38cc820 2024-03-18T15:03:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7facd1df1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5622d37d0920 con 0x5622d38c3130 2024-03-18T15:03:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7facd1df1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5622d3823ae0 con 0x5622d38c3a50 2024-03-18T15:03:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7facccfa8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38c3130 0x5622d38c3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7faccd7a9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38cc820 0x5622d38cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7facc7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38c3a50 0x5622d38cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7facccfa8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38c3130 0x5622d38c3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:60836/0 (socket says 172.21.15.76:60836) 2024-03-18T15:03:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7faccd7a9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38cc820 0x5622d38cec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51948/0 (socket says 172.21.15.76:51948) 2024-03-18T15:03:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.678+0000 7facccfa8640 1 -- 172.21.15.76:0/687981619 learned_addr learned my addr 172.21.15.76:0/687981619 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.679+0000 7facccfa8640 1 -- 172.21.15.76:0/687981619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38c3a50 msgr2=0x5622d38cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.679+0000 7facccfa8640 1 --2- 172.21.15.76:0/687981619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38c3a50 0x5622d38cc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.679+0000 7facccfa8640 1 -- 172.21.15.76:0/687981619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38cc820 msgr2=0x5622d38cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.679+0000 7facccfa8640 1 --2- 172.21.15.76:0/687981619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38cc820 0x5622d38cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.679+0000 7faccd7a9640 1 --2- 172.21.15.76:0/4265355382 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7facb80a2a10 0x7facb80a4ed0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7facc0005fd0 tx=0x7facc0013b60 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.679+0000 7facccfa8640 1 -- 172.21.15.76:0/687981619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622d37e8460 con 0x5622d38c3130 2024-03-18T15:03:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.679+0000 7facccfa8640 1 --2- 172.21.15.76:0/687981619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38c3130 0x5622d38c3510 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7facbc0a3ac0 tx=0x7facbc09b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d7180ab17a753ee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.679+0000 7facb2ffd640 1 -- 172.21.15.76:0/687981619 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc09b850 con 0x5622d38c3130 2024-03-18T15:03:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.679+0000 7facb2ffd640 1 -- 172.21.15.76:0/687981619 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facbc09f9a0 con 0x5622d38c3130 2024-03-18T15:03:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.679+0000 7facb2ffd640 1 -- 172.21.15.76:0/687981619 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc097ec0 con 0x5622d38c3130 2024-03-18T15:03:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.680+0000 7facd1df1740 1 -- 172.21.15.76:0/687981619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38c3130 msgr2=0x5622d38c3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.680+0000 7facd1df1740 1 --2- 172.21.15.76:0/687981619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38c3130 0x5622d38c3510 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7facbc0a3ac0 tx=0x7facbc09b390 comp rx=0 tx=0).stop 2024-03-18T15:03:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.680+0000 7facd1df1740 1 -- 172.21.15.76:0/687981619 shutdown_connections 2024-03-18T15:03:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.680+0000 7facd1df1740 1 --2- 172.21.15.76:0/687981619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38c3a50 0x5622d38cc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.680+0000 7facd1df1740 1 --2- 172.21.15.76:0/687981619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38cc820 0x5622d38cec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.680+0000 7facd1df1740 1 --2- 172.21.15.76:0/687981619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38c3130 0x5622d38c3510 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.680+0000 7facd1df1740 1 -- 172.21.15.76:0/687981619 >> 172.21.15.76:0/687981619 conn(0x5622d38bda50 msgr2=0x5622d38bde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.680+0000 7facd1df1740 1 -- 172.21.15.76:0/687981619 shutdown_connections 2024-03-18T15:03:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.680+0000 7facd1df1740 1 -- 172.21.15.76:0/687981619 wait complete. 2024-03-18T15:03:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.680+0000 7facd1df1740 1 -- start start 2024-03-18T15:03:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.680+0000 7facd1df1740 1 Processor -- start 2024-03-18T15:03:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7facd1df1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38c3130 0x5622d38c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7facd1df1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38c3a50 0x5622d38c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7facd1df1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38cc820 0x5622d38ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7facd1df1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5622d38cf450 con 0x5622d38cc820 2024-03-18T15:03:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7facd1df1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5622d37e3d70 con 0x5622d38c3a50 2024-03-18T15:03:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7facd1df1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5622d38cf5d0 con 0x5622d38c3130 2024-03-18T15:03:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7facd1df1740 10 client.0 _ll_register_callbacks cb 0x5622d353ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:03:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7faccd7a9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38cc820 0x5622d38ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7facc7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38c3a50 0x5622d38c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7facccfa8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38c3130 0x5622d38c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7faccd7a9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38cc820 0x5622d38ca330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51950/0 (socket says 172.21.15.76:51950) 2024-03-18T15:03:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7faccd7a9640 1 -- 172.21.15.76:0/1239306490 learned_addr learned my addr 172.21.15.76:0/1239306490 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7faccd7a9640 1 -- 172.21.15.76:0/1239306490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38c3130 msgr2=0x5622d38c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7faccd7a9640 1 --2- 172.21.15.76:0/1239306490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38c3130 0x5622d38c98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7faccd7a9640 1 -- 172.21.15.76:0/1239306490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38c3a50 msgr2=0x5622d38c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7faccd7a9640 1 --2- 172.21.15.76:0/1239306490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38c3a50 0x5622d38c9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7faccd7a9640 1 -- 172.21.15.76:0/1239306490 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622d353c210 con 0x5622d38cc820 2024-03-18T15:03:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7faccd7a9640 1 --2- 172.21.15.76:0/1239306490 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38cc820 0x5622d38ca330 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7facc000f790 tx=0x7facc0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abe6e4e557e2585c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7faca7fff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:03:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7faca7fff640 1 -- 172.21.15.76:0/1239306490 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc0003970 con 0x5622d38cc820 2024-03-18T15:03:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7faca7fff640 1 -- 172.21.15.76:0/1239306490 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8440+0+0 (secure 0 0 0) 0x7facc0003b10 con 0x5622d38cc820 2024-03-18T15:03:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.681+0000 7facd1df1740 1 -- 172.21.15.76:0/1239306490 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facbc09f9a0 con 0x5622d38cc820 2024-03-18T15:03:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.682+0000 7faca7fff640 1 -- 172.21.15.76:0/1239306490 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7facc002f020 con 0x5622d38cc820 2024-03-18T15:03:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.682+0000 7facd1df1740 10 client.16638 fetch_fsmap learned FSMap version 42 2024-03-18T15:03:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.682+0000 7facd1df1740 1 -- 172.21.15.76:0/1239306490 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5622d3608620 con 0x5622d38cc820 2024-03-18T15:03:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.682+0000 7faca7fff640 1 -- 172.21.15.76:0/1239306490 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facbc09f9a0 con 0x5622d38cc820 2024-03-18T15:03:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.682+0000 7faca7fff640 1 -- 172.21.15.76:0/1239306490 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7facc001f070 con 0x5622d38cc820 2024-03-18T15:03:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.682+0000 7facd1df1740 10 client.16638 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T15:03:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.682+0000 7facd1df1740 10 client.16638 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:03:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.682+0000 7facd1df1740 20 client.16638 populate_metadata read hostname 'smithi076' 2024-03-18T15:03:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.682+0000 7facd1df1740 1 --2- 172.21.15.76:0/1239306490 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5622d38dca00 0x5622d38dee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.682+0000 7facd1df1740 4 client.16638 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:03:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.682+0000 7facd1df1740 1 -- 172.21.15.76:0/1239306490 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622d3834f80 con 0x5622d38dca00 2024-03-18T15:03:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.682+0000 7facccfa8640 1 --2- 172.21.15.76:0/1239306490 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5622d38dca00 0x5622d38dee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.683+0000 7facccfa8640 1 --2- 172.21.15.76:0/1239306490 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5622d38dca00 0x5622d38dee30 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9fca872ed3e45cc server_cookie=5ac6d920fe980a40 in_seq=0 out_seq=0 2024-03-18T15:03:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.683+0000 7faca7fff640 10 client.16638 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.683+0000 7faca7fff640 1 -- 172.21.15.76:0/1239306490 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5622d3834f80 con 0x5622d38dca00 2024-03-18T15:03:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.683+0000 7faca7fff640 10 client.16638 handle_command_reply: tid=0 2024-03-18T15:03:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.683+0000 7facd1df1740 1 client.16638 shutdown 2024-03-18T15:03:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.684+0000 7facd1df1740 1 -- 172.21.15.76:0/1239306490 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38cc820 msgr2=0x5622d38ca330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.684+0000 7facd1df1740 1 --2- 172.21.15.76:0/1239306490 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38cc820 0x5622d38ca330 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7facc000f790 tx=0x7facc0004ab0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.684+0000 7facd1df1740 1 -- 172.21.15.76:0/1239306490 shutdown_connections 2024-03-18T15:03:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.684+0000 7facd1df1740 1 --2- 172.21.15.76:0/1239306490 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5622d38dca00 0x5622d38dee30 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.684+0000 7facd1df1740 1 --2- 172.21.15.76:0/1239306490 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38cc820 0x5622d38ca330 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.684+0000 7facd1df1740 1 --2- 172.21.15.76:0/1239306490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38c3a50 0x5622d38c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.684+0000 7faca7fff640 0 client.16638 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.684+0000 7facd1df1740 1 --2- 172.21.15.76:0/1239306490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38c3130 0x5622d38c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.684+0000 7facd1df1740 1 -- 172.21.15.76:0/1239306490 >> 172.21.15.76:0/1239306490 conn(0x5622d38bda50 msgr2=0x5622d38bf7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.684+0000 7facd1df1740 1 -- 172.21.15.76:0/1239306490 shutdown_connections 2024-03-18T15:03:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.684+0000 7facd1df1740 1 -- 172.21.15.76:0/1239306490 wait complete. 2024-03-18T15:03:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.684+0000 7facd1df1740 20 client.16638 trim_cache size 0 max 16384 2024-03-18T15:03:00.703 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.698+0000 7facd1df1740 1 Processor -- start 2024-03-18T15:03:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.698+0000 7facd1df1740 1 -- start start 2024-03-18T15:03:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.698+0000 7facd1df1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38c3130 0x5622d38cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.698+0000 7facd1df1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38c3a50 0x5622d38d81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.698+0000 7facd1df1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38cc820 0x5622d38e4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.698+0000 7facd1df1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7facc81afc10 con 0x5622d38c3a50 2024-03-18T15:03:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.698+0000 7facd1df1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7facc81ad4e0 con 0x5622d38c3130 2024-03-18T15:03:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.698+0000 7facd1df1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7facc81ab330 con 0x5622d38cc820 2024-03-18T15:03:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.699+0000 7facc7fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38c3a50 0x5622d38d81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.699+0000 7facc7fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38c3a50 0x5622d38d81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51952/0 (socket says 172.21.15.76:51952) 2024-03-18T15:03:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.699+0000 7facc7fff640 1 -- 172.21.15.76:0/2354439263 learned_addr learned my addr 172.21.15.76:0/2354439263 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.699+0000 7facccfa8640 1 --2- 172.21.15.76:0/2354439263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38c3130 0x5622d38cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.699+0000 7faccd7a9640 1 --2- 172.21.15.76:0/2354439263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38cc820 0x5622d38e4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.699+0000 7facc7fff640 1 -- 172.21.15.76:0/2354439263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38cc820 msgr2=0x5622d38e4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.699+0000 7facc7fff640 1 --2- 172.21.15.76:0/2354439263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38cc820 0x5622d38e4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.699+0000 7facc7fff640 1 -- 172.21.15.76:0/2354439263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38c3130 msgr2=0x5622d38cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.699+0000 7facc7fff640 1 --2- 172.21.15.76:0/2354439263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38c3130 0x5622d38cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.699+0000 7facc7fff640 1 -- 172.21.15.76:0/2354439263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622d38b9310 con 0x5622d38c3a50 2024-03-18T15:03:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.699+0000 7facc7fff640 1 --2- 172.21.15.76:0/2354439263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38c3a50 0x5622d38d81f0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7facb4009900 tx=0x7facb402c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4672f6cd31af0b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.699+0000 7facb27fc640 1 -- 172.21.15.76:0/2354439263 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb404c070 con 0x5622d38c3a50 2024-03-18T15:03:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.699+0000 7facb27fc640 1 -- 172.21.15.76:0/2354439263 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facb402c8b0 con 0x5622d38c3a50 2024-03-18T15:03:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.699+0000 7facb27fc640 1 -- 172.21.15.76:0/2354439263 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb4004ec0 con 0x5622d38c3a50 2024-03-18T15:03:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.700+0000 7facd1df1740 1 -- 172.21.15.76:0/2354439263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38c3a50 msgr2=0x5622d38d81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.700+0000 7facd1df1740 1 --2- 172.21.15.76:0/2354439263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38c3a50 0x5622d38d81f0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7facb4009900 tx=0x7facb402c710 comp rx=0 tx=0).stop 2024-03-18T15:03:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.700+0000 7facd1df1740 1 -- 172.21.15.76:0/2354439263 shutdown_connections 2024-03-18T15:03:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.700+0000 7facd1df1740 1 --2- 172.21.15.76:0/2354439263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38cc820 0x5622d38e4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.700+0000 7facd1df1740 1 --2- 172.21.15.76:0/2354439263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38c3a50 0x5622d38d81f0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.700+0000 7facd1df1740 1 --2- 172.21.15.76:0/2354439263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38c3130 0x5622d38cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.700+0000 7facd1df1740 1 -- 172.21.15.76:0/2354439263 >> 172.21.15.76:0/2354439263 conn(0x5622d38bda50 msgr2=0x5622d38bc320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:00.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.700+0000 7facd1df1740 1 -- 172.21.15.76:0/2354439263 shutdown_connections 2024-03-18T15:03:00.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.700+0000 7facd1df1740 1 -- 172.21.15.76:0/2354439263 wait complete. 2024-03-18T15:03:00.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.700+0000 7facd1df1740 1 -- start start 2024-03-18T15:03:00.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.700+0000 7facd1df1740 1 Processor -- start 2024-03-18T15:03:00.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facd1df1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38c3130 0x5622d38d4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facd1df1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38c3a50 0x5622d38d17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facd1df1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38cc820 0x5622d38d1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facd1df1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7facc81ab9a0 con 0x5622d38c3130 2024-03-18T15:03:00.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facd1df1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7facc81ace10 con 0x5622d38cc820 2024-03-18T15:03:00.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facd1df1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7facc81ac070 con 0x5622d38c3a50 2024-03-18T15:03:00.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facd1df1740 10 client.0 _ll_register_callbacks cb 0x7facc0012050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:03:00.706 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facccfa8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38c3130 0x5622d38d4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facccfa8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38c3130 0x5622d38d4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51966/0 (socket says 172.21.15.76:51966) 2024-03-18T15:03:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facccfa8640 1 -- 172.21.15.76:0/2024677901 learned_addr learned my addr 172.21.15.76:0/2024677901 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7faccd7a9640 1 --2- 172.21.15.76:0/2024677901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38cc820 0x5622d38d1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facc7fff640 1 --2- 172.21.15.76:0/2024677901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38c3a50 0x5622d38d17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facccfa8640 1 -- 172.21.15.76:0/2024677901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38c3a50 msgr2=0x5622d38d17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facccfa8640 1 --2- 172.21.15.76:0/2024677901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38c3a50 0x5622d38d17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facccfa8640 1 -- 172.21.15.76:0/2024677901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38cc820 msgr2=0x5622d38d1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facccfa8640 1 --2- 172.21.15.76:0/2024677901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38cc820 0x5622d38d1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facccfa8640 1 -- 172.21.15.76:0/2024677901 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622d38d22a0 con 0x5622d38c3130 2024-03-18T15:03:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facccfa8640 1 --2- 172.21.15.76:0/2024677901 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38c3130 0x5622d38d4820 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7facbc09b7e0 tx=0x7facbc009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f67ba5ba058fe4bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7faca7fff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:03:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7faca7fff640 1 -- 172.21.15.76:0/2024677901 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc0ae020 con 0x5622d38c3130 2024-03-18T15:03:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7faca7fff640 1 -- 172.21.15.76:0/2024677901 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8440+0+0 (secure 0 0 0) 0x7facbc097ec0 con 0x5622d38c3130 2024-03-18T15:03:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facd1df1740 1 -- 172.21.15.76:0/2024677901 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facb402c8b0 con 0x5622d38c3130 2024-03-18T15:03:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7faca7fff640 1 -- 172.21.15.76:0/2024677901 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7facbc09f9a0 con 0x5622d38c3130 2024-03-18T15:03:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facd1df1740 10 client.16644 fetch_fsmap learned FSMap version 42 2024-03-18T15:03:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7facd1df1740 1 -- 172.21.15.76:0/2024677901 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5622d3943b10 con 0x5622d38c3130 2024-03-18T15:03:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.701+0000 7faca7fff640 1 -- 172.21.15.76:0/2024677901 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facb402c8b0 con 0x5622d38c3130 2024-03-18T15:03:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.702+0000 7faca7fff640 1 -- 172.21.15.76:0/2024677901 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7facbc09cb80 con 0x5622d38c3130 2024-03-18T15:03:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.702+0000 7facd1df1740 10 client.16644 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T15:03:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.702+0000 7facd1df1740 10 client.16644 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:03:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.702+0000 7facd1df1740 20 client.16644 populate_metadata read hostname 'smithi076' 2024-03-18T15:03:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.702+0000 7facd1df1740 1 --2- 172.21.15.76:0/2024677901 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5622d3943da0 0x5622d3946190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.702+0000 7facd1df1740 4 client.16644 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:03:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.702+0000 7facd1df1740 1 -- 172.21.15.76:0/2024677901 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5622d382f2d0 con 0x5622d3943da0 2024-03-18T15:03:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.702+0000 7facc7fff640 1 --2- 172.21.15.76:0/2024677901 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5622d3943da0 0x5622d3946190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.702+0000 7facc7fff640 1 --2- 172.21.15.76:0/2024677901 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5622d3943da0 0x5622d3946190 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5da53d18a008dae2 server_cookie=290df8640425a651 in_seq=0 out_seq=0 2024-03-18T15:03:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.702+0000 7faca7fff640 10 client.16644 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:00.758 INFO:tasks.workunit.client.0.smithi076.stdout: 1 955933 9.42 MB/sec execute 421 sec latency 37.928 ms 2024-03-18T15:03:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.786+0000 7faca7fff640 1 -- 172.21.15.76:0/2024677901 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5622d382f2d0 con 0x5622d3943da0 2024-03-18T15:03:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.786+0000 7faca7fff640 10 client.16644 handle_command_reply: tid=0 2024-03-18T15:03:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.786+0000 7facd1df1740 1 client.16644 shutdown 2024-03-18T15:03:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.786+0000 7facd1df1740 1 -- 172.21.15.76:0/2024677901 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38c3130 msgr2=0x5622d38d4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.786+0000 7facd1df1740 1 --2- 172.21.15.76:0/2024677901 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38c3130 0x5622d38d4820 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7facbc09b7e0 tx=0x7facbc009310 comp rx=0 tx=0).stop 2024-03-18T15:03:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.786+0000 7facd1df1740 1 -- 172.21.15.76:0/2024677901 shutdown_connections 2024-03-18T15:03:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.786+0000 7facd1df1740 1 --2- 172.21.15.76:0/2024677901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622d38c3a50 0x5622d38d17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.786+0000 7facd1df1740 1 --2- 172.21.15.76:0/2024677901 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5622d3943da0 0x5622d3946190 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.786+0000 7facd1df1740 1 --2- 172.21.15.76:0/2024677901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d38cc820 0x5622d38d1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.786+0000 7faca7fff640 0 client.16644 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.786+0000 7facd1df1740 1 --2- 172.21.15.76:0/2024677901 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5622d38c3130 0x5622d38d4820 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.786+0000 7facd1df1740 1 -- 172.21.15.76:0/2024677901 >> 172.21.15.76:0/2024677901 conn(0x5622d38bda50 msgr2=0x5622d38bf5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.787+0000 7facd1df1740 1 -- 172.21.15.76:0/2024677901 shutdown_connections 2024-03-18T15:03:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.787+0000 7facd1df1740 1 -- 172.21.15.76:0/2024677901 wait complete. 2024-03-18T15:03:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.787+0000 7facd1df1740 20 client.16644 trim_cache size 0 max 16384 2024-03-18T15:03:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:03:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:03:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:03:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:03:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.787+0000 7faccf20c640 1 -- 172.21.15.76:0/4265355382 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7facb80a2a10 msgr2=0x7facb80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.787+0000 7faccf20c640 1 --2- 172.21.15.76:0/4265355382 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7facb80a2a10 0x7facb80a4ed0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7facc0005fd0 tx=0x7facc0013b60 comp rx=0 tx=0).stop 2024-03-18T15:03:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.787+0000 7faccf20c640 1 -- 172.21.15.76:0/4265355382 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7facc8100930 msgr2=0x7facc8198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.787+0000 7faccf20c640 1 --2- 172.21.15.76:0/4265355382 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7facc8100930 0x7facc8198100 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7facbc00e9f0 tx=0x7facbc00edf0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.787+0000 7faccf20c640 1 -- 172.21.15.76:0/4265355382 shutdown_connections 2024-03-18T15:03:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.787+0000 7faccf20c640 1 --2- 172.21.15.76:0/4265355382 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7facb80a2a10 0x7facb80a4ed0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.787+0000 7faccf20c640 1 --2- 172.21.15.76:0/4265355382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facc81012f0 0x7facc8198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.787+0000 7faccf20c640 1 --2- 172.21.15.76:0/4265355382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facc8105680 0x7facc819c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.787+0000 7faccf20c640 1 --2- 172.21.15.76:0/4265355382 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7facc8100930 0x7facc8198100 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.787+0000 7faccf20c640 1 -- 172.21.15.76:0/4265355382 >> 172.21.15.76:0/4265355382 conn(0x7facc8077530 msgr2=0x7facc8069200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.787+0000 7faccf20c640 1 -- 172.21.15.76:0/4265355382 shutdown_connections 2024-03-18T15:03:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:00.787+0000 7faccf20c640 1 -- 172.21.15.76:0/4265355382 wait complete. 2024-03-18T15:03:01.758 INFO:tasks.workunit.client.0.smithi076.stdout: 1 956994 9.41 MB/sec execute 422 sec latency 82.329 ms 2024-03-18T15:03:02.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:01 smithi076 ceph-mon[32205]: pgmap v308: 97 pgs: 97 active+clean; 276 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 201 op/s 2024-03-18T15:03:02.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:01 smithi174 ceph-mon[36787]: pgmap v308: 97 pgs: 97 active+clean; 276 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 201 op/s 2024-03-18T15:03:02.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:01 smithi077 ceph-mon[35865]: pgmap v308: 97 pgs: 97 active+clean; 276 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 201 op/s 2024-03-18T15:03:02.758 INFO:tasks.workunit.client.0.smithi076.stdout: 1 959395 9.42 MB/sec execute 423 sec latency 41.129 ms 2024-03-18T15:03:03.758 INFO:tasks.workunit.client.0.smithi076.stdout: 1 959658 9.40 MB/sec execute 424 sec latency 54.456 ms 2024-03-18T15:03:04.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:03 smithi076 ceph-mon[32205]: osdmap e103: 12 total, 12 up, 12 in 2024-03-18T15:03:04.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:03 smithi076 ceph-mon[32205]: pgmap v310: 97 pgs: 26 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 57 active+clean; 274 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 319 op/s 2024-03-18T15:03:04.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:03 smithi174 ceph-mon[36787]: osdmap e103: 12 total, 12 up, 12 in 2024-03-18T15:03:04.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:03 smithi174 ceph-mon[36787]: pgmap v310: 97 pgs: 26 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 57 active+clean; 274 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 319 op/s 2024-03-18T15:03:04.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:03 smithi077 ceph-mon[35865]: osdmap e103: 12 total, 12 up, 12 in 2024-03-18T15:03:04.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:03 smithi077 ceph-mon[35865]: pgmap v310: 97 pgs: 26 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 57 active+clean; 274 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 319 op/s 2024-03-18T15:03:04.758 INFO:tasks.workunit.client.0.smithi076.stdout: 1 961111 9.39 MB/sec execute 425 sec latency 21.941 ms 2024-03-18T15:03:05.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 963251 9.41 MB/sec execute 426 sec latency 13.875 ms 2024-03-18T15:03:06.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:05 smithi076 ceph-mon[32205]: pgmap v311: 97 pgs: 26 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 57 active+clean; 274 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 319 op/s 2024-03-18T15:03:06.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:05 smithi174 ceph-mon[36787]: pgmap v311: 97 pgs: 26 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 57 active+clean; 274 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 319 op/s 2024-03-18T15:03:06.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:05 smithi077 ceph-mon[35865]: pgmap v311: 97 pgs: 26 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 57 active+clean; 274 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 319 op/s 2024-03-18T15:03:06.758 INFO:tasks.workunit.client.0.smithi076.stdout: 1 964813 9.40 MB/sec execute 427 sec latency 25.548 ms 2024-03-18T15:03:07.758 INFO:tasks.workunit.client.0.smithi076.stdout: 1 966751 9.41 MB/sec execute 428 sec latency 45.698 ms 2024-03-18T15:03:08.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:07 smithi076 ceph-mon[32205]: pgmap v312: 97 pgs: 26 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 57 active+clean; 274 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 319 op/s 2024-03-18T15:03:08.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:07 smithi174 ceph-mon[36787]: pgmap v312: 97 pgs: 26 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 57 active+clean; 274 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 319 op/s 2024-03-18T15:03:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:07 smithi077 ceph-mon[35865]: pgmap v312: 97 pgs: 26 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 57 active+clean; 274 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 319 op/s 2024-03-18T15:03:08.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 968018 9.40 MB/sec execute 429 sec latency 45.792 ms 2024-03-18T15:03:09.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 970202 9.41 MB/sec execute 430 sec latency 16.110 ms 2024-03-18T15:03:10.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:09 smithi076 ceph-mon[32205]: pgmap v313: 97 pgs: 97 active+clean; 282 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 319 op/s 2024-03-18T15:03:10.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:09 smithi174 ceph-mon[36787]: pgmap v313: 97 pgs: 97 active+clean; 282 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 319 op/s 2024-03-18T15:03:10.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:09 smithi077 ceph-mon[35865]: pgmap v313: 97 pgs: 97 active+clean; 282 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 319 op/s 2024-03-18T15:03:10.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 971483 9.40 MB/sec execute 431 sec latency 20.333 ms 2024-03-18T15:03:11.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 973732 9.41 MB/sec execute 432 sec latency 17.071 ms 2024-03-18T15:03:12.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:11 smithi076 ceph-mon[32205]: pgmap v314: 97 pgs: 97 active+clean; 282 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 319 op/s 2024-03-18T15:03:12.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:11 smithi174 ceph-mon[36787]: pgmap v314: 97 pgs: 97 active+clean; 282 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 319 op/s 2024-03-18T15:03:12.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:11 smithi077 ceph-mon[35865]: pgmap v314: 97 pgs: 97 active+clean; 282 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 319 op/s 2024-03-18T15:03:12.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 974998 9.40 MB/sec execute 433 sec latency 26.120 ms 2024-03-18T15:03:13.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 977090 9.40 MB/sec execute 434 sec latency 83.020 ms 2024-03-18T15:03:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.879+0000 7f403f4ae640 1 -- 172.21.15.76:0/31388532 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4038100930 msgr2=0x7f4038100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.879+0000 7f403f4ae640 1 --2- 172.21.15.76:0/31388532 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4038100930 0x7f4038100db0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4028009900 tx=0x7f402802bc30 comp rx=0 tx=0).stop 2024-03-18T15:03:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.879+0000 7f403f4ae640 1 -- 172.21.15.76:0/31388532 shutdown_connections 2024-03-18T15:03:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.879+0000 7f403f4ae640 1 --2- 172.21.15.76:0/31388532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40381012f0 0x7f403810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.879+0000 7f403f4ae640 1 --2- 172.21.15.76:0/31388532 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4038100930 0x7f4038100db0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.879+0000 7f403f4ae640 1 --2- 172.21.15.76:0/31388532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4038105680 0x7f4038105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.879+0000 7f403f4ae640 1 -- 172.21.15.76:0/31388532 >> 172.21.15.76:0/31388532 conn(0x7f4038077530 msgr2=0x7f4038077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.879+0000 7f403f4ae640 1 -- 172.21.15.76:0/31388532 shutdown_connections 2024-03-18T15:03:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.879+0000 7f403f4ae640 1 -- 172.21.15.76:0/31388532 wait complete. 2024-03-18T15:03:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403f4ae640 1 Processor -- start 2024-03-18T15:03:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403f4ae640 1 -- start start 2024-03-18T15:03:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403f4ae640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4038100930 0x7f40381980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403f4ae640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40381012f0 0x7f4038198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403f4ae640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4038105680 0x7f403819c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403f4ae640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4038110ba0 con 0x7f4038100930 2024-03-18T15:03:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403f4ae640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4038110a20 con 0x7f40381012f0 2024-03-18T15:03:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403f4ae640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4038110d20 con 0x7f4038105680 2024-03-18T15:03:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403d24a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4038100930 0x7f40381980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403d24a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4038100930 0x7f40381980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58742/0 (socket says 172.21.15.76:58742) 2024-03-18T15:03:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403ca49640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40381012f0 0x7f4038198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403d24a640 1 -- 172.21.15.76:0/327257401 learned_addr learned my addr 172.21.15.76:0/327257401 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403da4b640 1 --2- 172.21.15.76:0/327257401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4038105680 0x7f403819c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403d24a640 1 -- 172.21.15.76:0/327257401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4038105680 msgr2=0x7f403819c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403d24a640 1 --2- 172.21.15.76:0/327257401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4038105680 0x7f403819c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403d24a640 1 -- 172.21.15.76:0/327257401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40381012f0 msgr2=0x7f4038198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403d24a640 1 --2- 172.21.15.76:0/327257401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40381012f0 0x7f4038198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.880+0000 7f403d24a640 1 -- 172.21.15.76:0/327257401 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f403819d0e0 con 0x7f4038100930 2024-03-18T15:03:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.881+0000 7f403d24a640 1 --2- 172.21.15.76:0/327257401 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4038100930 0x7f40381980f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f403400ea60 tx=0x7f403400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.881+0000 7f402e7fc640 1 -- 172.21.15.76:0/327257401 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403400cde0 con 0x7f4038100930 2024-03-18T15:03:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.881+0000 7f402e7fc640 1 -- 172.21.15.76:0/327257401 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40340035f0 con 0x7f4038100930 2024-03-18T15:03:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.881+0000 7f403f4ae640 1 -- 172.21.15.76:0/327257401 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f403819d3d0 con 0x7f4038100930 2024-03-18T15:03:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.881+0000 7f402e7fc640 1 -- 172.21.15.76:0/327257401 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4034010670 con 0x7f4038100930 2024-03-18T15:03:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.881+0000 7f403f4ae640 1 -- 172.21.15.76:0/327257401 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40381a4c60 con 0x7f4038100930 2024-03-18T15:03:13.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.884+0000 7f402e7fc640 1 -- 172.21.15.76:0/327257401 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f40340037a0 con 0x7f4038100930 2024-03-18T15:03:13.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.884+0000 7f4042170740 1 Processor -- start 2024-03-18T15:03:13.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.884+0000 7f4042170740 1 -- start start 2024-03-18T15:03:13.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.884+0000 7f4042170740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ad1130 0x55be15ad1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.884+0000 7f4042170740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ad1a50 0x55be15ada2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.890 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.884+0000 7f4042170740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ada820 0x55be15adcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.884+0000 7f4042170740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55be15a35a40 con 0x55be15ada820 2024-03-18T15:03:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.884+0000 7f4042170740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55be159de920 con 0x55be15ad1a50 2024-03-18T15:03:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.884+0000 7f4042170740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55be15a31ae0 con 0x55be15ad1130 2024-03-18T15:03:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f402e7fc640 1 --2- 172.21.15.76:0/327257401 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f40240a2a10 0x7f40240a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f403ca49640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ada820 0x55be15adcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f403d24a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ad1a50 0x55be15ada2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f403ca49640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ada820 0x55be15adcc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58744/0 (socket says 172.21.15.76:58744) 2024-03-18T15:03:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f403ca49640 1 -- 172.21.15.76:0/2480332465 learned_addr learned my addr 172.21.15.76:0/2480332465 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f403da4b640 1 --2- 172.21.15.76:0/2480332465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ad1130 0x55be15ad1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f403da4b640 1 --2- 172.21.15.76:0/327257401 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f40240a2a10 0x7f40240a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f402e7fc640 1 -- 172.21.15.76:0/327257401 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f40340cf0c0 con 0x7f4038100930 2024-03-18T15:03:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f403d24a640 1 -- 172.21.15.76:0/2480332465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ad1130 msgr2=0x55be15ad1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f403d24a640 1 --2- 172.21.15.76:0/2480332465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ad1130 0x55be15ad1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f403d24a640 1 -- 172.21.15.76:0/2480332465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ada820 msgr2=0x55be15adcc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f403d24a640 1 --2- 172.21.15.76:0/2480332465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ada820 0x55be15adcc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f403d24a640 1 -- 172.21.15.76:0/2480332465 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be159f6460 con 0x55be15ad1a50 2024-03-18T15:03:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f403ca49640 1 --2- 172.21.15.76:0/2480332465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ada820 0x55be15adcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:03:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f403da4b640 1 --2- 172.21.15.76:0/327257401 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f40240a2a10 0x7f40240a4ed0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4030005c40 tx=0x7f4030009500 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f403d24a640 1 --2- 172.21.15.76:0/2480332465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ad1a50 0x55be15ada2e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f40340cc830 tx=0x7f403409b640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b7a2e6d37143476 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f40237fe640 1 -- 172.21.15.76:0/2480332465 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403409be40 con 0x55be15ad1a50 2024-03-18T15:03:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f40237fe640 1 -- 172.21.15.76:0/2480332465 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f403409b740 con 0x55be15ad1a50 2024-03-18T15:03:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.885+0000 7f40237fe640 1 -- 172.21.15.76:0/2480332465 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4034003c50 con 0x55be15ad1a50 2024-03-18T15:03:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.886+0000 7f4042170740 1 -- 172.21.15.76:0/2480332465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ad1a50 msgr2=0x55be15ada2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.886+0000 7f4042170740 1 --2- 172.21.15.76:0/2480332465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ad1a50 0x55be15ada2e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f40340cc830 tx=0x7f403409b640 comp rx=0 tx=0).stop 2024-03-18T15:03:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.886+0000 7f4042170740 1 -- 172.21.15.76:0/2480332465 shutdown_connections 2024-03-18T15:03:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.886+0000 7f4042170740 1 --2- 172.21.15.76:0/2480332465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ada820 0x55be15adcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.886+0000 7f4042170740 1 --2- 172.21.15.76:0/2480332465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ad1a50 0x55be15ada2e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.886+0000 7f4042170740 1 --2- 172.21.15.76:0/2480332465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ad1130 0x55be15ad1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.886+0000 7f4042170740 1 -- 172.21.15.76:0/2480332465 >> 172.21.15.76:0/2480332465 conn(0x55be15acba50 msgr2=0x55be15acbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.886+0000 7f4042170740 1 -- 172.21.15.76:0/2480332465 shutdown_connections 2024-03-18T15:03:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.886+0000 7f4042170740 1 -- 172.21.15.76:0/2480332465 wait complete. 2024-03-18T15:03:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f4042170740 1 -- start start 2024-03-18T15:03:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f4042170740 1 Processor -- start 2024-03-18T15:03:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f4042170740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ad1130 0x55be15ad7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f4042170740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 0x55be15ad7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f4042170740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ada820 0x55be15ad83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f4042170740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55be15add390 con 0x55be15ad1a50 2024-03-18T15:03:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f4042170740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55be159f1d70 con 0x55be15ad1130 2024-03-18T15:03:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f4042170740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55be15add510 con 0x55be15ada820 2024-03-18T15:03:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f4042170740 10 client.0 _ll_register_callbacks cb 0x55be15749a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:03:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f403ca49640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 0x55be15ad7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f403d24a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ad1130 0x55be15ad7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f403d24a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ad1130 0x55be15ad7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:34366/0 (socket says 172.21.15.76:34366) 2024-03-18T15:03:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f403d24a640 1 -- 172.21.15.76:0/3889249136 learned_addr learned my addr 172.21.15.76:0/3889249136 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f403da4b640 1 --2- 172.21.15.76:0/3889249136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ada820 0x55be15ad83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f403d24a640 1 -- 172.21.15.76:0/3889249136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ada820 msgr2=0x55be15ad83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f403d24a640 1 --2- 172.21.15.76:0/3889249136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ada820 0x55be15ad83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f403d24a640 1 -- 172.21.15.76:0/3889249136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 msgr2=0x55be15ad7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f403d24a640 1 --2- 172.21.15.76:0/3889249136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 0x55be15ad7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.887+0000 7f403d24a640 1 -- 172.21.15.76:0/3889249136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be1574a210 con 0x55be15ad1130 2024-03-18T15:03:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.888+0000 7f403d24a640 1 --2- 172.21.15.76:0/3889249136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ad1130 0x55be15ad7920 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4034098110 tx=0x7f403409f890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc9b48d7fb2d6baf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.888+0000 7f4020ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T15:03:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.888+0000 7f4020ff9640 1 -- 172.21.15.76:0/3889249136 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40340a3ee0 con 0x55be15ad1130 2024-03-18T15:03:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.888+0000 7f4020ff9640 1 -- 172.21.15.76:0/3889249136 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f40340ccab0 con 0x55be15ad1130 2024-03-18T15:03:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.888+0000 7f4042170740 1 -- 172.21.15.76:0/3889249136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f403409b740 con 0x55be15ad1130 2024-03-18T15:03:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.888+0000 7f4020ff9640 1 -- 172.21.15.76:0/3889249136 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f4034003c30 con 0x55be15ad1130 2024-03-18T15:03:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.888+0000 7f4042170740 10 client.26519 fetch_fsmap learned FSMap version 42 2024-03-18T15:03:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.888+0000 7f4042170740 1 -- 172.21.15.76:0/3889249136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55be15816620 con 0x55be15ad1130 2024-03-18T15:03:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.888+0000 7f4020ff9640 1 -- 172.21.15.76:0/3889249136 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f403409b740 con 0x55be15ad1130 2024-03-18T15:03:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.888+0000 7f4020ff9640 1 -- 172.21.15.76:0/3889249136 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7f403409ab90 con 0x55be15ad1130 2024-03-18T15:03:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.888+0000 7f4042170740 10 client.26519 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T15:03:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.888+0000 7f4042170740 10 client.26519 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:03:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.888+0000 7f4042170740 20 client.26519 populate_metadata read hostname 'smithi076' 2024-03-18T15:03:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.888+0000 7f4042170740 1 --2- 172.21.15.76:0/3889249136 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55be15aea850 0x55be15aecc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.888+0000 7f4042170740 4 client.26519 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:03:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.888+0000 7f4042170740 1 -- 172.21.15.76:0/3889249136 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be15a42f80 con 0x55be15aea850 2024-03-18T15:03:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.889+0000 7f403ca49640 1 --2- 172.21.15.76:0/3889249136 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55be15aea850 0x55be15aecc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.889+0000 7f4020ff9640 10 client.26519 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.889+0000 7f403ca49640 1 --2- 172.21.15.76:0/3889249136 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55be15aea850 0x55be15aecc40 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62a14b2a0c17e58e server_cookie=3ef3cd50f48a9049 in_seq=0 out_seq=0 2024-03-18T15:03:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.891+0000 7f4020ff9640 1 -- 172.21.15.76:0/3889249136 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55be15a42f80 con 0x55be15aea850 2024-03-18T15:03:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.891+0000 7f4020ff9640 10 client.26519 handle_command_reply: tid=0 2024-03-18T15:03:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.891+0000 7f4042170740 1 client.26519 shutdown 2024-03-18T15:03:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.891+0000 7f4042170740 1 -- 172.21.15.76:0/3889249136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ad1130 msgr2=0x55be15ad7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.891+0000 7f4042170740 1 --2- 172.21.15.76:0/3889249136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ad1130 0x55be15ad7920 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4034098110 tx=0x7f403409f890 comp rx=0 tx=0).stop 2024-03-18T15:03:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.891+0000 7f4042170740 1 -- 172.21.15.76:0/3889249136 shutdown_connections 2024-03-18T15:03:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.891+0000 7f4042170740 1 --2- 172.21.15.76:0/3889249136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ada820 0x55be15ad83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.891+0000 7f4042170740 1 --2- 172.21.15.76:0/3889249136 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55be15aea850 0x55be15aecc40 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.891+0000 7f4042170740 1 --2- 172.21.15.76:0/3889249136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 0x55be15ad7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.891+0000 7f4042170740 1 --2- 172.21.15.76:0/3889249136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ad1130 0x55be15ad7920 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.891+0000 7f4020ff9640 0 client.26519 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.891+0000 7f4042170740 1 -- 172.21.15.76:0/3889249136 >> 172.21.15.76:0/3889249136 conn(0x55be15acba50 msgr2=0x55be15acd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.891+0000 7f4042170740 1 -- 172.21.15.76:0/3889249136 shutdown_connections 2024-03-18T15:03:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.891+0000 7f4042170740 1 -- 172.21.15.76:0/3889249136 wait complete. 2024-03-18T15:03:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.892+0000 7f4042170740 20 client.26519 trim_cache size 0 max 16384 2024-03-18T15:03:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.911+0000 7f4042170740 1 Processor -- start 2024-03-18T15:03:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.911+0000 7f4042170740 1 -- start start 2024-03-18T15:03:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.911+0000 7f4042170740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ad1130 0x55be15ada050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.911+0000 7f4042170740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 0x55be15ae5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.911+0000 7f4042170740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ada820 0x55be15aefc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.911+0000 7f4042170740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f40381afcc0 con 0x55be15ad1a50 2024-03-18T15:03:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.911+0000 7f4042170740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f40381ad590 con 0x55be15ada820 2024-03-18T15:03:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.911+0000 7f4042170740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f40381ab400 con 0x55be15ad1130 2024-03-18T15:03:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.911+0000 7f403ca49640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 0x55be15ae5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.911+0000 7f403ca49640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 0x55be15ae5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58764/0 (socket says 172.21.15.76:58764) 2024-03-18T15:03:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.911+0000 7f403da4b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ada820 0x55be15aefc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.911+0000 7f403d24a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ad1130 0x55be15ada050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.911+0000 7f403ca49640 1 -- 172.21.15.76:0/364975098 learned_addr learned my addr 172.21.15.76:0/364975098 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.912+0000 7f403ca49640 1 -- 172.21.15.76:0/364975098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ad1130 msgr2=0x55be15ada050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.912+0000 7f403ca49640 1 --2- 172.21.15.76:0/364975098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ad1130 0x55be15ada050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.912+0000 7f403ca49640 1 -- 172.21.15.76:0/364975098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ada820 msgr2=0x55be15aefc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.912+0000 7f403ca49640 1 --2- 172.21.15.76:0/364975098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ada820 0x55be15aefc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.912+0000 7f403ca49640 1 -- 172.21.15.76:0/364975098 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be15acb580 con 0x55be15ad1a50 2024-03-18T15:03:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.912+0000 7f403ca49640 1 --2- 172.21.15.76:0/364975098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 0x55be15ae5e70 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f4028036040 tx=0x7f4028005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eaa305973f0277a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.912+0000 7f4022ffd640 1 -- 172.21.15.76:0/364975098 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028004240 con 0x55be15ad1a50 2024-03-18T15:03:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.912+0000 7f4022ffd640 1 -- 172.21.15.76:0/364975098 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f402804f030 con 0x55be15ad1a50 2024-03-18T15:03:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.912+0000 7f4022ffd640 1 -- 172.21.15.76:0/364975098 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028004ec0 con 0x55be15ad1a50 2024-03-18T15:03:13.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.913+0000 7f4042170740 1 -- 172.21.15.76:0/364975098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 msgr2=0x55be15ae5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:13.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.913+0000 7f4042170740 1 --2- 172.21.15.76:0/364975098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 0x55be15ae5e70 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f4028036040 tx=0x7f4028005fd0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.913+0000 7f4042170740 1 -- 172.21.15.76:0/364975098 shutdown_connections 2024-03-18T15:03:13.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.913+0000 7f4042170740 1 --2- 172.21.15.76:0/364975098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ada820 0x55be15aefc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.913+0000 7f4042170740 1 --2- 172.21.15.76:0/364975098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 0x55be15ae5e70 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.913+0000 7f4042170740 1 --2- 172.21.15.76:0/364975098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ad1130 0x55be15ada050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.913+0000 7f4042170740 1 -- 172.21.15.76:0/364975098 >> 172.21.15.76:0/364975098 conn(0x55be15acba50 msgr2=0x55be15ae5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:13.918 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.913+0000 7f4042170740 1 -- 172.21.15.76:0/364975098 shutdown_connections 2024-03-18T15:03:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.913+0000 7f4042170740 1 -- 172.21.15.76:0/364975098 wait complete. 2024-03-18T15:03:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.913+0000 7f4042170740 1 -- start start 2024-03-18T15:03:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.913+0000 7f4042170740 1 Processor -- start 2024-03-18T15:03:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f4042170740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ad1130 0x55be15aedc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f4042170740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 0x55be15aee180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f4042170740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ada820 0x55be15b4f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f4042170740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f40381aba50 con 0x55be15ad1a50 2024-03-18T15:03:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f4042170740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f40381acec0 con 0x55be15ada820 2024-03-18T15:03:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f4042170740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f40381ac120 con 0x55be15ad1130 2024-03-18T15:03:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f4042170740 10 client.0 _ll_register_callbacks cb 0x7f40300093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:03:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f403ca49640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 0x55be15aee180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f403ca49640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 0x55be15aee180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58770/0 (socket says 172.21.15.76:58770) 2024-03-18T15:03:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f403ca49640 1 -- 172.21.15.76:0/2064592081 learned_addr learned my addr 172.21.15.76:0/2064592081 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f403d24a640 1 --2- 172.21.15.76:0/2064592081 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ad1130 0x55be15aedc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f403da4b640 1 --2- 172.21.15.76:0/2064592081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ada820 0x55be15b4f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f403ca49640 1 -- 172.21.15.76:0/2064592081 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ad1130 msgr2=0x55be15aedc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:13.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f403ca49640 1 --2- 172.21.15.76:0/2064592081 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ad1130 0x55be15aedc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f403ca49640 1 -- 172.21.15.76:0/2064592081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ada820 msgr2=0x55be15b4f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:13.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f403ca49640 1 --2- 172.21.15.76:0/2064592081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ada820 0x55be15b4f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f403ca49640 1 -- 172.21.15.76:0/2064592081 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be15b4fd30 con 0x55be15ad1a50 2024-03-18T15:03:13.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f403ca49640 1 --2- 172.21.15.76:0/2064592081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 0x55be15aee180 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f402802c170 tx=0x7f40280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e25aa48fd22651 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:13.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f4020ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:03:13.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f4020ff9640 1 -- 172.21.15.76:0/2064592081 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402802c8b0 con 0x55be15ad1a50 2024-03-18T15:03:13.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f4020ff9640 1 -- 172.21.15.76:0/2064592081 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f402802ca50 con 0x55be15ad1a50 2024-03-18T15:03:13.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f4042170740 1 -- 172.21.15.76:0/2064592081 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f402804f030 con 0x55be15ad1a50 2024-03-18T15:03:13.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f4020ff9640 1 -- 172.21.15.76:0/2064592081 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f4028063050 con 0x55be15ad1a50 2024-03-18T15:03:13.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.914+0000 7f4042170740 10 client.16662 fetch_fsmap learned FSMap version 42 2024-03-18T15:03:13.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.915+0000 7f4042170740 1 -- 172.21.15.76:0/2064592081 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55be15b50fb0 con 0x55be15ad1a50 2024-03-18T15:03:13.920 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.915+0000 7f4020ff9640 1 -- 172.21.15.76:0/2064592081 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f402804f030 con 0x55be15ad1a50 2024-03-18T15:03:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.915+0000 7f4020ff9640 1 -- 172.21.15.76:0/2064592081 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7f40280054a0 con 0x55be15ad1a50 2024-03-18T15:03:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.915+0000 7f4042170740 10 client.16662 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T15:03:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.915+0000 7f4042170740 10 client.16662 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:03:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.915+0000 7f4042170740 20 client.16662 populate_metadata read hostname 'smithi076' 2024-03-18T15:03:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.915+0000 7f4042170740 1 --2- 172.21.15.76:0/2064592081 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55be15b516a0 0x55be15b53ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.915+0000 7f4042170740 4 client.16662 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:03:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.915+0000 7f4042170740 1 -- 172.21.15.76:0/2064592081 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55be15a3d2d0 con 0x55be15b516a0 2024-03-18T15:03:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.915+0000 7f403d24a640 1 --2- 172.21.15.76:0/2064592081 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55be15b516a0 0x55be15b53ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.915+0000 7f403d24a640 1 --2- 172.21.15.76:0/2064592081 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55be15b516a0 0x55be15b53ac0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ddd7d7b793ee5ac server_cookie=1fcb5210bbc4ce2e in_seq=0 out_seq=0 2024-03-18T15:03:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.915+0000 7f4020ff9640 10 client.16662 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:13.996 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.995+0000 7f4020ff9640 1 -- 172.21.15.76:0/2064592081 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55be15a3d2d0 con 0x55be15b516a0 2024-03-18T15:03:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.995+0000 7f4020ff9640 10 client.16662 handle_command_reply: tid=0 2024-03-18T15:03:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.995+0000 7f4042170740 1 client.16662 shutdown 2024-03-18T15:03:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.995+0000 7f4042170740 1 -- 172.21.15.76:0/2064592081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 msgr2=0x55be15aee180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.995+0000 7f4042170740 1 --2- 172.21.15.76:0/2064592081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 0x55be15aee180 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f402802c170 tx=0x7f40280046c0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.995+0000 7f4042170740 1 -- 172.21.15.76:0/2064592081 shutdown_connections 2024-03-18T15:03:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.995+0000 7f4042170740 1 --2- 172.21.15.76:0/2064592081 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55be15b516a0 0x55be15b53ac0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.995+0000 7f4042170740 1 --2- 172.21.15.76:0/2064592081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55be15ada820 0x55be15b4f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.995+0000 7f4042170740 1 --2- 172.21.15.76:0/2064592081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55be15ad1a50 0x55be15aee180 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.995+0000 7f4042170740 1 --2- 172.21.15.76:0/2064592081 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55be15ad1130 0x55be15aedc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.995+0000 7f4042170740 1 -- 172.21.15.76:0/2064592081 >> 172.21.15.76:0/2064592081 conn(0x55be15acba50 msgr2=0x55be15ae5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.995+0000 7f4020ff9640 0 client.16662 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.996+0000 7f4042170740 1 -- 172.21.15.76:0/2064592081 shutdown_connections 2024-03-18T15:03:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.996+0000 7f4042170740 1 -- 172.21.15.76:0/2064592081 wait complete. 2024-03-18T15:03:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.996+0000 7f4042170740 20 client.16662 trim_cache size 0 max 16384 2024-03-18T15:03:13.998 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:03:13.998 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:03:13.998 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:03:13.998 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:03:13.998 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.997+0000 7f403f4ae640 1 -- 172.21.15.76:0/327257401 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f40240a2a10 msgr2=0x7f40240a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:13.998 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.997+0000 7f403f4ae640 1 --2- 172.21.15.76:0/327257401 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f40240a2a10 0x7f40240a4ed0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4030005c40 tx=0x7f4030009500 comp rx=0 tx=0).stop 2024-03-18T15:03:13.999 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.997+0000 7f403f4ae640 1 -- 172.21.15.76:0/327257401 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4038100930 msgr2=0x7f40381980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:13.999 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:13.997+0000 7f403f4ae640 1 --2- 172.21.15.76:0/327257401 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4038100930 0x7f40381980f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f403400ea60 tx=0x7f403400ef20 comp rx=0 tx=0).stop 2024-03-18T15:03:14.001 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:14.000+0000 7f403f4ae640 1 -- 172.21.15.76:0/327257401 shutdown_connections 2024-03-18T15:03:14.001 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:14.000+0000 7f403f4ae640 1 --2- 172.21.15.76:0/327257401 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f40240a2a10 0x7f40240a4ed0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:14.001 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:14.000+0000 7f403f4ae640 1 --2- 172.21.15.76:0/327257401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4038105680 0x7f403819c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:14.001 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:14.000+0000 7f403f4ae640 1 --2- 172.21.15.76:0/327257401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40381012f0 0x7f4038198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:14.001 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:14.000+0000 7f403f4ae640 1 --2- 172.21.15.76:0/327257401 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4038100930 0x7f40381980f0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:14.001 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:14.000+0000 7f403f4ae640 1 -- 172.21.15.76:0/327257401 >> 172.21.15.76:0/327257401 conn(0x7f4038077530 msgr2=0x7f40380691a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:14.001 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:14.000+0000 7f403f4ae640 1 -- 172.21.15.76:0/327257401 shutdown_connections 2024-03-18T15:03:14.001 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:14.000+0000 7f403f4ae640 1 -- 172.21.15.76:0/327257401 wait complete. 2024-03-18T15:03:14.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:14 smithi076 ceph-mon[32205]: osdmap e104: 12 total, 12 up, 12 in 2024-03-18T15:03:14.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:14 smithi076 ceph-mon[32205]: pgmap v316: 97 pgs: 97 active+clean; 310 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.2 MiB/s wr, 240 op/s 2024-03-18T15:03:14.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:14 smithi174 ceph-mon[36787]: osdmap e104: 12 total, 12 up, 12 in 2024-03-18T15:03:14.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:14 smithi174 ceph-mon[36787]: pgmap v316: 97 pgs: 97 active+clean; 310 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.2 MiB/s wr, 240 op/s 2024-03-18T15:03:14.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:14 smithi077 ceph-mon[35865]: osdmap e104: 12 total, 12 up, 12 in 2024-03-18T15:03:14.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:14 smithi077 ceph-mon[35865]: pgmap v316: 97 pgs: 97 active+clean; 310 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.2 MiB/s wr, 240 op/s 2024-03-18T15:03:14.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 977985 9.39 MB/sec execute 435 sec latency 62.370 ms 2024-03-18T15:03:15.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 980391 9.40 MB/sec execute 436 sec latency 21.424 ms 2024-03-18T15:03:16.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:15 smithi076 ceph-mon[32205]: pgmap v317: 97 pgs: 97 active+clean; 310 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.2 MiB/s wr, 240 op/s 2024-03-18T15:03:16.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:15 smithi174 ceph-mon[36787]: pgmap v317: 97 pgs: 97 active+clean; 310 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.2 MiB/s wr, 240 op/s 2024-03-18T15:03:16.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:15 smithi077 ceph-mon[35865]: pgmap v317: 97 pgs: 97 active+clean; 310 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.2 MiB/s wr, 240 op/s 2024-03-18T15:03:16.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 981615 9.39 MB/sec execute 437 sec latency 35.615 ms 2024-03-18T15:03:17.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 984102 9.41 MB/sec execute 438 sec latency 23.476 ms 2024-03-18T15:03:18.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:17 smithi076 ceph-mon[32205]: pgmap v318: 97 pgs: 97 active+clean; 310 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.2 MiB/s wr, 240 op/s 2024-03-18T15:03:18.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:17 smithi174 ceph-mon[36787]: pgmap v318: 97 pgs: 97 active+clean; 310 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.2 MiB/s wr, 240 op/s 2024-03-18T15:03:18.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:17 smithi077 ceph-mon[35865]: pgmap v318: 97 pgs: 97 active+clean; 310 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.2 MiB/s wr, 240 op/s 2024-03-18T15:03:18.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 985045 9.39 MB/sec execute 439 sec latency 56.478 ms 2024-03-18T15:03:19.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 987445 9.40 MB/sec execute 440 sec latency 28.512 ms 2024-03-18T15:03:20.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:19 smithi076 ceph-mon[32205]: pgmap v319: 97 pgs: 97 active+clean; 310 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.1 MiB/s wr, 268 op/s 2024-03-18T15:03:20.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:19 smithi174 ceph-mon[36787]: pgmap v319: 97 pgs: 97 active+clean; 310 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.1 MiB/s wr, 268 op/s 2024-03-18T15:03:20.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:19 smithi077 ceph-mon[35865]: pgmap v319: 97 pgs: 97 active+clean; 310 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.1 MiB/s wr, 268 op/s 2024-03-18T15:03:20.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 988451 9.39 MB/sec execute 441 sec latency 23.664 ms 2024-03-18T15:03:21.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 990656 9.40 MB/sec execute 442 sec latency 24.296 ms 2024-03-18T15:03:22.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:21 smithi076 ceph-mon[32205]: pgmap v320: 97 pgs: 97 active+clean; 310 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.1 MiB/s wr, 268 op/s 2024-03-18T15:03:22.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:21 smithi174 ceph-mon[36787]: pgmap v320: 97 pgs: 97 active+clean; 310 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.1 MiB/s wr, 268 op/s 2024-03-18T15:03:22.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:21 smithi077 ceph-mon[35865]: pgmap v320: 97 pgs: 97 active+clean; 310 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.1 MiB/s wr, 268 op/s 2024-03-18T15:03:22.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 991899 9.39 MB/sec execute 443 sec latency 24.373 ms 2024-03-18T15:03:23.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 993578 9.38 MB/sec execute 444 sec latency 82.428 ms 2024-03-18T15:03:24.018 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:23 smithi174 ceph-mon[36787]: pgmap v321: 97 pgs: 97 active+clean; 317 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 376 op/s 2024-03-18T15:03:24.028 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:23 smithi077 ceph-mon[35865]: pgmap v321: 97 pgs: 97 active+clean; 317 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 376 op/s 2024-03-18T15:03:24.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:23 smithi076 ceph-mon[32205]: pgmap v321: 97 pgs: 97 active+clean; 317 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 376 op/s 2024-03-18T15:03:24.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 995208 9.39 MB/sec execute 445 sec latency 29.355 ms 2024-03-18T15:03:25.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 997170 9.39 MB/sec execute 446 sec latency 17.197 ms 2024-03-18T15:03:26.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:25 smithi076 ceph-mon[32205]: pgmap v322: 97 pgs: 97 active+clean; 317 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 225 op/s 2024-03-18T15:03:26.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:25 smithi174 ceph-mon[36787]: pgmap v322: 97 pgs: 97 active+clean; 317 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 225 op/s 2024-03-18T15:03:26.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:25 smithi077 ceph-mon[35865]: pgmap v322: 97 pgs: 97 active+clean; 317 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 225 op/s 2024-03-18T15:03:26.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 998887 9.39 MB/sec execute 447 sec latency 12.996 ms 2024-03-18T15:03:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.092+0000 7ff6d39f4640 1 -- 172.21.15.76:0/20665925 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6cc105680 msgr2=0x7ff6cc105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.092+0000 7ff6d39f4640 1 --2- 172.21.15.76:0/20665925 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6cc105680 0x7ff6cc105a60 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009900 tx=0x7ff6c802bc30 comp rx=0 tx=0).stop 2024-03-18T15:03:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.093+0000 7ff6d39f4640 1 -- 172.21.15.76:0/20665925 shutdown_connections 2024-03-18T15:03:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.093+0000 7ff6d39f4640 1 --2- 172.21.15.76:0/20665925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6cc1012f0 0x7ff6cc10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.093+0000 7ff6d39f4640 1 --2- 172.21.15.76:0/20665925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc100930 0x7ff6cc100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.093+0000 7ff6d39f4640 1 --2- 172.21.15.76:0/20665925 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6cc105680 0x7ff6cc105a60 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.093+0000 7ff6d39f4640 1 -- 172.21.15.76:0/20665925 >> 172.21.15.76:0/20665925 conn(0x7ff6cc077530 msgr2=0x7ff6cc077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.093+0000 7ff6d39f4640 1 -- 172.21.15.76:0/20665925 shutdown_connections 2024-03-18T15:03:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.093+0000 7ff6d39f4640 1 -- 172.21.15.76:0/20665925 wait complete. 2024-03-18T15:03:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.093+0000 7ff6d39f4640 1 Processor -- start 2024-03-18T15:03:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.093+0000 7ff6d39f4640 1 -- start start 2024-03-18T15:03:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.093+0000 7ff6d39f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc100930 0x7ff6cc198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.093+0000 7ff6d39f4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6cc1012f0 0x7ff6cc1993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.093+0000 7ff6d39f4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6cc105680 0x7ff6cc1a3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.093+0000 7ff6d39f4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc110ba0 con 0x7ff6cc1012f0 2024-03-18T15:03:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.093+0000 7ff6d39f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc110a20 con 0x7ff6cc100930 2024-03-18T15:03:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.093+0000 7ff6d39f4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc110d20 con 0x7ff6cc105680 2024-03-18T15:03:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.094+0000 7ff6d0f8f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6cc1012f0 0x7ff6cc1993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.094+0000 7ff6d1790640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc100930 0x7ff6cc198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.094+0000 7ff6d0f8f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6cc1012f0 0x7ff6cc1993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59972/0 (socket says 172.21.15.76:59972) 2024-03-18T15:03:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.094+0000 7ff6d0f8f640 1 -- 172.21.15.76:0/2973086870 learned_addr learned my addr 172.21.15.76:0/2973086870 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.094+0000 7ff6d1f91640 1 --2- 172.21.15.76:0/2973086870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6cc105680 0x7ff6cc1a3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.094+0000 7ff6d0f8f640 1 -- 172.21.15.76:0/2973086870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6cc105680 msgr2=0x7ff6cc1a3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.094+0000 7ff6d0f8f640 1 --2- 172.21.15.76:0/2973086870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6cc105680 0x7ff6cc1a3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.094+0000 7ff6d0f8f640 1 -- 172.21.15.76:0/2973086870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc100930 msgr2=0x7ff6cc198e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.094+0000 7ff6d0f8f640 1 --2- 172.21.15.76:0/2973086870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc100930 0x7ff6cc198e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.094+0000 7ff6d0f8f640 1 -- 172.21.15.76:0/2973086870 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6cc1a4440 con 0x7ff6cc1012f0 2024-03-18T15:03:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.094+0000 7ff6d1790640 1 --2- 172.21.15.76:0/2973086870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc100930 0x7ff6cc198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:03:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.094+0000 7ff6d0f8f640 1 --2- 172.21.15.76:0/2973086870 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6cc1012f0 0x7ff6cc1993c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc00e9f0 tx=0x7ff6bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.094+0000 7ff6c27fc640 1 -- 172.21.15.76:0/2973086870 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc00cd80 con 0x7ff6cc1012f0 2024-03-18T15:03:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.094+0000 7ff6c27fc640 1 -- 172.21.15.76:0/2973086870 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6bc010910 con 0x7ff6cc1012f0 2024-03-18T15:03:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.094+0000 7ff6d39f4640 1 -- 172.21.15.76:0/2973086870 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6cc1a4610 con 0x7ff6cc1012f0 2024-03-18T15:03:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.094+0000 7ff6c27fc640 1 -- 172.21.15.76:0/2973086870 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc018990 con 0x7ff6cc1012f0 2024-03-18T15:03:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.094+0000 7ff6d39f4640 1 -- 172.21.15.76:0/2973086870 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6cc1a4b30 con 0x7ff6cc1012f0 2024-03-18T15:03:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.097+0000 7ff6c27fc640 1 -- 172.21.15.76:0/2973086870 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7ff6bc010ac0 con 0x7ff6cc1012f0 2024-03-18T15:03:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.097+0000 7ff6d65e8740 1 Processor -- start 2024-03-18T15:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.097+0000 7ff6d65e8740 1 -- start start 2024-03-18T15:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.097+0000 7ff6d65e8740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59130 0x55c12df59510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.097+0000 7ff6d65e8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df59a50 0x55c12df622e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.097+0000 7ff6d65e8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df62820 0x55c12df64c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.097+0000 7ff6d65e8740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c12debda40 con 0x55c12df59130 2024-03-18T15:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.097+0000 7ff6d65e8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c12de66920 con 0x55c12df62820 2024-03-18T15:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.097+0000 7ff6d65e8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c12deb9ae0 con 0x55c12df59a50 2024-03-18T15:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.097+0000 7ff6d1f91640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59130 0x55c12df59510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.097+0000 7ff6d0f8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df62820 0x55c12df64c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.097+0000 7ff6d1f91640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59130 0x55c12df59510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59982/0 (socket says 172.21.15.76:59982) 2024-03-18T15:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.097+0000 7ff6d1f91640 1 -- 172.21.15.76:0/2708507888 learned_addr learned my addr 172.21.15.76:0/2708507888 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.097+0000 7ff6d1790640 1 --2- 172.21.15.76:0/2708507888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df59a50 0x55c12df622e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.098+0000 7ff6c27fc640 1 --2- 172.21.15.76:0/2973086870 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6b80a2a10 0x7ff6b80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.098+0000 7ff6d1f91640 1 -- 172.21.15.76:0/2708507888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df59a50 msgr2=0x55c12df622e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.098+0000 7ff6d1f91640 1 --2- 172.21.15.76:0/2708507888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df59a50 0x55c12df622e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.098+0000 7ff6d1f91640 1 -- 172.21.15.76:0/2708507888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df62820 msgr2=0x55c12df64c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.098+0000 7ff6d1f91640 1 --2- 172.21.15.76:0/2708507888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df62820 0x55c12df64c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.098+0000 7ff6d1f91640 1 -- 172.21.15.76:0/2708507888 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c12de7e460 con 0x55c12df59130 2024-03-18T15:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.098+0000 7ff6d0f8f640 1 --2- 172.21.15.76:0/2708507888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df62820 0x55c12df64c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.098+0000 7ff6d1f91640 1 --2- 172.21.15.76:0/2973086870 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6b80a2a10 0x7ff6b80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.098+0000 7ff6c27fc640 1 -- 172.21.15.76:0/2973086870 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7ff6bc0cf1a0 con 0x7ff6cc1012f0 2024-03-18T15:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.098+0000 7ff6d1f91640 1 --2- 172.21.15.76:0/2708507888 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59130 0x55c12df59510 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4011590 tx=0x7ff6c4011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0bd0bfc33ca1ce9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.098+0000 7ff6b77fe640 1 -- 172.21.15.76:0/2708507888 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c401f020 con 0x55c12df59130 2024-03-18T15:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.098+0000 7ff6b77fe640 1 -- 172.21.15.76:0/2708507888 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6c401be20 con 0x55c12df59130 2024-03-18T15:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.098+0000 7ff6b77fe640 1 -- 172.21.15.76:0/2708507888 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4017050 con 0x55c12df59130 2024-03-18T15:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.099+0000 7ff6d65e8740 1 -- 172.21.15.76:0/2708507888 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59130 msgr2=0x55c12df59510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.099+0000 7ff6d1f91640 1 --2- 172.21.15.76:0/2973086870 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6b80a2a10 0x7ff6b80a4ed0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4020000 tx=0x7ff6c4021000 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.099+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/2708507888 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59130 0x55c12df59510 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4011590 tx=0x7ff6c4011a50 comp rx=0 tx=0).stop 2024-03-18T15:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.099+0000 7ff6d65e8740 1 -- 172.21.15.76:0/2708507888 shutdown_connections 2024-03-18T15:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.099+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/2708507888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df59a50 0x55c12df622e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.099+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/2708507888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df62820 0x55c12df64c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.099+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/2708507888 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59130 0x55c12df59510 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.099+0000 7ff6d65e8740 1 -- 172.21.15.76:0/2708507888 >> 172.21.15.76:0/2708507888 conn(0x55c12df53a50 msgr2=0x55c12df53e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.099+0000 7ff6d65e8740 1 -- 172.21.15.76:0/2708507888 shutdown_connections 2024-03-18T15:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.099+0000 7ff6d65e8740 1 -- 172.21.15.76:0/2708507888 wait complete. 2024-03-18T15:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.099+0000 7ff6d65e8740 1 -- start start 2024-03-18T15:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.099+0000 7ff6d65e8740 1 Processor -- start 2024-03-18T15:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d65e8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59130 0x55c12df5f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d65e8740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59a50 0x55c12df5fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d65e8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df62820 0x55c12df60310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d65e8740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c12df65450 con 0x55c12df59a50 2024-03-18T15:03:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d65e8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c12de79d70 con 0x55c12df59130 2024-03-18T15:03:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d65e8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c12df655d0 con 0x55c12df62820 2024-03-18T15:03:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d65e8740 10 client.0 _ll_register_callbacks cb 0x55c12dbd1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:03:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d0f8f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59a50 0x55c12df5fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d1790640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59130 0x55c12df5f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d0f8f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59a50 0x55c12df5fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59984/0 (socket says 172.21.15.76:59984) 2024-03-18T15:03:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d0f8f640 1 -- 172.21.15.76:0/194973728 learned_addr learned my addr 172.21.15.76:0/194973728 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d1f91640 1 --2- 172.21.15.76:0/194973728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df62820 0x55c12df60310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d0f8f640 1 -- 172.21.15.76:0/194973728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df62820 msgr2=0x55c12df60310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d0f8f640 1 --2- 172.21.15.76:0/194973728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df62820 0x55c12df60310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d0f8f640 1 -- 172.21.15.76:0/194973728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59130 msgr2=0x55c12df5f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d0f8f640 1 --2- 172.21.15.76:0/194973728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59130 0x55c12df5f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d0f8f640 1 -- 172.21.15.76:0/194973728 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c12dbd2210 con 0x55c12df59a50 2024-03-18T15:03:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6d0f8f640 1 --2- 172.21.15.76:0/194973728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59a50 0x55c12df5fdd0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0ccf10 tx=0x7ff6bc009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eaf763aa83acc5f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.100+0000 7ff6b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:03:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.101+0000 7ff6b4ff9640 1 -- 172.21.15.76:0/194973728 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc09b230 con 0x55c12df59a50 2024-03-18T15:03:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.101+0000 7ff6d65e8740 1 -- 172.21.15.76:0/194973728 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6c401be20 con 0x55c12df59a50 2024-03-18T15:03:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.101+0000 7ff6b4ff9640 1 -- 172.21.15.76:0/194973728 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7ff6bc09ad20 con 0x55c12df59a50 2024-03-18T15:03:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.101+0000 7ff6b4ff9640 1 -- 172.21.15.76:0/194973728 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7ff6bc09b8d0 con 0x55c12df59a50 2024-03-18T15:03:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.101+0000 7ff6d65e8740 10 client.16674 fetch_fsmap learned FSMap version 42 2024-03-18T15:03:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.101+0000 7ff6d65e8740 1 -- 172.21.15.76:0/194973728 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55c12dc9e620 con 0x55c12df59a50 2024-03-18T15:03:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.101+0000 7ff6b4ff9640 1 -- 172.21.15.76:0/194973728 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6c401be20 con 0x55c12df59a50 2024-03-18T15:03:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.101+0000 7ff6b4ff9640 1 -- 172.21.15.76:0/194973728 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7ff6bc097620 con 0x55c12df59a50 2024-03-18T15:03:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.101+0000 7ff6d65e8740 10 client.16674 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T15:03:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.101+0000 7ff6d65e8740 10 client.16674 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:03:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.101+0000 7ff6d65e8740 20 client.16674 populate_metadata read hostname 'smithi076' 2024-03-18T15:03:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.101+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/194973728 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55c12df729f0 0x55c12df74de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.101+0000 7ff6d65e8740 4 client.16674 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:03:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.101+0000 7ff6d65e8740 1 -- 172.21.15.76:0/194973728 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c12decaf80 con 0x55c12df729f0 2024-03-18T15:03:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.101+0000 7ff6d1790640 1 --2- 172.21.15.76:0/194973728 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55c12df729f0 0x55c12df74de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.102+0000 7ff6b4ff9640 10 client.16674 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.102+0000 7ff6d1790640 1 --2- 172.21.15.76:0/194973728 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55c12df729f0 0x55c12df74de0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96608a80deca7215 server_cookie=29340c658b643367 in_seq=0 out_seq=0 2024-03-18T15:03:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.103+0000 7ff6b4ff9640 1 -- 172.21.15.76:0/194973728 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c12decaf80 con 0x55c12df729f0 2024-03-18T15:03:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.103+0000 7ff6b4ff9640 10 client.16674 handle_command_reply: tid=0 2024-03-18T15:03:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.103+0000 7ff6d65e8740 1 client.16674 shutdown 2024-03-18T15:03:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.104+0000 7ff6d65e8740 1 -- 172.21.15.76:0/194973728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59a50 msgr2=0x55c12df5fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.104+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/194973728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59a50 0x55c12df5fdd0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0ccf10 tx=0x7ff6bc009310 comp rx=0 tx=0).stop 2024-03-18T15:03:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.104+0000 7ff6d65e8740 1 -- 172.21.15.76:0/194973728 shutdown_connections 2024-03-18T15:03:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.104+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/194973728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df62820 0x55c12df60310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.104+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/194973728 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55c12df729f0 0x55c12df74de0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.104+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/194973728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59a50 0x55c12df5fdd0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.104+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/194973728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59130 0x55c12df5f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.104+0000 7ff6d65e8740 1 -- 172.21.15.76:0/194973728 >> 172.21.15.76:0/194973728 conn(0x55c12df53a50 msgr2=0x55c12df557a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.104+0000 7ff6b4ff9640 0 client.16674 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.104+0000 7ff6d65e8740 1 -- 172.21.15.76:0/194973728 shutdown_connections 2024-03-18T15:03:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.104+0000 7ff6d65e8740 1 -- 172.21.15.76:0/194973728 wait complete. 2024-03-18T15:03:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.104+0000 7ff6d65e8740 20 client.16674 trim_cache size 0 max 16384 2024-03-18T15:03:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.119+0000 7ff6d65e8740 1 Processor -- start 2024-03-18T15:03:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.119+0000 7ff6d65e8740 1 -- start start 2024-03-18T15:03:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d65e8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59130 0x55c12df61fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d65e8740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59a50 0x55c12df4ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d65e8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df62820 0x55c12df6e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d65e8740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1afcf0 con 0x55c12df59a50 2024-03-18T15:03:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d65e8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1ad5c0 con 0x55c12df59130 2024-03-18T15:03:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d65e8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1ab430 con 0x55c12df62820 2024-03-18T15:03:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d0f8f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59a50 0x55c12df4ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d0f8f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59a50 0x55c12df4ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:59994/0 (socket says 172.21.15.76:59994) 2024-03-18T15:03:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d0f8f640 1 -- 172.21.15.76:0/3837510678 learned_addr learned my addr 172.21.15.76:0/3837510678 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d1790640 1 --2- 172.21.15.76:0/3837510678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59130 0x55c12df61fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d1f91640 1 --2- 172.21.15.76:0/3837510678 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df62820 0x55c12df6e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d0f8f640 1 -- 172.21.15.76:0/3837510678 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df62820 msgr2=0x55c12df6e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d0f8f640 1 --2- 172.21.15.76:0/3837510678 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df62820 0x55c12df6e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d0f8f640 1 -- 172.21.15.76:0/3837510678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59130 msgr2=0x55c12df61fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d0f8f640 1 --2- 172.21.15.76:0/3837510678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59130 0x55c12df61fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d0f8f640 1 -- 172.21.15.76:0/3837510678 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c12df5e840 con 0x55c12df59a50 2024-03-18T15:03:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d1790640 1 --2- 172.21.15.76:0/3837510678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59130 0x55c12df61fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:03:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6d0f8f640 1 --2- 172.21.15.76:0/3837510678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59a50 0x55c12df4ea40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc097d30 tx=0x7ff6bc00c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1da4215f92cf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6b6ffd640 1 -- 172.21.15.76:0/3837510678 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc09f970 con 0x55c12df59a50 2024-03-18T15:03:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6b6ffd640 1 -- 172.21.15.76:0/3837510678 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6bc0a3fc0 con 0x55c12df59a50 2024-03-18T15:03:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.120+0000 7ff6b6ffd640 1 -- 172.21.15.76:0/3837510678 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc09ad20 con 0x55c12df59a50 2024-03-18T15:03:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.121+0000 7ff6d65e8740 1 -- 172.21.15.76:0/3837510678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59a50 msgr2=0x55c12df4ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.121+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/3837510678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59a50 0x55c12df4ea40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc097d30 tx=0x7ff6bc00c7d0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.121+0000 7ff6d65e8740 1 -- 172.21.15.76:0/3837510678 shutdown_connections 2024-03-18T15:03:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.121+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/3837510678 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df62820 0x55c12df6e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.121+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/3837510678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df59a50 0x55c12df4ea40 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.121+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/3837510678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59130 0x55c12df61fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.121+0000 7ff6d65e8740 1 -- 172.21.15.76:0/3837510678 >> 172.21.15.76:0/3837510678 conn(0x55c12df53a50 msgr2=0x55c12df5f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.121+0000 7ff6d65e8740 1 -- 172.21.15.76:0/3837510678 shutdown_connections 2024-03-18T15:03:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.121+0000 7ff6d65e8740 1 -- 172.21.15.76:0/3837510678 wait complete. 2024-03-18T15:03:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.121+0000 7ff6d65e8740 1 -- start start 2024-03-18T15:03:27.125 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d65e8740 1 Processor -- start 2024-03-18T15:03:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d65e8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df59130 0x55c12df6a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d65e8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59a50 0x55c12df65e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d65e8740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df62820 0x55c12df66340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d65e8740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1aba80 con 0x55c12df62820 2024-03-18T15:03:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d65e8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1acef0 con 0x55c12df59a50 2024-03-18T15:03:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d65e8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1ac150 con 0x55c12df59130 2024-03-18T15:03:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d65e8740 10 client.0 _ll_register_callbacks cb 0x7ff6bc009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:03:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d1f91640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df62820 0x55c12df66340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d0f8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59a50 0x55c12df65e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d1f91640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df62820 0x55c12df66340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60010/0 (socket says 172.21.15.76:60010) 2024-03-18T15:03:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d0f8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59a50 0x55c12df65e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:56394/0 (socket says 172.21.15.76:56394) 2024-03-18T15:03:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d1f91640 1 -- 172.21.15.76:0/3744005626 learned_addr learned my addr 172.21.15.76:0/3744005626 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d1790640 1 --2- 172.21.15.76:0/3744005626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df59130 0x55c12df6a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d1f91640 1 -- 172.21.15.76:0/3744005626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df59130 msgr2=0x55c12df6a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d1f91640 1 --2- 172.21.15.76:0/3744005626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df59130 0x55c12df6a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d1f91640 1 -- 172.21.15.76:0/3744005626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59a50 msgr2=0x55c12df65e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d1f91640 1 --2- 172.21.15.76:0/3744005626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59a50 0x55c12df65e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d1f91640 1 -- 172.21.15.76:0/3744005626 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c12df66900 con 0x55c12df62820 2024-03-18T15:03:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.122+0000 7ff6d0f8f640 1 --2- 172.21.15.76:0/3744005626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59a50 0x55c12df65e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:03:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.123+0000 7ff6d1f91640 1 --2- 172.21.15.76:0/3744005626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df62820 0x55c12df66340 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4011530 tx=0x7ff6c4013030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b733fdc691c26a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.123+0000 7ff6b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:03:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.123+0000 7ff6b4ff9640 1 -- 172.21.15.76:0/3744005626 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4032030 con 0x55c12df62820 2024-03-18T15:03:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.123+0000 7ff6b4ff9640 1 -- 172.21.15.76:0/3744005626 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7ff6c4004bf0 con 0x55c12df62820 2024-03-18T15:03:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.123+0000 7ff6d65e8740 1 -- 172.21.15.76:0/3744005626 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6bc0a3fc0 con 0x55c12df62820 2024-03-18T15:03:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.123+0000 7ff6b4ff9640 1 -- 172.21.15.76:0/3744005626 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7ff6c4017020 con 0x55c12df62820 2024-03-18T15:03:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.123+0000 7ff6d65e8740 10 client.16680 fetch_fsmap learned FSMap version 42 2024-03-18T15:03:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.123+0000 7ff6d65e8740 1 -- 172.21.15.76:0/3744005626 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55c12dfd8a70 con 0x55c12df62820 2024-03-18T15:03:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.123+0000 7ff6b4ff9640 1 -- 172.21.15.76:0/3744005626 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6bc0a3fc0 con 0x55c12df62820 2024-03-18T15:03:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.123+0000 7ff6b4ff9640 1 -- 172.21.15.76:0/3744005626 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7ff6c403e050 con 0x55c12df62820 2024-03-18T15:03:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.123+0000 7ff6d65e8740 10 client.16680 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T15:03:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.123+0000 7ff6d65e8740 10 client.16680 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:03:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.123+0000 7ff6d65e8740 20 client.16680 populate_metadata read hostname 'smithi076' 2024-03-18T15:03:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.123+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/3744005626 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55c12dfd8f90 0x55c12dfdb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.123+0000 7ff6d65e8740 4 client.16680 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:03:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.123+0000 7ff6d65e8740 1 -- 172.21.15.76:0/3744005626 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c12dec52d0 con 0x55c12dfd8f90 2024-03-18T15:03:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.124+0000 7ff6d1790640 1 --2- 172.21.15.76:0/3744005626 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55c12dfd8f90 0x55c12dfdb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.124+0000 7ff6d1790640 1 --2- 172.21.15.76:0/3744005626 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55c12dfd8f90 0x55c12dfdb3b0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f99ea404d5cdcaa1 server_cookie=91db0a383f8e3efd in_seq=0 out_seq=0 2024-03-18T15:03:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.124+0000 7ff6b4ff9640 10 client.16680 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.220+0000 7ff6b4ff9640 1 -- 172.21.15.76:0/3744005626 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c12dec52d0 con 0x55c12dfd8f90 2024-03-18T15:03:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.220+0000 7ff6b4ff9640 10 client.16680 handle_command_reply: tid=0 2024-03-18T15:03:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.220+0000 7ff6d65e8740 1 client.16680 shutdown 2024-03-18T15:03:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.221+0000 7ff6d65e8740 1 -- 172.21.15.76:0/3744005626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df62820 msgr2=0x55c12df66340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.221+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/3744005626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df62820 0x55c12df66340 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4011530 tx=0x7ff6c4013030 comp rx=0 tx=0).stop 2024-03-18T15:03:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.221+0000 7ff6d65e8740 1 -- 172.21.15.76:0/3744005626 shutdown_connections 2024-03-18T15:03:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.221+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/3744005626 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55c12dfd8f90 0x55c12dfdb3b0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.221+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/3744005626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c12df62820 0x55c12df66340 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.221+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/3744005626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c12df59a50 0x55c12df65e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.221+0000 7ff6d65e8740 1 --2- 172.21.15.76:0/3744005626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c12df59130 0x55c12df6a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.221+0000 7ff6d65e8740 1 -- 172.21.15.76:0/3744005626 >> 172.21.15.76:0/3744005626 conn(0x55c12df53a50 msgr2=0x55c12df52bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.221+0000 7ff6b4ff9640 0 client.16680 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.221+0000 7ff6d65e8740 1 -- 172.21.15.76:0/3744005626 shutdown_connections 2024-03-18T15:03:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.221+0000 7ff6d65e8740 1 -- 172.21.15.76:0/3744005626 wait complete. 2024-03-18T15:03:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.221+0000 7ff6d65e8740 20 client.16680 trim_cache size 0 max 16384 2024-03-18T15:03:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:03:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:03:27.224 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:03:27.224 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:03:27.224 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.222+0000 7ff6d39f4640 1 -- 172.21.15.76:0/2973086870 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6b80a2a10 msgr2=0x7ff6b80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:27.224 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.222+0000 7ff6d39f4640 1 --2- 172.21.15.76:0/2973086870 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6b80a2a10 0x7ff6b80a4ed0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4020000 tx=0x7ff6c4021000 comp rx=0 tx=0).stop 2024-03-18T15:03:27.224 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.222+0000 7ff6d39f4640 1 -- 172.21.15.76:0/2973086870 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6cc1012f0 msgr2=0x7ff6cc1993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.222+0000 7ff6d39f4640 1 --2- 172.21.15.76:0/2973086870 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6cc1012f0 0x7ff6cc1993c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc00e9f0 tx=0x7ff6bc00eeb0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.222+0000 7ff6d39f4640 1 -- 172.21.15.76:0/2973086870 shutdown_connections 2024-03-18T15:03:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.222+0000 7ff6d39f4640 1 --2- 172.21.15.76:0/2973086870 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff6b80a2a10 0x7ff6b80a4ed0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.222+0000 7ff6d39f4640 1 --2- 172.21.15.76:0/2973086870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6cc105680 0x7ff6cc1a3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.222+0000 7ff6d39f4640 1 --2- 172.21.15.76:0/2973086870 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6cc1012f0 0x7ff6cc1993c0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.222+0000 7ff6d39f4640 1 --2- 172.21.15.76:0/2973086870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc100930 0x7ff6cc198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.222+0000 7ff6d39f4640 1 -- 172.21.15.76:0/2973086870 >> 172.21.15.76:0/2973086870 conn(0x7ff6cc077530 msgr2=0x7ff6cc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.222+0000 7ff6d39f4640 1 -- 172.21.15.76:0/2973086870 shutdown_connections 2024-03-18T15:03:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:27.222+0000 7ff6d39f4640 1 -- 172.21.15.76:0/2973086870 wait complete. 2024-03-18T15:03:27.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1000741 9.39 MB/sec execute 448 sec latency 31.040 ms 2024-03-18T15:03:28.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:27 smithi076 ceph-mon[32205]: pgmap v323: 97 pgs: 97 active+clean; 317 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 225 op/s 2024-03-18T15:03:28.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:27 smithi174 ceph-mon[36787]: pgmap v323: 97 pgs: 97 active+clean; 317 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 225 op/s 2024-03-18T15:03:28.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:27 smithi077 ceph-mon[35865]: pgmap v323: 97 pgs: 97 active+clean; 317 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 225 op/s 2024-03-18T15:03:28.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1002314 9.39 MB/sec execute 449 sec latency 75.794 ms 2024-03-18T15:03:29.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1003843 9.38 MB/sec execute 450 sec latency 28.572 ms 2024-03-18T15:03:30.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:29 smithi076 ceph-mon[32205]: pgmap v324: 97 pgs: 97 active+clean; 315 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.2 MiB/s wr, 364 op/s 2024-03-18T15:03:30.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:29 smithi174 ceph-mon[36787]: pgmap v324: 97 pgs: 97 active+clean; 315 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.2 MiB/s wr, 364 op/s 2024-03-18T15:03:30.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:29 smithi077 ceph-mon[35865]: pgmap v324: 97 pgs: 97 active+clean; 315 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.2 MiB/s wr, 364 op/s 2024-03-18T15:03:30.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1005802 9.39 MB/sec execute 451 sec latency 16.981 ms 2024-03-18T15:03:31.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1007225 9.38 MB/sec execute 452 sec latency 23.484 ms 2024-03-18T15:03:32.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:31 smithi076 ceph-mon[32205]: pgmap v325: 97 pgs: 97 active+clean; 315 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 241 op/s 2024-03-18T15:03:32.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:31 smithi174 ceph-mon[36787]: pgmap v325: 97 pgs: 97 active+clean; 315 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 241 op/s 2024-03-18T15:03:32.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:31 smithi077 ceph-mon[35865]: pgmap v325: 97 pgs: 97 active+clean; 315 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 241 op/s 2024-03-18T15:03:32.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1009227 9.39 MB/sec execute 453 sec latency 21.846 ms 2024-03-18T15:03:33.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:32 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:03:33.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:32 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:03:33.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:32 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:03:33.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1009926 9.37 MB/sec execute 454 sec latency 133.952 ms 2024-03-18T15:03:34.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1012320 9.38 MB/sec execute 455 sec latency 31.604 ms 2024-03-18T15:03:34.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:34 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:34.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:34 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:34.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:34 smithi076 ceph-mon[32205]: pgmap v326: 97 pgs: 97 active+clean; 319 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 335 op/s 2024-03-18T15:03:34.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:34 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:34.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:34 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:34.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:34 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:34 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:34 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:34 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:34 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:34 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:34 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:34 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:34 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:34.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:34 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:34.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:34 smithi174 ceph-mon[36787]: pgmap v326: 97 pgs: 97 active+clean; 319 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 335 op/s 2024-03-18T15:03:34.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:34 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:34.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:34 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:34.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:34 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:34 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:34 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:34 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:34 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:34 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:34 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:34 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:34 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:34.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:34 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:34.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:34 smithi077 ceph-mon[35865]: pgmap v326: 97 pgs: 97 active+clean; 319 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 335 op/s 2024-03-18T15:03:34.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:34 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:34.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:34 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:34.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:34 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:34 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:34 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:34 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:34 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:34 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:34 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:34.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:34 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:35.759 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1013441 9.38 MB/sec execute 456 sec latency 22.918 ms 2024-03-18T15:03:35.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:35 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-03-18T15:03:35.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:35 smithi076 ceph-mon[32205]: Unable to set osd_memory_target on smithi076 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-03-18T15:03:35.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:35 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi077 to 174.1M 2024-03-18T15:03:35.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:35 smithi076 ceph-mon[32205]: Unable to set osd_memory_target on smithi077 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-18T15:03:35.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:35 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:35.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:35 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:35.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:35 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:35.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:35 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:35.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:35 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:35.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:35 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:35.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:35 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:03:35.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:35 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:03:35.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:35 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:35.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:35 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-03-18T15:03:35.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:35 smithi174 ceph-mon[36787]: Unable to set osd_memory_target on smithi076 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-03-18T15:03:35.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:35 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi077 to 174.1M 2024-03-18T15:03:35.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:35 smithi174 ceph-mon[36787]: Unable to set osd_memory_target on smithi077 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-18T15:03:35.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:35 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:35.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:35 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:35.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:35 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:35.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:35 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:35.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:35 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:35.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:35 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:35.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:35 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:03:35.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:35 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:03:35.968 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:35 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:35.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:35 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-03-18T15:03:35.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:35 smithi077 ceph-mon[35865]: Unable to set osd_memory_target on smithi076 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-03-18T15:03:35.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:35 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi077 to 174.1M 2024-03-18T15:03:35.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:35 smithi077 ceph-mon[35865]: Unable to set osd_memory_target on smithi077 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-18T15:03:35.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:35 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:35.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:35 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:35.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:35 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:35.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:35 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:35.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:35 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:35.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:35 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:03:35.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:35 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:03:35.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:35 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:03:35.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:35 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:03:36.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1016023 9.39 MB/sec execute 457 sec latency 20.768 ms 2024-03-18T15:03:36.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:36 smithi076 ceph-mon[32205]: Adjusting osd_memory_target on smithi174 to 174.1M 2024-03-18T15:03:36.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:36 smithi076 ceph-mon[32205]: Unable to set osd_memory_target on smithi174 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-18T15:03:36.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:36 smithi076 ceph-mon[32205]: pgmap v327: 97 pgs: 97 active+clean; 319 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 233 op/s 2024-03-18T15:03:36.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:36 smithi174 ceph-mon[36787]: Adjusting osd_memory_target on smithi174 to 174.1M 2024-03-18T15:03:36.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:36 smithi174 ceph-mon[36787]: Unable to set osd_memory_target on smithi174 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-18T15:03:36.967 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:36 smithi174 ceph-mon[36787]: pgmap v327: 97 pgs: 97 active+clean; 319 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 233 op/s 2024-03-18T15:03:36.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:36 smithi077 ceph-mon[35865]: Adjusting osd_memory_target on smithi174 to 174.1M 2024-03-18T15:03:36.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:36 smithi077 ceph-mon[35865]: Unable to set osd_memory_target on smithi174 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-18T15:03:36.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:36 smithi077 ceph-mon[35865]: pgmap v327: 97 pgs: 97 active+clean; 319 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 233 op/s 2024-03-18T15:03:37.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1017270 9.38 MB/sec execute 458 sec latency 15.943 ms 2024-03-18T15:03:38.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:37 smithi076 ceph-mon[32205]: pgmap v328: 97 pgs: 97 active+clean; 319 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 233 op/s 2024-03-18T15:03:38.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:37 smithi174 ceph-mon[36787]: pgmap v328: 97 pgs: 97 active+clean; 319 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 233 op/s 2024-03-18T15:03:38.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:37 smithi077 ceph-mon[35865]: pgmap v328: 97 pgs: 97 active+clean; 319 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 233 op/s 2024-03-18T15:03:38.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1019535 9.39 MB/sec execute 459 sec latency 61.907 ms 2024-03-18T15:03:39.762 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1020389 9.38 MB/sec execute 460 sec latency 28.820 ms 2024-03-18T15:03:40.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:39 smithi076 ceph-mon[32205]: pgmap v329: 97 pgs: 97 active+clean; 327 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 329 op/s 2024-03-18T15:03:40.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:39 smithi174 ceph-mon[36787]: pgmap v329: 97 pgs: 97 active+clean; 327 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 329 op/s 2024-03-18T15:03:40.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:39 smithi077 ceph-mon[35865]: pgmap v329: 97 pgs: 97 active+clean; 327 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 329 op/s 2024-03-18T15:03:40.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.312+0000 7f3ea603d640 1 -- 172.21.15.76:0/635745707 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ea0100930 msgr2=0x7f3ea0100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:40.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.312+0000 7f3ea603d640 1 --2- 172.21.15.76:0/635745707 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ea0100930 0x7f3ea0100db0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f3e90009900 tx=0x7f3e9002bc30 comp rx=0 tx=0).stop 2024-03-18T15:03:40.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.312+0000 7f3ea603d640 1 -- 172.21.15.76:0/635745707 shutdown_connections 2024-03-18T15:03:40.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.312+0000 7f3ea603d640 1 --2- 172.21.15.76:0/635745707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ea01012f0 0x7f3ea010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.315 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.312+0000 7f3ea603d640 1 --2- 172.21.15.76:0/635745707 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ea0100930 0x7f3ea0100db0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.312+0000 7f3ea603d640 1 --2- 172.21.15.76:0/635745707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ea0105680 0x7f3ea0105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.312+0000 7f3ea603d640 1 -- 172.21.15.76:0/635745707 >> 172.21.15.76:0/635745707 conn(0x7f3ea0077530 msgr2=0x7f3ea0077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:40.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.312+0000 7f3ea603d640 1 -- 172.21.15.76:0/635745707 shutdown_connections 2024-03-18T15:03:40.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.312+0000 7f3ea603d640 1 -- 172.21.15.76:0/635745707 wait complete. 2024-03-18T15:03:40.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.312+0000 7f3ea603d640 1 Processor -- start 2024-03-18T15:03:40.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3ea603d640 1 -- start start 2024-03-18T15:03:40.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3ea603d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ea0100930 0x7f3ea01980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3ea603d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ea01012f0 0x7f3ea0198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3ea603d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ea0105680 0x7f3ea019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3ea603d640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea0110ba0 con 0x7f3ea01012f0 2024-03-18T15:03:40.316 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3ea603d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea0110a20 con 0x7f3ea0100930 2024-03-18T15:03:40.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3ea603d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea0110d20 con 0x7f3ea0105680 2024-03-18T15:03:40.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3e9effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ea01012f0 0x7f3ea0198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3e9effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ea01012f0 0x7f3ea0198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52200/0 (socket says 172.21.15.76:52200) 2024-03-18T15:03:40.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3e9effd640 1 -- 172.21.15.76:0/3623586593 learned_addr learned my addr 172.21.15.76:0/3623586593 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:40.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3e9f7fe640 1 --2- 172.21.15.76:0/3623586593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ea0100930 0x7f3ea01980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3e9ffff640 1 --2- 172.21.15.76:0/3623586593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ea0105680 0x7f3ea019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3e9effd640 1 -- 172.21.15.76:0/3623586593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ea0105680 msgr2=0x7f3ea019c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:40.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3e9effd640 1 --2- 172.21.15.76:0/3623586593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ea0105680 0x7f3ea019c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3e9effd640 1 -- 172.21.15.76:0/3623586593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ea0100930 msgr2=0x7f3ea01980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:40.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3e9effd640 1 --2- 172.21.15.76:0/3623586593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ea0100930 0x7f3ea01980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3e9effd640 1 -- 172.21.15.76:0/3623586593 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea019d080 con 0x7f3ea01012f0 2024-03-18T15:03:40.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3e9f7fe640 1 --2- 172.21.15.76:0/3623586593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ea0100930 0x7f3ea01980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:03:40.317 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3e9effd640 1 --2- 172.21.15.76:0/3623586593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ea01012f0 0x7f3ea0198620 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f3e90005940 tx=0x7f3e90004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:40.318 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.313+0000 7f3e9cff9640 1 -- 172.21.15.76:0/3623586593 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e90041070 con 0x7f3ea01012f0 2024-03-18T15:03:40.318 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.314+0000 7f3e9cff9640 1 -- 172.21.15.76:0/3623586593 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e900054e0 con 0x7f3ea01012f0 2024-03-18T15:03:40.318 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.314+0000 7f3ea603d640 1 -- 172.21.15.76:0/3623586593 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ea019d310 con 0x7f3ea01012f0 2024-03-18T15:03:40.318 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.314+0000 7f3e9cff9640 1 -- 172.21.15.76:0/3623586593 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e90003d00 con 0x7f3ea01012f0 2024-03-18T15:03:40.318 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.314+0000 7f3ea603d640 1 -- 172.21.15.76:0/3623586593 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ea01a4c00 con 0x7f3ea01012f0 2024-03-18T15:03:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.316+0000 7f3e9cff9640 1 -- 172.21.15.76:0/3623586593 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f3e90044070 con 0x7f3ea01012f0 2024-03-18T15:03:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.316+0000 7f3ea8c21740 1 Processor -- start 2024-03-18T15:03:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.316+0000 7f3ea8c21740 1 -- start start 2024-03-18T15:03:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.316+0000 7f3e9cff9640 1 --2- 172.21.15.76:0/3623586593 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3e880a2a10 0x7f3e880a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.316+0000 7f3ea8c21740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508c130 0x56241508c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.316+0000 7f3ea8c21740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 0x5624150952e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.316+0000 7f3e9ffff640 1 --2- 172.21.15.76:0/3623586593 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3e880a2a10 0x7f3e880a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.316+0000 7f3ea8c21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562415095820 0x562415097c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.316+0000 7f3ea8c21740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562414ff0a40 con 0x56241508ca50 2024-03-18T15:03:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.316+0000 7f3ea8c21740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562414f99920 con 0x56241508c130 2024-03-18T15:03:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.316+0000 7f3ea8c21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562414fecae0 con 0x562415095820 2024-03-18T15:03:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.316+0000 7f3e9cff9640 1 -- 172.21.15.76:0/3623586593 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f3e900f0970 con 0x7f3ea01012f0 2024-03-18T15:03:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.316+0000 7f3e9f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508c130 0x56241508c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.317+0000 7f3e9f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508c130 0x56241508c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:43616/0 (socket says 172.21.15.76:43616) 2024-03-18T15:03:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.317+0000 7f3e9f7fe640 1 -- 172.21.15.76:0/1781263190 learned_addr learned my addr 172.21.15.76:0/1781263190 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.317+0000 7f3e9effd640 1 --2- 172.21.15.76:0/1781263190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 0x5624150952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.317+0000 7f3e9ffff640 1 --2- 172.21.15.76:0/1781263190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562415095820 0x562415097c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.317+0000 7f3e9f7fe640 1 -- 172.21.15.76:0/1781263190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562415095820 msgr2=0x562415097c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.317+0000 7f3e9f7fe640 1 --2- 172.21.15.76:0/1781263190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562415095820 0x562415097c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.317+0000 7f3e9f7fe640 1 -- 172.21.15.76:0/1781263190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 msgr2=0x5624150952e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.317+0000 7f3e9f7fe640 1 --2- 172.21.15.76:0/1781263190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 0x5624150952e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.317+0000 7f3e9f7fe640 1 -- 172.21.15.76:0/1781263190 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562414fb1460 con 0x56241508c130 2024-03-18T15:03:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.317+0000 7f3e9ffff640 1 --2- 172.21.15.76:0/3623586593 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3e880a2a10 0x7f3e880a4ed0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009fd0 tx=0x7f3e98011bc0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.317+0000 7f3e9f7fe640 1 --2- 172.21.15.76:0/1781263190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508c130 0x56241508c510 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3e940115c0 tx=0x7f3e94011a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68965a792bc3c580 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.317+0000 7f3e85ffb640 1 -- 172.21.15.76:0/1781263190 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e9400f5c0 con 0x56241508c130 2024-03-18T15:03:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.317+0000 7f3e85ffb640 1 -- 172.21.15.76:0/1781263190 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e9400fc40 con 0x56241508c130 2024-03-18T15:03:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.317+0000 7f3e85ffb640 1 -- 172.21.15.76:0/1781263190 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e94019cb0 con 0x56241508c130 2024-03-18T15:03:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.318+0000 7f3ea8c21740 1 -- 172.21.15.76:0/1781263190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508c130 msgr2=0x56241508c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.318+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/1781263190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508c130 0x56241508c510 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3e940115c0 tx=0x7f3e94011a80 comp rx=0 tx=0).stop 2024-03-18T15:03:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.318+0000 7f3ea8c21740 1 -- 172.21.15.76:0/1781263190 shutdown_connections 2024-03-18T15:03:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.318+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/1781263190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562415095820 0x562415097c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.318+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/1781263190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 0x5624150952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.318+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/1781263190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508c130 0x56241508c510 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.318+0000 7f3ea8c21740 1 -- 172.21.15.76:0/1781263190 >> 172.21.15.76:0/1781263190 conn(0x562415086a50 msgr2=0x562415086e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.318+0000 7f3ea8c21740 1 -- 172.21.15.76:0/1781263190 shutdown_connections 2024-03-18T15:03:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.318+0000 7f3ea8c21740 1 -- 172.21.15.76:0/1781263190 wait complete. 2024-03-18T15:03:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.318+0000 7f3ea8c21740 1 -- start start 2024-03-18T15:03:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.318+0000 7f3ea8c21740 1 Processor -- start 2024-03-18T15:03:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3ea8c21740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508c130 0x562415092810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3ea8c21740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 0x562415092d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3ea8c21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562415095820 0x562415093290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3ea8c21740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562415098450 con 0x56241508ca50 2024-03-18T15:03:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3ea8c21740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562414facd70 con 0x56241508c130 2024-03-18T15:03:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3ea8c21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5624150985d0 con 0x562415095820 2024-03-18T15:03:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3ea8c21740 10 client.0 _ll_register_callbacks cb 0x562414d04a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:03:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3e9f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508c130 0x562415092810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3e9f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508c130 0x562415092810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:43628/0 (socket says 172.21.15.76:43628) 2024-03-18T15:03:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3e9f7fe640 1 -- 172.21.15.76:0/2181044905 learned_addr learned my addr 172.21.15.76:0/2181044905 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3e9effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 0x562415092d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3e9ffff640 1 --2- 172.21.15.76:0/2181044905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562415095820 0x562415093290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3e9f7fe640 1 -- 172.21.15.76:0/2181044905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562415095820 msgr2=0x562415093290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3e9f7fe640 1 --2- 172.21.15.76:0/2181044905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562415095820 0x562415093290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3e9f7fe640 1 -- 172.21.15.76:0/2181044905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 msgr2=0x562415092d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3e9f7fe640 1 --2- 172.21.15.76:0/2181044905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 0x562415092d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3e9f7fe640 1 -- 172.21.15.76:0/2181044905 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562414d05210 con 0x56241508c130 2024-03-18T15:03:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3e9effd640 1 --2- 172.21.15.76:0/2181044905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 0x562415092d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:03:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3e9f7fe640 1 --2- 172.21.15.76:0/2181044905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508c130 0x562415092810 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3e9400a010 tx=0x7f3e94009db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f64f590f4d867bc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3e7effd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-18T15:03:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3e7effd640 1 -- 172.21.15.76:0/2181044905 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e94004350 con 0x56241508c130 2024-03-18T15:03:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3e7effd640 1 -- 172.21.15.76:0/2181044905 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f3e940199b0 con 0x56241508c130 2024-03-18T15:03:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.319+0000 7f3ea8c21740 1 -- 172.21.15.76:0/2181044905 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e9400fc40 con 0x56241508c130 2024-03-18T15:03:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.320+0000 7f3e7effd640 1 -- 172.21.15.76:0/2181044905 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f3e94030050 con 0x56241508c130 2024-03-18T15:03:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.320+0000 7f3ea8c21740 10 client.26555 fetch_fsmap learned FSMap version 42 2024-03-18T15:03:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.320+0000 7f3ea8c21740 1 -- 172.21.15.76:0/2181044905 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x562414dd1620 con 0x56241508c130 2024-03-18T15:03:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.320+0000 7f3e7effd640 1 -- 172.21.15.76:0/2181044905 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e9400fc40 con 0x56241508c130 2024-03-18T15:03:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.320+0000 7f3e7effd640 1 -- 172.21.15.76:0/2181044905 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7f3e94005260 con 0x56241508c130 2024-03-18T15:03:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.320+0000 7f3ea8c21740 10 client.26555 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T15:03:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.320+0000 7f3ea8c21740 10 client.26555 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:03:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.320+0000 7f3ea8c21740 20 client.26555 populate_metadata read hostname 'smithi076' 2024-03-18T15:03:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.320+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/2181044905 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5624150a5880 0x5624150a7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.320+0000 7f3ea8c21740 4 client.26555 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:03:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.320+0000 7f3ea8c21740 1 -- 172.21.15.76:0/2181044905 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562414ffdf80 con 0x5624150a5880 2024-03-18T15:03:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.320+0000 7f3e9effd640 1 --2- 172.21.15.76:0/2181044905 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5624150a5880 0x5624150a7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.320+0000 7f3e7effd640 10 client.26555 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.320+0000 7f3e9effd640 1 --2- 172.21.15.76:0/2181044905 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5624150a5880 0x5624150a7c70 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abfc992779663986 server_cookie=4ca590a560e1d58f in_seq=0 out_seq=0 2024-03-18T15:03:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.321+0000 7f3e7effd640 1 -- 172.21.15.76:0/2181044905 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562414ffdf80 con 0x5624150a5880 2024-03-18T15:03:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.321+0000 7f3e7effd640 10 client.26555 handle_command_reply: tid=0 2024-03-18T15:03:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.321+0000 7f3ea8c21740 1 client.26555 shutdown 2024-03-18T15:03:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.321+0000 7f3ea8c21740 1 -- 172.21.15.76:0/2181044905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508c130 msgr2=0x562415092810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.321+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/2181044905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508c130 0x562415092810 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3e9400a010 tx=0x7f3e94009db0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.321+0000 7f3ea8c21740 1 -- 172.21.15.76:0/2181044905 shutdown_connections 2024-03-18T15:03:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.321+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/2181044905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562415095820 0x562415093290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.321+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/2181044905 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x5624150a5880 0x5624150a7c70 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.321+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/2181044905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 0x562415092d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.321+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/2181044905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508c130 0x562415092810 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.321+0000 7f3e7effd640 0 client.26555 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.321+0000 7f3ea8c21740 1 -- 172.21.15.76:0/2181044905 >> 172.21.15.76:0/2181044905 conn(0x562415086a50 msgr2=0x5624150887a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.321+0000 7f3ea8c21740 1 -- 172.21.15.76:0/2181044905 shutdown_connections 2024-03-18T15:03:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.321+0000 7f3ea8c21740 1 -- 172.21.15.76:0/2181044905 wait complete. 2024-03-18T15:03:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.322+0000 7f3ea8c21740 20 client.26555 trim_cache size 0 max 16384 2024-03-18T15:03:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.336+0000 7f3ea8c21740 1 Processor -- start 2024-03-18T15:03:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.336+0000 7f3ea8c21740 1 -- start start 2024-03-18T15:03:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.336+0000 7f3ea8c21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56241508c130 0x562415094f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.336+0000 7f3ea8c21740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 0x5624150a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.336+0000 7f3ea8c21740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562415095820 0x562415085a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.336+0000 7f3ea8c21740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea01afc30 con 0x56241508ca50 2024-03-18T15:03:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.337+0000 7f3ea8c21740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea01ad500 con 0x562415095820 2024-03-18T15:03:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.337+0000 7f3ea8c21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea01ab370 con 0x56241508c130 2024-03-18T15:03:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.337+0000 7f3e9effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 0x5624150a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.337+0000 7f3e9effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 0x5624150a11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52236/0 (socket says 172.21.15.76:52236) 2024-03-18T15:03:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.337+0000 7f3e9effd640 1 -- 172.21.15.76:0/1402628930 learned_addr learned my addr 172.21.15.76:0/1402628930 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.337+0000 7f3e9f7fe640 1 --2- 172.21.15.76:0/1402628930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56241508c130 0x562415094f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.337+0000 7f3e9ffff640 1 --2- 172.21.15.76:0/1402628930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562415095820 0x562415085a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.337+0000 7f3e9effd640 1 -- 172.21.15.76:0/1402628930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56241508c130 msgr2=0x562415094f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.337+0000 7f3e9effd640 1 --2- 172.21.15.76:0/1402628930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56241508c130 0x562415094f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.337+0000 7f3e9effd640 1 -- 172.21.15.76:0/1402628930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562415095820 msgr2=0x562415085a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.337+0000 7f3e9effd640 1 --2- 172.21.15.76:0/1402628930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562415095820 0x562415085a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.337+0000 7f3e9effd640 1 -- 172.21.15.76:0/1402628930 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562415086880 con 0x56241508ca50 2024-03-18T15:03:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.337+0000 7f3e9effd640 1 --2- 172.21.15.76:0/1402628930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 0x5624150a11f0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f3e900bd140 tx=0x7f3e900dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bad76e943fe3a0a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.337+0000 7f3e857fa640 1 -- 172.21.15.76:0/1402628930 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e9004b040 con 0x56241508ca50 2024-03-18T15:03:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.337+0000 7f3e857fa640 1 -- 172.21.15.76:0/1402628930 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e90005690 con 0x56241508ca50 2024-03-18T15:03:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.337+0000 7f3e857fa640 1 -- 172.21.15.76:0/1402628930 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e900bb4b0 con 0x56241508ca50 2024-03-18T15:03:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.338+0000 7f3ea8c21740 1 -- 172.21.15.76:0/1402628930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 msgr2=0x5624150a11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.338+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/1402628930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 0x5624150a11f0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f3e900bd140 tx=0x7f3e900dd040 comp rx=0 tx=0).stop 2024-03-18T15:03:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.338+0000 7f3ea8c21740 1 -- 172.21.15.76:0/1402628930 shutdown_connections 2024-03-18T15:03:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.338+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/1402628930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562415095820 0x562415085a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.338+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/1402628930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508ca50 0x5624150a11f0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.338+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/1402628930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56241508c130 0x562415094f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.338+0000 7f3ea8c21740 1 -- 172.21.15.76:0/1402628930 >> 172.21.15.76:0/1402628930 conn(0x562415086a50 msgr2=0x562415085320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.338+0000 7f3ea8c21740 1 -- 172.21.15.76:0/1402628930 shutdown_connections 2024-03-18T15:03:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.338+0000 7f3ea8c21740 1 -- 172.21.15.76:0/1402628930 wait complete. 2024-03-18T15:03:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.338+0000 7f3ea8c21740 1 -- start start 2024-03-18T15:03:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.338+0000 7f3ea8c21740 1 Processor -- start 2024-03-18T15:03:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3ea8c21740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508c130 0x56241509db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3ea8c21740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508ca50 0x562415098ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3ea8c21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562415095820 0x562415099400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3ea8c21740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea01ab9c0 con 0x56241508c130 2024-03-18T15:03:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3ea8c21740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea01ace30 con 0x56241508ca50 2024-03-18T15:03:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3ea8c21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea01ac090 con 0x562415095820 2024-03-18T15:03:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3ea8c21740 10 client.0 _ll_register_callbacks cb 0x7f3e900035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:03:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3e9f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508c130 0x56241509db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3e9f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508c130 0x56241509db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52242/0 (socket says 172.21.15.76:52242) 2024-03-18T15:03:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3e9f7fe640 1 -- 172.21.15.76:0/656934030 learned_addr learned my addr 172.21.15.76:0/656934030 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3e9effd640 1 --2- 172.21.15.76:0/656934030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508ca50 0x562415098ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3e9ffff640 1 --2- 172.21.15.76:0/656934030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562415095820 0x562415099400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3e9f7fe640 1 -- 172.21.15.76:0/656934030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562415095820 msgr2=0x562415099400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3e9f7fe640 1 --2- 172.21.15.76:0/656934030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562415095820 0x562415099400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3e9f7fe640 1 -- 172.21.15.76:0/656934030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508ca50 msgr2=0x562415098ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:40.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3e9f7fe640 1 --2- 172.21.15.76:0/656934030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508ca50 0x562415098ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3e9f7fe640 1 -- 172.21.15.76:0/656934030 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624150a81d0 con 0x56241508c130 2024-03-18T15:03:40.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3e9f7fe640 1 --2- 172.21.15.76:0/656934030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508c130 0x56241509db30 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3e940056b0 tx=0x7f3e94004820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c5f70ecea614072 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:40.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3e7effd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:03:40.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3e7effd640 1 -- 172.21.15.76:0/656934030 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e94039060 con 0x56241508c130 2024-03-18T15:03:40.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3e7effd640 1 -- 172.21.15.76:0/656934030 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7f3e94011040 con 0x56241508c130 2024-03-18T15:03:40.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.339+0000 7f3ea8c21740 1 -- 172.21.15.76:0/656934030 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56241508e410 con 0x56241508c130 2024-03-18T15:03:40.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.340+0000 7f3e7effd640 1 -- 172.21.15.76:0/656934030 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f3e94011260 con 0x56241508c130 2024-03-18T15:03:40.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.340+0000 7f3ea8c21740 10 client.16698 fetch_fsmap learned FSMap version 42 2024-03-18T15:03:40.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.340+0000 7f3ea8c21740 1 -- 172.21.15.76:0/656934030 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56241510b8f0 con 0x56241508c130 2024-03-18T15:03:40.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.340+0000 7f3e7effd640 1 -- 172.21.15.76:0/656934030 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56241508e410 con 0x56241508c130 2024-03-18T15:03:40.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.340+0000 7f3e7effd640 1 -- 172.21.15.76:0/656934030 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7f3e940198e0 con 0x56241508c130 2024-03-18T15:03:40.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.340+0000 7f3ea8c21740 10 client.16698 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T15:03:40.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.340+0000 7f3ea8c21740 10 client.16698 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:03:40.345 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.340+0000 7f3ea8c21740 20 client.16698 populate_metadata read hostname 'smithi076' 2024-03-18T15:03:40.346 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.340+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/656934030 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56241510bfe0 0x56241510e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:40.346 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.340+0000 7f3ea8c21740 4 client.16698 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:03:40.346 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.340+0000 7f3ea8c21740 1 -- 172.21.15.76:0/656934030 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562414ff82d0 con 0x56241510bfe0 2024-03-18T15:03:40.346 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.340+0000 7f3e9effd640 1 --2- 172.21.15.76:0/656934030 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56241510bfe0 0x56241510e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:40.346 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.340+0000 7f3e7effd640 10 client.16698 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:40.346 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.340+0000 7f3e9effd640 1 --2- 172.21.15.76:0/656934030 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56241510bfe0 0x56241510e400 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e539841cb148c4f server_cookie=672ed0abc851c628 in_seq=0 out_seq=0 2024-03-18T15:03:40.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.430+0000 7f3e7effd640 1 -- 172.21.15.76:0/656934030 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562414ff82d0 con 0x56241510bfe0 2024-03-18T15:03:40.431 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.430+0000 7f3e7effd640 10 client.16698 handle_command_reply: tid=0 2024-03-18T15:03:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.430+0000 7f3ea8c21740 1 client.16698 shutdown 2024-03-18T15:03:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.430+0000 7f3ea8c21740 1 -- 172.21.15.76:0/656934030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508c130 msgr2=0x56241509db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.430+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/656934030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508c130 0x56241509db30 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3e940056b0 tx=0x7f3e94004820 comp rx=0 tx=0).stop 2024-03-18T15:03:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.430+0000 7f3ea8c21740 1 -- 172.21.15.76:0/656934030 shutdown_connections 2024-03-18T15:03:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.430+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/656934030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562415095820 0x562415099400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.430+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/656934030 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x56241510bfe0 0x56241510e400 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.430+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/656934030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56241508ca50 0x562415098ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.430+0000 7f3e7effd640 0 client.16698 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.430+0000 7f3ea8c21740 1 --2- 172.21.15.76:0/656934030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56241508c130 0x56241509db30 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.430+0000 7f3ea8c21740 1 -- 172.21.15.76:0/656934030 >> 172.21.15.76:0/656934030 conn(0x562415086a50 msgr2=0x562415087ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.430+0000 7f3ea8c21740 1 -- 172.21.15.76:0/656934030 shutdown_connections 2024-03-18T15:03:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.431+0000 7f3ea8c21740 1 -- 172.21.15.76:0/656934030 wait complete. 2024-03-18T15:03:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.431+0000 7f3ea8c21740 20 client.16698 trim_cache size 0 max 16384 2024-03-18T15:03:40.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:03:40.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:03:40.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:03:40.433 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:03:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.431+0000 7f3ea603d640 1 -- 172.21.15.76:0/3623586593 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3e880a2a10 msgr2=0x7f3e880a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.431+0000 7f3ea603d640 1 --2- 172.21.15.76:0/3623586593 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3e880a2a10 0x7f3e880a4ed0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009fd0 tx=0x7f3e98011bc0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.431+0000 7f3ea603d640 1 -- 172.21.15.76:0/3623586593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ea01012f0 msgr2=0x7f3ea0198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.431+0000 7f3ea603d640 1 --2- 172.21.15.76:0/3623586593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ea01012f0 0x7f3ea0198620 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f3e90005940 tx=0x7f3e90004270 comp rx=0 tx=0).stop 2024-03-18T15:03:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.431+0000 7f3ea603d640 1 -- 172.21.15.76:0/3623586593 shutdown_connections 2024-03-18T15:03:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.431+0000 7f3ea603d640 1 --2- 172.21.15.76:0/3623586593 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f3e880a2a10 0x7f3e880a4ed0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.431+0000 7f3ea603d640 1 --2- 172.21.15.76:0/3623586593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ea0105680 0x7f3ea019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.431+0000 7f3ea603d640 1 --2- 172.21.15.76:0/3623586593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ea01012f0 0x7f3ea0198620 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.431+0000 7f3ea603d640 1 --2- 172.21.15.76:0/3623586593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ea0100930 0x7f3ea01980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.432+0000 7f3ea603d640 1 -- 172.21.15.76:0/3623586593 >> 172.21.15.76:0/3623586593 conn(0x7f3ea0077530 msgr2=0x7f3ea00690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.432+0000 7f3ea603d640 1 -- 172.21.15.76:0/3623586593 shutdown_connections 2024-03-18T15:03:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:40.432+0000 7f3ea603d640 1 -- 172.21.15.76:0/3623586593 wait complete. 2024-03-18T15:03:40.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1022206 9.37 MB/sec execute 461 sec latency 44.921 ms 2024-03-18T15:03:41.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1023617 9.38 MB/sec execute 462 sec latency 13.387 ms 2024-03-18T15:03:42.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:41 smithi076 ceph-mon[32205]: pgmap v330: 97 pgs: 97 active+clean; 327 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 189 op/s 2024-03-18T15:03:42.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:41 smithi174 ceph-mon[36787]: pgmap v330: 97 pgs: 97 active+clean; 327 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 189 op/s 2024-03-18T15:03:42.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:41 smithi077 ceph-mon[35865]: pgmap v330: 97 pgs: 97 active+clean; 327 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 189 op/s 2024-03-18T15:03:42.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1025032 9.37 MB/sec execute 463 sec latency 27.169 ms 2024-03-18T15:03:43.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1026948 9.38 MB/sec execute 464 sec latency 56.675 ms 2024-03-18T15:03:44.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:43 smithi076 ceph-mon[32205]: pgmap v331: 97 pgs: 97 active+clean; 323 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 320 op/s 2024-03-18T15:03:44.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:43 smithi174 ceph-mon[36787]: pgmap v331: 97 pgs: 97 active+clean; 323 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 320 op/s 2024-03-18T15:03:44.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:43 smithi077 ceph-mon[35865]: pgmap v331: 97 pgs: 97 active+clean; 323 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 320 op/s 2024-03-18T15:03:44.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1028098 9.37 MB/sec execute 465 sec latency 31.687 ms 2024-03-18T15:03:45.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1030337 9.37 MB/sec execute 466 sec latency 20.583 ms 2024-03-18T15:03:46.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:45 smithi076 ceph-mon[32205]: pgmap v332: 97 pgs: 97 active+clean; 323 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 226 op/s 2024-03-18T15:03:46.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:45 smithi174 ceph-mon[36787]: pgmap v332: 97 pgs: 97 active+clean; 323 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 226 op/s 2024-03-18T15:03:46.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:45 smithi077 ceph-mon[35865]: pgmap v332: 97 pgs: 97 active+clean; 323 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 226 op/s 2024-03-18T15:03:46.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1031211 9.36 MB/sec execute 467 sec latency 20.828 ms 2024-03-18T15:03:47.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1033417 9.37 MB/sec execute 468 sec latency 24.545 ms 2024-03-18T15:03:48.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:47 smithi076 ceph-mon[32205]: pgmap v333: 97 pgs: 97 active+clean; 323 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 226 op/s 2024-03-18T15:03:48.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:47 smithi174 ceph-mon[36787]: pgmap v333: 97 pgs: 97 active+clean; 323 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 226 op/s 2024-03-18T15:03:48.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:47 smithi077 ceph-mon[35865]: pgmap v333: 97 pgs: 97 active+clean; 323 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 226 op/s 2024-03-18T15:03:48.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1034354 9.36 MB/sec execute 469 sec latency 61.583 ms 2024-03-18T15:03:49.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1036133 9.35 MB/sec execute 470 sec latency 32.126 ms 2024-03-18T15:03:50.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:49 smithi076 ceph-mon[32205]: pgmap v334: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 313 op/s 2024-03-18T15:03:50.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:49 smithi174 ceph-mon[36787]: pgmap v334: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 313 op/s 2024-03-18T15:03:50.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:49 smithi077 ceph-mon[35865]: pgmap v334: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 313 op/s 2024-03-18T15:03:50.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1037713 9.36 MB/sec execute 471 sec latency 14.570 ms 2024-03-18T15:03:51.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1039016 9.35 MB/sec execute 472 sec latency 20.313 ms 2024-03-18T15:03:52.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:51 smithi076 ceph-mon[32205]: pgmap v335: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 217 op/s 2024-03-18T15:03:52.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:51 smithi174 ceph-mon[36787]: pgmap v335: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 217 op/s 2024-03-18T15:03:52.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:51 smithi077 ceph-mon[35865]: pgmap v335: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 217 op/s 2024-03-18T15:03:52.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1041153 9.36 MB/sec execute 473 sec latency 15.408 ms 2024-03-18T15:03:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.523+0000 7fb4fd627640 1 -- 172.21.15.76:0/1161211371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4f8101090 msgr2=0x7fb4f810d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.523+0000 7fb4fd627640 1 --2- 172.21.15.76:0/1161211371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4f8101090 0x7fb4f810d5a0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009cd0 tx=0x7fb4f002c0f0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.523+0000 7fb4fd627640 1 -- 172.21.15.76:0/1161211371 shutdown_connections 2024-03-18T15:03:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.523+0000 7fb4fd627640 1 --2- 172.21.15.76:0/1161211371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4f81006d0 0x7fb4f8100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.523+0000 7fb4fd627640 1 --2- 172.21.15.76:0/1161211371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4f8101090 0x7fb4f810d5a0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.523+0000 7fb4fd627640 1 --2- 172.21.15.76:0/1161211371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4f81066e0 0x7fb4f8106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.523+0000 7fb4fd627640 1 -- 172.21.15.76:0/1161211371 >> 172.21.15.76:0/1161211371 conn(0x7fb4f80778f0 msgr2=0x7fb4f8077d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.523+0000 7fb4fd627640 1 -- 172.21.15.76:0/1161211371 shutdown_connections 2024-03-18T15:03:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.523+0000 7fb4fd627640 1 -- 172.21.15.76:0/1161211371 wait complete. 2024-03-18T15:03:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4fd627640 1 Processor -- start 2024-03-18T15:03:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4fd627640 1 -- start start 2024-03-18T15:03:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4fd627640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4f81006d0 0x7fb4f8198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4fd627640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4f8101090 0x7fb4f8198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4fd627640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4f81066e0 0x7fb4f819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4fd627640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f8110af0 con 0x7fb4f8101090 2024-03-18T15:03:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4fd627640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f8110970 con 0x7fb4f81006d0 2024-03-18T15:03:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4fd627640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f8110c70 con 0x7fb4f81066e0 2024-03-18T15:03:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4effff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4f8101090 0x7fb4f8198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4effff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4f8101090 0x7fb4f8198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48892/0 (socket says 172.21.15.76:48892) 2024-03-18T15:03:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4effff640 1 -- 172.21.15.76:0/3360678859 learned_addr learned my addr 172.21.15.76:0/3360678859 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4f6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4f81006d0 0x7fb4f8198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4f77fe640 1 --2- 172.21.15.76:0/3360678859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4f81066e0 0x7fb4f819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4effff640 1 -- 172.21.15.76:0/3360678859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4f81066e0 msgr2=0x7fb4f819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4effff640 1 --2- 172.21.15.76:0/3360678859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4f81066e0 0x7fb4f819c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4effff640 1 -- 172.21.15.76:0/3360678859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4f81006d0 msgr2=0x7fb4f8198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4effff640 1 --2- 172.21.15.76:0/3360678859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4f81006d0 0x7fb4f8198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.524+0000 7fb4effff640 1 -- 172.21.15.76:0/3360678859 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f819cfc0 con 0x7fb4f8101090 2024-03-18T15:03:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.525+0000 7fb4effff640 1 --2- 172.21.15.76:0/3360678859 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4f8101090 0x7fb4f8198560 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb4e000e9f0 tx=0x7fb4e000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.525+0000 7fb4f4ff9640 1 -- 172.21.15.76:0/3360678859 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e000cd80 con 0x7fb4f8101090 2024-03-18T15:03:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.525+0000 7fb4f4ff9640 1 -- 172.21.15.76:0/3360678859 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4e0005500 con 0x7fb4f8101090 2024-03-18T15:03:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.525+0000 7fb4fd627640 1 -- 172.21.15.76:0/3360678859 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4f819d2b0 con 0x7fb4f8101090 2024-03-18T15:03:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.525+0000 7fb4f4ff9640 1 -- 172.21.15.76:0/3360678859 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e0010690 con 0x7fb4f8101090 2024-03-18T15:03:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.525+0000 7fb4fd627640 1 -- 172.21.15.76:0/3360678859 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4f81a4b40 con 0x7fb4f8101090 2024-03-18T15:03:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.527+0000 7fb4f4ff9640 1 -- 172.21.15.76:0/3360678859 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fb4e00046c0 con 0x7fb4f8101090 2024-03-18T15:03:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.527+0000 7fb500211740 1 Processor -- start 2024-03-18T15:03:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.527+0000 7fb500211740 1 -- start start 2024-03-18T15:03:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb500211740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be0c130 0x55a32be0c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb500211740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0ca50 0x55a32be152e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb500211740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be15820 0x55a32be17c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb500211740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a32bd70a40 con 0x55a32be15820 2024-03-18T15:03:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb500211740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a32bd19920 con 0x55a32be0ca50 2024-03-18T15:03:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb500211740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a32bd6cae0 con 0x55a32be0c130 2024-03-18T15:03:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb4f4ff9640 1 --2- 172.21.15.76:0/3360678859 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb4e40a2a10 0x7fb4e40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb4f6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0ca50 0x55a32be152e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb4f77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be0c130 0x55a32be0c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb4effff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be15820 0x55a32be17c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb4f6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0ca50 0x55a32be152e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:39798/0 (socket says 172.21.15.76:39798) 2024-03-18T15:03:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb4f6ffd640 1 -- 172.21.15.76:0/620151612 learned_addr learned my addr 172.21.15.76:0/620151612 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb4f77fe640 1 --2- 172.21.15.76:0/3360678859 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb4e40a2a10 0x7fb4e40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb4f4ff9640 1 -- 172.21.15.76:0/3360678859 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7fb4e00cf230 con 0x7fb4f8101090 2024-03-18T15:03:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb4f6ffd640 1 -- 172.21.15.76:0/620151612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be0c130 msgr2=0x55a32be0c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb4f6ffd640 1 --2- 172.21.15.76:0/620151612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be0c130 0x55a32be0c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb4f6ffd640 1 -- 172.21.15.76:0/620151612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be15820 msgr2=0x55a32be17c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb4f6ffd640 1 --2- 172.21.15.76:0/620151612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be15820 0x55a32be17c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb4f6ffd640 1 -- 172.21.15.76:0/620151612 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a32bd31460 con 0x55a32be0ca50 2024-03-18T15:03:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb4f77fe640 1 --2- 172.21.15.76:0/3360678859 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb4e40a2a10 0x7fb4e40a4ed0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0005cc0 tx=0x7fb4f002cec0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.528+0000 7fb4f6ffd640 1 --2- 172.21.15.76:0/620151612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0ca50 0x55a32be152e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8009c70 tx=0x7fb4e8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c3d9976ef580915 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.529+0000 7fb4ed7fa640 1 -- 172.21.15.76:0/620151612 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8004230 con 0x55a32be0ca50 2024-03-18T15:03:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.529+0000 7fb4ed7fa640 1 -- 172.21.15.76:0/620151612 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4e80043d0 con 0x55a32be0ca50 2024-03-18T15:03:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.529+0000 7fb4ed7fa640 1 -- 172.21.15.76:0/620151612 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e80197c0 con 0x55a32be0ca50 2024-03-18T15:03:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.529+0000 7fb500211740 1 -- 172.21.15.76:0/620151612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0ca50 msgr2=0x55a32be152e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.529+0000 7fb500211740 1 --2- 172.21.15.76:0/620151612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0ca50 0x55a32be152e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8009c70 tx=0x7fb4e8011810 comp rx=0 tx=0).stop 2024-03-18T15:03:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.529+0000 7fb500211740 1 -- 172.21.15.76:0/620151612 shutdown_connections 2024-03-18T15:03:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.529+0000 7fb500211740 1 --2- 172.21.15.76:0/620151612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be15820 0x55a32be17c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.529+0000 7fb500211740 1 --2- 172.21.15.76:0/620151612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0ca50 0x55a32be152e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.529+0000 7fb500211740 1 --2- 172.21.15.76:0/620151612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be0c130 0x55a32be0c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.529+0000 7fb500211740 1 -- 172.21.15.76:0/620151612 >> 172.21.15.76:0/620151612 conn(0x55a32be06a50 msgr2=0x55a32be06e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.529+0000 7fb500211740 1 -- 172.21.15.76:0/620151612 shutdown_connections 2024-03-18T15:03:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.529+0000 7fb500211740 1 -- 172.21.15.76:0/620151612 wait complete. 2024-03-18T15:03:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.530+0000 7fb500211740 1 -- start start 2024-03-18T15:03:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.530+0000 7fb500211740 1 Processor -- start 2024-03-18T15:03:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.530+0000 7fb500211740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be0c130 0x55a32be12810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.530+0000 7fb500211740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0ca50 0x55a32be12d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.530+0000 7fb500211740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be15820 0x55a32be13290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.530+0000 7fb500211740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a32be18450 con 0x55a32be0c130 2024-03-18T15:03:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.530+0000 7fb500211740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a32bd2cd70 con 0x55a32be0ca50 2024-03-18T15:03:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.530+0000 7fb500211740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a32be185d0 con 0x55a32be15820 2024-03-18T15:03:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.530+0000 7fb500211740 10 client.0 _ll_register_callbacks cb 0x55a32ba84a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:03:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.530+0000 7fb4f6ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be0c130 0x55a32be12810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.530+0000 7fb4effff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0ca50 0x55a32be12d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.530+0000 7fb4f6ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be0c130 0x55a32be12810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48902/0 (socket says 172.21.15.76:48902) 2024-03-18T15:03:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.530+0000 7fb4f6ffd640 1 -- 172.21.15.76:0/1754499783 learned_addr learned my addr 172.21.15.76:0/1754499783 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.530+0000 7fb4f77fe640 1 --2- 172.21.15.76:0/1754499783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be15820 0x55a32be13290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.530+0000 7fb4f6ffd640 1 -- 172.21.15.76:0/1754499783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be15820 msgr2=0x55a32be13290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.531+0000 7fb4f6ffd640 1 --2- 172.21.15.76:0/1754499783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be15820 0x55a32be13290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.531+0000 7fb4f6ffd640 1 -- 172.21.15.76:0/1754499783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0ca50 msgr2=0x55a32be12d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.531+0000 7fb4f6ffd640 1 --2- 172.21.15.76:0/1754499783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0ca50 0x55a32be12d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.531+0000 7fb4f6ffd640 1 -- 172.21.15.76:0/1754499783 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a32ba85210 con 0x55a32be0c130 2024-03-18T15:03:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.531+0000 7fb4effff640 1 --2- 172.21.15.76:0/1754499783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0ca50 0x55a32be12d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:03:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.531+0000 7fb4f6ffd640 1 --2- 172.21.15.76:0/1754499783 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be0c130 0x55a32be12810 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8003f80 tx=0x7fb4e8005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de7ebcaf2b5c95 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.531+0000 7fb4de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:03:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.531+0000 7fb4de7fc640 1 -- 172.21.15.76:0/1754499783 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8026070 con 0x55a32be0c130 2024-03-18T15:03:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.531+0000 7fb4de7fc640 1 -- 172.21.15.76:0/1754499783 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7fb4e80035f0 con 0x55a32be0c130 2024-03-18T15:03:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.531+0000 7fb500211740 1 -- 172.21.15.76:0/1754499783 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4e80043d0 con 0x55a32be0c130 2024-03-18T15:03:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.531+0000 7fb4de7fc640 1 -- 172.21.15.76:0/1754499783 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fb4e8032050 con 0x55a32be0c130 2024-03-18T15:03:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.531+0000 7fb500211740 10 client.16710 fetch_fsmap learned FSMap version 42 2024-03-18T15:03:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.531+0000 7fb500211740 1 -- 172.21.15.76:0/1754499783 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55a32bb51620 con 0x55a32be0c130 2024-03-18T15:03:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.531+0000 7fb4de7fc640 1 -- 172.21.15.76:0/1754499783 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4e80043d0 con 0x55a32be0c130 2024-03-18T15:03:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.531+0000 7fb4de7fc640 1 -- 172.21.15.76:0/1754499783 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7fb4e8018070 con 0x55a32be0c130 2024-03-18T15:03:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.531+0000 7fb500211740 10 client.16710 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T15:03:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.531+0000 7fb500211740 10 client.16710 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:03:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.532+0000 7fb500211740 20 client.16710 populate_metadata read hostname 'smithi076' 2024-03-18T15:03:53.541 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.532+0000 7fb500211740 1 --2- 172.21.15.76:0/1754499783 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a32be25840 0x55a32be27c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.541 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.532+0000 7fb500211740 4 client.16710 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:03:53.541 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.532+0000 7fb500211740 1 -- 172.21.15.76:0/1754499783 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a32bd7df80 con 0x55a32be25840 2024-03-18T15:03:53.541 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.532+0000 7fb4effff640 1 --2- 172.21.15.76:0/1754499783 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a32be25840 0x55a32be27c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.541 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.532+0000 7fb4de7fc640 10 client.16710 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:53.541 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.532+0000 7fb4effff640 1 --2- 172.21.15.76:0/1754499783 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a32be25840 0x55a32be27c30 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ce6824cef4fc29d server_cookie=a53cf83ff6ab392f in_seq=0 out_seq=0 2024-03-18T15:03:53.541 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.534+0000 7fb4de7fc640 1 -- 172.21.15.76:0/1754499783 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a32bd7df80 con 0x55a32be25840 2024-03-18T15:03:53.541 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.534+0000 7fb4de7fc640 10 client.16710 handle_command_reply: tid=0 2024-03-18T15:03:53.541 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.534+0000 7fb500211740 1 client.16710 shutdown 2024-03-18T15:03:53.541 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.534+0000 7fb500211740 1 -- 172.21.15.76:0/1754499783 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be0c130 msgr2=0x55a32be12810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:53.541 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.534+0000 7fb500211740 1 --2- 172.21.15.76:0/1754499783 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be0c130 0x55a32be12810 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8003f80 tx=0x7fb4e8005dc0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.541 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.534+0000 7fb500211740 1 -- 172.21.15.76:0/1754499783 shutdown_connections 2024-03-18T15:03:53.541 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.534+0000 7fb500211740 1 --2- 172.21.15.76:0/1754499783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be15820 0x55a32be13290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.541 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.534+0000 7fb500211740 1 --2- 172.21.15.76:0/1754499783 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a32be25840 0x55a32be27c30 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.541 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.534+0000 7fb500211740 1 --2- 172.21.15.76:0/1754499783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0ca50 0x55a32be12d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.541 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.534+0000 7fb500211740 1 --2- 172.21.15.76:0/1754499783 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be0c130 0x55a32be12810 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.542 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.534+0000 7fb4de7fc640 0 client.16710 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:53.542 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.534+0000 7fb500211740 1 -- 172.21.15.76:0/1754499783 >> 172.21.15.76:0/1754499783 conn(0x55a32be06a50 msgr2=0x55a32be087a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:53.542 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.534+0000 7fb500211740 1 -- 172.21.15.76:0/1754499783 shutdown_connections 2024-03-18T15:03:53.542 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.534+0000 7fb500211740 1 -- 172.21.15.76:0/1754499783 wait complete. 2024-03-18T15:03:53.542 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.534+0000 7fb500211740 20 client.16710 trim_cache size 0 max 16384 2024-03-18T15:03:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.548+0000 7fb500211740 1 Processor -- start 2024-03-18T15:03:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.548+0000 7fb500211740 1 -- start start 2024-03-18T15:03:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb500211740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0c130 0x55a32be211f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb500211740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be0ca50 0x55a32be2d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb500211740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be15820 0x55a32be07e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb500211740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f81afd10 con 0x55a32be0ca50 2024-03-18T15:03:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb500211740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f81ad5e0 con 0x55a32be0c130 2024-03-18T15:03:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb500211740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f81ab450 con 0x55a32be15820 2024-03-18T15:03:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb4effff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be0ca50 0x55a32be2d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb4f6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0c130 0x55a32be211f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb4effff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be0ca50 0x55a32be2d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48916/0 (socket says 172.21.15.76:48916) 2024-03-18T15:03:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb4f6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0c130 0x55a32be211f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:39820/0 (socket says 172.21.15.76:39820) 2024-03-18T15:03:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb4effff640 1 -- 172.21.15.76:0/577195632 learned_addr learned my addr 172.21.15.76:0/577195632 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb4f77fe640 1 --2- 172.21.15.76:0/577195632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be15820 0x55a32be07e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb4effff640 1 -- 172.21.15.76:0/577195632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be15820 msgr2=0x55a32be07e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb4effff640 1 --2- 172.21.15.76:0/577195632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be15820 0x55a32be07e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb4effff640 1 -- 172.21.15.76:0/577195632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0c130 msgr2=0x55a32be211f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb4effff640 1 --2- 172.21.15.76:0/577195632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0c130 0x55a32be211f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb4effff640 1 -- 172.21.15.76:0/577195632 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a32be21730 con 0x55a32be0ca50 2024-03-18T15:03:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb4f6ffd640 1 --2- 172.21.15.76:0/577195632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0c130 0x55a32be211f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:03:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.549+0000 7fb4effff640 1 --2- 172.21.15.76:0/577195632 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be0ca50 0x55a32be2d3b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb4e009b840 tx=0x7fb4e006e650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb50cfa45fbba20 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.550+0000 7fb4ecff9640 1 -- 172.21.15.76:0/577195632 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e009fa00 con 0x55a32be0ca50 2024-03-18T15:03:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.550+0000 7fb4ecff9640 1 -- 172.21.15.76:0/577195632 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4e00a4050 con 0x55a32be0ca50 2024-03-18T15:03:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.550+0000 7fb4ecff9640 1 -- 172.21.15.76:0/577195632 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e00a3bc0 con 0x55a32be0ca50 2024-03-18T15:03:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.550+0000 7fb500211740 1 -- 172.21.15.76:0/577195632 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be0ca50 msgr2=0x55a32be2d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.550+0000 7fb500211740 1 --2- 172.21.15.76:0/577195632 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be0ca50 0x55a32be2d3b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb4e009b840 tx=0x7fb4e006e650 comp rx=0 tx=0).stop 2024-03-18T15:03:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.550+0000 7fb500211740 1 -- 172.21.15.76:0/577195632 shutdown_connections 2024-03-18T15:03:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.550+0000 7fb500211740 1 --2- 172.21.15.76:0/577195632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be15820 0x55a32be07e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.550+0000 7fb500211740 1 --2- 172.21.15.76:0/577195632 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be0ca50 0x55a32be2d3b0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.550+0000 7fb500211740 1 --2- 172.21.15.76:0/577195632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0c130 0x55a32be211f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.550+0000 7fb500211740 1 -- 172.21.15.76:0/577195632 >> 172.21.15.76:0/577195632 conn(0x55a32be06a50 msgr2=0x55a32be11d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb500211740 1 -- 172.21.15.76:0/577195632 shutdown_connections 2024-03-18T15:03:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb500211740 1 -- 172.21.15.76:0/577195632 wait complete. 2024-03-18T15:03:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb500211740 1 -- start start 2024-03-18T15:03:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb500211740 1 Processor -- start 2024-03-18T15:03:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb500211740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0c130 0x55a32be1dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb500211740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be0ca50 0x55a32be0f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb500211740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be15820 0x55a32be0f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb500211740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f81abaa0 con 0x55a32be15820 2024-03-18T15:03:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb500211740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f81acf10 con 0x55a32be0c130 2024-03-18T15:03:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb500211740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f81ac170 con 0x55a32be0ca50 2024-03-18T15:03:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb500211740 10 client.0 _ll_register_callbacks cb 0x7fb4e009b3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:03:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb4f77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be15820 0x55a32be0f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb4f6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0c130 0x55a32be1dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb4f77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be15820 0x55a32be0f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48918/0 (socket says 172.21.15.76:48918) 2024-03-18T15:03:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb4f77fe640 1 -- 172.21.15.76:0/1381207532 learned_addr learned my addr 172.21.15.76:0/1381207532 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:03:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb4effff640 1 --2- 172.21.15.76:0/1381207532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be0ca50 0x55a32be0f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb4f77fe640 1 -- 172.21.15.76:0/1381207532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be0ca50 msgr2=0x55a32be0f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb4f77fe640 1 --2- 172.21.15.76:0/1381207532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be0ca50 0x55a32be0f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb4f77fe640 1 -- 172.21.15.76:0/1381207532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0c130 msgr2=0x55a32be1dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:03:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.551+0000 7fb4f77fe640 1 --2- 172.21.15.76:0/1381207532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0c130 0x55a32be1dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.552+0000 7fb4f77fe640 1 -- 172.21.15.76:0/1381207532 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a32be8a9e0 con 0x55a32be15820 2024-03-18T15:03:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.552+0000 7fb4f6ffd640 1 --2- 172.21.15.76:0/1381207532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0c130 0x55a32be1dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:03:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.552+0000 7fb4f77fe640 1 --2- 172.21.15.76:0/1381207532 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be15820 0x55a32be0f9b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009cd0 tx=0x7fb4f002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67a0e3bf4943dd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:03:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.552+0000 7fb4de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:03:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.552+0000 7fb4de7fc640 1 -- 172.21.15.76:0/1381207532 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0039040 con 0x55a32be15820 2024-03-18T15:03:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.552+0000 7fb4de7fc640 1 -- 172.21.15.76:0/1381207532 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8484+0+0 (secure 0 0 0) 0x7fb4f0004250 con 0x55a32be15820 2024-03-18T15:03:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.552+0000 7fb500211740 1 -- 172.21.15.76:0/1381207532 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4e00a4050 con 0x55a32be15820 2024-03-18T15:03:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.552+0000 7fb4de7fc640 1 -- 172.21.15.76:0/1381207532 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fb4f0004b40 con 0x55a32be15820 2024-03-18T15:03:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.552+0000 7fb500211740 10 client.16716 fetch_fsmap learned FSMap version 42 2024-03-18T15:03:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.552+0000 7fb500211740 1 -- 172.21.15.76:0/1381207532 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55a32be8bb90 con 0x55a32be15820 2024-03-18T15:03:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.552+0000 7fb4de7fc640 1 -- 172.21.15.76:0/1381207532 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4e00a4050 con 0x55a32be15820 2024-03-18T15:03:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.552+0000 7fb4de7fc640 1 -- 172.21.15.76:0/1381207532 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7fb4f003f040 con 0x55a32be15820 2024-03-18T15:03:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.552+0000 7fb500211740 10 client.16716 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T15:03:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.552+0000 7fb500211740 10 client.16716 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:03:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.552+0000 7fb500211740 20 client.16716 populate_metadata read hostname 'smithi076' 2024-03-18T15:03:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.552+0000 7fb500211740 1 --2- 172.21.15.76:0/1381207532 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a32be8c390 0x55a32be8e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:03:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.553+0000 7fb500211740 4 client.16716 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T15:03:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.553+0000 7fb500211740 1 -- 172.21.15.76:0/1381207532 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a32bd782d0 con 0x55a32be8c390 2024-03-18T15:03:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.553+0000 7fb4f6ffd640 1 --2- 172.21.15.76:0/1381207532 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a32be8c390 0x55a32be8e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:03:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.553+0000 7fb4de7fc640 10 client.16716 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.553+0000 7fb4f6ffd640 1 --2- 172.21.15.76:0/1381207532 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a32be8c390 0x55a32be8e7b0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2169669417f6085a server_cookie=f64139c40f6dcd63 in_seq=0 out_seq=0 2024-03-18T15:03:53.634 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.630+0000 7fb4de7fc640 1 -- 172.21.15.76:0/1381207532 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a32bd782d0 con 0x55a32be8c390 2024-03-18T15:03:53.634 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.630+0000 7fb4de7fc640 10 client.16716 handle_command_reply: tid=0 2024-03-18T15:03:53.634 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.630+0000 7fb500211740 1 client.16716 shutdown 2024-03-18T15:03:53.634 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.630+0000 7fb500211740 1 -- 172.21.15.76:0/1381207532 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be15820 msgr2=0x55a32be0f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:53.634 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.630+0000 7fb500211740 1 --2- 172.21.15.76:0/1381207532 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be15820 0x55a32be0f9b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009cd0 tx=0x7fb4f002e040 comp rx=0 tx=0).stop 2024-03-18T15:03:53.634 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.630+0000 7fb500211740 1 -- 172.21.15.76:0/1381207532 shutdown_connections 2024-03-18T15:03:53.634 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.630+0000 7fb500211740 1 --2- 172.21.15.76:0/1381207532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a32be0ca50 0x55a32be0f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.634 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.630+0000 7fb500211740 1 --2- 172.21.15.76:0/1381207532 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a32be8c390 0x55a32be8e7b0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.630+0000 7fb500211740 1 --2- 172.21.15.76:0/1381207532 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32be15820 0x55a32be0f9b0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.630+0000 7fb4de7fc640 0 client.16716 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:03:53.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.630+0000 7fb500211740 1 --2- 172.21.15.76:0/1381207532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a32be0c130 0x55a32be1dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.630+0000 7fb500211740 1 -- 172.21.15.76:0/1381207532 >> 172.21.15.76:0/1381207532 conn(0x55a32be06a50 msgr2=0x55a32be05bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:53.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.630+0000 7fb500211740 1 -- 172.21.15.76:0/1381207532 shutdown_connections 2024-03-18T15:03:53.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.630+0000 7fb500211740 1 -- 172.21.15.76:0/1381207532 wait complete. 2024-03-18T15:03:53.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.631+0000 7fb500211740 20 client.16716 trim_cache size 0 max 16384 2024-03-18T15:03:53.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:{ 2024-03-18T15:03:53.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "message": "", 2024-03-18T15:03:53.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout: "return_code": 0 2024-03-18T15:03:53.635 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stdout:} 2024-03-18T15:03:53.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.631+0000 7fb4fd627640 1 -- 172.21.15.76:0/3360678859 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb4e40a2a10 msgr2=0x7fb4e40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:53.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.631+0000 7fb4fd627640 1 --2- 172.21.15.76:0/3360678859 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb4e40a2a10 0x7fb4e40a4ed0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0005cc0 tx=0x7fb4f002cec0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.636 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.631+0000 7fb4fd627640 1 -- 172.21.15.76:0/3360678859 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4f8101090 msgr2=0x7fb4f8198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:03:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.631+0000 7fb4fd627640 1 --2- 172.21.15.76:0/3360678859 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4f8101090 0x7fb4f8198560 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb4e000e9f0 tx=0x7fb4e000edf0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.631+0000 7fb4fd627640 1 -- 172.21.15.76:0/3360678859 shutdown_connections 2024-03-18T15:03:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.631+0000 7fb4fd627640 1 --2- 172.21.15.76:0/3360678859 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fb4e40a2a10 0x7fb4e40a4ed0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.631+0000 7fb4fd627640 1 --2- 172.21.15.76:0/3360678859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4f81066e0 0x7fb4f819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.631+0000 7fb4fd627640 1 --2- 172.21.15.76:0/3360678859 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4f8101090 0x7fb4f8198560 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.631+0000 7fb4fd627640 1 --2- 172.21.15.76:0/3360678859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4f81006d0 0x7fb4f8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:03:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.631+0000 7fb4fd627640 1 -- 172.21.15.76:0/3360678859 >> 172.21.15.76:0/3360678859 conn(0x7fb4f80778f0 msgr2=0x7fb4f80ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:03:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.631+0000 7fb4fd627640 1 -- 172.21.15.76:0/3360678859 shutdown_connections 2024-03-18T15:03:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:03:53.631+0000 7fb4fd627640 1 -- 172.21.15.76:0/3360678859 wait complete. 2024-03-18T15:03:53.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1042323 9.35 MB/sec execute 474 sec latency 33.553 ms 2024-03-18T15:03:54.018 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:53 smithi174 ceph-mon[36787]: pgmap v336: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 337 op/s 2024-03-18T15:03:54.028 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:53 smithi077 ceph-mon[35865]: pgmap v336: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 337 op/s 2024-03-18T15:03:54.029 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:03:53 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setuser ceph since I am not root 2024-03-18T15:03:54.029 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:03:53 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setgroup ceph since I am not root 2024-03-18T15:03:54.029 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:03:53 smithi077 ceph-mds[65404]: -- 172.21.15.77:0/2041507473 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f64620fa40 con 0x55f6461e2800 2024-03-18T15:03:54.029 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:03:53 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: starting mds.e at 2024-03-18T15:03:54.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:53 smithi076 ceph-mon[32205]: pgmap v336: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 337 op/s 2024-03-18T15:03:54.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1044556 9.36 MB/sec execute 475 sec latency 59.083 ms 2024-03-18T15:03:55.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:54 smithi076 ceph-mon[32205]: daemon mds.e restarted 2024-03-18T15:03:55.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:54 smithi174 ceph-mon[36787]: daemon mds.e restarted 2024-03-18T15:03:55.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:54 smithi077 ceph-mon[35865]: daemon mds.e restarted 2024-03-18T15:03:55.761 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1045850 9.35 MB/sec execute 476 sec latency 20.876 ms 2024-03-18T15:03:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:55 smithi076 ceph-mon[32205]: osdmap e105: 12 total, 12 up, 12 in 2024-03-18T15:03:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:55 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.77:6834/2933679925,v1:172.21.15.77:6835/2933679925] up:boot 2024-03-18T15:03:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:55 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T15:03:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:55 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:03:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:55 smithi076 ceph-mon[32205]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T15:03:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:55 smithi076 ceph-mon[32205]: pgmap v338: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 248 op/s 2024-03-18T15:03:56.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:55 smithi174 ceph-mon[36787]: osdmap e105: 12 total, 12 up, 12 in 2024-03-18T15:03:56.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:55 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.77:6834/2933679925,v1:172.21.15.77:6835/2933679925] up:boot 2024-03-18T15:03:56.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:55 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T15:03:56.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:55 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:03:56.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:55 smithi174 ceph-mon[36787]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T15:03:56.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:55 smithi174 ceph-mon[36787]: pgmap v338: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 248 op/s 2024-03-18T15:03:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:55 smithi077 ceph-mon[35865]: osdmap e105: 12 total, 12 up, 12 in 2024-03-18T15:03:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:55 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.77:6834/2933679925,v1:172.21.15.77:6835/2933679925] up:boot 2024-03-18T15:03:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:55 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T15:03:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:55 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:03:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:55 smithi077 ceph-mon[35865]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T15:03:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:55 smithi077 ceph-mon[35865]: pgmap v338: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 248 op/s 2024-03-18T15:03:56.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1048202 9.36 MB/sec execute 477 sec latency 16.678 ms 2024-03-18T15:03:57.760 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1049573 9.35 MB/sec execute 478 sec latency 15.688 ms 2024-03-18T15:03:58.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:57 smithi076 ceph-mon[32205]: pgmap v339: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 248 op/s 2024-03-18T15:03:58.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:57 smithi174 ceph-mon[36787]: pgmap v339: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 248 op/s 2024-03-18T15:03:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:57 smithi077 ceph-mon[35865]: pgmap v339: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 248 op/s 2024-03-18T15:03:58.761 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1051723 9.36 MB/sec execute 479 sec latency 16.910 ms 2024-03-18T15:03:59.761 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1052452 9.34 MB/sec execute 480 sec latency 44.115 ms 2024-03-18T15:04:00.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:03:59 smithi076 ceph-mon[32205]: pgmap v340: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 265 op/s 2024-03-18T15:04:00.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:03:59 smithi174 ceph-mon[36787]: pgmap v340: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 265 op/s 2024-03-18T15:04:00.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:03:59 smithi077 ceph-mon[35865]: pgmap v340: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 265 op/s 2024-03-18T15:04:00.761 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1053709 9.34 MB/sec execute 481 sec latency 330.375 ms 2024-03-18T15:04:01.761 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1053709 9.32 MB/sec execute 482 sec latency 1330.434 ms 2024-03-18T15:04:02.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:01 smithi076 ceph-mon[32205]: pgmap v341: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 265 op/s 2024-03-18T15:04:02.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:01 smithi174 ceph-mon[36787]: pgmap v341: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 265 op/s 2024-03-18T15:04:02.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:01 smithi077 ceph-mon[35865]: pgmap v341: 97 pgs: 97 active+clean; 324 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 265 op/s 2024-03-18T15:04:02.761 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1053709 9.30 MB/sec execute 483 sec latency 2330.525 ms 2024-03-18T15:04:03.761 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1053709 9.28 MB/sec execute 484 sec latency 3330.573 ms 2024-03-18T15:04:04.761 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1055116 9.28 MB/sec execute 485 sec latency 3342.565 ms 2024-03-18T15:04:05.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:04 smithi076 ceph-mon[32205]: pgmap v342: 97 pgs: 97 active+clean; 325 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 200 op/s 2024-03-18T15:04:05.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:04 smithi174 ceph-mon[36787]: pgmap v342: 97 pgs: 97 active+clean; 325 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 200 op/s 2024-03-18T15:04:05.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:04 smithi077 ceph-mon[35865]: pgmap v342: 97 pgs: 97 active+clean; 325 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 200 op/s 2024-03-18T15:04:05.761 INFO:tasks.workunit.client.0.smithi076.stdout: 1 1055846 9.27 MB/sec execute 486 sec latency 52.320 ms 2024-03-18T15:04:06.176 INFO:tasks.workunit.client.0.smithi076.stdout:[1056580] write failed on handle 10221 (Resource temporarily unavailable) 2024-03-18T15:04:06.176 INFO:tasks.workunit.client.0.smithi076.stdout:Child failed with status 1 2024-03-18T15:04:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:04:06.177 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-03-18T15:04:06.177 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-18T15:04:06.492 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi076 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e19f704afd9957ea657ab858f9ecf76401fef6f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-18T15:04:06.729 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=72925a15d9c44b068fb7f231efa133ed Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi076 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e19f704afd9957ea657ab858f9ecf76401fef6f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-18T15:04:06.731 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-18T15:04:06.742 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi076.front.sepia.ceph.com 2024-03-18T15:04:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.721+0000 7f6979873640 1 -- 172.21.15.76:0/556528078 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974105680 msgr2=0x7f6974105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.721+0000 7f6979873640 1 --2- 172.21.15.76:0/556528078 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974105680 0x7f6974105a60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6964009ca0 tx=0x7f696402c0f0 comp rx=0 tx=0).stop 2024-03-18T15:04:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.721+0000 7f6979873640 1 -- 172.21.15.76:0/556528078 shutdown_connections 2024-03-18T15:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.721+0000 7f6979873640 1 --2- 172.21.15.76:0/556528078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69741012f0 0x7f697410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.721+0000 7f6979873640 1 --2- 172.21.15.76:0/556528078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6974100930 0x7f6974100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.721+0000 7f6979873640 1 --2- 172.21.15.76:0/556528078 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974105680 0x7f6974105a60 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.721+0000 7f6979873640 1 -- 172.21.15.76:0/556528078 >> 172.21.15.76:0/556528078 conn(0x7f6974077530 msgr2=0x7f6974077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.721+0000 7f6979873640 1 -- 172.21.15.76:0/556528078 shutdown_connections 2024-03-18T15:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.721+0000 7f6979873640 1 -- 172.21.15.76:0/556528078 wait complete. 2024-03-18T15:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.722+0000 7f6979873640 1 Processor -- start 2024-03-18T15:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.722+0000 7f6979873640 1 -- start start 2024-03-18T15:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.722+0000 7f6979873640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974100930 0x7f6974196f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.722+0000 7f6979873640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69741012f0 0x7f69741974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.722+0000 7f6979873640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6974105680 0x7f697419b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.722+0000 7f6979873640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6974110ba0 con 0x7f6974100930 2024-03-18T15:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.722+0000 7f6979873640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6974110a20 con 0x7f69741012f0 2024-03-18T15:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.722+0000 7f6979873640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6974110d20 con 0x7f6974105680 2024-03-18T15:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.722+0000 7f6972ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974100930 0x7f6974196f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.722+0000 7f6972ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974100930 0x7f6974196f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38520/0 (socket says 172.21.15.76:38520) 2024-03-18T15:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.722+0000 7f6972ffd640 1 -- 172.21.15.76:0/2955182410 learned_addr learned my addr 172.21.15.76:0/2955182410 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.722+0000 7f69727fc640 1 --2- 172.21.15.76:0/2955182410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69741012f0 0x7f69741974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.722+0000 7f69737fe640 1 --2- 172.21.15.76:0/2955182410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6974105680 0x7f697419b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.723+0000 7f6972ffd640 1 -- 172.21.15.76:0/2955182410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6974105680 msgr2=0x7f697419b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.723+0000 7f6972ffd640 1 --2- 172.21.15.76:0/2955182410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6974105680 0x7f697419b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.723+0000 7f6972ffd640 1 -- 172.21.15.76:0/2955182410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69741012f0 msgr2=0x7f69741974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.723+0000 7f6972ffd640 1 --2- 172.21.15.76:0/2955182410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69741012f0 0x7f69741974c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.723+0000 7f6972ffd640 1 -- 172.21.15.76:0/2955182410 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f697419bf20 con 0x7f6974100930 2024-03-18T15:04:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.723+0000 7f6972ffd640 1 --2- 172.21.15.76:0/2955182410 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974100930 0x7f6974196f80 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6964005e30 tx=0x7f6964032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.723+0000 7f6978871640 1 -- 172.21.15.76:0/2955182410 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6964042020 con 0x7f6974100930 2024-03-18T15:04:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.723+0000 7f6978871640 1 -- 172.21.15.76:0/2955182410 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6964034700 con 0x7f6974100930 2024-03-18T15:04:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.723+0000 7f6979873640 1 -- 172.21.15.76:0/2955182410 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f697419c1b0 con 0x7f6974100930 2024-03-18T15:04:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.723+0000 7f6978871640 1 -- 172.21.15.76:0/2955182410 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6964039040 con 0x7f6974100930 2024-03-18T15:04:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.723+0000 7f6979873640 1 -- 172.21.15.76:0/2955182410 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69741a3aa0 con 0x7f6974100930 2024-03-18T15:04:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.725+0000 7f6978871640 1 -- 172.21.15.76:0/2955182410 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f6964045020 con 0x7f6974100930 2024-03-18T15:04:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.725+0000 7f697c45d740 1 Processor -- start 2024-03-18T15:04:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.725+0000 7f6978871640 1 --2- 172.21.15.76:0/2955182410 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f695c0a2a10 0x7f695c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.725+0000 7f697c45d740 1 -- start start 2024-03-18T15:04:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f69737fe640 1 --2- 172.21.15.76:0/2955182410 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f695c0a2a10 0x7f695c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f697c45d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a028914af0 0x55a028914ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f6978871640 1 -- 172.21.15.76:0/2955182410 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8440+0+0 (secure 0 0 0) 0x7f69640f0c80 con 0x7f6974100930 2024-03-18T15:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f697c45d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a028915410 0x55a0289204f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f697c45d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a028920a30 0x55a028922e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f697c45d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a028879a40 con 0x55a028920a30 2024-03-18T15:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f697c45d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a028822920 con 0x55a028915410 2024-03-18T15:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f697c45d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a028875ae0 con 0x55a028914af0 2024-03-18T15:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f69737fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a028920a30 0x55a028922e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f69737fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a028920a30 0x55a028922e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38530/0 (socket says 172.21.15.76:38530) 2024-03-18T15:04:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f69737fe640 1 -- 172.21.15.76:0/636239821 learned_addr learned my addr 172.21.15.76:0/636239821 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f69727fc640 1 --2- 172.21.15.76:0/636239821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a028915410 0x55a0289204f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f6972ffd640 1 --2- 172.21.15.76:0/636239821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a028914af0 0x55a028914ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f69727fc640 1 -- 172.21.15.76:0/636239821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a028914af0 msgr2=0x55a028914ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:04:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f69727fc640 1 --2- 172.21.15.76:0/636239821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a028914af0 0x55a028914ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f69727fc640 1 -- 172.21.15.76:0/636239821 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a028920a30 msgr2=0x55a028922e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:04:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f69727fc640 1 --2- 172.21.15.76:0/636239821 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a028920a30 0x55a028922e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f69737fe640 1 --2- 172.21.15.76:0/2955182410 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f695c0a2a10 0x7f695c0a4ed0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f696c0134b0 tx=0x7f696c0138b0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f69727fc640 1 -- 172.21.15.76:0/636239821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a02883a460 con 0x55a028915410 2024-03-18T15:04:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f69727fc640 1 --2- 172.21.15.76:0/636239821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a028915410 0x55a0289204f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6968009bf0 tx=0x7f6968011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9585ed80c1420f35 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.726+0000 7f6959ffb640 1 -- 172.21.15.76:0/636239821 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6968011e50 con 0x55a028915410 2024-03-18T15:04:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.727+0000 7f6959ffb640 1 -- 172.21.15.76:0/636239821 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6968005020 con 0x55a028915410 2024-03-18T15:04:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.727+0000 7f6959ffb640 1 -- 172.21.15.76:0/636239821 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6968004c40 con 0x55a028915410 2024-03-18T15:04:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.727+0000 7f697c45d740 1 -- 172.21.15.76:0/636239821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a028915410 msgr2=0x55a0289204f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.727+0000 7f697c45d740 1 --2- 172.21.15.76:0/636239821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a028915410 0x55a0289204f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6968009bf0 tx=0x7f6968011440 comp rx=0 tx=0).stop 2024-03-18T15:04:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.727+0000 7f697c45d740 1 -- 172.21.15.76:0/636239821 shutdown_connections 2024-03-18T15:04:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.727+0000 7f697c45d740 1 --2- 172.21.15.76:0/636239821 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a028920a30 0x55a028922e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:06.770 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.727+0000 7f697c45d740 1 --2- 172.21.15.76:0/636239821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a028915410 0x55a0289204f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:06.770 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.727+0000 7f697c45d740 1 --2- 172.21.15.76:0/636239821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a028914af0 0x55a028914ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:06.770 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.727+0000 7f697c45d740 1 -- 172.21.15.76:0/636239821 >> 172.21.15.76:0/636239821 conn(0x55a02890fa50 msgr2=0x55a02890fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:06.770 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.727+0000 7f697c45d740 1 -- 172.21.15.76:0/636239821 shutdown_connections 2024-03-18T15:04:06.770 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.727+0000 7f697c45d740 1 -- 172.21.15.76:0/636239821 wait complete. 2024-03-18T15:04:06.770 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f697c45d740 1 -- start start 2024-03-18T15:04:06.770 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f697c45d740 1 Processor -- start 2024-03-18T15:04:06.770 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f697c45d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a028914af0 0x55a02891da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:06.770 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f697c45d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a028915410 0x55a02891df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f697c45d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a028920a30 0x55a02891e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f697c45d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a028923660 con 0x55a028914af0 2024-03-18T15:04:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f697c45d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a028835d70 con 0x55a028920a30 2024-03-18T15:04:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f697c45d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a0289237e0 con 0x55a028915410 2024-03-18T15:04:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f697c45d740 10 client.0 _ll_register_callbacks cb 0x55a02858da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:04:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f6972ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a028914af0 0x55a02891da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f6972ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a028914af0 0x55a02891da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38540/0 (socket says 172.21.15.76:38540) 2024-03-18T15:04:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f6972ffd640 1 -- 172.21.15.76:0/824552219 learned_addr learned my addr 172.21.15.76:0/824552219 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f69727fc640 1 --2- 172.21.15.76:0/824552219 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a028915410 0x55a02891df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f69737fe640 1 --2- 172.21.15.76:0/824552219 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a028920a30 0x55a02891e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f6972ffd640 1 -- 172.21.15.76:0/824552219 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a028915410 msgr2=0x55a02891df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:04:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f6972ffd640 1 --2- 172.21.15.76:0/824552219 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a028915410 0x55a02891df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f6972ffd640 1 -- 172.21.15.76:0/824552219 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a028920a30 msgr2=0x55a02891e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:04:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f6972ffd640 1 --2- 172.21.15.76:0/824552219 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a028920a30 0x55a02891e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.728+0000 7f6972ffd640 1 -- 172.21.15.76:0/824552219 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a02858e210 con 0x55a028914af0 2024-03-18T15:04:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.729+0000 7f6972ffd640 1 --2- 172.21.15.76:0/824552219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a028914af0 0x55a02891da30 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f6964006010 tx=0x7f69640bcf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca27ef245eee2de7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.729+0000 7f6952ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-18T15:04:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.729+0000 7f6952ffd640 1 -- 172.21.15.76:0/824552219 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69640d6060 con 0x55a028914af0 2024-03-18T15:04:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.729+0000 7f697c45d740 1 -- 172.21.15.76:0/824552219 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6968005020 con 0x55a028914af0 2024-03-18T15:04:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.729+0000 7f6952ffd640 1 -- 172.21.15.76:0/824552219 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8440+0+0 (secure 0 0 0) 0x7f69640d6200 con 0x55a028914af0 2024-03-18T15:04:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.729+0000 7f6952ffd640 1 -- 172.21.15.76:0/824552219 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f69640bd440 con 0x55a028914af0 2024-03-18T15:04:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.729+0000 7f697c45d740 10 client.16734 fetch_fsmap learned FSMap version 44 2024-03-18T15:04:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.729+0000 7f697c45d740 1 -- 172.21.15.76:0/824552219 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55a02865a620 con 0x55a028914af0 2024-03-18T15:04:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.729+0000 7f6952ffd640 1 -- 172.21.15.76:0/824552219 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6968005020 con 0x55a028914af0 2024-03-18T15:04:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.729+0000 7f6952ffd640 1 -- 172.21.15.76:0/824552219 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 44) ==== 8338+0+0 (secure 0 0 0) 0x7f69640b9ab0 con 0x55a028914af0 2024-03-18T15:04:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.729+0000 7f697c45d740 10 client.16734 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T15:04:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.729+0000 7f697c45d740 10 client.16734 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T15:04:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.729+0000 7f697c45d740 20 client.16734 populate_metadata read hostname 'smithi076' 2024-03-18T15:04:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.729+0000 7f697c45d740 1 --2- 172.21.15.76:0/824552219 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a028930ba0 0x55a028932f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.729+0000 7f697c45d740 4 client.16734 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:04:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.729+0000 7f697c45d740 1 -- 172.21.15.76:0/824552219 --> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a028886f80 con 0x55a028930ba0 2024-03-18T15:04:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.729+0000 7f69727fc640 1 --2- 172.21.15.76:0/824552219 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a028930ba0 0x55a028932f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.730+0000 7f6952ffd640 10 client.16734 ms_handle_connect on v2:172.21.15.77:6838/1685041985 2024-03-18T15:04:06.775 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.730+0000 7f69727fc640 1 --2- 172.21.15.76:0/824552219 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a028930ba0 0x55a028932f90 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cbcb24dbdcb8ece server_cookie=a93a26eaacbbfa0f in_seq=0 out_seq=0 2024-03-18T15:04:06.775 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.731+0000 7f6952ffd640 1 -- 172.21.15.76:0/824552219 <== mds.0 v2:172.21.15.77:6838/1685041985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a028886f80 con 0x55a028930ba0 2024-03-18T15:04:06.775 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.731+0000 7f6952ffd640 10 client.16734 handle_command_reply: tid=0 2024-03-18T15:04:06.775 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.731+0000 7f697c45d740 1 client.16734 shutdown 2024-03-18T15:04:06.775 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.731+0000 7f697c45d740 1 -- 172.21.15.76:0/824552219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a028914af0 msgr2=0x55a02891da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:06.775 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.731+0000 7f697c45d740 1 --2- 172.21.15.76:0/824552219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a028914af0 0x55a02891da30 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f6964006010 tx=0x7f69640bcf80 comp rx=0 tx=0).stop 2024-03-18T15:04:06.775 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.731+0000 7f697c45d740 1 -- 172.21.15.76:0/824552219 shutdown_connections 2024-03-18T15:04:06.775 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.731+0000 7f697c45d740 1 --2- 172.21.15.76:0/824552219 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a028915410 0x55a02891df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:06.775 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.731+0000 7f697c45d740 1 --2- 172.21.15.76:0/824552219 >> [v2:172.21.15.77:6838/1685041985,v1:172.21.15.77:6839/1685041985] conn(0x55a028930ba0 0x55a028932f90 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:06.776 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.731+0000 7f697c45d740 1 --2- 172.21.15.76:0/824552219 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a028920a30 0x55a02891e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:06.776 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.731+0000 7f697c45d740 1 --2- 172.21.15.76:0/824552219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a028914af0 0x55a02891da30 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:06.776 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.731+0000 7f697c45d740 1 -- 172.21.15.76:0/824552219 >> 172.21.15.76:0/824552219 conn(0x55a02890fa50 msgr2=0x55a0289117a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:06.776 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.731+0000 7f6952ffd640 0 client.16734 ms_handle_reset on v2:172.21.15.77:6838/1685041985 2024-03-18T15:04:06.776 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.731+0000 7f697c45d740 1 -- 172.21.15.76:0/824552219 shutdown_connections 2024-03-18T15:04:06.776 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.731+0000 7f697c45d740 1 -- 172.21.15.76:0/824552219 wait complete. 2024-03-18T15:04:06.776 INFO:teuthology.task.background_exec.ubuntu@smithi076.front.sepia.ceph.com.smithi076.stderr:2024-03-18T15:04:06.731+0000 7f697c45d740 20 client.16734 trim_cache size 0 max 16384 2024-03-18T15:04:06.777 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-18T15:04:06.787 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-18T15:04:06.787 INFO:teuthology.task.exec:Executing custom commands... 2024-03-18T15:04:06.787 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi076.front.sepia.ceph.com 2024-03-18T15:04:06.788 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-18T15:04:06.831 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-18T15:04:06.900 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-18T15:04:07.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.043+0000 7fa4c14d1640 1 -- 172.21.15.76:0/2926355211 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa4bc101790 msgr2=0x7fa4bc101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:07.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.043+0000 7fa4c14d1640 1 --2- 172.21.15.76:0/2926355211 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa4bc101790 0x7fa4bc101c10 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0009930 tx=0x7fa4b002bc30 comp rx=0 tx=0).stop 2024-03-18T15:04:07.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.043+0000 7fa4c14d1640 1 -- 172.21.15.76:0/2926355211 shutdown_connections 2024-03-18T15:04:07.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.043+0000 7fa4c14d1640 1 --2- 172.21.15.76:0/2926355211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4bc102150 0x7fa4bc10e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.043+0000 7fa4c14d1640 1 --2- 172.21.15.76:0/2926355211 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa4bc101790 0x7fa4bc101c10 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.043+0000 7fa4c14d1640 1 --2- 172.21.15.76:0/2926355211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4bc1077a0 0x7fa4bc107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.043+0000 7fa4c14d1640 1 -- 172.21.15.76:0/2926355211 >> 172.21.15.76:0/2926355211 conn(0x7fa4bc0fd2a0 msgr2=0x7fa4bc0ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:07.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.043+0000 7fa4c14d1640 1 -- 172.21.15.76:0/2926355211 shutdown_connections 2024-03-18T15:04:07.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.043+0000 7fa4c14d1640 1 -- 172.21.15.76:0/2926355211 wait complete. 2024-03-18T15:04:07.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4c14d1640 1 Processor -- start 2024-03-18T15:04:07.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4c14d1640 1 -- start start 2024-03-18T15:04:07.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4c14d1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa4bc101790 0x7fa4bc198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:07.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4c14d1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4bc102150 0x7fa4bc1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:07.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4c14d1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4bc1077a0 0x7fa4bc19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:07.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4c14d1640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa4bc111bb0 con 0x7fa4bc101790 2024-03-18T15:04:07.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4c14d1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa4bc111a30 con 0x7fa4bc102150 2024-03-18T15:04:07.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4c14d1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa4bc111d30 con 0x7fa4bc1077a0 2024-03-18T15:04:07.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4baffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa4bc101790 0x7fa4bc198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:07.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4baffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa4bc101790 0x7fa4bc198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38556/0 (socket says 172.21.15.76:38556) 2024-03-18T15:04:07.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4baffd640 1 -- 172.21.15.76:0/1713592044 learned_addr learned my addr 172.21.15.76:0/1713592044 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:07.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4ba7fc640 1 --2- 172.21.15.76:0/1713592044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4bc102150 0x7fa4bc1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:07.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4bb7fe640 1 --2- 172.21.15.76:0/1713592044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4bc1077a0 0x7fa4bc19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:07.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4baffd640 1 -- 172.21.15.76:0/1713592044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4bc1077a0 msgr2=0x7fa4bc19c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:07.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4baffd640 1 --2- 172.21.15.76:0/1713592044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4bc1077a0 0x7fa4bc19c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4baffd640 1 -- 172.21.15.76:0/1713592044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4bc102150 msgr2=0x7fa4bc1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:07.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4baffd640 1 --2- 172.21.15.76:0/1713592044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4bc102150 0x7fa4bc1985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4baffd640 1 -- 172.21.15.76:0/1713592044 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4bc19d010 con 0x7fa4bc101790 2024-03-18T15:04:07.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.044+0000 7fa4ba7fc640 1 --2- 172.21.15.76:0/1713592044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4bc102150 0x7fa4bc1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:04:07.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.045+0000 7fa4baffd640 1 --2- 172.21.15.76:0/1713592044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa4bc101790 0x7fa4bc198070 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00e9f0 tx=0x7fa4ac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:07.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.045+0000 7fa4a3fff640 1 -- 172.21.15.76:0/1713592044 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac00cd80 con 0x7fa4bc101790 2024-03-18T15:04:07.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.045+0000 7fa4c14d1640 1 -- 172.21.15.76:0/1713592044 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4bc19d300 con 0x7fa4bc101790 2024-03-18T15:04:07.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.045+0000 7fa4a3fff640 1 -- 172.21.15.76:0/1713592044 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4ac005500 con 0x7fa4bc101790 2024-03-18T15:04:07.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.045+0000 7fa4c14d1640 1 -- 172.21.15.76:0/1713592044 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4bc1a4b90 con 0x7fa4bc101790 2024-03-18T15:04:07.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.045+0000 7fa4a3fff640 1 -- 172.21.15.76:0/1713592044 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac010690 con 0x7fa4bc101790 2024-03-18T15:04:07.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.046+0000 7fa4c14d1640 1 -- 172.21.15.76:0/1713592044 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4bc102ed0 con 0x7fa4bc101790 2024-03-18T15:04:07.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.047+0000 7fa4a3fff640 1 -- 172.21.15.76:0/1713592044 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fa4ac0046c0 con 0x7fa4bc101790 2024-03-18T15:04:07.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.047+0000 7fa4a3fff640 1 --2- 172.21.15.76:0/1713592044 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa4a40a2a10 0x7fa4a40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:07.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.048+0000 7fa4a3fff640 1 -- 172.21.15.76:0/1713592044 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8440+0+0 (secure 0 0 0) 0x7fa4ac0cf360 con 0x7fa4bc101790 2024-03-18T15:04:07.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.048+0000 7fa4ba7fc640 1 --2- 172.21.15.76:0/1713592044 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa4a40a2a10 0x7fa4a40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:07.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.048+0000 7fa4ba7fc640 1 --2- 172.21.15.76:0/1713592044 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa4a40a2a10 0x7fa4a40a4ed0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0005ab0 tx=0x7fa4b0005fd0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:07.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.051+0000 7fa4a3fff640 1 -- 172.21.15.76:0/1713592044 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa4ac053740 con 0x7fa4bc101790 2024-03-18T15:04:07.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:06 smithi076 ceph-mon[32205]: pgmap v343: 97 pgs: 97 active+clean; 325 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 183 op/s 2024-03-18T15:04:07.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:06 smithi174 ceph-mon[36787]: pgmap v343: 97 pgs: 97 active+clean; 325 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 183 op/s 2024-03-18T15:04:07.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:06 smithi077 ceph-mon[35865]: pgmap v343: 97 pgs: 97 active+clean; 325 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 183 op/s 2024-03-18T15:04:07.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.248+0000 7fa4c14d1640 1 -- 172.21.15.76:0/1713592044 --> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fa4bc109d70 con 0x7fa4a40a2a10 2024-03-18T15:04:07.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.257+0000 7fa4a3fff640 1 -- 172.21.15.76:0/1713592044 <== mgr.25298 v2:172.21.15.76:6800/3052007196 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7fa4b0004a70 con 0x7fa4a40a2a10 2024-03-18T15:04:07.257 INFO:teuthology.orchestra.run.smithi076.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T14:53:18", "first": "2024-03-18T14:54:00", "last": "2024-03-18T15:04:00", "last_pruned": "2024-03-18T15:04:00", "created_count": 11, "pruned_count": 5, "active": true} 2024-03-18T15:04:07.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.259+0000 7fa4c14d1640 1 -- 172.21.15.76:0/1713592044 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa4a40a2a10 msgr2=0x7fa4a40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:07.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.259+0000 7fa4c14d1640 1 --2- 172.21.15.76:0/1713592044 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa4a40a2a10 0x7fa4a40a4ed0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0005ab0 tx=0x7fa4b0005fd0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.259+0000 7fa4c14d1640 1 -- 172.21.15.76:0/1713592044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa4bc101790 msgr2=0x7fa4bc198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:07.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.259+0000 7fa4c14d1640 1 --2- 172.21.15.76:0/1713592044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa4bc101790 0x7fa4bc198070 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00e9f0 tx=0x7fa4ac00edf0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.259+0000 7fa4c14d1640 1 -- 172.21.15.76:0/1713592044 shutdown_connections 2024-03-18T15:04:07.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.259+0000 7fa4c14d1640 1 --2- 172.21.15.76:0/1713592044 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa4a40a2a10 0x7fa4a40a4ed0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.259+0000 7fa4c14d1640 1 --2- 172.21.15.76:0/1713592044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4bc1077a0 0x7fa4bc19c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.259+0000 7fa4c14d1640 1 --2- 172.21.15.76:0/1713592044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4bc102150 0x7fa4bc1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.259+0000 7fa4c14d1640 1 --2- 172.21.15.76:0/1713592044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa4bc101790 0x7fa4bc198070 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.259+0000 7fa4c14d1640 1 -- 172.21.15.76:0/1713592044 >> 172.21.15.76:0/1713592044 conn(0x7fa4bc0fd2a0 msgr2=0x7fa4bc0fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:07.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.259+0000 7fa4c14d1640 1 -- 172.21.15.76:0/1713592044 shutdown_connections 2024-03-18T15:04:07.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.259+0000 7fa4c14d1640 1 -- 172.21.15.76:0/1713592044 wait complete. 2024-03-18T15:04:07.270 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-18T15:04:07.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.373+0000 7f7daee36640 1 -- 172.21.15.76:0/1538364280 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7da8100930 msgr2=0x7f7da8100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.373+0000 7f7daee36640 1 --2- 172.21.15.76:0/1538364280 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7da8100930 0x7f7da8100d90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c009900 tx=0x7f7d9c02bc30 comp rx=0 tx=0).stop 2024-03-18T15:04:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.373+0000 7f7daee36640 1 -- 172.21.15.76:0/1538364280 shutdown_connections 2024-03-18T15:04:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.373+0000 7f7daee36640 1 --2- 172.21.15.76:0/1538364280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7da81012d0 0x7f7da810d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.373+0000 7f7daee36640 1 --2- 172.21.15.76:0/1538364280 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7da8100930 0x7f7da8100d90 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.373+0000 7f7daee36640 1 --2- 172.21.15.76:0/1538364280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da8106760 0x7f7da8106b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.373+0000 7f7daee36640 1 -- 172.21.15.76:0/1538364280 >> 172.21.15.76:0/1538364280 conn(0x7f7da8077540 msgr2=0x7f7da8077950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.373+0000 7f7daee36640 1 -- 172.21.15.76:0/1538364280 shutdown_connections 2024-03-18T15:04:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.373+0000 7f7daee36640 1 -- 172.21.15.76:0/1538364280 wait complete. 2024-03-18T15:04:07.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.373+0000 7f7daee36640 1 Processor -- start 2024-03-18T15:04:07.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.373+0000 7f7daee36640 1 -- start start 2024-03-18T15:04:07.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7daee36640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da8100930 0x7f7da8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:07.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7daee36640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7da81012d0 0x7f7da8198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:07.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7daee36640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7da8106760 0x7f7da819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:07.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7daee36640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7da8110ae0 con 0x7f7da81012d0 2024-03-18T15:04:07.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7daee36640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7da8110960 con 0x7f7da8106760 2024-03-18T15:04:07.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7daee36640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7da8110c60 con 0x7f7da8100930 2024-03-18T15:04:07.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7da7fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7da81012d0 0x7f7da8198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:07.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7da7fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7da81012d0 0x7f7da8198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38568/0 (socket says 172.21.15.76:38568) 2024-03-18T15:04:07.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7da7fff640 1 -- 172.21.15.76:0/1379290984 learned_addr learned my addr 172.21.15.76:0/1379290984 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:07.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7dacbd2640 1 --2- 172.21.15.76:0/1379290984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da8100930 0x7f7da8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:07.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7dad3d3640 1 --2- 172.21.15.76:0/1379290984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7da8106760 0x7f7da819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:07.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7da7fff640 1 -- 172.21.15.76:0/1379290984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da8100930 msgr2=0x7f7da8197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:07.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7da7fff640 1 --2- 172.21.15.76:0/1379290984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da8100930 0x7f7da8197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7da7fff640 1 -- 172.21.15.76:0/1379290984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7da8106760 msgr2=0x7f7da819c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:07.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7da7fff640 1 --2- 172.21.15.76:0/1379290984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7da8106760 0x7f7da819c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7da7fff640 1 -- 172.21.15.76:0/1379290984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da819cf70 con 0x7f7da81012d0 2024-03-18T15:04:07.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7da7fff640 1 --2- 172.21.15.76:0/1379290984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7da81012d0 0x7f7da8198510 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c006010 tx=0x7f7d9c02cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:07.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7da5ffb640 1 -- 172.21.15.76:0/1379290984 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c041070 con 0x7f7da81012d0 2024-03-18T15:04:07.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7da5ffb640 1 -- 172.21.15.76:0/1379290984 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d9c004450 con 0x7f7da81012d0 2024-03-18T15:04:07.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7da5ffb640 1 -- 172.21.15.76:0/1379290984 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c003e00 con 0x7f7da81012d0 2024-03-18T15:04:07.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.374+0000 7f7daee36640 1 -- 172.21.15.76:0/1379290984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7da819d200 con 0x7f7da81012d0 2024-03-18T15:04:07.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.375+0000 7f7daee36640 1 -- 172.21.15.76:0/1379290984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7da81a4af0 con 0x7f7da81012d0 2024-03-18T15:04:07.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.376+0000 7f7daee36640 1 -- 172.21.15.76:0/1379290984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7da8101e40 con 0x7f7da81012d0 2024-03-18T15:04:07.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.377+0000 7f7da5ffb640 1 -- 172.21.15.76:0/1379290984 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f7d9c044070 con 0x7f7da81012d0 2024-03-18T15:04:07.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.377+0000 7f7da5ffb640 1 --2- 172.21.15.76:0/1379290984 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7d900a2a10 0x7f7d900a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:07.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.377+0000 7f7dacbd2640 1 --2- 172.21.15.76:0/1379290984 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7d900a2a10 0x7f7d900a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:07.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.377+0000 7f7da5ffb640 1 -- 172.21.15.76:0/1379290984 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8440+0+0 (secure 0 0 0) 0x7f7d9c0f09c0 con 0x7f7da81012d0 2024-03-18T15:04:07.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.378+0000 7f7dacbd2640 1 --2- 172.21.15.76:0/1379290984 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7d900a2a10 0x7f7d900a4ed0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7da0005fd0 tx=0x7f7da0009680 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:07.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.381+0000 7f7da5ffb640 1 -- 172.21.15.76:0/1379290984 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7d9c0bd230 con 0x7f7da81012d0 2024-03-18T15:04:07.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.578+0000 7f7daee36640 1 -- 172.21.15.76:0/1379290984 --> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f7da81044e0 con 0x7f7d900a2a10 2024-03-18T15:04:07.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.591+0000 7f7da5ffb640 1 -- 172.21.15.76:0/1379290984 <== mgr.25298 v2:172.21.15.76:6800/3052007196 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f7da81044e0 con 0x7f7d900a2a10 2024-03-18T15:04:07.592 INFO:teuthology.orchestra.run.smithi076.stdout:/ 1m 6m3h 2024-03-18T15:04:07.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.593+0000 7f7daee36640 1 -- 172.21.15.76:0/1379290984 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7d900a2a10 msgr2=0x7f7d900a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:07.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.593+0000 7f7daee36640 1 --2- 172.21.15.76:0/1379290984 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7d900a2a10 0x7f7d900a4ed0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7da0005fd0 tx=0x7f7da0009680 comp rx=0 tx=0).stop 2024-03-18T15:04:07.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.593+0000 7f7daee36640 1 -- 172.21.15.76:0/1379290984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7da81012d0 msgr2=0x7f7da8198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:07.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.593+0000 7f7daee36640 1 --2- 172.21.15.76:0/1379290984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7da81012d0 0x7f7da8198510 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c006010 tx=0x7f7d9c02cb30 comp rx=0 tx=0).stop 2024-03-18T15:04:07.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.593+0000 7f7daee36640 1 -- 172.21.15.76:0/1379290984 shutdown_connections 2024-03-18T15:04:07.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.593+0000 7f7daee36640 1 --2- 172.21.15.76:0/1379290984 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f7d900a2a10 0x7f7d900a4ed0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.593+0000 7f7daee36640 1 --2- 172.21.15.76:0/1379290984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7da8106760 0x7f7da819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.593+0000 7f7daee36640 1 --2- 172.21.15.76:0/1379290984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7da81012d0 0x7f7da8198510 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.593+0000 7f7daee36640 1 --2- 172.21.15.76:0/1379290984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da8100930 0x7f7da8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.593+0000 7f7daee36640 1 -- 172.21.15.76:0/1379290984 >> 172.21.15.76:0/1379290984 conn(0x7f7da8077540 msgr2=0x7f7da80ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:07.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.594+0000 7f7daee36640 1 -- 172.21.15.76:0/1379290984 shutdown_connections 2024-03-18T15:04:07.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.594+0000 7f7daee36640 1 -- 172.21.15.76:0/1379290984 wait complete. 2024-03-18T15:04:07.605 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-18T15:04:07.615 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-18T15:04:07.616 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-03-18T15:04:07.620 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-18T15:04:07.630 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:04:07.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.735+0000 7f8423612640 1 -- 172.21.15.76:0/1655041292 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f841c069e90 msgr2=0x7f841c06a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:07.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.735+0000 7f8423612640 1 --2- 172.21.15.76:0/1655041292 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f841c069e90 0x7f841c06a2f0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f840c009900 tx=0x7f840c02bcd0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.735+0000 7f8423612640 1 -- 172.21.15.76:0/1655041292 shutdown_connections 2024-03-18T15:04:07.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.735+0000 7f8423612640 1 --2- 172.21.15.76:0/1655041292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f841c06a830 0x7f841c10e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.735+0000 7f8423612640 1 --2- 172.21.15.76:0/1655041292 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f841c069e90 0x7f841c06a2f0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.735+0000 7f8423612640 1 --2- 172.21.15.76:0/1655041292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f841c1077e0 0x7f841c107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.735+0000 7f8423612640 1 -- 172.21.15.76:0/1655041292 >> 172.21.15.76:0/1655041292 conn(0x7f841c077780 msgr2=0x7f841c077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:07.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.735+0000 7f8423612640 1 -- 172.21.15.76:0/1655041292 shutdown_connections 2024-03-18T15:04:07.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.735+0000 7f8423612640 1 -- 172.21.15.76:0/1655041292 wait complete. 2024-03-18T15:04:07.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8423612640 1 Processor -- start 2024-03-18T15:04:07.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8423612640 1 -- start start 2024-03-18T15:04:07.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8423612640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f841c069e90 0x7f841c100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:07.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8423612640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f841c06a830 0x7f841c100c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:07.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8423612640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f841c1077e0 0x7f841c101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:07.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8423612640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f841c111ba0 con 0x7f841c1077e0 2024-03-18T15:04:07.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8423612640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f841c111a20 con 0x7f841c06a830 2024-03-18T15:04:07.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8423612640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f841c111d20 con 0x7f841c069e90 2024-03-18T15:04:07.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8421baf640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f841c1077e0 0x7f841c101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:07.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f84213ae640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f841c069e90 0x7f841c100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:07.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8421baf640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f841c1077e0 0x7f841c101190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38598/0 (socket says 172.21.15.76:38598) 2024-03-18T15:04:07.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8421baf640 1 -- 172.21.15.76:0/2526586299 learned_addr learned my addr 172.21.15.76:0/2526586299 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:07.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8420bad640 1 --2- 172.21.15.76:0/2526586299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f841c06a830 0x7f841c100c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:07.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8421baf640 1 -- 172.21.15.76:0/2526586299 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f841c069e90 msgr2=0x7f841c100710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:07.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8421baf640 1 --2- 172.21.15.76:0/2526586299 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f841c069e90 0x7f841c100710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8421baf640 1 -- 172.21.15.76:0/2526586299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f841c06a830 msgr2=0x7f841c100c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:07.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8421baf640 1 --2- 172.21.15.76:0/2526586299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f841c06a830 0x7f841c100c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:07.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8421baf640 1 -- 172.21.15.76:0/2526586299 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f841c1a6540 con 0x7f841c1077e0 2024-03-18T15:04:07.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f8420bad640 1 --2- 172.21.15.76:0/2526586299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f841c06a830 0x7f841c100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:04:07.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.736+0000 7f84213ae640 1 --2- 172.21.15.76:0/2526586299 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f841c069e90 0x7f841c100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:04:07.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.737+0000 7f8421baf640 1 --2- 172.21.15.76:0/2526586299 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f841c1077e0 0x7f841c101190 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f841400eb10 tx=0x7f841400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:07.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.737+0000 7f84127fc640 1 -- 172.21.15.76:0/2526586299 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8414004b20 con 0x7f841c1077e0 2024-03-18T15:04:07.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.737+0000 7f84127fc640 1 -- 172.21.15.76:0/2526586299 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8414004cc0 con 0x7f841c1077e0 2024-03-18T15:04:07.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.737+0000 7f8423612640 1 -- 172.21.15.76:0/2526586299 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f841c1a6830 con 0x7f841c1077e0 2024-03-18T15:04:07.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.737+0000 7f84127fc640 1 -- 172.21.15.76:0/2526586299 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8414010760 con 0x7f841c1077e0 2024-03-18T15:04:07.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.737+0000 7f8423612640 1 -- 172.21.15.76:0/2526586299 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f841c1a6d50 con 0x7f841c1077e0 2024-03-18T15:04:07.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.738+0000 7f8423612640 1 -- 172.21.15.76:0/2526586299 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f841c102f00 con 0x7f841c1077e0 2024-03-18T15:04:07.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.739+0000 7f84127fc640 1 -- 172.21.15.76:0/2526586299 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f8414005020 con 0x7f841c1077e0 2024-03-18T15:04:07.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.739+0000 7f84127fc640 1 --2- 172.21.15.76:0/2526586299 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f84080a2970 0x7f84080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:07.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.739+0000 7f84213ae640 1 --2- 172.21.15.76:0/2526586299 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f84080a2970 0x7f84080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:07.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.739+0000 7f84127fc640 1 -- 172.21.15.76:0/2526586299 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8440+0+0 (secure 0 0 0) 0x7f84140cf500 con 0x7f841c1077e0 2024-03-18T15:04:07.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.740+0000 7f84213ae640 1 --2- 172.21.15.76:0/2526586299 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f84080a2970 0x7f84080a4e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8418009f40 tx=0x7f8418005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:07.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:07.744+0000 7f84127fc640 1 -- 172.21.15.76:0/2526586299 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f841409c2b0 con 0x7f841c1077e0 2024-03-18T15:04:07.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:07 smithi076 ceph-mon[32205]: from='client.16740 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:04:07.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:07 smithi076 ceph-mon[32205]: from='client.16746 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:04:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.086+0000 7f8423612640 1 -- 172.21.15.76:0/2526586299 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f841c109cb0 con 0x7f841c1077e0 2024-03-18T15:04:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.088+0000 7f84127fc640 1 -- 172.21.15.76:0/2526586299 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+23116 (secure 0 0 0) 0x7f841409bbb0 con 0x7f841c1077e0 2024-03-18T15:04:08.089 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T15:04:08.089 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":106,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T15:04:07.755812+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":106,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3010549595":"2024-03-19T14:53:17.537054+0000","172.21.15.76:6800/3010549595":"2024-03-19T14:53:17.537054+0000","172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/880938501":"2024-03-19T14:53:17.537054+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/2082819586":"2024-03-19T14:53:17.537054+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/3349561522":"2024-03-19T14:53:17.537054+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/788250061":"2024-03-19T14:53:17.537054+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:04:08.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.091+0000 7f8423612640 1 -- 172.21.15.76:0/2526586299 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f84080a2970 msgr2=0x7f84080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:08.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.091+0000 7f8423612640 1 --2- 172.21.15.76:0/2526586299 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f84080a2970 0x7f84080a4e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8418009f40 tx=0x7f8418005f90 comp rx=0 tx=0).stop 2024-03-18T15:04:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.092+0000 7f8423612640 1 -- 172.21.15.76:0/2526586299 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f841c1077e0 msgr2=0x7f841c101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.092+0000 7f8423612640 1 --2- 172.21.15.76:0/2526586299 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f841c1077e0 0x7f841c101190 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f841400eb10 tx=0x7f841400c2d0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.092+0000 7f8423612640 1 -- 172.21.15.76:0/2526586299 shutdown_connections 2024-03-18T15:04:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.092+0000 7f8423612640 1 --2- 172.21.15.76:0/2526586299 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f84080a2970 0x7f84080a4e30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.092+0000 7f8423612640 1 --2- 172.21.15.76:0/2526586299 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f841c1077e0 0x7f841c101190 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.092+0000 7f8423612640 1 --2- 172.21.15.76:0/2526586299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f841c06a830 0x7f841c100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.092+0000 7f8423612640 1 --2- 172.21.15.76:0/2526586299 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f841c069e90 0x7f841c100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.092+0000 7f8423612640 1 -- 172.21.15.76:0/2526586299 >> 172.21.15.76:0/2526586299 conn(0x7f841c077780 msgr2=0x7f841c0ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:08.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.092+0000 7f8423612640 1 -- 172.21.15.76:0/2526586299 shutdown_connections 2024-03-18T15:04:08.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.092+0000 7f8423612640 1 -- 172.21.15.76:0/2526586299 wait complete. 2024-03-18T15:04:08.111 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:46:26.588384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:50:59.606852+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:51:00.465325+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 106, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T15:04:08.112 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:04:08.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:07 smithi174 ceph-mon[36787]: from='client.16740 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:04:08.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:07 smithi174 ceph-mon[36787]: from='client.16746 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:04:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:07 smithi077 ceph-mon[35865]: from='client.16740 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:04:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:07 smithi077 ceph-mon[35865]: from='client.16746 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:04:08.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.284+0000 7ff9ad92f640 1 -- 172.21.15.76:0/3867480148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a8100900 msgr2=0x7ff9a8106430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:08.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.284+0000 7ff9ad92f640 1 --2- 172.21.15.76:0/3867480148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a8100900 0x7ff9a8106430 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff990009900 tx=0x7ff99002bc30 comp rx=0 tx=0).stop 2024-03-18T15:04:08.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.284+0000 7ff9ad92f640 1 -- 172.21.15.76:0/3867480148 shutdown_connections 2024-03-18T15:04:08.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.285+0000 7ff9ad92f640 1 --2- 172.21.15.76:0/3867480148 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9a8106b60 0x7ff9a810e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.285+0000 7ff9ad92f640 1 --2- 172.21.15.76:0/3867480148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a8100900 0x7ff9a8106430 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.285+0000 7ff9ad92f640 1 --2- 172.21.15.76:0/3867480148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9a80fff50 0x7ff9a8100330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.285+0000 7ff9ad92f640 1 -- 172.21.15.76:0/3867480148 >> 172.21.15.76:0/3867480148 conn(0x7ff9a8076ca0 msgr2=0x7ff9a80fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:08.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.285+0000 7ff9ad92f640 1 -- 172.21.15.76:0/3867480148 shutdown_connections 2024-03-18T15:04:08.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.285+0000 7ff9ad92f640 1 -- 172.21.15.76:0/3867480148 wait complete. 2024-03-18T15:04:08.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.286+0000 7ff9ad92f640 1 Processor -- start 2024-03-18T15:04:08.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.287+0000 7ff9ad92f640 1 -- start start 2024-03-18T15:04:08.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.287+0000 7ff9ad92f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9a80fff50 0x7ff9a810cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:08.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.287+0000 7ff9ad92f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a8106b60 0x7ff9a8073260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:08.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.287+0000 7ff9ad92f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9a810d050 0x7ff9a810d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:08.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.287+0000 7ff9ad92f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a810fa40 con 0x7ff9a810d050 2024-03-18T15:04:08.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.287+0000 7ff9ad92f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a810f8c0 con 0x7ff9a8106b60 2024-03-18T15:04:08.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.287+0000 7ff9ad92f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a810fbc0 con 0x7ff9a80fff50 2024-03-18T15:04:08.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.288+0000 7ff9a77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9a810d050 0x7ff9a810d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:08.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.288+0000 7ff9a77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9a810d050 0x7ff9a810d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38618/0 (socket says 172.21.15.76:38618) 2024-03-18T15:04:08.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.288+0000 7ff9a77fe640 1 -- 172.21.15.76:0/2770282911 learned_addr learned my addr 172.21.15.76:0/2770282911 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:08.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.288+0000 7ff9a6ffd640 1 --2- 172.21.15.76:0/2770282911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9a80fff50 0x7ff9a810cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:08.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.288+0000 7ff99e7fc640 1 --2- 172.21.15.76:0/2770282911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a8106b60 0x7ff9a8073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:08.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.288+0000 7ff9a77fe640 1 -- 172.21.15.76:0/2770282911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9a80fff50 msgr2=0x7ff9a810cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:08.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.288+0000 7ff9a77fe640 1 --2- 172.21.15.76:0/2770282911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9a80fff50 0x7ff9a810cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.288+0000 7ff9a77fe640 1 -- 172.21.15.76:0/2770282911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a8106b60 msgr2=0x7ff9a8073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:08.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.288+0000 7ff9a77fe640 1 --2- 172.21.15.76:0/2770282911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a8106b60 0x7ff9a8073260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.288+0000 7ff9a77fe640 1 -- 172.21.15.76:0/2770282911 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a81131b0 con 0x7ff9a810d050 2024-03-18T15:04:08.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.289+0000 7ff9a77fe640 1 --2- 172.21.15.76:0/2770282911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9a810d050 0x7ff9a810d4d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ff9a000eb10 tx=0x7ff9a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:08.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.289+0000 7ff9ac92d640 1 -- 172.21.15.76:0/2770282911 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a0004b20 con 0x7ff9a810d050 2024-03-18T15:04:08.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.289+0000 7ff9ad92f640 1 -- 172.21.15.76:0/2770282911 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a81134a0 con 0x7ff9a810d050 2024-03-18T15:04:08.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.289+0000 7ff9ad92f640 1 -- 172.21.15.76:0/2770282911 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a81139c0 con 0x7ff9a810d050 2024-03-18T15:04:08.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.289+0000 7ff9ac92d640 1 -- 172.21.15.76:0/2770282911 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9a0004cc0 con 0x7ff9a810d050 2024-03-18T15:04:08.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.289+0000 7ff9ac92d640 1 -- 172.21.15.76:0/2770282911 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a0010760 con 0x7ff9a810d050 2024-03-18T15:04:08.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.291+0000 7ff9ad92f640 1 -- 172.21.15.76:0/2770282911 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff988005290 con 0x7ff9a810d050 2024-03-18T15:04:08.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.292+0000 7ff9ac92d640 1 -- 172.21.15.76:0/2770282911 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7ff9a0005020 con 0x7ff9a810d050 2024-03-18T15:04:08.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.293+0000 7ff9ac92d640 1 --2- 172.21.15.76:0/2770282911 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff9940a2a30 0x7ff9940a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:08.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.293+0000 7ff9ac92d640 1 -- 172.21.15.76:0/2770282911 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8528+0+0 (secure 0 0 0) 0x7ff9a00cf2d0 con 0x7ff9a810d050 2024-03-18T15:04:08.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.295+0000 7ff9a6ffd640 1 --2- 172.21.15.76:0/2770282911 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff9940a2a30 0x7ff9940a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:08.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.295+0000 7ff9a6ffd640 1 --2- 172.21.15.76:0/2770282911 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff9940a2a30 0x7ff9940a4ef0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff998005fd0 tx=0x7ff99800b390 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:08.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.301+0000 7ff9ac92d640 1 -- 172.21.15.76:0/2770282911 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff9a009bff0 con 0x7ff9a810d050 2024-03-18T15:04:08.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.631+0000 7ff9ad92f640 1 -- 172.21.15.76:0/2770282911 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff988005850 con 0x7ff9a810d050 2024-03-18T15:04:08.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.632+0000 7ff9ac92d640 1 -- 172.21.15.76:0/2770282911 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) ==== 93+0+10 (secure 0 0 0) 0x7ff9a009b8f0 con 0x7ff9a810d050 2024-03-18T15:04:08.633 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-18T15:04:08.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.635+0000 7ff9ad92f640 1 -- 172.21.15.76:0/2770282911 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff9940a2a30 msgr2=0x7ff9940a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:08.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.635+0000 7ff9ad92f640 1 --2- 172.21.15.76:0/2770282911 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff9940a2a30 0x7ff9940a4ef0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff998005fd0 tx=0x7ff99800b390 comp rx=0 tx=0).stop 2024-03-18T15:04:08.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.635+0000 7ff9ad92f640 1 -- 172.21.15.76:0/2770282911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9a810d050 msgr2=0x7ff9a810d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:08.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.635+0000 7ff9ad92f640 1 --2- 172.21.15.76:0/2770282911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9a810d050 0x7ff9a810d4d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ff9a000eb10 tx=0x7ff9a000c2d0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.636+0000 7ff9ad92f640 1 -- 172.21.15.76:0/2770282911 shutdown_connections 2024-03-18T15:04:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.636+0000 7ff9ad92f640 1 --2- 172.21.15.76:0/2770282911 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff9940a2a30 0x7ff9940a4ef0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.636+0000 7ff9ad92f640 1 --2- 172.21.15.76:0/2770282911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9a810d050 0x7ff9a810d4d0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.636+0000 7ff9ad92f640 1 --2- 172.21.15.76:0/2770282911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a8106b60 0x7ff9a8073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.636+0000 7ff9ad92f640 1 --2- 172.21.15.76:0/2770282911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9a80fff50 0x7ff9a810cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.636+0000 7ff9ad92f640 1 -- 172.21.15.76:0/2770282911 >> 172.21.15.76:0/2770282911 conn(0x7ff9a8076ca0 msgr2=0x7ff9a8105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.636+0000 7ff9ad92f640 1 -- 172.21.15.76:0/2770282911 shutdown_connections 2024-03-18T15:04:08.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.637+0000 7ff9ad92f640 1 -- 172.21.15.76:0/2770282911 wait complete. 2024-03-18T15:04:08.656 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T15:04:08.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.803+0000 7f48ae485640 1 -- 172.21.15.76:0/1019497978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a80689e0 msgr2=0x7f48a8068e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:08.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.803+0000 7f48ae485640 1 --2- 172.21.15.76:0/1019497978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a80689e0 0x7f48a8068e60 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f489c009930 tx=0x7f489c02bc30 comp rx=0 tx=0).stop 2024-03-18T15:04:08.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.803+0000 7f48ae485640 1 -- 172.21.15.76:0/1019497978 shutdown_connections 2024-03-18T15:04:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.803+0000 7f48ae485640 1 --2- 172.21.15.76:0/1019497978 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f48a80693a0 0x7f48a810e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.803+0000 7f48ae485640 1 --2- 172.21.15.76:0/1019497978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a80689e0 0x7f48a8068e60 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.803+0000 7f48ae485640 1 --2- 172.21.15.76:0/1019497978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48a8107830 0x7f48a8107c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.803+0000 7f48ae485640 1 -- 172.21.15.76:0/1019497978 >> 172.21.15.76:0/1019497978 conn(0x7f48a80777e0 msgr2=0x7f48a8077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.803+0000 7f48ae485640 1 -- 172.21.15.76:0/1019497978 shutdown_connections 2024-03-18T15:04:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.803+0000 7f48ae485640 1 -- 172.21.15.76:0/1019497978 wait complete. 2024-03-18T15:04:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.804+0000 7f48ae485640 1 Processor -- start 2024-03-18T15:04:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.804+0000 7f48ae485640 1 -- start start 2024-03-18T15:04:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.804+0000 7f48ae485640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f48a80689e0 0x7f48a819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.804+0000 7f48ae485640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48a80693a0 0x7f48a819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.804+0000 7f48ae485640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a8107830 0x7f48a819eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.804+0000 7f48ae485640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f48a8111b40 con 0x7f48a8107830 2024-03-18T15:04:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.804+0000 7f48ae485640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f48a81119c0 con 0x7f48a80693a0 2024-03-18T15:04:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.804+0000 7f48ae485640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f48a8111cc0 con 0x7f48a80689e0 2024-03-18T15:04:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.804+0000 7f48aca22640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a8107830 0x7f48a819eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.804+0000 7f48a7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f48a80689e0 0x7f48a819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.804+0000 7f48a77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48a80693a0 0x7f48a819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.804+0000 7f48a7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f48a80689e0 0x7f48a819a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:38388/0 (socket says 172.21.15.76:38388) 2024-03-18T15:04:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.804+0000 7f48a77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48a80693a0 0x7f48a819a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:52614/0 (socket says 172.21.15.76:52614) 2024-03-18T15:04:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.804+0000 7f48a77fe640 1 -- 172.21.15.76:0/773906698 learned_addr learned my addr 172.21.15.76:0/773906698 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.805+0000 7f48a77fe640 1 -- 172.21.15.76:0/773906698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f48a80689e0 msgr2=0x7f48a819a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.805+0000 7f48a77fe640 1 --2- 172.21.15.76:0/773906698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f48a80689e0 0x7f48a819a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.805+0000 7f48a77fe640 1 -- 172.21.15.76:0/773906698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a8107830 msgr2=0x7f48a819eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.805+0000 7f48a77fe640 1 --2- 172.21.15.76:0/773906698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a8107830 0x7f48a819eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.805+0000 7f48a77fe640 1 -- 172.21.15.76:0/773906698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a819f230 con 0x7f48a80693a0 2024-03-18T15:04:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.805+0000 7f48aca22640 1 --2- 172.21.15.76:0/773906698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a8107830 0x7f48a819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:04:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.805+0000 7f48a77fe640 1 --2- 172.21.15.76:0/773906698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48a80693a0 0x7f48a819a780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f489c009230 tx=0x7f489c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.805+0000 7f48a57fa640 1 -- 172.21.15.76:0/773906698 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c041070 con 0x7f48a80693a0 2024-03-18T15:04:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.805+0000 7f48ae485640 1 -- 172.21.15.76:0/773906698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48a819f4c0 con 0x7f48a80693a0 2024-03-18T15:04:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.805+0000 7f48a57fa640 1 -- 172.21.15.76:0/773906698 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f489c0054e0 con 0x7f48a80693a0 2024-03-18T15:04:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.805+0000 7f48ae485640 1 -- 172.21.15.76:0/773906698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48a81a6db0 con 0x7f48a80693a0 2024-03-18T15:04:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.805+0000 7f48a57fa640 1 -- 172.21.15.76:0/773906698 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c003c90 con 0x7f48a80693a0 2024-03-18T15:04:08.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.806+0000 7f48ae485640 1 -- 172.21.15.76:0/773906698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48a819b590 con 0x7f48a80693a0 2024-03-18T15:04:08.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.808+0000 7f48a57fa640 1 -- 172.21.15.76:0/773906698 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f489c044070 con 0x7f48a80693a0 2024-03-18T15:04:08.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.808+0000 7f48a57fa640 1 --2- 172.21.15.76:0/773906698 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f48900a2970 0x7f48900a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:08.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.809+0000 7f48a7fff640 1 --2- 172.21.15.76:0/773906698 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f48900a2970 0x7f48900a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:08.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.809+0000 7f48a57fa640 1 -- 172.21.15.76:0/773906698 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8528+0+0 (secure 0 0 0) 0x7f489c0f0ca0 con 0x7f48a80693a0 2024-03-18T15:04:08.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.809+0000 7f48a7fff640 1 --2- 172.21.15.76:0/773906698 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f48900a2970 0x7f48900a4e30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4898009bd0 tx=0x7f4898005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:08.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.816+0000 7f48a57fa640 1 -- 172.21.15.76:0/773906698 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f489c0bd4c0 con 0x7f48a80693a0 2024-03-18T15:04:08.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:08 smithi076 ceph-mon[32205]: pgmap v344: 97 pgs: 97 active+clean; 325 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.2 MiB/s wr, 167 op/s 2024-03-18T15:04:08.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:08 smithi076 ceph-mon[32205]: osdmap e106: 12 total, 12 up, 12 in 2024-03-18T15:04:08.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:08 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2526586299' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:04:08.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:08 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/2770282911' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:04:08.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.992+0000 7f48ae485640 1 -- 172.21.15.76:0/773906698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f48a8102eb0 con 0x7f48a80693a0 2024-03-18T15:04:08.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.992+0000 7f48a57fa640 1 -- 172.21.15.76:0/773906698 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) ==== 104+0+11 (secure 0 0 0) 0x7f489c0c1560 con 0x7f48a80693a0 2024-03-18T15:04:08.993 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-18T15:04:08.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.994+0000 7f48ae485640 1 -- 172.21.15.76:0/773906698 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f48900a2970 msgr2=0x7f48900a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:08.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.994+0000 7f48ae485640 1 --2- 172.21.15.76:0/773906698 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f48900a2970 0x7f48900a4e30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4898009bd0 tx=0x7f4898005f90 comp rx=0 tx=0).stop 2024-03-18T15:04:08.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.994+0000 7f48ae485640 1 -- 172.21.15.76:0/773906698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48a80693a0 msgr2=0x7f48a819a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:08.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.994+0000 7f48ae485640 1 --2- 172.21.15.76:0/773906698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48a80693a0 0x7f48a819a780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f489c009230 tx=0x7f489c004270 comp rx=0 tx=0).stop 2024-03-18T15:04:08.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.995+0000 7f48ae485640 1 -- 172.21.15.76:0/773906698 shutdown_connections 2024-03-18T15:04:08.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.995+0000 7f48ae485640 1 --2- 172.21.15.76:0/773906698 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f48900a2970 0x7f48900a4e30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.995+0000 7f48ae485640 1 --2- 172.21.15.76:0/773906698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a8107830 0x7f48a819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.995+0000 7f48ae485640 1 --2- 172.21.15.76:0/773906698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48a80693a0 0x7f48a819a780 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.995+0000 7f48ae485640 1 --2- 172.21.15.76:0/773906698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f48a80689e0 0x7f48a819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:08.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.995+0000 7f48ae485640 1 -- 172.21.15.76:0/773906698 >> 172.21.15.76:0/773906698 conn(0x7f48a80777e0 msgr2=0x7f48a810e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:08.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.995+0000 7f48ae485640 1 -- 172.21.15.76:0/773906698 shutdown_connections 2024-03-18T15:04:08.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:08.995+0000 7f48ae485640 1 -- 172.21.15.76:0/773906698 wait complete. 2024-03-18T15:04:09.007 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T15:04:09.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.135+0000 7f50a0722640 1 -- 172.21.15.76:0/1895277027 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f509810aaa0 msgr2=0x7f50981131f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:09.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.135+0000 7f50a0722640 1 --2- 172.21.15.76:0/1895277027 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f509810aaa0 0x7f50981131f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f5088009900 tx=0x7f508802bcd0 comp rx=0 tx=0).stop 2024-03-18T15:04:09.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.135+0000 7f50a0722640 1 -- 172.21.15.76:0/1895277027 shutdown_connections 2024-03-18T15:04:09.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.135+0000 7f50a0722640 1 --2- 172.21.15.76:0/1895277027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5098113730 0x7f5098115b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:09.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.135+0000 7f50a0722640 1 --2- 172.21.15.76:0/1895277027 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f509810aaa0 0x7f50981131f0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:09.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.135+0000 7f50a0722640 1 --2- 172.21.15.76:0/1895277027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f509810a180 0x7f509810a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:09.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.135+0000 7f50a0722640 1 -- 172.21.15.76:0/1895277027 >> 172.21.15.76:0/1895277027 conn(0x7f509806cd20 msgr2=0x7f509806d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:09.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.135+0000 7f50a0722640 1 -- 172.21.15.76:0/1895277027 shutdown_connections 2024-03-18T15:04:09.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.135+0000 7f50a0722640 1 -- 172.21.15.76:0/1895277027 wait complete. 2024-03-18T15:04:09.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.136+0000 7f50a0722640 1 Processor -- start 2024-03-18T15:04:09.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.136+0000 7f50a0722640 1 -- start start 2024-03-18T15:04:09.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.136+0000 7f50a0722640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f509810a180 0x7f50981a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:09.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.136+0000 7f50a0722640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f509810aaa0 0x7f50981a6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:09.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.136+0000 7f50a0722640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5098113730 0x7f50981a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:09.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.136+0000 7f50a0722640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5098119200 con 0x7f509810a180 2024-03-18T15:04:09.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.136+0000 7f50a0722640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5098119080 con 0x7f5098113730 2024-03-18T15:04:09.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.136+0000 7f50a0722640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5098119380 con 0x7f509810aaa0 2024-03-18T15:04:09.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.136+0000 7f509e4be640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f509810a180 0x7f50981a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:09.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.136+0000 7f509ecbf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5098113730 0x7f50981a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:09.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.136+0000 7f509e4be640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f509810a180 0x7f50981a6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35242/0 (socket says 172.21.15.76:35242) 2024-03-18T15:04:09.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.136+0000 7f509e4be640 1 -- 172.21.15.76:0/218072147 learned_addr learned my addr 172.21.15.76:0/218072147 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:09.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.136+0000 7f509dcbd640 1 --2- 172.21.15.76:0/218072147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f509810aaa0 0x7f50981a6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:09.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.137+0000 7f509e4be640 1 -- 172.21.15.76:0/218072147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f509810aaa0 msgr2=0x7f50981a6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:09.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.137+0000 7f509e4be640 1 --2- 172.21.15.76:0/218072147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f509810aaa0 0x7f50981a6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:09.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.137+0000 7f509e4be640 1 -- 172.21.15.76:0/218072147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5098113730 msgr2=0x7f50981a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:09.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.137+0000 7f509e4be640 1 --2- 172.21.15.76:0/218072147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5098113730 0x7f50981a0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:09.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.137+0000 7f509e4be640 1 -- 172.21.15.76:0/218072147 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50981a0fe0 con 0x7f509810a180 2024-03-18T15:04:09.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.137+0000 7f509ecbf640 1 --2- 172.21.15.76:0/218072147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5098113730 0x7f50981a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:04:09.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.137+0000 7f509e4be640 1 --2- 172.21.15.76:0/218072147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f509810a180 0x7f50981a6610 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f509400ea60 tx=0x7f509400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:09.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.137+0000 7f508f7fe640 1 -- 172.21.15.76:0/218072147 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509400cde0 con 0x7f509810a180 2024-03-18T15:04:09.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.137+0000 7f50a0722640 1 -- 172.21.15.76:0/218072147 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50981a1270 con 0x7f509810a180 2024-03-18T15:04:09.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.137+0000 7f508f7fe640 1 -- 172.21.15.76:0/218072147 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50940035f0 con 0x7f509810a180 2024-03-18T15:04:09.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.137+0000 7f508f7fe640 1 -- 172.21.15.76:0/218072147 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5094010670 con 0x7f509810a180 2024-03-18T15:04:09.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.137+0000 7f50a0722640 1 -- 172.21.15.76:0/218072147 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5098070480 con 0x7f509810a180 2024-03-18T15:04:09.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.138+0000 7f50a0722640 1 -- 172.21.15.76:0/218072147 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50981a1600 con 0x7f509810a180 2024-03-18T15:04:09.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.139+0000 7f508f7fe640 1 -- 172.21.15.76:0/218072147 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f50940037a0 con 0x7f509810a180 2024-03-18T15:04:09.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.140+0000 7f508f7fe640 1 --2- 172.21.15.76:0/218072147 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f50840a2970 0x7f50840a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:09.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.140+0000 7f509dcbd640 1 --2- 172.21.15.76:0/218072147 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f50840a2970 0x7f50840a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:09.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.140+0000 7f508f7fe640 1 -- 172.21.15.76:0/218072147 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8528+0+0 (secure 0 0 0) 0x7f50940cf0e0 con 0x7f509810a180 2024-03-18T15:04:09.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.140+0000 7f509dcbd640 1 --2- 172.21.15.76:0/218072147 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f50840a2970 0x7f50840a4e30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f508802efd0 tx=0x7f508802cf90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:09.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.144+0000 7f508f7fe640 1 -- 172.21.15.76:0/218072147 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f50981a1600 con 0x7f509810a180 2024-03-18T15:04:09.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:08 smithi174 ceph-mon[36787]: pgmap v344: 97 pgs: 97 active+clean; 325 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.2 MiB/s wr, 167 op/s 2024-03-18T15:04:09.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:08 smithi174 ceph-mon[36787]: osdmap e106: 12 total, 12 up, 12 in 2024-03-18T15:04:09.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:08 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2526586299' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:04:09.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:08 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/2770282911' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:04:09.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:08 smithi077 ceph-mon[35865]: pgmap v344: 97 pgs: 97 active+clean; 325 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.2 MiB/s wr, 167 op/s 2024-03-18T15:04:09.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:08 smithi077 ceph-mon[35865]: osdmap e106: 12 total, 12 up, 12 in 2024-03-18T15:04:09.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:08 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2526586299' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:04:09.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:08 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/2770282911' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:04:09.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.332+0000 7f50a0722640 1 -- 172.21.15.76:0/218072147 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f50981a1c00 con 0x7f509810a180 2024-03-18T15:04:09.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.333+0000 7f508f7fe640 1 -- 172.21.15.76:0/218072147 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) ==== 100+0+11 (secure 0 0 0) 0x7f509409be40 con 0x7f509810a180 2024-03-18T15:04:09.334 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-18T15:04:09.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.335+0000 7f50a0722640 1 -- 172.21.15.76:0/218072147 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f50840a2970 msgr2=0x7f50840a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:09.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.335+0000 7f50a0722640 1 --2- 172.21.15.76:0/218072147 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f50840a2970 0x7f50840a4e30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f508802efd0 tx=0x7f508802cf90 comp rx=0 tx=0).stop 2024-03-18T15:04:09.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.335+0000 7f50a0722640 1 -- 172.21.15.76:0/218072147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f509810a180 msgr2=0x7f50981a6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:09.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.335+0000 7f50a0722640 1 --2- 172.21.15.76:0/218072147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f509810a180 0x7f50981a6610 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f509400ea60 tx=0x7f509400ef20 comp rx=0 tx=0).stop 2024-03-18T15:04:09.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.335+0000 7f50a0722640 1 -- 172.21.15.76:0/218072147 shutdown_connections 2024-03-18T15:04:09.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.335+0000 7f50a0722640 1 --2- 172.21.15.76:0/218072147 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f50840a2970 0x7f50840a4e30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:09.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.335+0000 7f50a0722640 1 --2- 172.21.15.76:0/218072147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f509810aaa0 0x7f50981a6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:09.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.335+0000 7f50a0722640 1 --2- 172.21.15.76:0/218072147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5098113730 0x7f50981a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:09.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.335+0000 7f50a0722640 1 --2- 172.21.15.76:0/218072147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f509810a180 0x7f50981a6610 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:09.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.335+0000 7f50a0722640 1 -- 172.21.15.76:0/218072147 >> 172.21.15.76:0/218072147 conn(0x7f509806cd20 msgr2=0x7f5098109450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:09.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.335+0000 7f50a0722640 1 -- 172.21.15.76:0/218072147 shutdown_connections 2024-03-18T15:04:09.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.335+0000 7f50a0722640 1 -- 172.21.15.76:0/218072147 wait complete. 2024-03-18T15:04:09.346 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-18T15:04:09.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.447+0000 7fe6128e8640 1 -- 172.21.15.76:0/3261972307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe60c100760 msgr2=0x7fe60c100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:09.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.447+0000 7fe6128e8640 1 --2- 172.21.15.76:0/3261972307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe60c100760 0x7fe60c100be0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fe600009930 tx=0x7fe60002bc30 comp rx=0 tx=0).stop 2024-03-18T15:04:09.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.447+0000 7fe6128e8640 1 -- 172.21.15.76:0/3261972307 shutdown_connections 2024-03-18T15:04:09.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.447+0000 7fe6128e8640 1 --2- 172.21.15.76:0/3261972307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe60c101120 0x7fe60c10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:09.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.447+0000 7fe6128e8640 1 --2- 172.21.15.76:0/3261972307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe60c100760 0x7fe60c100be0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:09.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.447+0000 7fe6128e8640 1 --2- 172.21.15.76:0/3261972307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60c106720 0x7fe60c106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:09.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.447+0000 7fe6128e8640 1 -- 172.21.15.76:0/3261972307 >> 172.21.15.76:0/3261972307 conn(0x7fe60c077840 msgr2=0x7fe60c077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:09.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.447+0000 7fe6128e8640 1 -- 172.21.15.76:0/3261972307 shutdown_connections 2024-03-18T15:04:09.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.447+0000 7fe6128e8640 1 -- 172.21.15.76:0/3261972307 wait complete. 2024-03-18T15:04:09.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe6128e8640 1 Processor -- start 2024-03-18T15:04:09.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe6128e8640 1 -- start start 2024-03-18T15:04:09.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe6128e8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe60c100760 0x7fe60c198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:09.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe6128e8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe60c101120 0x7fe60c199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:09.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe6128e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60c106720 0x7fe60c1a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:09.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe6128e8640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c110b30 con 0x7fe60c101120 2024-03-18T15:04:09.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe6128e8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c1109b0 con 0x7fe60c106720 2024-03-18T15:04:09.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe6128e8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c110cb0 con 0x7fe60c100760 2024-03-18T15:04:09.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe60b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe60c101120 0x7fe60c199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:09.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe60b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe60c101120 0x7fe60c199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35270/0 (socket says 172.21.15.76:35270) 2024-03-18T15:04:09.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe60b7fe640 1 -- 172.21.15.76:0/3244418721 learned_addr learned my addr 172.21.15.76:0/3244418721 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:09.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe610e85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60c106720 0x7fe60c1a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:09.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe60bfff640 1 --2- 172.21.15.76:0/3244418721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe60c100760 0x7fe60c198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:09.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe60b7fe640 1 -- 172.21.15.76:0/3244418721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe60c100760 msgr2=0x7fe60c198f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:09.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe60b7fe640 1 --2- 172.21.15.76:0/3244418721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe60c100760 0x7fe60c198f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:09.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe60b7fe640 1 -- 172.21.15.76:0/3244418721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60c106720 msgr2=0x7fe60c1a3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:09.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe60b7fe640 1 --2- 172.21.15.76:0/3244418721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60c106720 0x7fe60c1a3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:09.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.448+0000 7fe60b7fe640 1 -- 172.21.15.76:0/3244418721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60c1a44f0 con 0x7fe60c101120 2024-03-18T15:04:09.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.449+0000 7fe60b7fe640 1 --2- 172.21.15.76:0/3244418721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe60c101120 0x7fe60c199470 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe600009230 tx=0x7fe60002cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:09.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.449+0000 7fe6097fa640 1 -- 172.21.15.76:0/3244418721 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe600041070 con 0x7fe60c101120 2024-03-18T15:04:09.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.449+0000 7fe6097fa640 1 -- 172.21.15.76:0/3244418721 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe600004450 con 0x7fe60c101120 2024-03-18T15:04:09.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.449+0000 7fe6128e8640 1 -- 172.21.15.76:0/3244418721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe60c1a46c0 con 0x7fe60c101120 2024-03-18T15:04:09.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.449+0000 7fe6097fa640 1 -- 172.21.15.76:0/3244418721 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe600003e50 con 0x7fe60c101120 2024-03-18T15:04:09.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.449+0000 7fe6128e8640 1 -- 172.21.15.76:0/3244418721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe60c1a4b30 con 0x7fe60c101120 2024-03-18T15:04:09.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.450+0000 7fe6128e8640 1 -- 172.21.15.76:0/3244418721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe60c068a20 con 0x7fe60c101120 2024-03-18T15:04:09.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.451+0000 7fe6097fa640 1 -- 172.21.15.76:0/3244418721 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fe600044070 con 0x7fe60c101120 2024-03-18T15:04:09.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.451+0000 7fe6097fa640 1 --2- 172.21.15.76:0/3244418721 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe5f40a2b10 0x7fe5f40a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:09.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.452+0000 7fe60bfff640 1 --2- 172.21.15.76:0/3244418721 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe5f40a2b10 0x7fe5f40a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:09.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.452+0000 7fe6097fa640 1 -- 172.21.15.76:0/3244418721 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8528+0+0 (secure 0 0 0) 0x7fe6000f0620 con 0x7fe60c101120 2024-03-18T15:04:09.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.452+0000 7fe60bfff640 1 --2- 172.21.15.76:0/3244418721 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe5f40a2b10 0x7fe5f40a4fd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009b50 tx=0x7fe5fc005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:09.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.456+0000 7fe6097fa640 1 -- 172.21.15.76:0/3244418721 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe6000bce40 con 0x7fe60c101120 2024-03-18T15:04:09.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.678+0000 7fe6128e8640 1 -- 172.21.15.76:0/3244418721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fe60c108ca0 con 0x7fe60c101120 2024-03-18T15:04:09.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.687+0000 7fe6097fa640 1 -- 172.21.15.76:0/3244418721 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+740363 (secure 0 0 0) 0x7fe6000c0ee0 con 0x7fe60c101120 2024-03-18T15:04:09.698 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T15:04:09.698 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T15:04:09.698 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 25, 2024-03-18T15:04:09.698 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.698 INFO:teuthology.orchestra.run.smithi076.stdout: "active_gid": 25298, 2024-03-18T15:04:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: "active_name": "x", 2024-03-18T15:04:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: "active_addrs": { 2024-03-18T15:04:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-03-18T15:04:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-03-18T15:04:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6800", 2024-03-18T15:04:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3052007196 2024-03-18T15:04:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-03-18T15:04:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6801", 2024-03-18T15:04:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3052007196 2024-03-18T15:04:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-18T15:04:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: "active_addr": "172.21.15.76:6801/3052007196", 2024-03-18T15:04:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: "active_change": "2024-03-18T14:53:17.537260+0000", 2024-03-18T15:04:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: "active_mgr_features": 4540701547738038271, 2024-03-18T15:04:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: "available": true, 2024-03-18T15:04:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: "standbys": [ 2024-03-18T15:04:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: "gid": 25288, 2024-03-18T15:04:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "z", 2024-03-18T15:04:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_features": 4540701547738038271, 2024-03-18T15:04:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: "available_modules": [ 2024-03-18T15:04:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "alerts", 2024-03-18T15:04:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-18T15:04:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-18T15:04:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-18T15:04:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T15:04:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_destination": { 2024-03-18T15:04:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_destination", 2024-03-18T15:04:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Email address to send alerts to", 2024-03-18T15:04:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_from_name": { 2024-03-18T15:04:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_from_name", 2024-03-18T15:04:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "Ceph", 2024-03-18T15:04:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Email From: name", 2024-03-18T15:04:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_host": { 2024-03-18T15:04:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_host", 2024-03-18T15:04:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP server", 2024-03-18T15:04:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_password": { 2024-03-18T15:04:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_password", 2024-03-18T15:04:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Password to authenticate with", 2024-03-18T15:04:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_port": { 2024-03-18T15:04:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_port", 2024-03-18T15:04:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "465", 2024-03-18T15:04:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP port", 2024-03-18T15:04:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_sender": { 2024-03-18T15:04:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_sender", 2024-03-18T15:04:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP envelope sender", 2024-03-18T15:04:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_ssl": { 2024-03-18T15:04:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_ssl", 2024-03-18T15:04:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T15:04:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_user": { 2024-03-18T15:04:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_user", 2024-03-18T15:04:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "User to authenticate as", 2024-03-18T15:04:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "balancer", 2024-03-18T15:04:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: "active": { 2024-03-18T15:04:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "active", 2024-03-18T15:04:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T15:04:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: "begin_time": { 2024-03-18T15:04:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "begin_time", 2024-03-18T15:04:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0000", 2024-03-18T15:04:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T15:04:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T15:04:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: "begin_weekday": { 2024-03-18T15:04:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "begin_weekday", 2024-03-18T15:04:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-18T15:04:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-18T15:04:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0", 2024-03-18T15:04:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "6", 2024-03-18T15:04:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T15:04:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T15:04:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_max_iterations": { 2024-03-18T15:04:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_max_iterations", 2024-03-18T15:04:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-18T15:04:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "25", 2024-03-18T15:04:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-18T15:04:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "250", 2024-03-18T15:04:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T15:04:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_metrics": { 2024-03-18T15:04:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_metrics", 2024-03-18T15:04:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T15:04:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T15:04:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T15:04:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_step": { 2024-03-18T15:04:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_step", 2024-03-18T15:04:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0.5", 2024-03-18T15:04:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0.001", 2024-03-18T15:04:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "0.999", 2024-03-18T15:04:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "aggressiveness of optimization", 2024-03-18T15:04:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T15:04:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: "end_time": { 2024-03-18T15:04:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "end_time", 2024-03-18T15:04:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "2359", 2024-03-18T15:04:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T15:04:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T15:04:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: "end_weekday": { 2024-03-18T15:04:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "end_weekday", 2024-03-18T15:04:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-18T15:04:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-18T15:04:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0", 2024-03-18T15:04:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "6", 2024-03-18T15:04:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T15:04:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T15:04:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: "min_score": { 2024-03-18T15:04:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "min_score", 2024-03-18T15:04:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-18T15:04:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T15:04:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": { 2024-03-18T15:04:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mode", 2024-03-18T15:04:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "upmap", 2024-03-18T15:04:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-compat", 2024-03-18T15:04:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: "none", 2024-03-18T15:04:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: "read", 2024-03-18T15:04:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap", 2024-03-18T15:04:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap-read" 2024-03-18T15:04:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Balancer mode", 2024-03-18T15:04:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_ids": { 2024-03-18T15:04:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pool_ids", 2024-03-18T15:04:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T15:04:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-18T15:04:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-18T15:04:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-18T15:04:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T15:04:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap_max_deviation": { 2024-03-18T15:04:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "upmap_max_deviation", 2024-03-18T15:04:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-18T15:04:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-18T15:04:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T15:04:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T15:04:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap_max_optimizations": { 2024-03-18T15:04:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "upmap_max_optimizations", 2024-03-18T15:04:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-18T15:04:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T15:04:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cephadm", 2024-03-18T15:04:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_down_multiplier": { 2024-03-18T15:04:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_down_multiplier", 2024-03-18T15:04:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3.0", 2024-03-18T15:04:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T15:04:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_refresh_rate": { 2024-03-18T15:04:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_refresh_rate", 2024-03-18T15:04:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "20", 2024-03-18T15:04:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T15:04:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_starting_port": { 2024-03-18T15:04:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_starting_port", 2024-03-18T15:04:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "4721", 2024-03-18T15:04:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T15:04:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_ptrace": { 2024-03-18T15:04:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_ptrace", 2024-03-18T15:04:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T15:04:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T15:04:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: "autotune_interval": { 2024-03-18T15:04:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "autotune_interval", 2024-03-18T15:04:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-18T15:04:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T15:04:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.767 INFO:teuthology.orchestra.run.smithi076.stdout: "autotune_memory_target_ratio": { 2024-03-18T15:04:09.767 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T15:04:09.767 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:09.767 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.767 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.767 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0.7", 2024-03-18T15:04:09.767 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.767 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.767 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.769 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T15:04:09.769 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.769 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.769 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.770 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm_log_destination": { 2024-03-18T15:04:09.770 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cephadm_log_destination", 2024-03-18T15:04:09.770 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.770 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.770 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.770 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.770 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.770 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.770 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:09.771 INFO:teuthology.orchestra.run.smithi076.stdout: "file", 2024-03-18T15:04:09.771 INFO:teuthology.orchestra.run.smithi076.stdout: "file,syslog", 2024-03-18T15:04:09.771 INFO:teuthology.orchestra.run.smithi076.stdout: "syslog" 2024-03-18T15:04:09.771 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:09.771 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T15:04:09.771 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.771 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.771 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.772 INFO:teuthology.orchestra.run.smithi076.stdout: "cgroups_split": { 2024-03-18T15:04:09.772 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cgroups_split", 2024-03-18T15:04:09.772 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.772 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.772 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.772 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.772 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.772 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.773 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.773 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T15:04:09.773 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.773 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.773 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.773 INFO:teuthology.orchestra.run.smithi076.stdout: "config_checks_enabled": { 2024-03-18T15:04:09.773 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_checks_enabled", 2024-03-18T15:04:09.773 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.774 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.774 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.774 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.774 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.774 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.774 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.774 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T15:04:09.774 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.774 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.775 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.775 INFO:teuthology.orchestra.run.smithi076.stdout: "config_dashboard": { 2024-03-18T15:04:09.775 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_dashboard", 2024-03-18T15:04:09.775 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.775 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.775 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.775 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.775 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.776 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.776 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.776 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T15:04:09.776 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.776 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.776 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.776 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_alertmanager": { 2024-03-18T15:04:09.777 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_alertmanager", 2024-03-18T15:04:09.777 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.777 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.777 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.777 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T15:04:09.777 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.777 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.777 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.778 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-18T15:04:09.778 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.778 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.778 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.778 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_base": { 2024-03-18T15:04:09.778 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_base", 2024-03-18T15:04:09.779 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.779 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.779 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.779 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T15:04:09.779 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.779 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.780 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.780 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Container image name, without the tag", 2024-03-18T15:04:09.780 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.780 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.780 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.781 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_elasticsearch": { 2024-03-18T15:04:09.781 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_elasticsearch", 2024-03-18T15:04:09.781 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.782 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.782 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.782 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T15:04:09.782 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.782 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.782 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.783 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "elasticsearch container image", 2024-03-18T15:04:09.783 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.783 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.783 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.783 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_grafana": { 2024-03-18T15:04:09.783 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_grafana", 2024-03-18T15:04:09.784 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.784 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.784 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.784 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T15:04:09.784 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.785 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.785 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.785 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-18T15:04:09.785 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.785 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.785 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.786 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_haproxy": { 2024-03-18T15:04:09.786 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_haproxy", 2024-03-18T15:04:09.786 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.786 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.786 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.787 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T15:04:09.787 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.787 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.787 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.787 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "HAproxy container image", 2024-03-18T15:04:09.788 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.788 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.788 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.788 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_agent": { 2024-03-18T15:04:09.788 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_agent", 2024-03-18T15:04:09.789 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.789 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.789 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.789 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T15:04:09.789 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.789 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.789 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.789 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger agent container image", 2024-03-18T15:04:09.790 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.790 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.790 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.790 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_collector": { 2024-03-18T15:04:09.790 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_collector", 2024-03-18T15:04:09.790 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.791 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.791 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.791 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T15:04:09.791 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.791 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.791 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.791 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger collector container image", 2024-03-18T15:04:09.792 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.792 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.792 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.792 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_query": { 2024-03-18T15:04:09.792 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_query", 2024-03-18T15:04:09.792 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.792 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.793 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.793 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T15:04:09.793 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.793 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.793 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.793 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger query container image", 2024-03-18T15:04:09.793 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.793 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.794 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.794 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_keepalived": { 2024-03-18T15:04:09.794 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_keepalived", 2024-03-18T15:04:09.794 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.794 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.795 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.795 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T15:04:09.795 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.795 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.795 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.795 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Keepalived container image", 2024-03-18T15:04:09.795 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.795 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.796 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.796 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_loki": { 2024-03-18T15:04:09.796 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_loki", 2024-03-18T15:04:09.796 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.796 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.796 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.796 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T15:04:09.797 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.797 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.797 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.797 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Loki container image", 2024-03-18T15:04:09.797 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.797 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.797 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.798 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_node_exporter": { 2024-03-18T15:04:09.798 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_node_exporter", 2024-03-18T15:04:09.798 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.798 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.798 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.798 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T15:04:09.799 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.799 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.799 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.799 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-18T15:04:09.799 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.800 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.800 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.800 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_nvmeof": { 2024-03-18T15:04:09.800 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_nvmeof", 2024-03-18T15:04:09.800 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.801 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.801 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.801 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T15:04:09.801 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.801 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.801 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.802 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Nvme-of container image", 2024-03-18T15:04:09.802 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.802 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.802 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.802 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_prometheus": { 2024-03-18T15:04:09.802 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_prometheus", 2024-03-18T15:04:09.802 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T15:04:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-18T15:04:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_promtail": { 2024-03-18T15:04:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_promtail", 2024-03-18T15:04:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T15:04:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Promtail container image", 2024-03-18T15:04:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_snmp_gateway": { 2024-03-18T15:04:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_snmp_gateway", 2024-03-18T15:04:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T15:04:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SNMP Gateway container image", 2024-03-18T15:04:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: "container_init": { 2024-03-18T15:04:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_init", 2024-03-18T15:04:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T15:04:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: "daemon_cache_timeout": { 2024-03-18T15:04:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "daemon_cache_timeout", 2024-03-18T15:04:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-18T15:04:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T15:04:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: "default_cephadm_command_timeout": { 2024-03-18T15:04:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T15:04:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "900", 2024-03-18T15:04:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T15:04:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: "default_registry": { 2024-03-18T15:04:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "default_registry", 2024-03-18T15:04:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io", 2024-03-18T15:04:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T15:04:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: "device_cache_timeout": { 2024-03-18T15:04:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_cache_timeout", 2024-03-18T15:04:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1800", 2024-03-18T15:04:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache device inventory", 2024-03-18T15:04:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: "device_enhanced_scan": { 2024-03-18T15:04:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_enhanced_scan", 2024-03-18T15:04:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T15:04:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: "facts_cache_timeout": { 2024-03-18T15:04:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "facts_cache_timeout", 2024-03-18T15:04:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-18T15:04:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache host facts data", 2024-03-18T15:04:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: "host_check_interval": { 2024-03-18T15:04:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "host_check_interval", 2024-03-18T15:04:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-18T15:04:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to perform a host check", 2024-03-18T15:04:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: "hw_monitoring": { 2024-03-18T15:04:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "hw_monitoring", 2024-03-18T15:04:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T15:04:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: "inventory_list_all": { 2024-03-18T15:04:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "inventory_list_all", 2024-03-18T15:04:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T15:04:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: "log_refresh_metadata": { 2024-03-18T15:04:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_refresh_metadata", 2024-03-18T15:04:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T15:04:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T15:04:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.835 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.835 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.835 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.835 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.835 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.835 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.835 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.835 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T15:04:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T15:04:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T15:04:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T15:04:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T15:04:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "*", 2024-03-18T15:04:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T15:04:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: "max_count_per_host": { 2024-03-18T15:04:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_count_per_host", 2024-03-18T15:04:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "max number of daemons per service per host", 2024-03-18T15:04:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd_draining_count": { 2024-03-18T15:04:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_osd_draining_count", 2024-03-18T15:04:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T15:04:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: "migration_current": { 2024-03-18T15:04:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "migration_current", 2024-03-18T15:04:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "internal - do not modify", 2024-03-18T15:04:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": { 2024-03-18T15:04:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mode", 2024-03-18T15:04:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "root", 2024-03-18T15:04:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm-package", 2024-03-18T15:04:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "root" 2024-03-18T15:04:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T15:04:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: "oob_default_addr": { 2024-03-18T15:04:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "oob_default_addr", 2024-03-18T15:04:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "169.254.1.1", 2024-03-18T15:04:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T15:04:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: "prometheus_alerts_path": { 2024-03-18T15:04:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prometheus_alerts_path", 2024-03-18T15:04:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.848 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.848 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.848 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T15:04:09.848 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.848 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.848 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.848 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T15:04:09.848 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_insecure": { 2024-03-18T15:04:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_insecure", 2024-03-18T15:04:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T15:04:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_password": { 2024-03-18T15:04:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_password", 2024-03-18T15:04:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T15:04:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_url": { 2024-03-18T15:04:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_url", 2024-03-18T15:04:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T15:04:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_username": { 2024-03-18T15:04:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_username", 2024-03-18T15:04:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T15:04:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: "secure_monitoring_stack": { 2024-03-18T15:04:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "secure_monitoring_stack", 2024-03-18T15:04:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T15:04:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: "service_discovery_port": { 2024-03-18T15:04:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "service_discovery_port", 2024-03-18T15:04:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8765", 2024-03-18T15:04:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "cephadm service discovery port", 2024-03-18T15:04:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: "ssh_config_file": { 2024-03-18T15:04:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssh_config_file", 2024-03-18T15:04:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T15:04:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: "use_agent": { 2024-03-18T15:04:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "use_agent", 2024-03-18T15:04:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T15:04:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: "use_repo_digest": { 2024-03-18T15:04:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "use_repo_digest", 2024-03-18T15:04:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T15:04:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_failed_host_check": { 2024-03-18T15:04:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_failed_host_check", 2024-03-18T15:04:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T15:04:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_stray_daemons": { 2024-03-18T15:04:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_stray_daemons", 2024-03-18T15:04:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T15:04:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_stray_hosts": { 2024-03-18T15:04:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_stray_hosts", 2024-03-18T15:04:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T15:04:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:09.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:09.869 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crash", 2024-03-18T15:04:09.869 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:09.869 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:09.869 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:09.869 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:09.869 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:09.869 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.870 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.870 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.870 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.870 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.870 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.870 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:09.870 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:09.870 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:09.871 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:09.871 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:09.871 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:09.871 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:09.871 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:09.871 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.871 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.871 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.871 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.872 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:09.872 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:09.872 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.872 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.872 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.872 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.872 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.872 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.873 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.873 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.873 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.873 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.873 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.873 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:09.873 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:09.874 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.874 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.874 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.874 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:09.874 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.874 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.874 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:09.874 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:09.875 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:09.875 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:09.875 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:09.875 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:09.875 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:09.875 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:09.875 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.875 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.875 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.876 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.876 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:09.876 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:09.876 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.876 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.876 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.876 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.876 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: "retain_interval": { 2024-03-18T15:04:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "retain_interval", 2024-03-18T15:04:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "31536000", 2024-03-18T15:04:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T15:04:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_recent_interval": { 2024-03-18T15:04:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_recent_interval", 2024-03-18T15:04:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1209600", 2024-03-18T15:04:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T15:04:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "dashboard", 2024-03-18T15:04:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T15:04:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T15:04:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T15:04:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T15:04:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.884 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.884 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.884 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.884 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.884 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.884 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.884 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.884 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T15:04:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T15:04:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.886 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.886 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.886 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.886 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.886 INFO:teuthology.orchestra.run.smithi076.stdout: "AUDIT_API_ENABLED": { 2024-03-18T15:04:09.886 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T15:04:09.886 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.886 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.887 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.887 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.887 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.887 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.887 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.887 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.887 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.887 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.887 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.888 INFO:teuthology.orchestra.run.smithi076.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T15:04:09.888 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T15:04:09.888 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.888 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.888 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.888 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.888 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.889 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.889 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.889 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.889 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.889 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.889 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.889 INFO:teuthology.orchestra.run.smithi076.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T15:04:09.889 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T15:04:09.890 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.890 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.890 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.890 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.890 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.890 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.890 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.890 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.891 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.891 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.891 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.891 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T15:04:09.891 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T15:04:09.891 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.891 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T15:04:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T15:04:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T15:04:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T15:04:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T15:04:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T15:04:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T15:04:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T15:04:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T15:04:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T15:04:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T15:04:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T15:04:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T15:04:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T15:04:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T15:04:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T15:04:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-18T15:04:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T15:04:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T15:04:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_URL": { 2024-03-18T15:04:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_URL", 2024-03-18T15:04:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T15:04:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T15:04:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-18T15:04:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T15:04:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T15:04:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T15:04:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T15:04:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T15:04:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T15:04:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T15:04:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T15:04:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T15:04:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T15:04:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "{}", 2024-03-18T15:04:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T15:04:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T15:04:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T15:04:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T15:04:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T15:04:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T15:04:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.923 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T15:04:09.923 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T15:04:09.923 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.923 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.923 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.923 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.923 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.924 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.924 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.924 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.924 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.924 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.924 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.924 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T15:04:09.924 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T15:04:09.925 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.925 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.925 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.925 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.925 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.925 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.925 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.925 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.925 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.926 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.926 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.926 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T15:04:09.926 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T15:04:09.926 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.926 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.926 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.927 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.927 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.927 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.927 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.927 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.927 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.927 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.927 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.928 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T15:04:09.928 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T15:04:09.928 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.928 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.928 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.928 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.928 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.928 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.928 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.929 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.929 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.929 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.929 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.929 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T15:04:09.929 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T15:04:09.929 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.930 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.930 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.930 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.930 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.930 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.930 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.930 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.930 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.930 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.931 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.931 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T15:04:09.931 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T15:04:09.931 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.931 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.931 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.931 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.931 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.932 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.932 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.932 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.932 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.932 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.932 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.932 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T15:04:09.932 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T15:04:09.933 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.933 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.933 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.933 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.933 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.933 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.933 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.933 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.934 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.934 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.934 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.934 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T15:04:09.934 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T15:04:09.934 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.934 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.934 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.935 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T15:04:09.935 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.935 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.935 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.935 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.935 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.935 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.935 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.936 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T15:04:09.936 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T15:04:09.936 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.936 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.936 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.936 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:09.936 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.936 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.937 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.937 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.937 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.937 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.937 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.937 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T15:04:09.937 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T15:04:09.937 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.938 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.938 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.938 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8", 2024-03-18T15:04:09.938 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.938 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.938 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.938 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.938 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.938 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.939 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.939 INFO:teuthology.orchestra.run.smithi076.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T15:04:09.939 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T15:04:09.939 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.939 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.939 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.939 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "45", 2024-03-18T15:04:09.939 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.940 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.940 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.940 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.940 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.940 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.940 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.940 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T15:04:09.940 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T15:04:09.941 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.941 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.941 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.941 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.941 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.941 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.941 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.941 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.941 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.942 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.942 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.942 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T15:04:09.942 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T15:04:09.942 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.942 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.942 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.942 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-18T15:04:09.943 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.943 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.943 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.943 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.943 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.943 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.943 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.943 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T15:04:09.944 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T15:04:09.944 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.944 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.944 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.944 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.944 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.944 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.944 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.944 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.945 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.945 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.945 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.945 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T15:04:09.945 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T15:04:09.945 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.945 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.945 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.946 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.946 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.946 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.946 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.946 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.946 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.946 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.946 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.947 INFO:teuthology.orchestra.run.smithi076.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T15:04:09.947 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T15:04:09.947 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.947 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.947 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.947 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.947 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.947 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.947 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.948 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.948 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.948 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.948 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.948 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T15:04:09.948 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T15:04:09.948 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.949 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.949 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.949 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-18T15:04:09.949 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.949 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.949 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.949 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.949 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.949 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.949 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.950 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T15:04:09.950 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T15:04:09.950 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.950 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.950 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.950 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:09.950 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.950 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.951 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.951 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.951 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.951 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.951 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.951 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T15:04:09.951 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T15:04:09.951 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.952 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.952 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.952 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-18T15:04:09.952 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.952 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.952 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.952 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.952 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.952 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.953 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.953 INFO:teuthology.orchestra.run.smithi076.stdout: "cross_origin_url": { 2024-03-18T15:04:09.953 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cross_origin_url", 2024-03-18T15:04:09.953 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.953 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.953 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.953 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.953 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.954 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.954 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.954 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.954 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.954 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.954 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.954 INFO:teuthology.orchestra.run.smithi076.stdout: "crt_file": { 2024-03-18T15:04:09.955 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crt_file", 2024-03-18T15:04:09.955 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.955 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.955 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.955 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.955 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.955 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.955 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.956 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.956 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.956 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.956 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.956 INFO:teuthology.orchestra.run.smithi076.stdout: "debug": { 2024-03-18T15:04:09.956 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "debug", 2024-03-18T15:04:09.956 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.956 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.957 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.957 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.957 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.957 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.957 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.957 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable/disable debug options", 2024-03-18T15:04:09.957 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.957 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.957 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.958 INFO:teuthology.orchestra.run.smithi076.stdout: "jwt_token_ttl": { 2024-03-18T15:04:09.958 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "jwt_token_ttl", 2024-03-18T15:04:09.958 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.958 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.958 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.958 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "28800", 2024-03-18T15:04:09.958 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.959 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.959 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.959 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.959 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.959 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.959 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.959 INFO:teuthology.orchestra.run.smithi076.stdout: "key_file": { 2024-03-18T15:04:09.959 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "key_file", 2024-03-18T15:04:09.960 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.960 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.960 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.960 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.960 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.960 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.960 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.960 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.960 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.960 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.961 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.961 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:09.961 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:09.961 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.961 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.961 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.961 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.962 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.962 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.962 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:09.962 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:09.962 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:09.962 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:09.962 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:09.962 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:09.962 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:09.963 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:09.963 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.963 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.963 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.963 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.963 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:09.963 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:09.963 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.964 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.964 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.964 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.964 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.964 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.964 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.964 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.964 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.964 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.965 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.965 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:09.965 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:09.965 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.965 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.965 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.965 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:09.966 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.966 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.966 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:09.966 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:09.966 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:09.966 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:09.966 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:09.966 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:09.966 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:09.967 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:09.967 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.967 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.967 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.967 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.967 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:09.967 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:09.967 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.968 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.968 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.968 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.968 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.968 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.968 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.968 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.968 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.968 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.969 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.969 INFO:teuthology.orchestra.run.smithi076.stdout: "motd": { 2024-03-18T15:04:09.969 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "motd", 2024-03-18T15:04:09.969 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.969 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.969 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.969 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.970 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.970 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.970 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.970 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "The message of the day", 2024-03-18T15:04:09.970 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.970 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.970 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.970 INFO:teuthology.orchestra.run.smithi076.stdout: "redirect_resolve_ip_addr": { 2024-03-18T15:04:09.971 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T15:04:09.971 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.971 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.974 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.974 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.974 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.974 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.974 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.974 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.974 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.975 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.975 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.975 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-18T15:04:09.975 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-18T15:04:09.975 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.975 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.975 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.975 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "::", 2024-03-18T15:04:09.976 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.976 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.976 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.976 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.976 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.976 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.976 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.977 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-18T15:04:09.977 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-18T15:04:09.977 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.977 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.977 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.977 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8080", 2024-03-18T15:04:09.977 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.977 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.978 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.978 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.978 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.978 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.978 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.978 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl": { 2024-03-18T15:04:09.978 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl", 2024-03-18T15:04:09.978 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.979 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.979 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.979 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.979 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.979 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.979 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.979 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.979 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.979 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.980 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.980 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl_server_port": { 2024-03-18T15:04:09.980 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl_server_port", 2024-03-18T15:04:09.980 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.980 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.980 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.980 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8443", 2024-03-18T15:04:09.981 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.981 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.981 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.981 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.981 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.981 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.981 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.981 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_behaviour": { 2024-03-18T15:04:09.982 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_behaviour", 2024-03-18T15:04:09.982 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.982 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.982 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.982 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "redirect", 2024-03-18T15:04:09.982 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.982 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.982 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:09.983 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:09.983 INFO:teuthology.orchestra.run.smithi076.stdout: "redirect" 2024-03-18T15:04:09.983 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:09.983 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.983 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.983 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.983 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.983 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_error_status_code": { 2024-03-18T15:04:09.984 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_error_status_code", 2024-03-18T15:04:09.984 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:09.984 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.984 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.984 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "500", 2024-03-18T15:04:09.984 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "400", 2024-03-18T15:04:09.984 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "599", 2024-03-18T15:04:09.984 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.985 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.985 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.985 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.985 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.985 INFO:teuthology.orchestra.run.smithi076.stdout: "url_prefix": { 2024-03-18T15:04:09.985 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "url_prefix", 2024-03-18T15:04:09.985 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.985 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.986 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:09.986 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.986 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.986 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.986 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.986 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.986 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.986 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.986 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.987 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:09.987 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:09.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:09.987 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "devicehealth", 2024-03-18T15:04:09.987 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:09.987 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:09.987 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:09.987 INFO:teuthology.orchestra.run.smithi076.stdout: "enable_monitoring": { 2024-03-18T15:04:09.988 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enable_monitoring", 2024-03-18T15:04:09.988 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.988 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.988 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.988 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:09.988 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.988 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.988 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.988 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "monitor device health metrics", 2024-03-18T15:04:09.989 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.989 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.989 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.989 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:09.989 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:09.989 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.989 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.989 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.990 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:09.990 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.990 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.990 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:09.990 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:09.990 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:09.990 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:09.990 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:09.990 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:09.991 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:09.991 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:09.991 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.991 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.991 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.991 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.991 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:09.992 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:09.992 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.992 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.992 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.992 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.992 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.992 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.992 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.992 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.993 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.993 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.993 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.993 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:09.993 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:09.993 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.993 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.993 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.994 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:09.994 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.994 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.994 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:09.994 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:09.994 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:09.994 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:09.994 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:09.994 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:09.995 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:09.995 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:09.995 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.995 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.995 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.995 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.995 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:09.995 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:09.996 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:09.996 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.996 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.996 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:09.996 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.996 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.996 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.996 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:09.996 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.997 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.997 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.997 INFO:teuthology.orchestra.run.smithi076.stdout: "mark_out_threshold": { 2024-03-18T15:04:09.997 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mark_out_threshold", 2024-03-18T15:04:09.997 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:09.997 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.997 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.997 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "2419200", 2024-03-18T15:04:09.998 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.998 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.998 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.998 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T15:04:09.998 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:09.998 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:09.998 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:09.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:09.998 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": { 2024-03-18T15:04:09.999 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pool_name", 2024-03-18T15:04:09.999 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:09.999 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:09.999 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:09.999 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "device_health_metrics", 2024-03-18T15:04:09.999 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:09.999 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:09.999 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:09.999 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T15:04:10.000 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.000 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.000 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.000 INFO:teuthology.orchestra.run.smithi076.stdout: "retention_period": { 2024-03-18T15:04:10.000 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "retention_period", 2024-03-18T15:04:10.000 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.000 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.000 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.001 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15552000", 2024-03-18T15:04:10.001 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.001 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.001 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.001 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long to retain device health metrics", 2024-03-18T15:04:10.001 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.001 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.001 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.002 INFO:teuthology.orchestra.run.smithi076.stdout: "scrape_frequency": { 2024-03-18T15:04:10.002 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "scrape_frequency", 2024-03-18T15:04:10.002 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.002 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.002 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.002 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "86400", 2024-03-18T15:04:10.002 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.002 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.002 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.003 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T15:04:10.003 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.003 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.003 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.003 INFO:teuthology.orchestra.run.smithi076.stdout: "self_heal": { 2024-03-18T15:04:10.003 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "self_heal", 2024-03-18T15:04:10.003 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.003 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.004 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.004 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.004 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.004 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.004 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.004 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T15:04:10.004 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.004 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.004 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.005 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-18T15:04:10.005 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-18T15:04:10.005 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.005 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.005 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.005 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-18T15:04:10.005 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.006 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.006 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.006 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T15:04:10.006 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.006 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.006 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.006 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_threshold": { 2024-03-18T15:04:10.006 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_threshold", 2024-03-18T15:04:10.007 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.007 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.007 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.007 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "7257600", 2024-03-18T15:04:10.007 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.007 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.007 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.007 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T15:04:10.007 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.008 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.008 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.008 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.008 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.008 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "diskprediction_local", 2024-03-18T15:04:10.008 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.008 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.009 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.009 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.009 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.009 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.009 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.009 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.009 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.009 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.010 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.010 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.010 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.010 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.010 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.010 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.010 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.010 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.010 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.011 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.011 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.011 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.011 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.011 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.011 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.011 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.012 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.012 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.012 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.012 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.012 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.012 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.012 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.012 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.012 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.013 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.013 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.013 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.013 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.013 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.013 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.013 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.013 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.014 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.014 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.014 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.014 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.014 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.014 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.014 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.014 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.015 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.015 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.015 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.015 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.015 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.015 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.015 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.015 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.015 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.016 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.016 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.016 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.016 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.016 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.016 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.016 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.016 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.017 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.017 INFO:teuthology.orchestra.run.smithi076.stdout: "predict_interval": { 2024-03-18T15:04:10.017 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "predict_interval", 2024-03-18T15:04:10.017 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.017 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.017 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.017 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "86400", 2024-03-18T15:04:10.017 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.018 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.018 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.018 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.018 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.018 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.018 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.018 INFO:teuthology.orchestra.run.smithi076.stdout: "predictor_model": { 2024-03-18T15:04:10.018 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "predictor_model", 2024-03-18T15:04:10.019 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.019 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.019 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.019 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "prophetstor", 2024-03-18T15:04:10.019 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.019 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.019 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.019 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.020 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.020 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.020 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.020 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-18T15:04:10.020 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-18T15:04:10.020 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.020 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.020 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.021 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-18T15:04:10.021 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.021 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.021 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.021 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.021 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.021 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.021 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.021 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.022 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.022 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "influx", 2024-03-18T15:04:10.022 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": false, 2024-03-18T15:04:10.022 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "influxdb python module not found", 2024-03-18T15:04:10.022 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.022 INFO:teuthology.orchestra.run.smithi076.stdout: "batch_size": { 2024-03-18T15:04:10.022 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "batch_size", 2024-03-18T15:04:10.023 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.023 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.023 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.023 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5000", 2024-03-18T15:04:10.023 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.023 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.023 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.023 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T15:04:10.023 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.024 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.024 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.024 INFO:teuthology.orchestra.run.smithi076.stdout: "database": { 2024-03-18T15:04:10.024 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "database", 2024-03-18T15:04:10.024 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.024 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.025 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.025 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "ceph", 2024-03-18T15:04:10.025 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.025 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.025 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.025 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T15:04:10.025 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.025 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.025 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.026 INFO:teuthology.orchestra.run.smithi076.stdout: "hostname": { 2024-03-18T15:04:10.026 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "hostname", 2024-03-18T15:04:10.026 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.026 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.026 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.026 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.026 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.026 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.027 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.027 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB server hostname", 2024-03-18T15:04:10.027 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.027 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.027 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.027 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-18T15:04:10.027 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-18T15:04:10.028 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.028 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.028 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.028 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "30", 2024-03-18T15:04:10.028 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "5", 2024-03-18T15:04:10.028 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.028 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.028 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T15:04:10.028 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.029 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.029 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.029 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.029 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.029 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.029 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.029 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.029 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.030 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.030 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.030 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.030 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.030 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.030 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.030 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.030 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.030 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.031 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.031 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.031 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.031 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.031 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.031 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.031 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.031 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.032 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.032 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.032 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.032 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.032 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.032 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.032 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.032 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.032 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.033 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.033 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.033 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.033 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.033 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.033 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.033 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.033 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.035 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.035 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.035 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.035 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.035 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.035 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.035 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.035 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.036 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.036 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.036 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.036 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.036 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.036 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.036 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.036 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.036 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.037 INFO:teuthology.orchestra.run.smithi076.stdout: "password": { 2024-03-18T15:04:10.037 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "password", 2024-03-18T15:04:10.037 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.037 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.037 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.037 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.037 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.037 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.038 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.038 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "password of InfluxDB server user", 2024-03-18T15:04:10.038 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.038 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.038 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.038 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.038 INFO:teuthology.orchestra.run.smithi076.stdout: "port": { 2024-03-18T15:04:10.038 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "port", 2024-03-18T15:04:10.038 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.039 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.039 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.039 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8086", 2024-03-18T15:04:10.039 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.039 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.039 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.039 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB server port", 2024-03-18T15:04:10.039 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.039 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.040 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.040 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl": { 2024-03-18T15:04:10.040 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl", 2024-03-18T15:04:10.040 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.040 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.040 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.040 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "false", 2024-03-18T15:04:10.040 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.041 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.041 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.041 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T15:04:10.041 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.041 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.041 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.041 INFO:teuthology.orchestra.run.smithi076.stdout: "threads": { 2024-03-18T15:04:10.041 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "threads", 2024-03-18T15:04:10.042 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.042 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.042 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.042 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-18T15:04:10.042 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-18T15:04:10.042 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "32", 2024-03-18T15:04:10.042 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.042 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T15:04:10.043 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.043 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.043 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.043 INFO:teuthology.orchestra.run.smithi076.stdout: "username": { 2024-03-18T15:04:10.043 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "username", 2024-03-18T15:04:10.043 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.043 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.043 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.044 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.044 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.044 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.044 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.044 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "username of InfluxDB server user", 2024-03-18T15:04:10.044 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.044 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.044 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.045 INFO:teuthology.orchestra.run.smithi076.stdout: "verify_ssl": { 2024-03-18T15:04:10.045 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "verify_ssl", 2024-03-18T15:04:10.045 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.045 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.045 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.045 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "true", 2024-03-18T15:04:10.045 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.045 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.046 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.046 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T15:04:10.046 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.046 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.046 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.046 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.046 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.047 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "insights", 2024-03-18T15:04:10.047 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.047 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.047 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.047 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.047 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.047 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.047 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.048 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.048 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.048 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.048 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.048 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.048 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.048 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.048 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.048 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.049 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.049 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.049 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.049 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.049 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.049 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.049 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.049 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.050 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.050 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.050 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.050 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.050 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.050 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.050 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.050 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.050 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.051 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.051 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.051 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.051 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.051 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.051 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.051 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.051 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.052 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.052 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.052 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.052 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.052 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.052 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.052 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.052 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.052 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.053 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.053 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.053 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.053 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.053 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.053 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.053 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.053 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.054 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.054 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.054 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.054 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.054 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.054 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.054 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.054 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.055 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.055 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.055 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.055 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.055 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.055 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "iostat", 2024-03-18T15:04:10.056 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.056 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.056 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.056 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.056 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.056 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.056 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.056 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.056 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.057 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.057 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.057 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.057 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.057 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.057 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.057 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.057 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.057 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.058 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.058 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.058 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.058 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.058 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.058 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.058 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.059 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.059 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.059 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.059 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.059 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.059 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.059 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.059 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.059 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.060 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.060 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.060 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.060 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.060 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.060 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.060 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.060 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.061 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.061 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.061 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.061 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.061 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.061 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.061 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.061 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.061 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.062 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.062 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.062 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.062 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.062 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.062 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.062 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.063 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.063 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.063 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.063 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.063 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.063 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.063 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.063 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.063 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.064 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.064 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.064 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.064 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.064 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "k8sevents", 2024-03-18T15:04:10.064 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.065 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.065 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.065 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_event_retention_days": { 2024-03-18T15:04:10.065 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ceph_event_retention_days", 2024-03-18T15:04:10.065 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.065 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.065 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.065 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "7", 2024-03-18T15:04:10.065 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.066 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.066 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.066 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T15:04:10.066 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.066 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.066 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.066 INFO:teuthology.orchestra.run.smithi076.stdout: "config_check_secs": { 2024-03-18T15:04:10.066 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_check_secs", 2024-03-18T15:04:10.067 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.067 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.067 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.067 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:10.067 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "10", 2024-03-18T15:04:10.067 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.067 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.067 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T15:04:10.068 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.068 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.068 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.068 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.068 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.068 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.068 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.069 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.069 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.069 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.069 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.069 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.069 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.069 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.069 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.069 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.070 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.070 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.070 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.070 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.070 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.070 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.070 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.071 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.071 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.071 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.071 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.071 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.071 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.071 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.071 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.072 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.072 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.072 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.072 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.072 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.072 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.072 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.072 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.073 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.073 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.073 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.073 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.073 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.073 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.073 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.073 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.074 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.074 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.074 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.074 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.074 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.074 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.074 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.074 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.074 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.075 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.075 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.075 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.075 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.075 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.075 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.075 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.076 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.076 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.076 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.076 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.076 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.076 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.076 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.076 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.077 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.077 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "localpool", 2024-03-18T15:04:10.077 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.077 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.077 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.077 INFO:teuthology.orchestra.run.smithi076.stdout: "failure_domain": { 2024-03-18T15:04:10.077 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "failure_domain", 2024-03-18T15:04:10.077 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.078 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.078 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.078 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "host", 2024-03-18T15:04:10.078 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.078 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.078 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.078 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "failure domain for any created local pool", 2024-03-18T15:04:10.078 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T15:04:10.079 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.079 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.079 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.079 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.079 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.079 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.079 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.079 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.079 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.080 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.080 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.080 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.080 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.080 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.080 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.080 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.080 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.080 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.081 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.081 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.081 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.081 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.081 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.081 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.081 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.081 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.082 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.082 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.082 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.082 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.082 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.082 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.082 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.082 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.083 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.083 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.083 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.083 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.083 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.083 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.083 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.083 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.084 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.084 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.084 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.084 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.084 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.084 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.084 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.084 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.084 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.085 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.085 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.085 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.085 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.085 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.085 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.085 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.085 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.086 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.086 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.086 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.086 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.086 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.086 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.086 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.086 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.086 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.087 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": { 2024-03-18T15:04:10.087 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "min_size", 2024-03-18T15:04:10.087 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.087 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.087 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.087 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.087 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.088 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.088 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.088 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default min_size for any created local pool", 2024-03-18T15:04:10.088 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T15:04:10.088 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.088 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.088 INFO:teuthology.orchestra.run.smithi076.stdout: "num_rep": { 2024-03-18T15:04:10.088 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "num_rep", 2024-03-18T15:04:10.089 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.089 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.089 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.089 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3", 2024-03-18T15:04:10.089 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.089 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.089 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.089 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default replica count for any created local pool", 2024-03-18T15:04:10.089 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.090 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.090 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.090 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": { 2024-03-18T15:04:10.090 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pg_num", 2024-03-18T15:04:10.090 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.090 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.090 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.091 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "128", 2024-03-18T15:04:10.091 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.091 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.091 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.091 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T15:04:10.091 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.091 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.091 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.092 INFO:teuthology.orchestra.run.smithi076.stdout: "prefix": { 2024-03-18T15:04:10.092 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prefix", 2024-03-18T15:04:10.092 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.092 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.092 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.092 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.092 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.092 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.092 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.093 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "name prefix for any created local pool", 2024-03-18T15:04:10.093 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.093 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.093 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.093 INFO:teuthology.orchestra.run.smithi076.stdout: "subtree": { 2024-03-18T15:04:10.093 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "subtree", 2024-03-18T15:04:10.093 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.093 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.094 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.094 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "rack", 2024-03-18T15:04:10.094 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.094 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.094 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.094 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T15:04:10.094 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T15:04:10.094 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.095 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.095 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.095 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.095 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mds_autoscaler", 2024-03-18T15:04:10.095 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.095 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.095 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.096 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.096 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.096 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.096 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.096 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.096 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.096 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.096 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.096 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.097 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.097 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.097 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.097 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.097 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.097 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.097 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.097 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.098 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.098 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.098 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.098 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.098 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.098 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.098 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.098 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.098 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.099 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.099 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.099 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.099 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.099 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.099 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.099 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.099 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.100 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.100 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.100 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.100 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.100 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.100 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.100 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.100 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.101 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.101 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.101 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.101 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.101 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.101 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.101 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.101 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.101 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.102 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.102 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.102 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.102 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.102 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.102 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.102 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.102 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.103 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.103 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.103 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.103 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.103 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.103 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.103 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.103 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.103 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.104 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.104 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mirroring", 2024-03-18T15:04:10.104 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.104 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.104 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.104 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.104 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.105 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.105 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.105 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.105 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.105 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.105 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.105 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.105 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.105 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.106 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.106 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.106 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.106 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.106 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.106 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.106 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.106 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.106 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.107 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.107 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.107 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.107 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.107 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.107 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.107 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.108 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.108 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.108 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.108 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.108 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.108 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.108 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.109 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.109 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.109 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.109 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.109 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.109 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.109 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.109 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.109 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.110 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.110 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.110 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.110 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.110 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.110 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.110 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.110 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.111 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.111 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.111 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.111 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.111 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.111 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.111 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.112 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.112 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.112 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.112 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.112 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.112 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.112 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.112 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.113 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.113 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.113 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "nfs", 2024-03-18T15:04:10.113 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.113 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.113 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.113 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.114 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.114 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.114 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.114 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.114 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.114 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.114 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.114 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.114 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.115 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.115 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.115 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.115 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.115 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.115 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.115 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.115 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.115 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.116 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.116 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.116 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.116 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.116 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.116 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.116 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.116 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.117 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.117 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.117 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.117 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.117 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.117 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.117 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.117 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.117 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.118 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.118 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.118 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.118 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.118 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.118 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.118 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.118 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.119 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.119 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.119 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.119 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.119 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.119 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.119 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.119 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.119 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.120 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.120 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.120 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.120 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.120 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.120 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.120 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.120 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.120 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.121 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.121 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.121 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.121 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.121 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.121 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.121 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "orchestrator", 2024-03-18T15:04:10.122 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.122 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.122 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.122 INFO:teuthology.orchestra.run.smithi076.stdout: "fail_fs": { 2024-03-18T15:04:10.122 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "fail_fs", 2024-03-18T15:04:10.122 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.122 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.123 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.123 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.123 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.123 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.123 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.123 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T15:04:10.123 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.123 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.123 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.124 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.124 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.124 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.124 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.124 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.124 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.124 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.124 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.124 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.124 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.125 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.125 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.125 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.125 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.125 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.125 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.125 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.125 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.125 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.126 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.126 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.126 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.126 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.126 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.126 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.126 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.127 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.127 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.127 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.127 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.127 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.127 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.127 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.127 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.128 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.128 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.128 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.128 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.128 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.128 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.128 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.128 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.128 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.128 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.129 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.129 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.129 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.129 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.129 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.129 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.129 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.129 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.129 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.130 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.130 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.130 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.130 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.130 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.130 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.130 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.131 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.131 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.131 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.131 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.131 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.131 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.131 INFO:teuthology.orchestra.run.smithi076.stdout: "orchestrator": { 2024-03-18T15:04:10.131 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "orchestrator", 2024-03-18T15:04:10.132 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.132 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.132 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.132 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.132 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.132 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.132 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.132 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm", 2024-03-18T15:04:10.133 INFO:teuthology.orchestra.run.smithi076.stdout: "rook", 2024-03-18T15:04:10.133 INFO:teuthology.orchestra.run.smithi076.stdout: "test_orchestrator" 2024-03-18T15:04:10.133 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.133 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Orchestrator backend", 2024-03-18T15:04:10.133 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.133 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.133 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.133 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.133 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.134 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "osd_perf_query", 2024-03-18T15:04:10.134 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.134 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.134 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.134 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.134 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.135 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.135 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.135 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.135 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.135 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.135 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.135 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.135 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.135 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.136 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.136 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.136 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.136 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.136 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.136 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.136 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.136 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.136 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.137 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.137 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.137 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.137 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.137 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.137 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.137 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.137 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.138 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.138 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.138 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.138 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.138 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.138 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.138 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.138 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.139 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.139 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.139 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.139 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.139 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.139 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.139 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.139 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.140 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.140 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.140 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.140 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.140 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.140 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.140 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.140 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.140 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.141 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.141 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.141 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.141 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.141 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.141 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.141 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.141 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.142 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.142 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.142 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.142 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.142 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.142 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.142 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.143 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "osd_support", 2024-03-18T15:04:10.143 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.143 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.143 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.143 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.143 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.143 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.143 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.143 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.144 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.144 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.144 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.144 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.144 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.144 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.144 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.144 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.144 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.145 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.145 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.145 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.145 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.145 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.145 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.145 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.145 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.146 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.146 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.146 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.146 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.146 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.146 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.146 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.146 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.146 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.147 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.147 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.147 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.147 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.147 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.147 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.147 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.147 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.147 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.148 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.148 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.148 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.148 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.148 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.148 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.148 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.148 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.149 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.149 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.149 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.149 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.149 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.149 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.149 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.149 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.150 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.150 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.150 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.150 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.150 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.150 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.150 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.150 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.151 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.151 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.151 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.151 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.151 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pg_autoscaler", 2024-03-18T15:04:10.151 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.151 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.152 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.152 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.152 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.152 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.152 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.152 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.152 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.152 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.153 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.153 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.153 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.153 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.153 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.153 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.153 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.153 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.153 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.154 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.154 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.154 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.154 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.154 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.154 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.154 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.154 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.155 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.155 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.155 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.155 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.155 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.155 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.155 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.155 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.155 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.156 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.156 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.156 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.156 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.156 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.156 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.156 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.157 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.157 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.157 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.157 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.157 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.157 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.157 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.157 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.157 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.158 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.158 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.158 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.158 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.158 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.158 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.158 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.158 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.159 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.159 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.159 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.159 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.159 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.159 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.159 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.159 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.159 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.160 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-18T15:04:10.160 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-18T15:04:10.160 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.160 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.160 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.160 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-18T15:04:10.160 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.161 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.161 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.161 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.161 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.161 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.161 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.161 INFO:teuthology.orchestra.run.smithi076.stdout: "threshold": { 2024-03-18T15:04:10.162 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "threshold", 2024-03-18T15:04:10.162 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:10.162 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.162 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.162 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3.0", 2024-03-18T15:04:10.162 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1.0", 2024-03-18T15:04:10.162 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.162 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.162 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "scaling threshold", 2024-03-18T15:04:10.163 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T15:04:10.163 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.163 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.163 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.163 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.163 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "progress", 2024-03-18T15:04:10.163 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.163 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.164 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.164 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_pg_recovery_event": { 2024-03-18T15:04:10.164 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_pg_recovery_event", 2024-03-18T15:04:10.164 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.164 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.164 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.164 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.164 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.164 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.165 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.165 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T15:04:10.165 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.165 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.165 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.165 INFO:teuthology.orchestra.run.smithi076.stdout: "enabled": { 2024-03-18T15:04:10.165 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enabled", 2024-03-18T15:04:10.165 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.166 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.166 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.166 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.166 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.166 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.166 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.166 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.166 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.166 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.167 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.167 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.167 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.167 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.167 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.167 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.167 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.167 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.167 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.168 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.168 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.168 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.168 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.168 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.168 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.168 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.168 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.168 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.169 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.169 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.169 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.169 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.169 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.169 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.169 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.169 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.170 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.170 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.170 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.170 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.170 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.170 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.170 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.170 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.170 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.171 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.171 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.171 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.171 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.171 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.171 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.171 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.171 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.171 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.172 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.172 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.172 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.172 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.172 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.172 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.172 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.172 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.172 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.173 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.173 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.173 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.173 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.173 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.173 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.173 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.173 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.174 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.174 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.174 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.174 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.174 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.174 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: "max_completed_events": { 2024-03-18T15:04:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_completed_events", 2024-03-18T15:04:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "50", 2024-03-18T15:04:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "number of past completed events to remember", 2024-03-18T15:04:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-18T15:04:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-18T15:04:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-18T15:04:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long the module is going to sleep", 2024-03-18T15:04:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prometheus", 2024-03-18T15:04:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "cache": { 2024-03-18T15:04:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cache", 2024-03-18T15:04:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: "exclude_perf_counters": { 2024-03-18T15:04:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "exclude_perf_counters", 2024-03-18T15:04:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T15:04:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T15:04:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_stats_pools": { 2024-03-18T15:04:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_stats_pools", 2024-03-18T15:04:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T15:04:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T15:04:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "300", 2024-03-18T15:04:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: "scrape_interval": { 2024-03-18T15:04:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "scrape_interval", 2024-03-18T15:04:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15.0", 2024-03-18T15:04:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-18T15:04:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-18T15:04:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "::", 2024-03-18T15:04:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T15:04:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-18T15:04:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-18T15:04:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "9283", 2024-03-18T15:04:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T15:04:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: "stale_cache_strategy": { 2024-03-18T15:04:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "stale_cache_strategy", 2024-03-18T15:04:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "log", 2024-03-18T15:04:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_behaviour": { 2024-03-18T15:04:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_behaviour", 2024-03-18T15:04:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "default", 2024-03-18T15:04:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: "default", 2024-03-18T15:04:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: "error" 2024-03-18T15:04:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_error_status_code": { 2024-03-18T15:04:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_error_status_code", 2024-03-18T15:04:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "500", 2024-03-18T15:04:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "400", 2024-03-18T15:04:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "599", 2024-03-18T15:04:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_support", 2024-03-18T15:04:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: "max_concurrent_snap_create": { 2024-03-18T15:04:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_concurrent_snap_create", 2024-03-18T15:04:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: "mirror_snapshot_schedule": { 2024-03-18T15:04:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T15:04:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: "trash_purge_schedule": { 2024-03-18T15:04:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "trash_purge_schedule", 2024-03-18T15:04:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "restful", 2024-03-18T15:04:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: "enable_auth": { 2024-03-18T15:04:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enable_auth", 2024-03-18T15:04:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: "key_file": { 2024-03-18T15:04:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "key_file", 2024-03-18T15:04:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.230 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.230 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.230 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.230 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.230 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.230 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.230 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.230 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.231 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.231 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.231 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.231 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.231 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.231 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.231 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.231 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-18T15:04:10.232 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-18T15:04:10.232 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.232 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.232 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.232 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.232 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.232 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.232 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.232 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-18T15:04:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-18T15:04:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rgw", 2024-03-18T15:04:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rook", 2024-03-18T15:04:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: "storage_class": { 2024-03-18T15:04:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "storage_class", 2024-03-18T15:04:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "local", 2024-03-18T15:04:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T15:04:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "selftest", 2024-03-18T15:04:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "roption1": { 2024-03-18T15:04:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "roption1", 2024-03-18T15:04:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: "roption2": { 2024-03-18T15:04:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "roption2", 2024-03-18T15:04:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "xyz", 2024-03-18T15:04:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption1": { 2024-03-18T15:04:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption1", 2024-03-18T15:04:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption2": { 2024-03-18T15:04:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption2", 2024-03-18T15:04:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption3": { 2024-03-18T15:04:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption3", 2024-03-18T15:04:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption4": { 2024-03-18T15:04:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption4", 2024-03-18T15:04:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption5": { 2024-03-18T15:04:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption5", 2024-03-18T15:04:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption6": { 2024-03-18T15:04:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption6", 2024-03-18T15:04:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption7": { 2024-03-18T15:04:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption7", 2024-03-18T15:04:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-18T15:04:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "42", 2024-03-18T15:04:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: "testkey": { 2024-03-18T15:04:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testkey", 2024-03-18T15:04:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: "testlkey": { 2024-03-18T15:04:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testlkey", 2024-03-18T15:04:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: "testnewline": { 2024-03-18T15:04:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testnewline", 2024-03-18T15:04:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snap_schedule", 2024-03-18T15:04:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_m_granularity": { 2024-03-18T15:04:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_m_granularity", 2024-03-18T15:04:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T15:04:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: "dump_on_update": { 2024-03-18T15:04:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "dump_on_update", 2024-03-18T15:04:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "dump database to debug log on update", 2024-03-18T15:04:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.292 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.292 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.292 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.292 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.292 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.292 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.292 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.292 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.292 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.293 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.293 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.293 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.293 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.293 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.293 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.293 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.293 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.293 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.294 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.294 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.294 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.294 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.294 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.294 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.294 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.295 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "stats", 2024-03-18T15:04:10.295 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.295 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.295 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.295 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.295 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.295 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.295 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.296 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.296 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.296 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.296 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.296 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.296 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.296 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.296 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.296 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.297 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.297 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.297 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.297 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.297 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.297 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.297 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.297 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.298 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.298 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.298 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.298 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.298 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.298 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.298 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.298 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.298 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.299 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.299 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.299 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.299 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.299 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.299 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.299 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.299 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.300 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.300 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.300 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.300 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.300 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.300 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.300 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.300 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.300 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.301 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.301 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.301 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.301 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.301 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.301 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.301 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.302 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.302 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.302 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.302 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.302 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.302 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.302 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.302 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.302 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.303 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.303 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.303 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.303 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.303 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.303 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "status", 2024-03-18T15:04:10.303 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.304 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.304 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.304 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.304 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.304 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.304 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.304 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.304 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.305 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.305 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.305 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.305 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.305 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.305 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.305 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.305 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.305 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.305 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.306 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.306 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.306 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.306 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.306 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.306 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.306 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.306 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.307 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.307 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.307 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.307 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.307 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.307 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.307 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.307 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.307 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.308 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.308 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.308 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.308 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.308 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.308 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.308 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.308 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.308 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.309 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.309 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.309 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.309 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.309 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.309 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.309 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.309 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.309 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.310 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.310 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.310 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.310 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.310 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.310 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.310 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.310 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.311 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.311 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.311 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.311 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.311 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.311 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.311 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.311 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.312 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.312 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.312 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.312 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "telegraf", 2024-03-18T15:04:10.312 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.312 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.312 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.313 INFO:teuthology.orchestra.run.smithi076.stdout: "address": { 2024-03-18T15:04:10.313 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "address", 2024-03-18T15:04:10.313 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.313 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.313 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.313 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T15:04:10.313 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.313 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.313 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.314 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.314 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.314 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.314 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.314 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-18T15:04:10.314 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-18T15:04:10.314 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.315 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.315 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.315 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15", 2024-03-18T15:04:10.315 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.315 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.315 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.315 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.315 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.316 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.316 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.316 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.316 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.316 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.316 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.316 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.316 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.317 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.317 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.317 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.317 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.317 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.317 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.317 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.317 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.318 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.318 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.318 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.318 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.318 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.318 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.318 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.319 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.319 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.319 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.319 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.319 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.319 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.319 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.319 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.319 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.320 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.320 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.320 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.320 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.320 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.320 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.320 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.320 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.321 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.321 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.321 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.321 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.321 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.321 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.321 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.321 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.321 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.322 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.322 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.322 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.322 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.322 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.322 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.322 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.323 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.323 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.323 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.323 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.323 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.323 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.323 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.323 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.323 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.324 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.324 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.324 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.324 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.324 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.324 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "telemetry", 2024-03-18T15:04:10.325 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.325 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.325 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.325 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_basic": { 2024-03-18T15:04:10.325 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_basic", 2024-03-18T15:04:10.325 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.325 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.325 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.325 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.326 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.326 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.326 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.326 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T15:04:10.326 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.326 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.326 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.327 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_crash": { 2024-03-18T15:04:10.327 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_crash", 2024-03-18T15:04:10.327 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.327 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.327 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.327 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.327 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.327 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.328 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.328 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T15:04:10.328 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.328 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.328 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.328 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_device": { 2024-03-18T15:04:10.328 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_device", 2024-03-18T15:04:10.328 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.329 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.329 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.329 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.329 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.329 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.329 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.329 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T15:04:10.329 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.329 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.330 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.330 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_ident": { 2024-03-18T15:04:10.330 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_ident", 2024-03-18T15:04:10.330 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.330 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.330 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.330 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.330 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.331 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.331 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.331 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T15:04:10.331 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.331 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.331 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.331 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_perf": { 2024-03-18T15:04:10.331 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_perf", 2024-03-18T15:04:10.332 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.332 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.332 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.332 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.332 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.332 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.332 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.332 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T15:04:10.332 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.333 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.333 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.333 INFO:teuthology.orchestra.run.smithi076.stdout: "contact": { 2024-03-18T15:04:10.333 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "contact", 2024-03-18T15:04:10.333 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.333 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.333 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.333 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.334 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.334 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.334 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.334 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.334 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.334 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.334 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.335 INFO:teuthology.orchestra.run.smithi076.stdout: "description": { 2024-03-18T15:04:10.335 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "description", 2024-03-18T15:04:10.335 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.335 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.335 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.335 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.335 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.335 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.336 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.336 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.336 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.336 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.336 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.336 INFO:teuthology.orchestra.run.smithi076.stdout: "device_url": { 2024-03-18T15:04:10.336 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_url", 2024-03-18T15:04:10.336 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.337 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.337 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.337 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T15:04:10.337 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.337 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.337 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.337 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.337 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.337 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.338 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.338 INFO:teuthology.orchestra.run.smithi076.stdout: "enabled": { 2024-03-18T15:04:10.338 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enabled", 2024-03-18T15:04:10.338 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.338 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.338 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.338 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.339 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.339 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.339 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.339 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.339 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.339 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.339 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.339 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-18T15:04:10.340 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-18T15:04:10.340 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.340 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.340 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.340 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "24", 2024-03-18T15:04:10.340 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "8", 2024-03-18T15:04:10.340 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.340 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.340 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.341 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.341 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.341 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.341 INFO:teuthology.orchestra.run.smithi076.stdout: "last_opt_revision": { 2024-03-18T15:04:10.341 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "last_opt_revision", 2024-03-18T15:04:10.341 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.341 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.341 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.342 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1", 2024-03-18T15:04:10.342 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.342 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.342 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.342 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.342 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.342 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.342 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.343 INFO:teuthology.orchestra.run.smithi076.stdout: "leaderboard": { 2024-03-18T15:04:10.343 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "leaderboard", 2024-03-18T15:04:10.343 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.343 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.343 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.343 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.343 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.343 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.344 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.344 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.344 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.344 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.344 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.344 INFO:teuthology.orchestra.run.smithi076.stdout: "leaderboard_description": { 2024-03-18T15:04:10.344 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "leaderboard_description", 2024-03-18T15:04:10.344 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.344 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.345 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.345 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.345 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.345 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.345 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.345 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.345 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.345 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.345 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.346 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.346 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.346 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.346 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.346 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.346 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.346 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.346 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.347 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.347 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.347 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.347 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.347 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.347 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.347 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.347 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.347 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.348 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.348 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.348 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.348 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.348 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.348 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.348 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.348 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.349 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.349 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.349 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.349 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.349 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.349 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.349 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.349 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.350 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.350 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.350 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.350 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.350 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.350 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.350 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.350 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.350 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.350 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.351 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.351 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.351 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.351 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.351 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.351 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.351 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.351 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.351 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.352 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.352 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.352 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.352 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.352 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.352 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.352 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.352 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.353 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.353 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.353 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.353 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.353 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.353 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.353 INFO:teuthology.orchestra.run.smithi076.stdout: "organization": { 2024-03-18T15:04:10.353 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "organization", 2024-03-18T15:04:10.354 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.354 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.354 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.354 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.354 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.354 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.354 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.354 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.355 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.355 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.355 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.355 INFO:teuthology.orchestra.run.smithi076.stdout: "proxy": { 2024-03-18T15:04:10.355 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "proxy", 2024-03-18T15:04:10.355 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.355 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.355 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.355 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.356 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.356 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.356 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.356 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.356 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.356 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.356 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.356 INFO:teuthology.orchestra.run.smithi076.stdout: "url": { 2024-03-18T15:04:10.357 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "url", 2024-03-18T15:04:10.357 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.357 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.357 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.357 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T15:04:10.357 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.357 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.357 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.358 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.358 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.358 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.358 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.358 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.358 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.358 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "test_orchestrator", 2024-03-18T15:04:10.359 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.359 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.359 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.359 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.359 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.359 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.359 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.359 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.359 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.360 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.360 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.360 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.360 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.360 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.360 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.360 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.360 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.360 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.361 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.361 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.361 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.361 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.361 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.361 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.361 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.361 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.362 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.362 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.362 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.362 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.362 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.362 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.362 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.362 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.362 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.363 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.363 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.363 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.363 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.363 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.363 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.363 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.364 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.364 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.364 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.364 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.364 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.364 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.364 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.364 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.364 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.364 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.365 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.365 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.365 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.365 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.365 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.365 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.365 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.365 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.366 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.366 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.366 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.366 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.366 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.366 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.366 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.366 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.367 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.367 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.367 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.367 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "volumes", 2024-03-18T15:04:10.367 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.367 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.367 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.368 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.368 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.368 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.368 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.368 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.368 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.368 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.368 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.368 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.369 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.369 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.369 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.369 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.369 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.369 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.369 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.369 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.369 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.370 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.370 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.370 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.370 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.370 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.370 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.370 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.371 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.371 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.371 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.371 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.371 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.371 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.371 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.371 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.371 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.372 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.372 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.372 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.372 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.372 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.372 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.372 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.372 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.372 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.373 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.373 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.373 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.373 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.373 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.373 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.373 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.373 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.373 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.374 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.374 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.374 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.374 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.374 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.374 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.374 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.375 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.375 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.375 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.375 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.375 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.375 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.375 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.375 INFO:teuthology.orchestra.run.smithi076.stdout: "max_concurrent_clones": { 2024-03-18T15:04:10.375 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_concurrent_clones", 2024-03-18T15:04:10.376 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.376 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.376 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.376 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "4", 2024-03-18T15:04:10.376 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.376 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.376 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.376 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T15:04:10.376 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.377 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.377 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.377 INFO:teuthology.orchestra.run.smithi076.stdout: "periodic_async_work": { 2024-03-18T15:04:10.377 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "periodic_async_work", 2024-03-18T15:04:10.377 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.377 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.377 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.378 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.378 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.378 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.378 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.378 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Periodically check for async work", 2024-03-18T15:04:10.378 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.378 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.378 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.379 INFO:teuthology.orchestra.run.smithi076.stdout: "snapshot_clone_delay": { 2024-03-18T15:04:10.379 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snapshot_clone_delay", 2024-03-18T15:04:10.381 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.381 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.381 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.382 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-18T15:04:10.382 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.382 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.382 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.382 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T15:04:10.382 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.382 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.382 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.383 INFO:teuthology.orchestra.run.smithi076.stdout: "snapshot_clone_no_wait": { 2024-03-18T15:04:10.383 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T15:04:10.383 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.383 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.383 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.383 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.383 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.383 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.383 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.384 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T15:04:10.384 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.384 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.384 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.384 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.384 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.384 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix", 2024-03-18T15:04:10.385 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.385 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.385 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.385 INFO:teuthology.orchestra.run.smithi076.stdout: "discovery_interval": { 2024-03-18T15:04:10.385 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "discovery_interval", 2024-03-18T15:04:10.385 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-18T15:04:10.385 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.385 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.386 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "100", 2024-03-18T15:04:10.386 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.386 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.386 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.386 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.386 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.386 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.386 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.387 INFO:teuthology.orchestra.run.smithi076.stdout: "identifier": { 2024-03-18T15:04:10.387 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "identifier", 2024-03-18T15:04:10.387 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.387 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.387 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.387 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.387 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.387 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.388 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.388 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.388 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.388 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.388 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.388 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-18T15:04:10.388 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-18T15:04:10.388 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.389 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.389 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.389 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-18T15:04:10.389 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.389 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.389 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.389 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.389 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.389 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.390 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.390 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.390 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.390 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.390 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.390 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.390 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.390 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.391 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.391 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.391 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.391 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.391 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.391 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.391 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.391 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.391 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.392 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.392 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.392 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.392 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.392 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.392 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.392 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.393 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.393 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.393 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.393 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.393 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.393 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.393 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.393 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.393 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.394 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.394 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.394 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.394 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.394 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.394 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.394 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.395 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.395 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.395 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.395 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.395 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.395 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.395 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.395 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.395 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.396 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.396 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.396 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.396 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.396 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.396 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.396 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.396 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.397 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.397 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.397 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.397 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.397 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.397 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.397 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.397 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.397 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.397 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.398 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_host": { 2024-03-18T15:04:10.398 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_host", 2024-03-18T15:04:10.398 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.398 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.398 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.398 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.398 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.399 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.399 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.399 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.399 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.399 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.399 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.399 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_port": { 2024-03-18T15:04:10.399 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_port", 2024-03-18T15:04:10.400 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.400 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.400 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.400 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10051", 2024-03-18T15:04:10.400 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.400 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.400 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.400 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.400 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.401 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.401 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.401 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_sender": { 2024-03-18T15:04:10.401 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_sender", 2024-03-18T15:04:10.401 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.401 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.401 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.402 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T15:04:10.402 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.402 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.402 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.402 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.402 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.402 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.402 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.402 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.403 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.403 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.403 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-18T15:04:10.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.403 INFO:teuthology.orchestra.run.smithi076.stdout: "gid": 25289, 2024-03-18T15:04:10.403 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "y", 2024-03-18T15:04:10.403 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_features": 4540701547738038271, 2024-03-18T15:04:10.403 INFO:teuthology.orchestra.run.smithi076.stdout: "available_modules": [ 2024-03-18T15:04:10.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.404 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "alerts", 2024-03-18T15:04:10.404 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.404 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.404 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.404 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-18T15:04:10.404 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-18T15:04:10.404 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.404 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.405 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.405 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-18T15:04:10.405 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.405 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.405 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.405 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T15:04:10.405 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.405 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.405 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.406 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.406 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.406 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.406 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.406 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.406 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.406 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.407 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.407 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.407 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.407 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.407 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.407 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.407 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.407 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.407 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.408 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.408 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.408 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.408 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.408 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.408 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.408 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.408 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.409 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.409 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.409 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.409 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.409 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.409 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.409 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.409 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.409 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.410 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.410 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.410 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.410 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.410 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.410 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.410 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.410 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.411 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.411 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.411 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.411 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.411 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.411 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.411 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.411 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.411 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.412 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.412 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.412 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.412 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.412 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.412 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.412 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.412 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.413 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.413 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.413 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.413 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.413 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.413 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.413 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.413 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.414 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_destination": { 2024-03-18T15:04:10.414 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_destination", 2024-03-18T15:04:10.414 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.414 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.414 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.414 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.414 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.414 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.414 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.415 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Email address to send alerts to", 2024-03-18T15:04:10.415 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.415 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.415 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.415 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_from_name": { 2024-03-18T15:04:10.415 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_from_name", 2024-03-18T15:04:10.415 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.415 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.416 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.416 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "Ceph", 2024-03-18T15:04:10.416 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.416 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.416 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.416 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Email From: name", 2024-03-18T15:04:10.416 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.416 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.416 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.417 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_host": { 2024-03-18T15:04:10.417 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_host", 2024-03-18T15:04:10.417 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.417 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.417 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.417 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.417 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.417 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.418 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.418 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP server", 2024-03-18T15:04:10.418 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.418 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.418 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.418 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_password": { 2024-03-18T15:04:10.418 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_password", 2024-03-18T15:04:10.418 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.419 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.419 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.419 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.419 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.419 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.419 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.419 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Password to authenticate with", 2024-03-18T15:04:10.419 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.419 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.420 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.420 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_port": { 2024-03-18T15:04:10.420 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_port", 2024-03-18T15:04:10.420 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.420 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.420 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.420 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "465", 2024-03-18T15:04:10.420 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.421 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.421 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.421 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP port", 2024-03-18T15:04:10.421 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.421 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.421 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.421 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_sender": { 2024-03-18T15:04:10.422 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_sender", 2024-03-18T15:04:10.422 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.422 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.422 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.422 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.422 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.422 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.422 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.422 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP envelope sender", 2024-03-18T15:04:10.423 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.423 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.423 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.423 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_ssl": { 2024-03-18T15:04:10.423 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_ssl", 2024-03-18T15:04:10.423 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.423 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.423 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.424 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.424 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.424 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.424 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.424 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T15:04:10.424 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.424 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.424 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.425 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_user": { 2024-03-18T15:04:10.425 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_user", 2024-03-18T15:04:10.425 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.425 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.425 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.425 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.425 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.425 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.425 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.426 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "User to authenticate as", 2024-03-18T15:04:10.426 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.426 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.426 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.426 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.426 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.427 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "balancer", 2024-03-18T15:04:10.427 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.427 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.427 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.427 INFO:teuthology.orchestra.run.smithi076.stdout: "active": { 2024-03-18T15:04:10.427 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "active", 2024-03-18T15:04:10.427 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.427 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.427 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.428 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.428 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.428 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.428 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.428 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T15:04:10.428 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.428 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.428 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.429 INFO:teuthology.orchestra.run.smithi076.stdout: "begin_time": { 2024-03-18T15:04:10.429 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "begin_time", 2024-03-18T15:04:10.429 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.429 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.429 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.429 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0000", 2024-03-18T15:04:10.429 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.429 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.429 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.430 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T15:04:10.430 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T15:04:10.430 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.430 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.430 INFO:teuthology.orchestra.run.smithi076.stdout: "begin_weekday": { 2024-03-18T15:04:10.430 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "begin_weekday", 2024-03-18T15:04:10.430 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-18T15:04:10.430 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.431 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.431 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-18T15:04:10.431 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0", 2024-03-18T15:04:10.431 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "6", 2024-03-18T15:04:10.431 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.431 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T15:04:10.431 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T15:04:10.431 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.431 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.432 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_max_iterations": { 2024-03-18T15:04:10.432 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_max_iterations", 2024-03-18T15:04:10.432 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-18T15:04:10.432 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.432 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.432 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "25", 2024-03-18T15:04:10.432 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-18T15:04:10.432 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "250", 2024-03-18T15:04:10.433 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.433 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T15:04:10.433 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.433 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.433 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.433 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_metrics": { 2024-03-18T15:04:10.433 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_metrics", 2024-03-18T15:04:10.433 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.434 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.434 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.434 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T15:04:10.434 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.434 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.434 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.434 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T15:04:10.434 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T15:04:10.435 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.435 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.435 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_step": { 2024-03-18T15:04:10.435 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_step", 2024-03-18T15:04:10.435 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:10.435 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.435 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.435 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0.5", 2024-03-18T15:04:10.436 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0.001", 2024-03-18T15:04:10.436 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "0.999", 2024-03-18T15:04:10.436 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.436 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "aggressiveness of optimization", 2024-03-18T15:04:10.436 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T15:04:10.436 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.436 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.437 INFO:teuthology.orchestra.run.smithi076.stdout: "end_time": { 2024-03-18T15:04:10.437 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "end_time", 2024-03-18T15:04:10.437 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.437 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.437 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.437 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "2359", 2024-03-18T15:04:10.437 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.437 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.437 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.438 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T15:04:10.438 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T15:04:10.438 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.438 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.438 INFO:teuthology.orchestra.run.smithi076.stdout: "end_weekday": { 2024-03-18T15:04:10.438 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "end_weekday", 2024-03-18T15:04:10.438 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-18T15:04:10.439 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.439 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.439 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-18T15:04:10.439 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0", 2024-03-18T15:04:10.439 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "6", 2024-03-18T15:04:10.439 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.439 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T15:04:10.439 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T15:04:10.439 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.440 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.440 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.440 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.440 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.440 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.440 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.440 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.440 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.441 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.441 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.441 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.441 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.441 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.441 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.441 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.441 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.441 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.442 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.442 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.442 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.442 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.442 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.442 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.442 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.443 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.443 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.443 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.443 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.443 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.443 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.443 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.443 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.443 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.444 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.444 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.444 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.444 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.444 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.444 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.444 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.444 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.445 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.445 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.445 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.445 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.445 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.445 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.445 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.445 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.446 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.446 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.446 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.446 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.446 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.446 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.446 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.447 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.447 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.447 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.447 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.447 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.447 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.447 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.447 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.447 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.448 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.448 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.448 INFO:teuthology.orchestra.run.smithi076.stdout: "min_score": { 2024-03-18T15:04:10.448 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "min_score", 2024-03-18T15:04:10.448 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:10.448 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.448 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.448 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-18T15:04:10.449 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.449 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.449 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.449 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T15:04:10.449 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.449 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.449 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.449 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": { 2024-03-18T15:04:10.449 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mode", 2024-03-18T15:04:10.450 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.450 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.450 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.450 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "upmap", 2024-03-18T15:04:10.450 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.450 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.450 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.450 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-compat", 2024-03-18T15:04:10.451 INFO:teuthology.orchestra.run.smithi076.stdout: "none", 2024-03-18T15:04:10.451 INFO:teuthology.orchestra.run.smithi076.stdout: "read", 2024-03-18T15:04:10.451 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap", 2024-03-18T15:04:10.451 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap-read" 2024-03-18T15:04:10.451 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.451 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Balancer mode", 2024-03-18T15:04:10.451 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.451 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.451 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.452 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_ids": { 2024-03-18T15:04:10.452 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pool_ids", 2024-03-18T15:04:10.452 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.452 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.452 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.452 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.452 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.452 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.453 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.453 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T15:04:10.453 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.453 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.453 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.453 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-18T15:04:10.453 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-18T15:04:10.453 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.454 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.454 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.454 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-18T15:04:10.454 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.454 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.454 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.454 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T15:04:10.454 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.455 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.455 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.455 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap_max_deviation": { 2024-03-18T15:04:10.455 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "upmap_max_deviation", 2024-03-18T15:04:10.455 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.455 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.455 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.455 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-18T15:04:10.456 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-18T15:04:10.456 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.456 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.456 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T15:04:10.456 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T15:04:10.456 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.456 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.457 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap_max_optimizations": { 2024-03-18T15:04:10.457 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "upmap_max_optimizations", 2024-03-18T15:04:10.457 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-18T15:04:10.457 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.457 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.457 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:10.457 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.457 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.457 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.458 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T15:04:10.458 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.458 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.458 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.458 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.458 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.458 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cephadm", 2024-03-18T15:04:10.459 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.459 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.459 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.459 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_down_multiplier": { 2024-03-18T15:04:10.459 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_down_multiplier", 2024-03-18T15:04:10.459 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:10.459 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.459 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.459 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3.0", 2024-03-18T15:04:10.460 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.460 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.460 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.460 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T15:04:10.460 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.460 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.460 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.461 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_refresh_rate": { 2024-03-18T15:04:10.461 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_refresh_rate", 2024-03-18T15:04:10.461 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.461 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.461 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.461 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "20", 2024-03-18T15:04:10.461 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.461 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.461 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.462 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T15:04:10.462 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.462 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.462 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.462 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_starting_port": { 2024-03-18T15:04:10.462 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_starting_port", 2024-03-18T15:04:10.462 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.462 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.463 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.463 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "4721", 2024-03-18T15:04:10.463 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.463 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.463 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.463 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T15:04:10.463 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.463 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.463 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.464 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_ptrace": { 2024-03-18T15:04:10.464 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_ptrace", 2024-03-18T15:04:10.464 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.464 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.464 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.464 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.464 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.464 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.465 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.465 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T15:04:10.465 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T15:04:10.465 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.465 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.465 INFO:teuthology.orchestra.run.smithi076.stdout: "autotune_interval": { 2024-03-18T15:04:10.465 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "autotune_interval", 2024-03-18T15:04:10.466 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.466 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.466 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.466 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-18T15:04:10.466 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.466 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.466 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.466 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T15:04:10.467 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.467 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.467 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.467 INFO:teuthology.orchestra.run.smithi076.stdout: "autotune_memory_target_ratio": { 2024-03-18T15:04:10.467 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T15:04:10.467 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:10.467 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.467 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.468 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0.7", 2024-03-18T15:04:10.468 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.468 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.468 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.468 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T15:04:10.468 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.468 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.468 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.469 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm_log_destination": { 2024-03-18T15:04:10.469 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cephadm_log_destination", 2024-03-18T15:04:10.469 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.469 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.469 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.469 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.469 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.469 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.469 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.469 INFO:teuthology.orchestra.run.smithi076.stdout: "file", 2024-03-18T15:04:10.470 INFO:teuthology.orchestra.run.smithi076.stdout: "file,syslog", 2024-03-18T15:04:10.470 INFO:teuthology.orchestra.run.smithi076.stdout: "syslog" 2024-03-18T15:04:10.470 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.470 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T15:04:10.470 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.470 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.470 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.470 INFO:teuthology.orchestra.run.smithi076.stdout: "cgroups_split": { 2024-03-18T15:04:10.471 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cgroups_split", 2024-03-18T15:04:10.471 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.471 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.471 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.471 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.471 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.471 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.471 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.471 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T15:04:10.472 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.472 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.472 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.472 INFO:teuthology.orchestra.run.smithi076.stdout: "config_checks_enabled": { 2024-03-18T15:04:10.472 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_checks_enabled", 2024-03-18T15:04:10.472 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.472 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.473 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.473 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.473 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.473 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.473 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.473 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T15:04:10.473 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.473 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.473 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.474 INFO:teuthology.orchestra.run.smithi076.stdout: "config_dashboard": { 2024-03-18T15:04:10.474 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_dashboard", 2024-03-18T15:04:10.474 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.474 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.474 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.474 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.474 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.475 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.475 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.475 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T15:04:10.475 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.475 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.475 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.475 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_alertmanager": { 2024-03-18T15:04:10.475 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_alertmanager", 2024-03-18T15:04:10.476 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.476 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.476 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.476 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T15:04:10.476 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.476 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.476 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.476 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-18T15:04:10.477 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.477 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.477 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.477 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_base": { 2024-03-18T15:04:10.477 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_base", 2024-03-18T15:04:10.477 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.477 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.477 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.478 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T15:04:10.478 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.478 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.478 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.478 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Container image name, without the tag", 2024-03-18T15:04:10.478 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.478 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.478 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.479 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_elasticsearch": { 2024-03-18T15:04:10.479 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_elasticsearch", 2024-03-18T15:04:10.479 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.479 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.479 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.479 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T15:04:10.479 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.479 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.480 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.480 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "elasticsearch container image", 2024-03-18T15:04:10.480 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.480 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.480 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.480 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_grafana": { 2024-03-18T15:04:10.480 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_grafana", 2024-03-18T15:04:10.481 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.481 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.481 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.481 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T15:04:10.481 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.481 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.481 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.481 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-18T15:04:10.481 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.482 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.482 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.482 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_haproxy": { 2024-03-18T15:04:10.482 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_haproxy", 2024-03-18T15:04:10.482 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.482 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.482 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.483 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T15:04:10.483 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.483 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.483 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.483 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "HAproxy container image", 2024-03-18T15:04:10.483 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.483 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.483 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.484 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_agent": { 2024-03-18T15:04:10.484 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_agent", 2024-03-18T15:04:10.484 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.484 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.484 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.484 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T15:04:10.484 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.485 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.485 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.485 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger agent container image", 2024-03-18T15:04:10.485 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.485 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.485 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.485 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_collector": { 2024-03-18T15:04:10.486 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_collector", 2024-03-18T15:04:10.486 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.486 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.486 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.486 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T15:04:10.486 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.486 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.486 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.487 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger collector container image", 2024-03-18T15:04:10.487 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.487 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.487 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.487 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_query": { 2024-03-18T15:04:10.487 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_query", 2024-03-18T15:04:10.487 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.487 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.488 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.488 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T15:04:10.488 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.488 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.488 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.488 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger query container image", 2024-03-18T15:04:10.488 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.489 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.489 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.489 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_keepalived": { 2024-03-18T15:04:10.489 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_keepalived", 2024-03-18T15:04:10.489 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.489 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.489 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.489 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T15:04:10.490 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.490 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.490 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.490 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Keepalived container image", 2024-03-18T15:04:10.490 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.490 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.490 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.491 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_loki": { 2024-03-18T15:04:10.491 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_loki", 2024-03-18T15:04:10.491 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.491 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.491 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.491 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T15:04:10.491 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.491 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.491 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.492 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Loki container image", 2024-03-18T15:04:10.492 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.492 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.492 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.492 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_node_exporter": { 2024-03-18T15:04:10.492 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_node_exporter", 2024-03-18T15:04:10.492 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.492 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.493 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.493 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T15:04:10.493 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.493 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.493 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.493 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-18T15:04:10.493 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.493 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.494 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.494 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_nvmeof": { 2024-03-18T15:04:10.494 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_nvmeof", 2024-03-18T15:04:10.494 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.494 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.494 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.494 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T15:04:10.495 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.495 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.495 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.495 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Nvme-of container image", 2024-03-18T15:04:10.495 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.495 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.495 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.495 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_prometheus": { 2024-03-18T15:04:10.496 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_prometheus", 2024-03-18T15:04:10.496 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.496 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.496 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.496 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T15:04:10.496 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.496 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.496 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.496 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-18T15:04:10.497 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.497 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.497 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.497 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_promtail": { 2024-03-18T15:04:10.497 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_promtail", 2024-03-18T15:04:10.497 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.497 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.498 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.498 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T15:04:10.498 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.498 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.498 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.498 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Promtail container image", 2024-03-18T15:04:10.498 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.498 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.499 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.499 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_snmp_gateway": { 2024-03-18T15:04:10.499 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_snmp_gateway", 2024-03-18T15:04:10.499 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.499 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.499 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.499 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T15:04:10.499 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.500 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.500 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.500 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SNMP Gateway container image", 2024-03-18T15:04:10.500 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.500 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.500 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.500 INFO:teuthology.orchestra.run.smithi076.stdout: "container_init": { 2024-03-18T15:04:10.500 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_init", 2024-03-18T15:04:10.501 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.501 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.501 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.501 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.501 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.501 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.501 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.501 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T15:04:10.501 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.502 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.502 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.502 INFO:teuthology.orchestra.run.smithi076.stdout: "daemon_cache_timeout": { 2024-03-18T15:04:10.502 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "daemon_cache_timeout", 2024-03-18T15:04:10.502 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.502 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.502 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.502 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-18T15:04:10.503 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.503 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.503 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.503 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T15:04:10.503 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.503 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.503 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.503 INFO:teuthology.orchestra.run.smithi076.stdout: "default_cephadm_command_timeout": { 2024-03-18T15:04:10.504 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T15:04:10.504 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.504 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.504 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.504 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "900", 2024-03-18T15:04:10.504 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.504 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.504 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.505 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T15:04:10.505 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.505 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.505 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.505 INFO:teuthology.orchestra.run.smithi076.stdout: "default_registry": { 2024-03-18T15:04:10.505 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "default_registry", 2024-03-18T15:04:10.505 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.505 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.506 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.506 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io", 2024-03-18T15:04:10.506 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.506 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.506 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.506 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T15:04:10.506 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.506 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.507 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.507 INFO:teuthology.orchestra.run.smithi076.stdout: "device_cache_timeout": { 2024-03-18T15:04:10.507 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_cache_timeout", 2024-03-18T15:04:10.507 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.507 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.507 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.507 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1800", 2024-03-18T15:04:10.507 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.508 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.508 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.508 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache device inventory", 2024-03-18T15:04:10.508 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.508 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.508 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.508 INFO:teuthology.orchestra.run.smithi076.stdout: "device_enhanced_scan": { 2024-03-18T15:04:10.508 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_enhanced_scan", 2024-03-18T15:04:10.509 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.509 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.509 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.509 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.509 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.509 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.509 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.509 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T15:04:10.509 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.510 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.510 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.510 INFO:teuthology.orchestra.run.smithi076.stdout: "facts_cache_timeout": { 2024-03-18T15:04:10.510 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "facts_cache_timeout", 2024-03-18T15:04:10.510 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.510 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.510 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.511 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-18T15:04:10.511 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.511 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.511 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.511 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache host facts data", 2024-03-18T15:04:10.511 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.511 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.511 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.511 INFO:teuthology.orchestra.run.smithi076.stdout: "host_check_interval": { 2024-03-18T15:04:10.512 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "host_check_interval", 2024-03-18T15:04:10.512 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.512 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.512 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.512 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-18T15:04:10.512 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.512 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.512 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.513 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to perform a host check", 2024-03-18T15:04:10.513 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.513 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.513 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.513 INFO:teuthology.orchestra.run.smithi076.stdout: "hw_monitoring": { 2024-03-18T15:04:10.513 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "hw_monitoring", 2024-03-18T15:04:10.513 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.513 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.514 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.514 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.514 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.514 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.514 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.514 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T15:04:10.514 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.514 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.514 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.515 INFO:teuthology.orchestra.run.smithi076.stdout: "inventory_list_all": { 2024-03-18T15:04:10.515 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "inventory_list_all", 2024-03-18T15:04:10.515 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.515 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.515 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.515 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.515 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.516 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.516 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.516 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T15:04:10.516 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.516 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.516 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.516 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.516 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.517 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.517 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.517 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.517 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.517 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.517 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.517 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.517 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.517 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.518 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.518 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.518 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.518 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.518 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.518 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.518 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.518 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.519 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.519 INFO:teuthology.orchestra.run.smithi076.stdout: "log_refresh_metadata": { 2024-03-18T15:04:10.519 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_refresh_metadata", 2024-03-18T15:04:10.519 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.519 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.519 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.519 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.519 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.520 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.520 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.520 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T15:04:10.520 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.520 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.520 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.520 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.520 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.521 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.521 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.521 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.521 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.521 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.521 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.521 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.521 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T15:04:10.522 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.522 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.522 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.522 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.522 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.522 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.522 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.522 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.523 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.523 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.523 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.523 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.523 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.523 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.523 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.523 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.524 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.524 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.524 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.524 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.524 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.524 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.524 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.524 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.524 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.525 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.525 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.525 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.525 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.525 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.525 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.525 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.525 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.525 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.526 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.526 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.526 INFO:teuthology.orchestra.run.smithi076.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T15:04:10.526 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T15:04:10.526 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.526 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.526 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.526 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.527 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.527 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.527 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.527 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T15:04:10.527 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.527 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.527 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.528 INFO:teuthology.orchestra.run.smithi076.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T15:04:10.528 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T15:04:10.528 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.528 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.528 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.528 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "*", 2024-03-18T15:04:10.528 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.528 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.528 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.529 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T15:04:10.529 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.529 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.529 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.529 INFO:teuthology.orchestra.run.smithi076.stdout: "max_count_per_host": { 2024-03-18T15:04:10.529 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_count_per_host", 2024-03-18T15:04:10.529 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.529 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.530 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.530 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:10.530 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.530 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.530 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.530 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "max number of daemons per service per host", 2024-03-18T15:04:10.530 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.530 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.530 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.531 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd_draining_count": { 2024-03-18T15:04:10.531 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_osd_draining_count", 2024-03-18T15:04:10.531 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.531 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.531 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.531 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:10.531 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.532 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.532 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.532 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T15:04:10.532 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.532 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.532 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.532 INFO:teuthology.orchestra.run.smithi076.stdout: "migration_current": { 2024-03-18T15:04:10.532 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "migration_current", 2024-03-18T15:04:10.533 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.533 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.533 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.533 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.533 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.533 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.533 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.533 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "internal - do not modify", 2024-03-18T15:04:10.533 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.534 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.534 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.534 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": { 2024-03-18T15:04:10.534 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mode", 2024-03-18T15:04:10.534 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.534 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.534 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.534 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "root", 2024-03-18T15:04:10.535 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.535 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.535 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.535 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm-package", 2024-03-18T15:04:10.535 INFO:teuthology.orchestra.run.smithi076.stdout: "root" 2024-03-18T15:04:10.535 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.535 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T15:04:10.535 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.535 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.536 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.536 INFO:teuthology.orchestra.run.smithi076.stdout: "oob_default_addr": { 2024-03-18T15:04:10.536 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "oob_default_addr", 2024-03-18T15:04:10.536 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.536 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.536 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.536 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "169.254.1.1", 2024-03-18T15:04:10.536 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.537 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.537 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.537 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T15:04:10.537 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.537 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.537 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.537 INFO:teuthology.orchestra.run.smithi076.stdout: "prometheus_alerts_path": { 2024-03-18T15:04:10.537 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prometheus_alerts_path", 2024-03-18T15:04:10.538 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.538 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.538 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.538 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T15:04:10.538 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.538 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.538 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.538 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T15:04:10.539 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.539 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.539 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.539 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_insecure": { 2024-03-18T15:04:10.539 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_insecure", 2024-03-18T15:04:10.539 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.539 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.539 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.540 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.540 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.540 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.540 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.540 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T15:04:10.540 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.540 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.540 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.541 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_password": { 2024-03-18T15:04:10.541 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_password", 2024-03-18T15:04:10.541 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.541 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.541 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.541 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.541 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.541 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.542 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.542 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T15:04:10.542 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.542 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.542 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.542 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_url": { 2024-03-18T15:04:10.542 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_url", 2024-03-18T15:04:10.543 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.543 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.543 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.543 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.543 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.543 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.543 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.543 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T15:04:10.543 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.544 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.544 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.544 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_username": { 2024-03-18T15:04:10.544 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_username", 2024-03-18T15:04:10.544 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.544 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.544 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.545 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.545 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.545 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.545 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.545 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T15:04:10.545 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.545 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.545 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.546 INFO:teuthology.orchestra.run.smithi076.stdout: "secure_monitoring_stack": { 2024-03-18T15:04:10.546 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "secure_monitoring_stack", 2024-03-18T15:04:10.546 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.546 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.546 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.546 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.546 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.546 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.546 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.547 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T15:04:10.547 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.547 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.547 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.547 INFO:teuthology.orchestra.run.smithi076.stdout: "service_discovery_port": { 2024-03-18T15:04:10.547 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "service_discovery_port", 2024-03-18T15:04:10.547 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.548 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.548 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.548 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8765", 2024-03-18T15:04:10.548 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.548 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.548 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.548 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "cephadm service discovery port", 2024-03-18T15:04:10.548 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.548 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.549 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.549 INFO:teuthology.orchestra.run.smithi076.stdout: "ssh_config_file": { 2024-03-18T15:04:10.549 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssh_config_file", 2024-03-18T15:04:10.549 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.549 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.549 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.549 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.549 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.550 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.550 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.550 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T15:04:10.550 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.550 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.550 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.550 INFO:teuthology.orchestra.run.smithi076.stdout: "use_agent": { 2024-03-18T15:04:10.550 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "use_agent", 2024-03-18T15:04:10.551 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.551 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.551 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.551 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.551 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.551 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.551 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.551 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T15:04:10.552 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.552 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.552 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.552 INFO:teuthology.orchestra.run.smithi076.stdout: "use_repo_digest": { 2024-03-18T15:04:10.552 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "use_repo_digest", 2024-03-18T15:04:10.552 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.552 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.552 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.553 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.553 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.553 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.553 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.553 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T15:04:10.553 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.553 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.553 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.554 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_failed_host_check": { 2024-03-18T15:04:10.554 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_failed_host_check", 2024-03-18T15:04:10.554 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.554 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.554 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.554 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.554 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.554 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.554 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.555 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T15:04:10.555 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.555 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.555 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.555 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_stray_daemons": { 2024-03-18T15:04:10.555 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_stray_daemons", 2024-03-18T15:04:10.555 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.556 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.556 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.556 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.556 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.556 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.556 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.556 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T15:04:10.556 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.557 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.557 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.557 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_stray_hosts": { 2024-03-18T15:04:10.557 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_stray_hosts", 2024-03-18T15:04:10.557 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.557 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.557 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.557 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.558 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.558 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.558 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.558 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T15:04:10.558 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.558 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.558 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.558 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.558 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.559 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crash", 2024-03-18T15:04:10.559 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.559 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.559 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.559 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.559 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.559 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.560 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.560 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.560 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.560 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.560 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.560 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.560 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.560 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.560 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.561 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.561 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.561 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.561 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.561 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.561 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.561 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.561 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.562 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.562 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.562 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.562 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.562 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.562 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.562 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.562 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.563 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.563 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.563 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.563 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.563 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.563 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.563 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.564 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.564 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.564 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.564 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.564 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.564 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.564 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.566 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.566 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.566 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.566 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.566 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.566 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.566 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.566 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.567 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.567 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.567 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.567 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.567 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.567 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.567 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.567 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.568 INFO:teuthology.orchestra.run.smithi076.stdout: "retain_interval": { 2024-03-18T15:04:10.568 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "retain_interval", 2024-03-18T15:04:10.568 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.568 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.568 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.568 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "31536000", 2024-03-18T15:04:10.568 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.569 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.569 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.569 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T15:04:10.569 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.569 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.569 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.569 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.569 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_recent_interval": { 2024-03-18T15:04:10.569 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_recent_interval", 2024-03-18T15:04:10.570 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.570 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.570 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.570 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1209600", 2024-03-18T15:04:10.570 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.570 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.570 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.570 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T15:04:10.570 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.571 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.571 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.571 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.571 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.571 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "dashboard", 2024-03-18T15:04:10.571 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.571 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.572 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.572 INFO:teuthology.orchestra.run.smithi076.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T15:04:10.572 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T15:04:10.572 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.572 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.572 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.572 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:10.572 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.572 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.573 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.573 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.573 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.573 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.573 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.573 INFO:teuthology.orchestra.run.smithi076.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T15:04:10.573 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T15:04:10.574 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.574 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.574 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.574 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.574 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.574 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.574 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.574 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.574 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.575 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.575 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.575 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.575 INFO:teuthology.orchestra.run.smithi076.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T15:04:10.575 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T15:04:10.575 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.575 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.575 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.575 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.576 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.576 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.576 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.576 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.576 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.576 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.576 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.576 INFO:teuthology.orchestra.run.smithi076.stdout: "AUDIT_API_ENABLED": { 2024-03-18T15:04:10.576 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T15:04:10.577 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.577 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.577 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.577 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.577 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.577 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.577 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.577 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.578 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.578 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.578 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.578 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.578 INFO:teuthology.orchestra.run.smithi076.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T15:04:10.578 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T15:04:10.578 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.578 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.578 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.579 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.579 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.579 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.579 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.579 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.579 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.579 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.579 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.580 INFO:teuthology.orchestra.run.smithi076.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T15:04:10.580 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T15:04:10.580 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.580 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.580 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.580 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.580 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.580 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.580 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.581 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.581 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.581 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.581 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.581 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.581 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T15:04:10.581 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T15:04:10.581 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.581 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.581 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.582 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.582 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.582 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.582 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.582 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.582 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.582 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.582 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.582 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.583 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T15:04:10.583 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T15:04:10.583 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.583 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.583 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.583 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.583 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.583 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.584 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.584 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.584 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.584 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.584 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.584 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.584 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T15:04:10.584 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T15:04:10.584 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.585 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.585 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.585 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.585 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.585 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.585 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.585 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.585 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.585 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.586 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.586 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.586 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T15:04:10.586 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T15:04:10.586 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.586 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.586 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.586 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.587 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.587 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.587 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.587 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.587 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.587 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.587 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.587 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.587 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T15:04:10.588 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T15:04:10.588 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.588 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.588 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.588 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.588 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.588 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.588 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.588 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T15:04:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T15:04:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.591 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.591 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T15:04:10.591 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T15:04:10.591 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.591 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.591 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.591 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.591 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.591 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.592 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.592 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.592 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.592 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.592 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.592 INFO:teuthology.orchestra.run.smithi076.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T15:04:10.592 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T15:04:10.592 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.593 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.593 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.593 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.593 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.593 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.593 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.593 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.593 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.593 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.594 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.594 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T15:04:10.594 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T15:04:10.594 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.594 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.594 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.594 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-18T15:04:10.594 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.595 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.595 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.595 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.595 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.595 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.595 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.595 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T15:04:10.596 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T15:04:10.596 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.596 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.596 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.596 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.596 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.596 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.596 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.596 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.596 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.597 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.597 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.597 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_URL": { 2024-03-18T15:04:10.597 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_URL", 2024-03-18T15:04:10.597 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.597 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.597 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.597 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.598 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.598 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.598 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.598 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.598 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.598 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.598 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.598 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T15:04:10.599 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T15:04:10.599 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.599 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.599 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.599 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-18T15:04:10.599 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.599 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.599 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.600 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.600 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.600 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.600 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.600 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T15:04:10.600 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T15:04:10.600 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.600 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.601 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.601 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.601 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.601 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.601 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.601 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.601 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.601 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.601 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.602 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T15:04:10.602 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T15:04:10.602 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.602 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.602 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.602 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.602 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.602 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.603 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.603 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.603 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.603 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.603 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.603 INFO:teuthology.orchestra.run.smithi076.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T15:04:10.603 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T15:04:10.603 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.604 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.604 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.604 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.604 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.604 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.604 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.604 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.604 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.604 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.604 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.605 INFO:teuthology.orchestra.run.smithi076.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T15:04:10.605 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T15:04:10.605 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.605 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.605 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.605 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.605 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.605 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.606 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.606 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.606 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.606 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.606 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.606 INFO:teuthology.orchestra.run.smithi076.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T15:04:10.606 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T15:04:10.606 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.607 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.607 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.607 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "{}", 2024-03-18T15:04:10.607 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.607 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.607 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.607 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.607 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.607 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.608 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.608 INFO:teuthology.orchestra.run.smithi076.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T15:04:10.608 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T15:04:10.608 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.608 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.608 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.608 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.608 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.609 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.609 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.609 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.609 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.609 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.609 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.609 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.609 INFO:teuthology.orchestra.run.smithi076.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T15:04:10.609 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T15:04:10.609 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.610 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.610 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.610 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.610 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.610 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.610 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.610 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.610 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.610 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.610 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.611 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T15:04:10.611 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T15:04:10.611 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.611 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.611 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.611 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.611 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.611 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.612 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.612 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.612 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.612 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.612 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.612 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T15:04:10.612 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T15:04:10.612 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.613 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.613 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.613 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.613 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.613 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.613 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.613 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.613 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.613 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T15:04:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T15:04:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T15:04:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T15:04:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T15:04:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T15:04:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T15:04:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T15:04:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T15:04:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T15:04:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T15:04:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T15:04:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T15:04:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T15:04:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T15:04:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T15:04:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T15:04:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T15:04:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T15:04:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8", 2024-03-18T15:04:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T15:04:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T15:04:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "45", 2024-03-18T15:04:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T15:04:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T15:04:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T15:04:10.632 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T15:04:10.632 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.632 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.632 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.632 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-18T15:04:10.632 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.632 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.632 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.632 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.633 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.633 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.633 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.633 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T15:04:10.633 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T15:04:10.633 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.633 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.633 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.634 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.634 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.634 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.634 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.634 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.634 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.634 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.634 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.635 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T15:04:10.635 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T15:04:10.635 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.635 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.635 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.635 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.635 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.635 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.635 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.636 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.636 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.636 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.636 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.636 INFO:teuthology.orchestra.run.smithi076.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T15:04:10.636 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T15:04:10.636 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.636 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.637 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.637 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.637 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.637 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.637 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.637 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.637 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.637 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.638 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.638 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T15:04:10.638 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T15:04:10.638 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.638 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.638 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.638 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-18T15:04:10.638 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.639 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.639 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.639 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.639 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.639 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.639 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.639 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T15:04:10.639 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T15:04:10.639 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.640 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.640 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.640 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:10.640 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.640 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.640 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.640 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.640 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.641 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.641 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.641 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T15:04:10.641 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T15:04:10.641 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.641 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.641 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.641 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-18T15:04:10.642 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.642 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.642 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.642 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.642 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.642 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.642 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.642 INFO:teuthology.orchestra.run.smithi076.stdout: "cross_origin_url": { 2024-03-18T15:04:10.643 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cross_origin_url", 2024-03-18T15:04:10.643 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.643 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.643 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.643 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.643 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.643 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.643 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.643 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.644 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.644 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.644 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.644 INFO:teuthology.orchestra.run.smithi076.stdout: "crt_file": { 2024-03-18T15:04:10.644 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crt_file", 2024-03-18T15:04:10.644 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.644 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.644 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.645 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.645 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.645 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.645 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.645 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.645 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.645 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.645 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.646 INFO:teuthology.orchestra.run.smithi076.stdout: "debug": { 2024-03-18T15:04:10.646 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "debug", 2024-03-18T15:04:10.646 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.646 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.646 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.646 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.646 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.646 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.646 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.647 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable/disable debug options", 2024-03-18T15:04:10.647 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.647 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.647 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.647 INFO:teuthology.orchestra.run.smithi076.stdout: "jwt_token_ttl": { 2024-03-18T15:04:10.647 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "jwt_token_ttl", 2024-03-18T15:04:10.647 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.647 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.648 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.648 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "28800", 2024-03-18T15:04:10.648 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.648 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.648 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.648 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.648 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.648 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.649 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.649 INFO:teuthology.orchestra.run.smithi076.stdout: "key_file": { 2024-03-18T15:04:10.649 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "key_file", 2024-03-18T15:04:10.649 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.649 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.649 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.649 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.650 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.650 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.650 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.650 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.650 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.650 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.650 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.650 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.651 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.651 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.651 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.651 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.651 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.651 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.651 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.651 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.651 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.652 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.652 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.652 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.652 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.652 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.652 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.652 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.652 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.653 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.653 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.653 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.653 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.653 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.653 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.653 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.653 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.654 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.654 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.654 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.654 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.654 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.654 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.654 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.655 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.655 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.655 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.655 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.655 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.655 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.655 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.655 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.655 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.656 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.656 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.656 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.656 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.656 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.656 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.656 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.656 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.657 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.657 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.657 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.657 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.657 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.657 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.657 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.658 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.658 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.658 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.658 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.658 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.658 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.658 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.658 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.658 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.659 INFO:teuthology.orchestra.run.smithi076.stdout: "motd": { 2024-03-18T15:04:10.659 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "motd", 2024-03-18T15:04:10.659 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.659 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.659 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.659 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.659 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.660 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.660 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.660 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "The message of the day", 2024-03-18T15:04:10.660 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.660 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.660 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.660 INFO:teuthology.orchestra.run.smithi076.stdout: "redirect_resolve_ip_addr": { 2024-03-18T15:04:10.660 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T15:04:10.661 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.661 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.661 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.661 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.661 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.661 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.661 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.661 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.661 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.662 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.662 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.662 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-18T15:04:10.662 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-18T15:04:10.662 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.662 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.663 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.663 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "::", 2024-03-18T15:04:10.663 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.663 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.663 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.663 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.663 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.663 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.663 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.664 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-18T15:04:10.664 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-18T15:04:10.664 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.664 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.664 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.664 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8080", 2024-03-18T15:04:10.664 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.664 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.665 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.665 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.665 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.665 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.665 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.665 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl": { 2024-03-18T15:04:10.665 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl", 2024-03-18T15:04:10.665 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.666 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.666 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.666 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.666 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.666 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.666 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.666 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.666 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.667 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.667 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.667 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl_server_port": { 2024-03-18T15:04:10.667 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl_server_port", 2024-03-18T15:04:10.667 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.667 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.667 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.667 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8443", 2024-03-18T15:04:10.668 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.668 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.668 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.668 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.668 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.668 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.668 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.669 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_behaviour": { 2024-03-18T15:04:10.669 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_behaviour", 2024-03-18T15:04:10.669 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.669 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.669 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.669 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "redirect", 2024-03-18T15:04:10.669 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.669 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.669 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.670 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.670 INFO:teuthology.orchestra.run.smithi076.stdout: "redirect" 2024-03-18T15:04:10.670 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.670 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.670 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.670 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.670 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.670 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_error_status_code": { 2024-03-18T15:04:10.671 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_error_status_code", 2024-03-18T15:04:10.671 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.671 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.671 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.671 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "500", 2024-03-18T15:04:10.671 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "400", 2024-03-18T15:04:10.671 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "599", 2024-03-18T15:04:10.671 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.672 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.672 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.672 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.672 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.672 INFO:teuthology.orchestra.run.smithi076.stdout: "url_prefix": { 2024-03-18T15:04:10.672 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "url_prefix", 2024-03-18T15:04:10.672 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.672 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.673 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.673 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.673 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.673 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.673 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.673 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.673 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.673 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.673 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.674 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.674 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.674 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "devicehealth", 2024-03-18T15:04:10.674 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.674 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.674 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.674 INFO:teuthology.orchestra.run.smithi076.stdout: "enable_monitoring": { 2024-03-18T15:04:10.675 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enable_monitoring", 2024-03-18T15:04:10.675 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.675 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.675 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.675 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.675 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.675 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.675 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.675 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "monitor device health metrics", 2024-03-18T15:04:10.676 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.676 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.676 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.676 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.676 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.676 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.676 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.677 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.677 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.677 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.677 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.677 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.677 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.677 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.677 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.677 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.678 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.678 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.678 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.678 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.678 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.678 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.678 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.679 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.679 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.679 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.679 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.679 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.679 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.679 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.679 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.680 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.680 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.680 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.680 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.680 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.680 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.680 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.680 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.681 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.681 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.681 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.681 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.681 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.681 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.681 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.681 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.682 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.682 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.682 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.682 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.682 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.682 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.682 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.682 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.683 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.683 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.683 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.683 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.683 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.683 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.683 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.684 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.684 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.684 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.684 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.684 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.684 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.684 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.684 INFO:teuthology.orchestra.run.smithi076.stdout: "mark_out_threshold": { 2024-03-18T15:04:10.685 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mark_out_threshold", 2024-03-18T15:04:10.685 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.685 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.685 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.685 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "2419200", 2024-03-18T15:04:10.685 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.685 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.685 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.686 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T15:04:10.686 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.686 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.686 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.686 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": { 2024-03-18T15:04:10.686 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pool_name", 2024-03-18T15:04:10.686 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.687 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.687 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.687 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "device_health_metrics", 2024-03-18T15:04:10.687 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.687 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.687 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.687 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T15:04:10.688 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.688 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.688 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.688 INFO:teuthology.orchestra.run.smithi076.stdout: "retention_period": { 2024-03-18T15:04:10.688 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "retention_period", 2024-03-18T15:04:10.688 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.688 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.688 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.689 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15552000", 2024-03-18T15:04:10.689 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.689 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.689 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.689 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long to retain device health metrics", 2024-03-18T15:04:10.689 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.689 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.689 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.690 INFO:teuthology.orchestra.run.smithi076.stdout: "scrape_frequency": { 2024-03-18T15:04:10.690 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "scrape_frequency", 2024-03-18T15:04:10.690 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.690 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.690 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.690 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "86400", 2024-03-18T15:04:10.690 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.690 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.690 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.691 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T15:04:10.691 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.691 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.691 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.691 INFO:teuthology.orchestra.run.smithi076.stdout: "self_heal": { 2024-03-18T15:04:10.691 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "self_heal", 2024-03-18T15:04:10.691 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.691 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.692 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.692 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.692 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.692 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.692 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.692 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T15:04:10.692 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.692 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.693 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.693 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-18T15:04:10.693 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-18T15:04:10.693 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.693 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.693 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.693 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-18T15:04:10.693 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.694 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.694 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.694 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T15:04:10.694 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.694 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.694 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.694 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_threshold": { 2024-03-18T15:04:10.694 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_threshold", 2024-03-18T15:04:10.695 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.695 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.695 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.695 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "7257600", 2024-03-18T15:04:10.695 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.695 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.695 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.695 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T15:04:10.695 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.696 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.696 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.696 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.696 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.696 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "diskprediction_local", 2024-03-18T15:04:10.696 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.696 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.697 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.697 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.697 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.697 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.697 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.697 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.697 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.697 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.697 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.698 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.698 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.698 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.698 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.698 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.698 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.698 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.698 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.699 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.699 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.699 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.699 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.699 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.699 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.699 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.699 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.700 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.700 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.700 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.700 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.700 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.700 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.700 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.700 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.701 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.701 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.701 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.701 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.701 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.702 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.702 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.702 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.702 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.702 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.702 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.702 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.702 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.703 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.703 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.703 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.703 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.703 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.703 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.703 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.703 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.704 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.704 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.704 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.704 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.704 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.704 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.704 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.704 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.705 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.705 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.705 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.705 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.705 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.705 INFO:teuthology.orchestra.run.smithi076.stdout: "predict_interval": { 2024-03-18T15:04:10.705 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "predict_interval", 2024-03-18T15:04:10.706 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.706 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.706 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.706 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "86400", 2024-03-18T15:04:10.706 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.706 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.706 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.706 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.707 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.707 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.707 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.707 INFO:teuthology.orchestra.run.smithi076.stdout: "predictor_model": { 2024-03-18T15:04:10.707 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "predictor_model", 2024-03-18T15:04:10.707 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.707 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.707 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.708 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "prophetstor", 2024-03-18T15:04:10.708 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.708 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.708 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.708 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.708 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.708 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.708 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.709 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-18T15:04:10.709 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-18T15:04:10.709 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.709 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.709 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.709 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-18T15:04:10.709 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.709 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.710 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.710 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.710 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.710 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.710 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.710 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.710 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.711 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "influx", 2024-03-18T15:04:10.711 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": false, 2024-03-18T15:04:10.711 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "influxdb python module not found", 2024-03-18T15:04:10.711 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.711 INFO:teuthology.orchestra.run.smithi076.stdout: "batch_size": { 2024-03-18T15:04:10.711 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "batch_size", 2024-03-18T15:04:10.711 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.711 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.712 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.712 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5000", 2024-03-18T15:04:10.712 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.712 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.712 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.712 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T15:04:10.712 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.712 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.713 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.713 INFO:teuthology.orchestra.run.smithi076.stdout: "database": { 2024-03-18T15:04:10.713 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "database", 2024-03-18T15:04:10.713 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.713 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.713 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.713 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "ceph", 2024-03-18T15:04:10.713 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.714 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.714 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.714 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T15:04:10.714 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.714 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.714 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.714 INFO:teuthology.orchestra.run.smithi076.stdout: "hostname": { 2024-03-18T15:04:10.715 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "hostname", 2024-03-18T15:04:10.715 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.715 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.715 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.715 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.715 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.715 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.715 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.715 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB server hostname", 2024-03-18T15:04:10.716 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.716 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.716 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.716 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-18T15:04:10.716 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-18T15:04:10.716 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.716 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.716 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.717 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "30", 2024-03-18T15:04:10.717 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "5", 2024-03-18T15:04:10.717 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.717 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.717 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T15:04:10.717 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.717 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.717 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.718 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.718 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.718 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.718 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.718 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.718 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.718 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.718 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.719 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.719 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.719 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.719 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.719 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.719 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.719 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.719 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.719 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.720 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.720 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.720 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.720 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.720 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.720 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.720 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.720 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.721 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.721 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.721 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.721 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.721 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.721 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.721 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.721 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.722 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.722 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.722 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.722 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.722 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.722 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.722 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.723 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.723 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.723 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.723 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.723 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.723 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.723 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.723 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.723 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.724 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.724 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.724 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.724 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.724 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.724 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.724 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.724 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.725 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.725 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.725 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.725 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.725 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.725 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.725 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.725 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.725 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.726 INFO:teuthology.orchestra.run.smithi076.stdout: "password": { 2024-03-18T15:04:10.726 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "password", 2024-03-18T15:04:10.726 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.726 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.726 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.726 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.726 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.727 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.727 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.727 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "password of InfluxDB server user", 2024-03-18T15:04:10.727 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.727 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.727 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.727 INFO:teuthology.orchestra.run.smithi076.stdout: "port": { 2024-03-18T15:04:10.728 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "port", 2024-03-18T15:04:10.728 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.728 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.728 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.728 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8086", 2024-03-18T15:04:10.728 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.728 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.728 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.728 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB server port", 2024-03-18T15:04:10.729 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.729 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.729 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.729 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl": { 2024-03-18T15:04:10.729 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl", 2024-03-18T15:04:10.729 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.729 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.730 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.730 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "false", 2024-03-18T15:04:10.730 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.730 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.730 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.730 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T15:04:10.730 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.730 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.731 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.731 INFO:teuthology.orchestra.run.smithi076.stdout: "threads": { 2024-03-18T15:04:10.731 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "threads", 2024-03-18T15:04:10.731 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.731 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.731 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.731 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-18T15:04:10.731 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-18T15:04:10.732 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "32", 2024-03-18T15:04:10.732 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.732 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T15:04:10.732 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.732 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.732 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.732 INFO:teuthology.orchestra.run.smithi076.stdout: "username": { 2024-03-18T15:04:10.732 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "username", 2024-03-18T15:04:10.733 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.733 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.733 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.733 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.733 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.733 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.733 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.733 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "username of InfluxDB server user", 2024-03-18T15:04:10.734 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.734 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.734 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.734 INFO:teuthology.orchestra.run.smithi076.stdout: "verify_ssl": { 2024-03-18T15:04:10.734 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "verify_ssl", 2024-03-18T15:04:10.734 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.734 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.735 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.735 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "true", 2024-03-18T15:04:10.735 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.735 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.735 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.735 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T15:04:10.735 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.735 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.735 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.736 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.736 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.736 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "insights", 2024-03-18T15:04:10.736 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.736 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.736 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.737 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.737 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.737 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.737 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.737 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.737 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.737 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.737 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.738 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.738 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.738 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.738 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.738 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.738 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.738 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.738 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.739 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.739 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.739 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.739 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.739 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.739 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.739 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.740 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.740 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.740 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.740 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.740 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.740 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.740 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.740 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.740 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.741 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.741 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.741 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.741 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.741 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.741 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.741 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.742 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.742 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.742 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.742 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.742 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.742 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.742 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.742 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.743 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.743 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.743 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.743 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.743 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.743 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.743 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.743 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.744 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.744 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.744 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.744 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.744 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.744 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.744 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.744 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.745 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.745 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.745 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.745 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.745 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.745 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "iostat", 2024-03-18T15:04:10.745 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.746 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.746 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.746 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.746 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.746 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.746 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.746 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.747 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.747 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.747 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.747 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.747 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.747 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.747 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.747 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.747 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.748 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.748 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.748 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.748 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.748 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.748 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.748 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.748 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.749 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.749 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.749 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.749 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.749 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.749 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.749 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.749 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.750 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.750 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.750 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.750 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.750 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.750 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.750 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.750 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.751 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.751 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.751 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.751 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.751 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.751 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.751 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.751 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.752 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.752 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.752 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.752 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.752 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.752 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.752 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.752 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.753 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.753 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.753 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.753 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.753 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.753 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.753 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.753 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.754 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.754 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.754 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.754 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.754 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.754 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.755 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "k8sevents", 2024-03-18T15:04:10.755 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.755 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.755 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.755 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_event_retention_days": { 2024-03-18T15:04:10.755 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ceph_event_retention_days", 2024-03-18T15:04:10.755 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.755 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.755 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.756 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "7", 2024-03-18T15:04:10.756 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.756 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.756 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.756 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T15:04:10.756 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.756 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.756 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.757 INFO:teuthology.orchestra.run.smithi076.stdout: "config_check_secs": { 2024-03-18T15:04:10.757 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_check_secs", 2024-03-18T15:04:10.757 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.757 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.757 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.757 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:10.757 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "10", 2024-03-18T15:04:10.757 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.758 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.758 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T15:04:10.758 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.758 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.758 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.758 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.758 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.759 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.759 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.759 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.759 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.759 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.759 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.759 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.759 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.759 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.760 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.760 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.760 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.760 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.760 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.760 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.760 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.760 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.761 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.761 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.761 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.761 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.761 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.761 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.761 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.761 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.762 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.762 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.762 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.762 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.762 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.762 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.762 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.763 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.763 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.763 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.763 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.763 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.763 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.763 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.763 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.764 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.764 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.764 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.764 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.764 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.764 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.764 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.764 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.764 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.765 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.765 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.765 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.765 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.765 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.765 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.765 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.766 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.766 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.766 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.766 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.766 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.766 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.766 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.766 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.767 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.767 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "localpool", 2024-03-18T15:04:10.767 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.767 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.767 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.767 INFO:teuthology.orchestra.run.smithi076.stdout: "failure_domain": { 2024-03-18T15:04:10.767 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "failure_domain", 2024-03-18T15:04:10.768 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.768 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.768 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.768 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "host", 2024-03-18T15:04:10.768 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.768 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.768 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.768 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "failure domain for any created local pool", 2024-03-18T15:04:10.769 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T15:04:10.769 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.769 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.769 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.769 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.769 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.769 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.769 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.769 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.770 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.770 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.770 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.770 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.770 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.770 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.770 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.770 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.771 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.771 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.771 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.771 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.771 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.771 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.771 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.772 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.772 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.772 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.772 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.772 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.772 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.772 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.772 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.772 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.773 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.773 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.773 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.773 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.773 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.773 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.773 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.774 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.774 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.774 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.774 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.774 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.774 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.774 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.774 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.775 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.775 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.775 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.775 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.775 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.775 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.775 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.775 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.776 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.776 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.776 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.776 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.776 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.776 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.776 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.776 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.776 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.777 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.777 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.777 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.777 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.777 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": { 2024-03-18T15:04:10.777 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "min_size", 2024-03-18T15:04:10.777 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.778 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.778 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.778 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.778 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.778 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.778 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.778 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default min_size for any created local pool", 2024-03-18T15:04:10.778 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T15:04:10.779 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.779 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.779 INFO:teuthology.orchestra.run.smithi076.stdout: "num_rep": { 2024-03-18T15:04:10.779 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "num_rep", 2024-03-18T15:04:10.779 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.779 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.779 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.779 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3", 2024-03-18T15:04:10.780 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.780 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.780 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.780 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default replica count for any created local pool", 2024-03-18T15:04:10.780 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.780 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.780 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.780 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": { 2024-03-18T15:04:10.781 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pg_num", 2024-03-18T15:04:10.781 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.781 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.781 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.781 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "128", 2024-03-18T15:04:10.781 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.781 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.781 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.782 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T15:04:10.782 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.782 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.782 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.782 INFO:teuthology.orchestra.run.smithi076.stdout: "prefix": { 2024-03-18T15:04:10.783 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prefix", 2024-03-18T15:04:10.783 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.783 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.783 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.783 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.783 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.783 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.783 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.784 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "name prefix for any created local pool", 2024-03-18T15:04:10.784 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.784 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.784 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.784 INFO:teuthology.orchestra.run.smithi076.stdout: "subtree": { 2024-03-18T15:04:10.784 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "subtree", 2024-03-18T15:04:10.784 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.785 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.785 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.785 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "rack", 2024-03-18T15:04:10.785 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.785 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.785 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.785 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T15:04:10.785 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T15:04:10.786 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.786 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.786 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.786 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.789 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mds_autoscaler", 2024-03-18T15:04:10.790 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.790 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.790 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.790 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.790 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.790 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.790 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.791 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.791 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.791 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.791 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.791 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.791 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.791 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.791 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.791 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.792 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.792 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.792 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.792 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.792 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.792 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.792 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.793 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.793 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.793 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.793 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.793 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.793 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.793 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.793 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.793 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.794 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.794 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.794 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.794 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.794 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.794 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.794 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.794 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.795 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.795 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.795 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.795 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.795 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.795 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.795 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.795 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.795 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.795 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.796 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.796 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.796 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.796 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.796 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.796 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.796 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.796 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.797 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.797 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.797 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.797 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.797 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.797 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.797 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.797 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.797 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.798 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.798 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.798 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.798 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.798 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mirroring", 2024-03-18T15:04:10.798 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.798 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.799 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.799 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.799 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.799 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.799 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.799 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.799 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.799 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.799 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.800 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.800 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.800 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.800 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.800 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.800 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.800 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.800 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.800 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.801 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.801 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.801 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.801 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.801 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.801 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.801 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.802 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.802 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.802 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.802 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.802 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.802 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.802 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.802 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.802 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.803 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.803 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.803 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.803 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.803 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.803 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.803 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.803 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.804 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.804 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.804 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.804 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.804 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.804 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.804 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.804 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.804 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.804 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.805 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.805 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.805 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.805 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.805 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.805 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.805 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.805 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.805 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.806 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.806 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.806 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.806 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.806 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.806 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.806 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.806 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.807 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "nfs", 2024-03-18T15:04:10.807 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.807 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.807 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.807 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.807 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.807 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.807 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.808 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.808 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.808 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.808 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.808 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.808 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.808 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.808 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.808 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.809 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.809 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.809 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.809 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.809 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.809 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.809 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.809 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.810 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.810 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.810 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.810 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.810 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.810 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.810 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.810 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.810 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.811 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.811 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.811 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.811 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.811 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.811 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.811 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.812 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.812 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.812 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.812 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.812 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.812 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.812 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.812 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.812 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.812 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.813 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.813 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.813 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.813 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.813 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.813 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.814 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.814 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.814 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.814 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.814 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.814 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.814 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.814 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.814 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.815 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.815 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.815 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.815 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.815 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.815 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.815 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "orchestrator", 2024-03-18T15:04:10.815 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.816 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.816 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.816 INFO:teuthology.orchestra.run.smithi076.stdout: "fail_fs": { 2024-03-18T15:04:10.816 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "fail_fs", 2024-03-18T15:04:10.816 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.816 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.816 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.816 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.816 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.817 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.817 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.817 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T15:04:10.817 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.817 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.817 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.817 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.817 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.818 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.818 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.818 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.818 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.818 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.818 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.818 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.818 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.818 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.819 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.819 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.819 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.819 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.819 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.819 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.819 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.819 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.819 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.820 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.820 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.820 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.820 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.820 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.820 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.820 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.820 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.821 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.821 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.821 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.821 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.821 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.821 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.821 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.821 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.822 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.822 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.822 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.822 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.822 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.822 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.822 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.822 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.822 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.823 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.823 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.823 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.823 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.823 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.823 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.823 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.823 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.824 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.824 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.824 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.824 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.824 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.824 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.824 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.824 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.824 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.825 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.825 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.825 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.825 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.825 INFO:teuthology.orchestra.run.smithi076.stdout: "orchestrator": { 2024-03-18T15:04:10.825 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "orchestrator", 2024-03-18T15:04:10.825 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.825 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.826 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.826 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.826 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.826 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.826 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.826 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm", 2024-03-18T15:04:10.826 INFO:teuthology.orchestra.run.smithi076.stdout: "rook", 2024-03-18T15:04:10.826 INFO:teuthology.orchestra.run.smithi076.stdout: "test_orchestrator" 2024-03-18T15:04:10.827 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.827 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Orchestrator backend", 2024-03-18T15:04:10.827 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.827 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.827 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.827 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.827 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.827 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "osd_perf_query", 2024-03-18T15:04:10.828 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.828 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.828 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.828 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.828 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.828 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.828 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.828 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.828 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.829 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.829 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.829 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.829 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.829 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.829 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.829 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.829 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.829 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.830 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.830 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.830 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.830 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.830 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.830 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.830 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.830 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.831 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.831 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.831 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.831 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.831 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.831 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.831 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.831 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.831 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.832 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.832 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.832 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.832 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.832 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.832 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.832 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.833 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.833 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.833 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.833 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.833 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.833 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.833 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.833 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.834 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.834 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.834 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.834 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.834 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.834 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.834 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.835 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.835 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.835 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.835 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.835 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.835 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.835 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.835 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.835 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.836 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.836 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.836 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.836 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.836 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.836 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "osd_support", 2024-03-18T15:04:10.836 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.837 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.837 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.837 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.837 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.837 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.837 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.837 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.837 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.837 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.838 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.838 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.838 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.838 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.838 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.838 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.838 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.838 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.839 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.839 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.839 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.839 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.839 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.839 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.839 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.839 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.840 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.840 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.840 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.840 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.840 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.840 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.840 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.840 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.840 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.841 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.841 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.841 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.841 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.841 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.841 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.841 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.842 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.842 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.842 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.842 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.842 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.842 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.842 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.842 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.842 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.843 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.843 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.843 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.843 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.843 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.843 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.843 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.843 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.844 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.844 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.844 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.844 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.844 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.844 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.844 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.844 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.845 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.845 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.845 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.845 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.845 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pg_autoscaler", 2024-03-18T15:04:10.845 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.845 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.846 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.846 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.846 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.846 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.846 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.846 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.846 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.846 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.846 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.847 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.847 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.847 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.847 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.847 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.847 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.847 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.847 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.848 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.848 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.848 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.848 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.848 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.848 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.848 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.848 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.849 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.849 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.849 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.849 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.849 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.849 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.849 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.849 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.849 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.850 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.850 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.850 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.850 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.850 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.850 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.850 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.851 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.851 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.851 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.851 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.851 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.851 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.851 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.851 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.851 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.852 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.852 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.852 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.852 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.852 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.852 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.852 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.853 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.853 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.853 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.853 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.853 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.853 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.853 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.853 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.854 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.854 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.854 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-18T15:04:10.854 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-18T15:04:10.854 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.854 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.854 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.854 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-18T15:04:10.855 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.855 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.855 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.855 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.855 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.855 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.855 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.855 INFO:teuthology.orchestra.run.smithi076.stdout: "threshold": { 2024-03-18T15:04:10.856 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "threshold", 2024-03-18T15:04:10.856 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:10.856 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.856 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.856 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3.0", 2024-03-18T15:04:10.856 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1.0", 2024-03-18T15:04:10.856 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.856 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.856 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "scaling threshold", 2024-03-18T15:04:10.857 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T15:04:10.857 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.857 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.857 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.857 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.857 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "progress", 2024-03-18T15:04:10.858 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.858 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.858 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.858 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_pg_recovery_event": { 2024-03-18T15:04:10.858 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_pg_recovery_event", 2024-03-18T15:04:10.858 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.858 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.858 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.859 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.859 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.859 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.859 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.859 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T15:04:10.859 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.859 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.859 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.860 INFO:teuthology.orchestra.run.smithi076.stdout: "enabled": { 2024-03-18T15:04:10.860 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enabled", 2024-03-18T15:04:10.860 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.860 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.860 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.860 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.860 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.860 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.861 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.861 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.861 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.861 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.861 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.861 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.861 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.861 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.862 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.862 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.862 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.862 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.862 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.862 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.862 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.862 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.863 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.863 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.863 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.863 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.863 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.863 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.863 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.863 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.863 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.864 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.864 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.864 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.864 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.864 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.864 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.864 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.864 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.865 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.865 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.865 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.865 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.865 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.865 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.865 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.865 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.865 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.866 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.866 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.866 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.866 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.866 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.866 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.866 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.866 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.867 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.867 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.867 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.867 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.867 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.867 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.867 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.867 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.868 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.868 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.868 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.868 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.868 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.868 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.868 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.868 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.868 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.869 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.869 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.869 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.869 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.869 INFO:teuthology.orchestra.run.smithi076.stdout: "max_completed_events": { 2024-03-18T15:04:10.869 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_completed_events", 2024-03-18T15:04:10.869 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.869 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.870 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.870 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "50", 2024-03-18T15:04:10.870 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.870 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.870 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.870 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "number of past completed events to remember", 2024-03-18T15:04:10.870 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.870 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.871 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.871 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-18T15:04:10.871 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-18T15:04:10.871 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:10.871 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.871 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.871 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-18T15:04:10.871 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.871 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.872 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.872 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long the module is going to sleep", 2024-03-18T15:04:10.872 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.872 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.872 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.872 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.872 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.873 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prometheus", 2024-03-18T15:04:10.873 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.873 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.873 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.873 INFO:teuthology.orchestra.run.smithi076.stdout: "cache": { 2024-03-18T15:04:10.873 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cache", 2024-03-18T15:04:10.873 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.873 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.874 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.874 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.874 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.874 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.874 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.874 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.874 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.874 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.875 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.875 INFO:teuthology.orchestra.run.smithi076.stdout: "exclude_perf_counters": { 2024-03-18T15:04:10.875 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "exclude_perf_counters", 2024-03-18T15:04:10.875 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.875 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.875 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.875 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.875 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.876 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.876 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.876 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T15:04:10.876 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T15:04:10.876 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.876 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.876 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.876 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.877 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.877 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.877 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.877 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.877 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.877 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.877 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.877 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.878 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.878 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.878 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.878 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.878 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.878 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.878 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.878 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.878 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.879 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.879 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.879 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.879 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.879 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.879 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.879 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.880 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.880 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.880 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.880 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.880 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.880 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.880 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.880 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.881 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.881 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.881 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.881 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.881 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.881 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.881 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.881 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.881 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.881 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.882 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.882 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.882 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.882 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.882 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.882 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.882 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.882 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.883 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.883 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.883 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.883 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.883 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.883 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.883 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.884 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.884 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.884 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.884 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.884 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.884 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.884 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.884 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_stats_pools": { 2024-03-18T15:04:10.885 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_stats_pools", 2024-03-18T15:04:10.885 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.885 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.885 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.885 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.885 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.885 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.885 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.886 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.886 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.886 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.886 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.886 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T15:04:10.886 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T15:04:10.886 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.887 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.887 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.887 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "300", 2024-03-18T15:04:10.887 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.887 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.887 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.887 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.887 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.888 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.888 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.888 INFO:teuthology.orchestra.run.smithi076.stdout: "scrape_interval": { 2024-03-18T15:04:10.888 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "scrape_interval", 2024-03-18T15:04:10.888 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:10.888 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.888 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.888 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15.0", 2024-03-18T15:04:10.889 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.889 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.889 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.889 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.889 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.889 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.889 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.890 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-18T15:04:10.890 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-18T15:04:10.890 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.890 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.890 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.890 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "::", 2024-03-18T15:04:10.890 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.890 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.891 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.891 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T15:04:10.891 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.891 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.891 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.891 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-18T15:04:10.891 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-18T15:04:10.891 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.892 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.892 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.892 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "9283", 2024-03-18T15:04:10.892 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.892 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.892 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.892 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T15:04:10.892 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.893 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.893 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.893 INFO:teuthology.orchestra.run.smithi076.stdout: "stale_cache_strategy": { 2024-03-18T15:04:10.893 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "stale_cache_strategy", 2024-03-18T15:04:10.893 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.893 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.893 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.893 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "log", 2024-03-18T15:04:10.894 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.894 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.894 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.894 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.894 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.894 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.894 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.895 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_behaviour": { 2024-03-18T15:04:10.895 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_behaviour", 2024-03-18T15:04:10.895 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.895 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.895 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.895 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "default", 2024-03-18T15:04:10.895 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.895 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.896 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.896 INFO:teuthology.orchestra.run.smithi076.stdout: "default", 2024-03-18T15:04:10.896 INFO:teuthology.orchestra.run.smithi076.stdout: "error" 2024-03-18T15:04:10.896 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.896 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.896 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.896 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.896 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.897 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_error_status_code": { 2024-03-18T15:04:10.897 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_error_status_code", 2024-03-18T15:04:10.897 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.897 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.897 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.897 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "500", 2024-03-18T15:04:10.897 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "400", 2024-03-18T15:04:10.898 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "599", 2024-03-18T15:04:10.898 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.898 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.898 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.898 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.898 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.899 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_support", 2024-03-18T15:04:10.899 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.899 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.899 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.899 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.899 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.899 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.900 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.900 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.900 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.900 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.900 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.900 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.900 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.900 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.900 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.901 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.901 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.901 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.901 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.901 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.901 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.901 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.902 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.902 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.902 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.902 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.902 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.902 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.902 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.903 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.903 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.903 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.903 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.903 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.903 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.903 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.904 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.904 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.904 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.904 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.904 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.904 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.904 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.904 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.904 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.905 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.905 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.905 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.905 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.905 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.905 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.905 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.905 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.906 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.906 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.906 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.906 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.906 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.906 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.906 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.906 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.907 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.907 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.907 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.907 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.907 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.907 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.907 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.907 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.908 INFO:teuthology.orchestra.run.smithi076.stdout: "max_concurrent_snap_create": { 2024-03-18T15:04:10.908 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_concurrent_snap_create", 2024-03-18T15:04:10.908 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.908 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.908 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.908 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:10.908 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.909 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.909 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.909 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.909 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.909 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.909 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.909 INFO:teuthology.orchestra.run.smithi076.stdout: "mirror_snapshot_schedule": { 2024-03-18T15:04:10.909 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T15:04:10.910 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.910 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.910 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.910 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.910 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.910 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.910 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.910 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.910 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.911 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.911 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.911 INFO:teuthology.orchestra.run.smithi076.stdout: "trash_purge_schedule": { 2024-03-18T15:04:10.911 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "trash_purge_schedule", 2024-03-18T15:04:10.911 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.911 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.911 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.912 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.912 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.912 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.912 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.912 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.912 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.912 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.912 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.913 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.913 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.913 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "restful", 2024-03-18T15:04:10.913 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.913 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.913 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.913 INFO:teuthology.orchestra.run.smithi076.stdout: "enable_auth": { 2024-03-18T15:04:10.914 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enable_auth", 2024-03-18T15:04:10.914 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.914 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.914 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.914 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.914 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.914 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.914 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.915 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.915 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.915 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.915 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.915 INFO:teuthology.orchestra.run.smithi076.stdout: "key_file": { 2024-03-18T15:04:10.915 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "key_file", 2024-03-18T15:04:10.915 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.916 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.916 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.916 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.916 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.916 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.916 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.916 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.916 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.916 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.917 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.917 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.917 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.917 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.917 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.917 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.917 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.918 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.918 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.918 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.918 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.918 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.918 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.918 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.918 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.919 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.919 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.919 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.919 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.919 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.919 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.919 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.920 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.920 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.920 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.920 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.920 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.920 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.920 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.920 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.921 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.921 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.921 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.921 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.921 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.921 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.921 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.922 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.922 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.922 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.922 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.922 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.922 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.922 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.922 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.922 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.923 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.923 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.923 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.923 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.923 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.923 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.923 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.923 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.924 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.924 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.924 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.924 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.924 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.924 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.924 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.924 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.924 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.925 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.925 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.925 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.925 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.925 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-18T15:04:10.925 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-18T15:04:10.925 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.925 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.926 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.926 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.926 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.926 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.926 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.926 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.926 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.926 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.926 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.927 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-18T15:04:10.927 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-18T15:04:10.927 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.927 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.927 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.927 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.927 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.927 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.928 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.928 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.928 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.928 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.928 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.928 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.928 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.929 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rgw", 2024-03-18T15:04:10.929 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.929 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.929 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.929 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.929 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.929 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.929 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.929 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.930 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.930 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.930 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.930 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.930 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.930 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.930 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.930 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.931 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.931 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.931 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.931 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.931 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.931 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.931 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.932 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.932 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.932 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.932 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.932 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.932 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.932 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.932 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.932 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.933 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.933 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.933 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.933 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.933 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.933 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.933 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.934 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.934 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.934 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.934 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.934 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.934 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.934 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.934 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.934 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.935 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.935 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.935 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.935 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.935 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.935 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.935 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.935 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.936 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.936 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.936 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.936 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.936 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.936 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.936 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.936 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.937 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.937 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.937 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.937 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.937 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.938 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rook", 2024-03-18T15:04:10.938 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.938 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.938 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.938 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.938 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.938 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.939 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.939 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.939 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.939 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.939 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.939 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.939 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.939 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.939 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.939 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.940 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.940 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.940 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.940 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.940 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.940 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.940 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.941 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.941 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.941 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.941 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.941 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.941 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.941 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.941 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.941 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.942 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.942 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.942 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.942 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.942 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.942 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.942 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.943 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.943 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.943 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.943 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.943 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.943 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.943 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.943 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.944 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.944 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.944 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.944 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.944 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.944 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.944 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.944 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.944 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.945 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.945 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.945 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.945 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.945 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.945 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.945 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.945 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.946 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.946 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.946 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.946 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.946 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.946 INFO:teuthology.orchestra.run.smithi076.stdout: "storage_class": { 2024-03-18T15:04:10.946 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "storage_class", 2024-03-18T15:04:10.947 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.947 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.947 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.947 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "local", 2024-03-18T15:04:10.947 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.947 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.947 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.947 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T15:04:10.948 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.948 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.948 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.948 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.948 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.948 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "selftest", 2024-03-18T15:04:10.949 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.949 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.949 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.949 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.949 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.949 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.949 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.949 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.950 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.950 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.950 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.950 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.950 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.950 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.950 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.950 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.951 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.951 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.951 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.951 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.951 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.951 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.951 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.952 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.952 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.952 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.952 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.952 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.952 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.952 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.952 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.953 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.953 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.953 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.953 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.953 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.953 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.953 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.954 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.954 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.954 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.954 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.954 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.954 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.954 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.954 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.955 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.955 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.955 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.955 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.955 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.955 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.955 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.956 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.956 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.956 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.956 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.956 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.956 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.956 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.957 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.957 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.957 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.957 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.957 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.957 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.957 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.957 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.957 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.958 INFO:teuthology.orchestra.run.smithi076.stdout: "roption1": { 2024-03-18T15:04:10.958 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "roption1", 2024-03-18T15:04:10.958 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.958 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.958 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.958 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.958 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.959 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.959 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.959 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.959 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.959 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.959 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.959 INFO:teuthology.orchestra.run.smithi076.stdout: "roption2": { 2024-03-18T15:04:10.959 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "roption2", 2024-03-18T15:04:10.960 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.960 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.960 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.960 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "xyz", 2024-03-18T15:04:10.960 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.960 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.960 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.960 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.960 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.961 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.961 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.961 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption1": { 2024-03-18T15:04:10.961 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption1", 2024-03-18T15:04:10.961 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.961 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.961 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.961 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.962 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.962 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.962 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.962 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.962 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.962 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.962 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.963 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption2": { 2024-03-18T15:04:10.963 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption2", 2024-03-18T15:04:10.963 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.963 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.963 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.963 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.963 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.963 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.963 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.963 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.964 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.964 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.964 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.964 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption3": { 2024-03-18T15:04:10.964 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption3", 2024-03-18T15:04:10.964 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:10.964 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.964 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.965 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.965 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.965 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.965 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.965 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.965 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.965 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.965 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.966 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption4": { 2024-03-18T15:04:10.966 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption4", 2024-03-18T15:04:10.966 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.966 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.966 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.966 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.966 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.966 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.966 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.967 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.967 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.967 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.967 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.967 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption5": { 2024-03-18T15:04:10.967 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption5", 2024-03-18T15:04:10.967 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.967 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.968 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.968 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.968 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.968 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.968 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.968 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.968 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.968 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.968 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.969 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption6": { 2024-03-18T15:04:10.969 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption6", 2024-03-18T15:04:10.969 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.969 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.969 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.969 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:10.969 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.969 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.969 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.970 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.970 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.970 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.970 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.970 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption7": { 2024-03-18T15:04:10.970 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption7", 2024-03-18T15:04:10.970 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:10.970 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.971 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.971 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.971 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-18T15:04:10.971 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "42", 2024-03-18T15:04:10.971 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.971 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.971 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.971 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.971 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.972 INFO:teuthology.orchestra.run.smithi076.stdout: "testkey": { 2024-03-18T15:04:10.972 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testkey", 2024-03-18T15:04:10.972 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.972 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.972 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.972 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.972 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.972 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.973 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.973 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.973 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.973 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.973 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.973 INFO:teuthology.orchestra.run.smithi076.stdout: "testlkey": { 2024-03-18T15:04:10.973 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testlkey", 2024-03-18T15:04:10.974 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.974 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.974 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.974 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.974 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.974 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.974 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.974 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.974 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.975 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.975 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.975 INFO:teuthology.orchestra.run.smithi076.stdout: "testnewline": { 2024-03-18T15:04:10.975 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testnewline", 2024-03-18T15:04:10.975 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.975 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.975 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:10.975 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.975 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.976 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.976 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.976 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.976 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.976 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.976 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.976 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.976 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.977 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snap_schedule", 2024-03-18T15:04:10.977 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.977 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.977 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.977 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_m_granularity": { 2024-03-18T15:04:10.977 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_m_granularity", 2024-03-18T15:04:10.977 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.978 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.978 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.978 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.978 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.978 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.978 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.978 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T15:04:10.978 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.979 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.979 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.979 INFO:teuthology.orchestra.run.smithi076.stdout: "dump_on_update": { 2024-03-18T15:04:10.979 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "dump_on_update", 2024-03-18T15:04:10.979 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.979 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.979 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.979 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.980 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.980 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.980 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.980 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "dump database to debug log on update", 2024-03-18T15:04:10.980 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.980 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.980 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.980 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.981 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.981 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.981 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.981 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.981 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.981 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.981 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.981 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.981 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.982 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.982 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.982 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.982 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.982 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.982 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.982 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.982 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.983 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.983 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.983 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.983 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.983 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.983 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.983 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.983 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.984 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.984 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.984 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.984 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.984 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.984 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.984 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.984 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.985 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.985 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.985 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.985 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.985 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.985 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.985 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.985 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.985 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.986 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.986 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.986 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.986 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.986 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.986 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.986 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.986 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.987 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.987 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.987 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.987 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.987 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.987 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.987 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.987 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.988 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.988 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.988 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.988 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.988 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.988 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.988 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.988 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.988 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:10.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:10.989 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "stats", 2024-03-18T15:04:10.989 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:10.989 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:10.989 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:10.989 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:10.989 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:10.989 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.990 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.990 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.990 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:10.990 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.990 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.990 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.990 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.990 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.991 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.991 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.993 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.993 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.993 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.994 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.994 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.994 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.994 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.994 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:10.994 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:10.994 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.995 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.995 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.995 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.995 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.995 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.995 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.995 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.995 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.995 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.996 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.996 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:10.996 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:10.996 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:10.996 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.996 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.996 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:10.996 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.997 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.997 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:10.997 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:10.997 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:10.997 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:10.997 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:10.997 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:10.997 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:10.998 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:10.998 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.998 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:10.998 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:10.998 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:10.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:10.998 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:10.998 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:10.999 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:10.999 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:10.999 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:10.999 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:10.999 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:10.999 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:10.999 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:10.999 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:10.999 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.000 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.000 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.000 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.000 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.000 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "status", 2024-03-18T15:04:11.000 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.000 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.001 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.001 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.001 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.001 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.001 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.001 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.001 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.001 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.002 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.002 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.002 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.002 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.002 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.002 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.002 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.002 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.002 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.003 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.003 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.003 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.003 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.003 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.003 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.003 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.004 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.004 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.004 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.004 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.004 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.004 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.004 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.004 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.004 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.005 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.005 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.005 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.005 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.005 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.005 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.005 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.005 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.006 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.006 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.006 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.006 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.006 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.006 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.006 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.006 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.007 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.007 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.007 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.007 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.007 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.007 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.007 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.007 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.008 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.008 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.008 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.008 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.008 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.008 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.008 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.008 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.008 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.009 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.009 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.009 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.009 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "telegraf", 2024-03-18T15:04:11.009 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.009 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.009 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.010 INFO:teuthology.orchestra.run.smithi076.stdout: "address": { 2024-03-18T15:04:11.010 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "address", 2024-03-18T15:04:11.010 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.010 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.010 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.010 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T15:04:11.010 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.010 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.011 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.011 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.011 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.011 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.011 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.011 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-18T15:04:11.011 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-18T15:04:11.011 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.012 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.012 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.012 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15", 2024-03-18T15:04:11.012 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.012 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.012 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.012 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.012 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.013 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.013 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.013 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.013 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.013 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.013 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.013 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.013 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.014 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.014 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.014 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.014 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.014 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.014 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.014 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.014 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.015 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.015 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.015 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.015 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.015 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.015 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.015 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.015 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.016 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.016 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.016 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.016 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.016 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.016 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.016 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.016 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.016 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.017 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.017 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.017 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.017 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.017 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.017 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.017 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.017 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.018 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.018 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.018 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.018 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.018 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.018 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.018 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.018 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.019 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.019 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.019 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.019 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.019 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.019 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.019 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.019 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.020 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.020 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.020 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.020 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.020 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.020 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.020 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.020 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.020 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.021 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.021 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.021 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.021 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.021 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "telemetry", 2024-03-18T15:04:11.021 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.021 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.022 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.022 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_basic": { 2024-03-18T15:04:11.022 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_basic", 2024-03-18T15:04:11.022 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.022 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.022 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.022 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.022 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.022 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.023 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.023 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T15:04:11.023 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.023 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.023 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.023 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_crash": { 2024-03-18T15:04:11.023 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_crash", 2024-03-18T15:04:11.024 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.024 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.024 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.024 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.024 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.024 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.024 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.024 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T15:04:11.024 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.024 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.025 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.025 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_device": { 2024-03-18T15:04:11.025 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_device", 2024-03-18T15:04:11.025 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.025 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.025 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.025 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.025 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.026 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.026 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.026 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T15:04:11.026 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.026 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.026 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.026 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_ident": { 2024-03-18T15:04:11.027 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_ident", 2024-03-18T15:04:11.027 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.027 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.027 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.027 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.027 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.027 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.027 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.027 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T15:04:11.028 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.028 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.028 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.028 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_perf": { 2024-03-18T15:04:11.028 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_perf", 2024-03-18T15:04:11.028 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.028 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.028 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.029 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.029 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.029 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.029 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.029 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T15:04:11.029 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.029 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.029 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.030 INFO:teuthology.orchestra.run.smithi076.stdout: "contact": { 2024-03-18T15:04:11.030 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "contact", 2024-03-18T15:04:11.030 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.030 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.030 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.030 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.030 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.030 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.030 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.031 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.031 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.031 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.031 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.031 INFO:teuthology.orchestra.run.smithi076.stdout: "description": { 2024-03-18T15:04:11.031 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "description", 2024-03-18T15:04:11.031 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.031 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.032 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.032 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.032 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.032 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.032 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.032 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.032 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.032 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.032 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.033 INFO:teuthology.orchestra.run.smithi076.stdout: "device_url": { 2024-03-18T15:04:11.033 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_url", 2024-03-18T15:04:11.033 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.033 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.033 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.033 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T15:04:11.033 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.033 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.034 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.034 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.034 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.034 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.034 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.034 INFO:teuthology.orchestra.run.smithi076.stdout: "enabled": { 2024-03-18T15:04:11.034 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enabled", 2024-03-18T15:04:11.035 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.035 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.035 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.035 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.035 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.035 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.035 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.035 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.035 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.036 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.036 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.036 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-18T15:04:11.036 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-18T15:04:11.036 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.036 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.036 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.036 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "24", 2024-03-18T15:04:11.037 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "8", 2024-03-18T15:04:11.037 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.037 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.037 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.037 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.037 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.037 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.038 INFO:teuthology.orchestra.run.smithi076.stdout: "last_opt_revision": { 2024-03-18T15:04:11.038 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "last_opt_revision", 2024-03-18T15:04:11.038 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.038 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.038 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.038 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1", 2024-03-18T15:04:11.038 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.038 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.038 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.039 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.039 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.039 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.039 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.039 INFO:teuthology.orchestra.run.smithi076.stdout: "leaderboard": { 2024-03-18T15:04:11.039 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "leaderboard", 2024-03-18T15:04:11.039 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.039 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.040 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.040 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.040 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.040 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.040 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.040 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.040 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.040 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.040 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.041 INFO:teuthology.orchestra.run.smithi076.stdout: "leaderboard_description": { 2024-03-18T15:04:11.041 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "leaderboard_description", 2024-03-18T15:04:11.041 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.041 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.041 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.041 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.041 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.041 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.042 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.042 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.042 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.042 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.042 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.042 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.042 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.042 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.043 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.043 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.043 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.043 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.043 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.043 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.043 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.043 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.044 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.044 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.044 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.044 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.044 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.044 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.044 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.044 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.044 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.045 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.045 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.045 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.045 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.045 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.045 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.045 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.045 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.046 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.046 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.046 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.046 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.046 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.046 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.046 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.047 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.047 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.047 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.047 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.047 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.047 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.047 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.047 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.047 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.048 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.048 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.048 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.048 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.048 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.048 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.048 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.048 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.048 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.049 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.049 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.049 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.049 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.049 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.049 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.049 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.049 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.050 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.050 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.050 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.050 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.050 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.050 INFO:teuthology.orchestra.run.smithi076.stdout: "organization": { 2024-03-18T15:04:11.050 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "organization", 2024-03-18T15:04:11.050 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.051 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.051 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.051 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.051 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.051 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.051 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.051 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.051 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.051 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.052 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.052 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.052 INFO:teuthology.orchestra.run.smithi076.stdout: "proxy": { 2024-03-18T15:04:11.052 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "proxy", 2024-03-18T15:04:11.052 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.052 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.052 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.052 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.052 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.053 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.053 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.053 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.053 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.053 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.053 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.053 INFO:teuthology.orchestra.run.smithi076.stdout: "url": { 2024-03-18T15:04:11.053 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "url", 2024-03-18T15:04:11.054 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.054 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.054 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.054 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T15:04:11.054 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.054 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.054 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.054 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.055 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.055 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.055 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.055 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.055 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.055 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "test_orchestrator", 2024-03-18T15:04:11.055 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.056 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.056 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.056 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.056 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.056 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.056 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.056 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.056 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.057 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.057 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.057 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.057 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.057 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.057 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.057 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.057 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.057 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.058 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.058 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.058 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.058 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.058 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.058 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.058 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.058 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.059 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.059 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.059 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.059 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.059 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.059 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.059 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.059 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.059 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.060 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.060 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.060 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.060 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.060 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.060 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.060 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.061 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.061 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.061 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.061 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.061 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.061 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.061 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.061 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.062 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.062 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.062 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.062 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.062 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.062 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.062 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.062 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.063 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.063 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.063 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.063 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.063 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.063 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.063 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.063 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.063 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.064 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.064 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.064 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.064 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.064 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "volumes", 2024-03-18T15:04:11.064 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.064 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.065 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.065 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.065 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.065 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.065 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.065 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.065 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.065 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.066 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.066 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.066 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.066 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.066 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.066 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.066 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.066 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.067 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.067 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.067 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.067 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.067 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.067 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.067 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.068 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.068 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.068 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.068 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.068 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.068 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.068 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.068 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.068 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.069 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.069 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.069 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.069 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.069 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.069 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.069 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.070 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.070 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.070 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.070 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.070 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.070 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.070 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.070 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.070 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.071 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.071 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.071 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.071 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.071 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.071 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.071 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.072 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.072 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.072 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.072 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.072 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.072 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.072 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.072 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.072 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.073 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.073 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.073 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.073 INFO:teuthology.orchestra.run.smithi076.stdout: "max_concurrent_clones": { 2024-03-18T15:04:11.073 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_concurrent_clones", 2024-03-18T15:04:11.073 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.073 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.073 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.074 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "4", 2024-03-18T15:04:11.074 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.074 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.074 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.074 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T15:04:11.074 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.074 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.074 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.075 INFO:teuthology.orchestra.run.smithi076.stdout: "periodic_async_work": { 2024-03-18T15:04:11.075 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "periodic_async_work", 2024-03-18T15:04:11.075 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.075 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.075 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.075 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.075 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.076 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.076 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.076 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Periodically check for async work", 2024-03-18T15:04:11.076 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.076 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.076 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.076 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.076 INFO:teuthology.orchestra.run.smithi076.stdout: "snapshot_clone_delay": { 2024-03-18T15:04:11.076 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snapshot_clone_delay", 2024-03-18T15:04:11.077 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.077 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.077 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.077 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-18T15:04:11.077 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.077 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.077 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.077 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T15:04:11.077 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.077 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.078 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.078 INFO:teuthology.orchestra.run.smithi076.stdout: "snapshot_clone_no_wait": { 2024-03-18T15:04:11.078 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T15:04:11.078 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.078 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.078 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.078 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.079 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.079 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.079 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.079 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T15:04:11.079 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.079 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.079 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.079 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.080 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.080 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix", 2024-03-18T15:04:11.080 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.080 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.080 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.080 INFO:teuthology.orchestra.run.smithi076.stdout: "discovery_interval": { 2024-03-18T15:04:11.080 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "discovery_interval", 2024-03-18T15:04:11.081 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-18T15:04:11.081 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.081 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.081 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "100", 2024-03-18T15:04:11.081 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.081 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.081 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.081 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.082 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.082 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.082 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.082 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.082 INFO:teuthology.orchestra.run.smithi076.stdout: "identifier": { 2024-03-18T15:04:11.082 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "identifier", 2024-03-18T15:04:11.082 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.082 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.083 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.083 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.083 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.083 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.083 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.083 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.083 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.083 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.084 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.084 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-18T15:04:11.084 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-18T15:04:11.084 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.084 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.084 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.084 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-18T15:04:11.084 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.085 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.085 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.085 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.085 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.085 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.085 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.085 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.085 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.085 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.086 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.086 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.086 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.086 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.086 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.086 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.086 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.086 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.087 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.087 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.087 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.087 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.087 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.087 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.087 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.087 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.087 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.088 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.088 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.088 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.088 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.088 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.088 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.088 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.088 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.089 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.089 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.089 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.089 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.089 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.089 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.089 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.090 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.090 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.090 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.090 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.090 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.090 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.090 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.090 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.090 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.091 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.091 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.091 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.091 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.091 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.091 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.091 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.091 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.092 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.092 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.092 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.092 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.092 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.092 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.092 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.093 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.093 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.093 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.093 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.093 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.093 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.093 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.093 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_host": { 2024-03-18T15:04:11.094 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_host", 2024-03-18T15:04:11.094 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.094 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.094 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.094 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.094 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.094 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.094 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.095 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.095 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.095 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.095 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.095 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_port": { 2024-03-18T15:04:11.095 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_port", 2024-03-18T15:04:11.095 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.096 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.096 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.096 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10051", 2024-03-18T15:04:11.096 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.096 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.096 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.096 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.096 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.096 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.097 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.097 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_sender": { 2024-03-18T15:04:11.097 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_sender", 2024-03-18T15:04:11.097 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.097 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.097 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.097 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T15:04:11.097 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.098 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.098 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.098 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.098 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.098 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.098 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.098 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.098 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.098 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.099 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-18T15:04:11.099 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.099 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.099 INFO:teuthology.orchestra.run.smithi076.stdout: "modules": [ 2024-03-18T15:04:11.099 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm", 2024-03-18T15:04:11.099 INFO:teuthology.orchestra.run.smithi076.stdout: "dashboard", 2024-03-18T15:04:11.099 INFO:teuthology.orchestra.run.smithi076.stdout: "iostat", 2024-03-18T15:04:11.099 INFO:teuthology.orchestra.run.smithi076.stdout: "nfs", 2024-03-18T15:04:11.099 INFO:teuthology.orchestra.run.smithi076.stdout: "restful", 2024-03-18T15:04:11.100 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_schedule" 2024-03-18T15:04:11.100 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.100 INFO:teuthology.orchestra.run.smithi076.stdout: "available_modules": [ 2024-03-18T15:04:11.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.100 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "alerts", 2024-03-18T15:04:11.100 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.100 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.100 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.100 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-18T15:04:11.101 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-18T15:04:11.101 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.101 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.101 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.101 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-18T15:04:11.101 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.101 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.101 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.101 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T15:04:11.102 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.102 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.102 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.102 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.102 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.102 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.102 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.103 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.103 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.103 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.103 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.103 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.103 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.103 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.103 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.104 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.104 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.104 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.104 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.104 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.104 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.104 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.104 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.105 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.105 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.105 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.105 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.105 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.105 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.105 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.105 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.105 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.106 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.106 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.106 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.106 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.106 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.106 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.106 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.107 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.107 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.107 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.107 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.107 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.107 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.107 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.107 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.107 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.108 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.108 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.108 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.108 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.108 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.108 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.108 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.108 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.109 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.109 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.109 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.109 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.109 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.109 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.109 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.109 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.109 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.110 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.110 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.110 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.110 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.110 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_destination": { 2024-03-18T15:04:11.110 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_destination", 2024-03-18T15:04:11.110 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.110 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.111 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.111 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.111 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.111 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.111 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.111 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Email address to send alerts to", 2024-03-18T15:04:11.111 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.111 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.112 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.112 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_from_name": { 2024-03-18T15:04:11.112 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_from_name", 2024-03-18T15:04:11.112 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.112 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.112 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.112 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "Ceph", 2024-03-18T15:04:11.112 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.113 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.113 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.113 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Email From: name", 2024-03-18T15:04:11.113 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.113 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.113 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.113 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_host": { 2024-03-18T15:04:11.113 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_host", 2024-03-18T15:04:11.114 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.114 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.114 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.114 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.114 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.114 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.114 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.114 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP server", 2024-03-18T15:04:11.115 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.115 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.115 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.115 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_password": { 2024-03-18T15:04:11.115 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_password", 2024-03-18T15:04:11.115 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.115 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.115 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.115 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.116 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.116 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.116 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.116 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Password to authenticate with", 2024-03-18T15:04:11.116 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.116 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.116 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.116 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_port": { 2024-03-18T15:04:11.117 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_port", 2024-03-18T15:04:11.117 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.117 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.117 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.117 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "465", 2024-03-18T15:04:11.117 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.117 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.117 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.117 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP port", 2024-03-18T15:04:11.117 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.118 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.118 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.118 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_sender": { 2024-03-18T15:04:11.118 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_sender", 2024-03-18T15:04:11.118 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.118 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.118 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.119 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.119 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.119 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.119 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.119 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP envelope sender", 2024-03-18T15:04:11.119 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.119 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.119 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.119 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_ssl": { 2024-03-18T15:04:11.120 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_ssl", 2024-03-18T15:04:11.120 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.120 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.120 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.120 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.120 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.120 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.120 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.120 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T15:04:11.121 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.121 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.121 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.121 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_user": { 2024-03-18T15:04:11.121 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_user", 2024-03-18T15:04:11.121 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.121 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.122 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.122 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.122 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.122 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.122 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.122 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "User to authenticate as", 2024-03-18T15:04:11.122 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.122 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.122 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.123 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.123 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.123 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "balancer", 2024-03-18T15:04:11.123 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.123 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.123 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.124 INFO:teuthology.orchestra.run.smithi076.stdout: "active": { 2024-03-18T15:04:11.124 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "active", 2024-03-18T15:04:11.124 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.124 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.124 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.124 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.124 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.124 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.124 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.125 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T15:04:11.125 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.125 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.125 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.125 INFO:teuthology.orchestra.run.smithi076.stdout: "begin_time": { 2024-03-18T15:04:11.125 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "begin_time", 2024-03-18T15:04:11.125 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.125 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.126 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.126 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0000", 2024-03-18T15:04:11.126 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.126 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.126 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.126 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T15:04:11.126 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T15:04:11.126 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.126 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.127 INFO:teuthology.orchestra.run.smithi076.stdout: "begin_weekday": { 2024-03-18T15:04:11.127 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "begin_weekday", 2024-03-18T15:04:11.127 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-18T15:04:11.127 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.127 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.127 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-18T15:04:11.127 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0", 2024-03-18T15:04:11.127 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "6", 2024-03-18T15:04:11.127 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.128 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T15:04:11.128 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T15:04:11.128 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.128 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.128 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_max_iterations": { 2024-03-18T15:04:11.128 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_max_iterations", 2024-03-18T15:04:11.128 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-18T15:04:11.128 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.129 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.129 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "25", 2024-03-18T15:04:11.129 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-18T15:04:11.129 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "250", 2024-03-18T15:04:11.129 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.129 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T15:04:11.129 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.129 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.129 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.130 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_metrics": { 2024-03-18T15:04:11.130 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_metrics", 2024-03-18T15:04:11.130 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.130 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.130 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.130 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T15:04:11.130 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.130 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.131 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.131 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T15:04:11.131 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T15:04:11.131 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.131 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.131 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_step": { 2024-03-18T15:04:11.131 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_step", 2024-03-18T15:04:11.132 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:11.132 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.132 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.132 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0.5", 2024-03-18T15:04:11.132 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0.001", 2024-03-18T15:04:11.132 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "0.999", 2024-03-18T15:04:11.132 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.132 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "aggressiveness of optimization", 2024-03-18T15:04:11.132 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T15:04:11.133 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.133 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.133 INFO:teuthology.orchestra.run.smithi076.stdout: "end_time": { 2024-03-18T15:04:11.133 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "end_time", 2024-03-18T15:04:11.133 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.133 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.133 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.134 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "2359", 2024-03-18T15:04:11.134 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.134 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.134 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.134 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T15:04:11.134 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T15:04:11.134 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.134 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.135 INFO:teuthology.orchestra.run.smithi076.stdout: "end_weekday": { 2024-03-18T15:04:11.135 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "end_weekday", 2024-03-18T15:04:11.135 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-18T15:04:11.135 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.135 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.135 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-18T15:04:11.135 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0", 2024-03-18T15:04:11.135 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "6", 2024-03-18T15:04:11.136 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.136 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T15:04:11.136 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T15:04:11.136 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.136 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.136 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.136 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.136 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.137 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.137 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.137 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.137 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.137 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.137 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.137 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.137 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.137 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.138 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.138 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.138 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.138 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.138 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.138 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.138 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.138 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.139 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.139 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.139 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.139 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.139 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.139 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.139 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.139 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.140 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.140 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.140 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.140 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.140 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.140 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.140 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.141 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.141 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.141 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.141 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.141 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.141 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.141 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.141 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.141 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.142 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.142 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.142 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.142 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.142 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.142 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.142 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.142 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.143 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.143 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.143 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.143 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.143 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.143 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.143 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.143 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.144 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.144 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.144 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.144 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.144 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.144 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.144 INFO:teuthology.orchestra.run.smithi076.stdout: "min_score": { 2024-03-18T15:04:11.144 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "min_score", 2024-03-18T15:04:11.144 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:11.145 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.145 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.145 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-18T15:04:11.145 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.145 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.145 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.145 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T15:04:11.145 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.145 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.146 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.146 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": { 2024-03-18T15:04:11.146 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mode", 2024-03-18T15:04:11.146 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.146 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.146 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.146 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "upmap", 2024-03-18T15:04:11.146 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.147 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.147 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.147 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-compat", 2024-03-18T15:04:11.147 INFO:teuthology.orchestra.run.smithi076.stdout: "none", 2024-03-18T15:04:11.147 INFO:teuthology.orchestra.run.smithi076.stdout: "read", 2024-03-18T15:04:11.147 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap", 2024-03-18T15:04:11.147 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap-read" 2024-03-18T15:04:11.147 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.148 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Balancer mode", 2024-03-18T15:04:11.148 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.148 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.148 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.148 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_ids": { 2024-03-18T15:04:11.148 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pool_ids", 2024-03-18T15:04:11.148 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.148 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.149 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.149 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.149 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.149 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.149 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.149 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T15:04:11.149 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.149 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.150 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.150 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-18T15:04:11.150 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-18T15:04:11.150 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.150 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.150 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.150 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-18T15:04:11.151 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.151 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.151 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.151 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T15:04:11.151 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.151 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.151 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.151 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap_max_deviation": { 2024-03-18T15:04:11.152 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "upmap_max_deviation", 2024-03-18T15:04:11.152 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.152 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.152 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.152 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-18T15:04:11.152 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-18T15:04:11.152 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.152 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.153 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T15:04:11.153 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T15:04:11.153 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.153 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.153 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap_max_optimizations": { 2024-03-18T15:04:11.153 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "upmap_max_optimizations", 2024-03-18T15:04:11.153 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-18T15:04:11.153 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.154 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.154 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:11.154 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.154 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.154 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.154 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T15:04:11.154 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.154 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.155 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.155 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.155 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.155 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cephadm", 2024-03-18T15:04:11.155 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.155 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.155 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.156 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_down_multiplier": { 2024-03-18T15:04:11.156 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_down_multiplier", 2024-03-18T15:04:11.156 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:11.156 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.156 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.156 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3.0", 2024-03-18T15:04:11.156 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.156 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.157 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.157 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T15:04:11.157 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.157 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.157 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.157 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_refresh_rate": { 2024-03-18T15:04:11.157 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_refresh_rate", 2024-03-18T15:04:11.157 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.158 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.158 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.158 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "20", 2024-03-18T15:04:11.158 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.158 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.158 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.158 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T15:04:11.158 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.158 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.159 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.159 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_starting_port": { 2024-03-18T15:04:11.159 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_starting_port", 2024-03-18T15:04:11.159 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.159 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.159 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.159 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "4721", 2024-03-18T15:04:11.159 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.159 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.160 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.160 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T15:04:11.160 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.160 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.160 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.160 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_ptrace": { 2024-03-18T15:04:11.160 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_ptrace", 2024-03-18T15:04:11.161 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.161 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.161 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.161 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.161 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.161 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.161 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.161 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T15:04:11.161 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T15:04:11.162 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.162 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.162 INFO:teuthology.orchestra.run.smithi076.stdout: "autotune_interval": { 2024-03-18T15:04:11.162 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "autotune_interval", 2024-03-18T15:04:11.162 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.162 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.162 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.162 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-18T15:04:11.163 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.163 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.163 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.163 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T15:04:11.163 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.163 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.163 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.163 INFO:teuthology.orchestra.run.smithi076.stdout: "autotune_memory_target_ratio": { 2024-03-18T15:04:11.164 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T15:04:11.164 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:11.164 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.164 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.164 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0.7", 2024-03-18T15:04:11.164 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.164 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.164 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.164 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T15:04:11.165 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.165 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.165 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.165 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm_log_destination": { 2024-03-18T15:04:11.165 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cephadm_log_destination", 2024-03-18T15:04:11.165 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.165 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.165 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.165 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.166 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.166 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.166 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.166 INFO:teuthology.orchestra.run.smithi076.stdout: "file", 2024-03-18T15:04:11.166 INFO:teuthology.orchestra.run.smithi076.stdout: "file,syslog", 2024-03-18T15:04:11.166 INFO:teuthology.orchestra.run.smithi076.stdout: "syslog" 2024-03-18T15:04:11.166 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.166 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T15:04:11.167 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.167 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.167 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.167 INFO:teuthology.orchestra.run.smithi076.stdout: "cgroups_split": { 2024-03-18T15:04:11.167 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cgroups_split", 2024-03-18T15:04:11.167 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.167 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.168 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.168 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.168 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.168 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.168 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.168 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T15:04:11.168 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.168 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.168 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.169 INFO:teuthology.orchestra.run.smithi076.stdout: "config_checks_enabled": { 2024-03-18T15:04:11.169 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_checks_enabled", 2024-03-18T15:04:11.169 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.169 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.169 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.169 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.169 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.170 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.170 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.170 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T15:04:11.170 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.170 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.170 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.170 INFO:teuthology.orchestra.run.smithi076.stdout: "config_dashboard": { 2024-03-18T15:04:11.171 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_dashboard", 2024-03-18T15:04:11.171 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.171 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.171 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.171 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.171 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.171 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.171 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.171 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T15:04:11.172 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.172 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.172 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.172 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_alertmanager": { 2024-03-18T15:04:11.172 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_alertmanager", 2024-03-18T15:04:11.172 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.172 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.172 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.173 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T15:04:11.173 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.173 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.173 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.173 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-18T15:04:11.173 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.173 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.173 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.174 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_base": { 2024-03-18T15:04:11.174 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_base", 2024-03-18T15:04:11.174 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.174 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.174 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.174 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T15:04:11.174 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.174 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.175 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.175 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Container image name, without the tag", 2024-03-18T15:04:11.175 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.175 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.175 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.175 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_elasticsearch": { 2024-03-18T15:04:11.175 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_elasticsearch", 2024-03-18T15:04:11.175 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.176 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.176 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.176 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T15:04:11.176 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.176 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.176 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.176 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "elasticsearch container image", 2024-03-18T15:04:11.176 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.177 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.177 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.177 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_grafana": { 2024-03-18T15:04:11.177 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_grafana", 2024-03-18T15:04:11.177 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.177 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.177 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.177 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T15:04:11.178 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.178 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.178 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.178 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-18T15:04:11.178 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.178 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.178 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.178 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_haproxy": { 2024-03-18T15:04:11.179 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_haproxy", 2024-03-18T15:04:11.179 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.179 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.179 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.179 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T15:04:11.179 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.179 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.179 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.179 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "HAproxy container image", 2024-03-18T15:04:11.180 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.180 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.180 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.180 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_agent": { 2024-03-18T15:04:11.180 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_agent", 2024-03-18T15:04:11.180 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.180 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.180 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.181 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T15:04:11.181 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.181 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.181 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.181 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger agent container image", 2024-03-18T15:04:11.181 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.181 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.181 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.182 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_collector": { 2024-03-18T15:04:11.182 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_collector", 2024-03-18T15:04:11.182 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.182 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.182 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.182 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T15:04:11.182 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.182 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.182 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.183 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger collector container image", 2024-03-18T15:04:11.183 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.183 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.183 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.183 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_query": { 2024-03-18T15:04:11.183 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_query", 2024-03-18T15:04:11.183 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.183 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.184 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.184 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T15:04:11.184 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.184 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.184 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.184 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger query container image", 2024-03-18T15:04:11.184 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.184 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.184 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.185 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_keepalived": { 2024-03-18T15:04:11.185 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_keepalived", 2024-03-18T15:04:11.185 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.185 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.185 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.185 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T15:04:11.185 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.185 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.185 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.186 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Keepalived container image", 2024-03-18T15:04:11.186 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.186 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.186 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.186 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_loki": { 2024-03-18T15:04:11.186 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_loki", 2024-03-18T15:04:11.186 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.186 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.187 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.187 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T15:04:11.187 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.187 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.187 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.187 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Loki container image", 2024-03-18T15:04:11.187 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.187 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.187 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.188 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_node_exporter": { 2024-03-18T15:04:11.188 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_node_exporter", 2024-03-18T15:04:11.188 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.188 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.188 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.188 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T15:04:11.188 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.188 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.189 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.189 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-18T15:04:11.189 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.189 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.189 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.189 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_nvmeof": { 2024-03-18T15:04:11.189 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_nvmeof", 2024-03-18T15:04:11.189 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.189 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T15:04:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Nvme-of container image", 2024-03-18T15:04:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_prometheus": { 2024-03-18T15:04:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_prometheus", 2024-03-18T15:04:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T15:04:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-18T15:04:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_promtail": { 2024-03-18T15:04:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_promtail", 2024-03-18T15:04:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T15:04:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Promtail container image", 2024-03-18T15:04:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_snmp_gateway": { 2024-03-18T15:04:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_snmp_gateway", 2024-03-18T15:04:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.195 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T15:04:11.195 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.195 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.195 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SNMP Gateway container image", 2024-03-18T15:04:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "container_init": { 2024-03-18T15:04:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_init", 2024-03-18T15:04:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T15:04:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: "daemon_cache_timeout": { 2024-03-18T15:04:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "daemon_cache_timeout", 2024-03-18T15:04:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-18T15:04:11.201 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.201 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.201 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.201 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T15:04:11.201 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.201 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.201 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.201 INFO:teuthology.orchestra.run.smithi076.stdout: "default_cephadm_command_timeout": { 2024-03-18T15:04:11.202 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T15:04:11.202 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.202 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.202 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.202 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "900", 2024-03-18T15:04:11.202 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.202 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.202 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.202 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T15:04:11.203 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.203 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.203 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.203 INFO:teuthology.orchestra.run.smithi076.stdout: "default_registry": { 2024-03-18T15:04:11.203 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "default_registry", 2024-03-18T15:04:11.203 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.203 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.204 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.204 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io", 2024-03-18T15:04:11.204 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.204 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.204 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.204 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T15:04:11.204 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.204 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.204 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.205 INFO:teuthology.orchestra.run.smithi076.stdout: "device_cache_timeout": { 2024-03-18T15:04:11.205 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_cache_timeout", 2024-03-18T15:04:11.205 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.205 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.205 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.205 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1800", 2024-03-18T15:04:11.205 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.205 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.205 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.205 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache device inventory", 2024-03-18T15:04:11.206 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.206 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.206 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.206 INFO:teuthology.orchestra.run.smithi076.stdout: "device_enhanced_scan": { 2024-03-18T15:04:11.206 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_enhanced_scan", 2024-03-18T15:04:11.206 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.206 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.207 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.207 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.207 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.207 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.207 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.207 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T15:04:11.207 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.207 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.207 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.208 INFO:teuthology.orchestra.run.smithi076.stdout: "facts_cache_timeout": { 2024-03-18T15:04:11.208 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "facts_cache_timeout", 2024-03-18T15:04:11.208 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.208 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.208 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.208 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-18T15:04:11.208 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.209 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.209 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.209 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache host facts data", 2024-03-18T15:04:11.209 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.209 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.209 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.209 INFO:teuthology.orchestra.run.smithi076.stdout: "host_check_interval": { 2024-03-18T15:04:11.209 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "host_check_interval", 2024-03-18T15:04:11.210 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.210 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.210 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.210 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-18T15:04:11.210 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.210 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.210 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.210 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to perform a host check", 2024-03-18T15:04:11.211 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.211 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.211 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.211 INFO:teuthology.orchestra.run.smithi076.stdout: "hw_monitoring": { 2024-03-18T15:04:11.211 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "hw_monitoring", 2024-03-18T15:04:11.211 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.211 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.211 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.212 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.212 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.212 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.212 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.212 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T15:04:11.212 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.212 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.212 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.213 INFO:teuthology.orchestra.run.smithi076.stdout: "inventory_list_all": { 2024-03-18T15:04:11.213 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "inventory_list_all", 2024-03-18T15:04:11.213 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.213 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.213 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.213 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.213 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.213 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.213 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.213 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T15:04:11.214 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.214 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.214 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.214 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.214 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.214 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.214 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.215 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.215 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.215 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.215 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.215 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.215 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.215 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.215 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.215 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.216 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.216 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.216 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.216 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.216 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.216 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.216 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.216 INFO:teuthology.orchestra.run.smithi076.stdout: "log_refresh_metadata": { 2024-03-18T15:04:11.217 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_refresh_metadata", 2024-03-18T15:04:11.217 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.217 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.217 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.217 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.217 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.217 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.217 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.217 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T15:04:11.218 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.218 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.218 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.218 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.218 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.218 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.218 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.219 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.219 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.219 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.219 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.219 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.219 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T15:04:11.219 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.219 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.219 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.220 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.220 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.220 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.220 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.220 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.220 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.220 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.220 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.220 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.221 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.221 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.221 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.221 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.221 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.221 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.221 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.221 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.221 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.222 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.222 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.222 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.222 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.222 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.222 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.222 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.222 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.223 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.223 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.223 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.223 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.223 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.223 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.223 INFO:teuthology.orchestra.run.smithi076.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T15:04:11.223 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T15:04:11.224 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.224 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.224 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.224 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.224 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.224 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.224 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.224 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T15:04:11.224 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.225 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.225 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.225 INFO:teuthology.orchestra.run.smithi076.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T15:04:11.225 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T15:04:11.225 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.225 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.225 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.225 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "*", 2024-03-18T15:04:11.225 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.226 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.226 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.226 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T15:04:11.226 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.226 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.226 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.226 INFO:teuthology.orchestra.run.smithi076.stdout: "max_count_per_host": { 2024-03-18T15:04:11.227 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_count_per_host", 2024-03-18T15:04:11.227 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.227 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.227 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.227 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:11.227 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.227 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.227 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.228 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "max number of daemons per service per host", 2024-03-18T15:04:11.228 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.228 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.228 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.228 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd_draining_count": { 2024-03-18T15:04:11.228 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_osd_draining_count", 2024-03-18T15:04:11.228 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.229 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.229 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.229 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:11.229 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.229 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.229 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.229 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T15:04:11.229 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.229 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.230 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.230 INFO:teuthology.orchestra.run.smithi076.stdout: "migration_current": { 2024-03-18T15:04:11.230 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "migration_current", 2024-03-18T15:04:11.230 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.230 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.230 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.230 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.231 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.231 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.231 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.231 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "internal - do not modify", 2024-03-18T15:04:11.231 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.231 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.231 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.231 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": { 2024-03-18T15:04:11.232 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mode", 2024-03-18T15:04:11.232 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.232 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.232 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.232 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "root", 2024-03-18T15:04:11.232 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.232 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.232 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.233 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm-package", 2024-03-18T15:04:11.233 INFO:teuthology.orchestra.run.smithi076.stdout: "root" 2024-03-18T15:04:11.233 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.233 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T15:04:11.233 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.233 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.233 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.234 INFO:teuthology.orchestra.run.smithi076.stdout: "oob_default_addr": { 2024-03-18T15:04:11.234 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "oob_default_addr", 2024-03-18T15:04:11.234 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.234 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.234 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.234 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "169.254.1.1", 2024-03-18T15:04:11.234 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.234 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.235 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.235 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T15:04:11.235 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.235 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.235 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.235 INFO:teuthology.orchestra.run.smithi076.stdout: "prometheus_alerts_path": { 2024-03-18T15:04:11.235 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prometheus_alerts_path", 2024-03-18T15:04:11.236 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.236 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.236 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.236 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T15:04:11.236 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.236 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.236 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.236 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T15:04:11.236 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.237 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.237 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.237 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_insecure": { 2024-03-18T15:04:11.237 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_insecure", 2024-03-18T15:04:11.237 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.237 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.237 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.237 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.238 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.238 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.238 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.238 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T15:04:11.238 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.238 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.238 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.238 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_password": { 2024-03-18T15:04:11.239 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_password", 2024-03-18T15:04:11.239 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.239 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.239 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.239 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.239 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.239 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.239 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.239 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T15:04:11.240 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.240 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.240 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.240 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_url": { 2024-03-18T15:04:11.240 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_url", 2024-03-18T15:04:11.240 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.240 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.240 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.241 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.241 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.241 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.241 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.241 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T15:04:11.241 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.241 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.241 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.242 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_username": { 2024-03-18T15:04:11.242 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_username", 2024-03-18T15:04:11.242 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.242 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.242 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.242 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.242 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.242 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.242 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.243 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T15:04:11.243 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.243 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.243 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.243 INFO:teuthology.orchestra.run.smithi076.stdout: "secure_monitoring_stack": { 2024-03-18T15:04:11.243 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "secure_monitoring_stack", 2024-03-18T15:04:11.243 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.244 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.244 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.244 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.244 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.244 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.244 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.244 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T15:04:11.244 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.244 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.245 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.245 INFO:teuthology.orchestra.run.smithi076.stdout: "service_discovery_port": { 2024-03-18T15:04:11.245 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "service_discovery_port", 2024-03-18T15:04:11.245 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.245 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.245 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.245 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8765", 2024-03-18T15:04:11.245 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.245 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.246 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.246 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "cephadm service discovery port", 2024-03-18T15:04:11.246 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.246 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.246 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.246 INFO:teuthology.orchestra.run.smithi076.stdout: "ssh_config_file": { 2024-03-18T15:04:11.246 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssh_config_file", 2024-03-18T15:04:11.246 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.247 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.247 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.247 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.247 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.247 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.247 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.247 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T15:04:11.247 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.247 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.248 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.248 INFO:teuthology.orchestra.run.smithi076.stdout: "use_agent": { 2024-03-18T15:04:11.248 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "use_agent", 2024-03-18T15:04:11.248 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.248 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.248 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.248 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.249 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.249 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.249 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.249 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T15:04:11.249 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.249 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.249 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.250 INFO:teuthology.orchestra.run.smithi076.stdout: "use_repo_digest": { 2024-03-18T15:04:11.250 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "use_repo_digest", 2024-03-18T15:04:11.250 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.250 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.250 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.250 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.250 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.250 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.251 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.251 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T15:04:11.251 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.251 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.251 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.251 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_failed_host_check": { 2024-03-18T15:04:11.251 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_failed_host_check", 2024-03-18T15:04:11.251 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.252 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.252 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.252 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.252 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.252 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.252 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.252 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T15:04:11.252 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.252 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.253 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.253 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_stray_daemons": { 2024-03-18T15:04:11.253 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_stray_daemons", 2024-03-18T15:04:11.253 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.253 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.253 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.253 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.253 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.254 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.254 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.254 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T15:04:11.254 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.254 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.254 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.254 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_stray_hosts": { 2024-03-18T15:04:11.255 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_stray_hosts", 2024-03-18T15:04:11.255 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.255 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.255 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.255 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.255 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.255 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.255 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.255 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T15:04:11.255 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.256 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.256 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.256 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.256 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.256 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crash", 2024-03-18T15:04:11.256 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.256 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.257 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.257 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.257 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.257 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.257 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.257 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.257 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.257 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.257 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.258 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.258 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.258 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.258 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.258 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.258 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.258 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.258 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.258 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.259 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.259 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.259 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.259 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.259 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.259 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.259 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.259 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.260 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.260 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.260 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.260 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.260 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.260 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.260 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.260 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.261 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.261 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.261 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.261 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.261 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.261 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.261 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.261 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.262 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.262 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.262 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.262 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.262 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.262 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.262 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.262 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.263 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.263 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.263 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.263 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.263 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.263 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.263 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.263 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.264 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.264 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.264 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.264 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.264 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.264 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.264 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.264 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.264 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.265 INFO:teuthology.orchestra.run.smithi076.stdout: "retain_interval": { 2024-03-18T15:04:11.265 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "retain_interval", 2024-03-18T15:04:11.265 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.265 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.265 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.265 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "31536000", 2024-03-18T15:04:11.265 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.265 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.266 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.266 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T15:04:11.266 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.266 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.266 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.266 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_recent_interval": { 2024-03-18T15:04:11.266 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_recent_interval", 2024-03-18T15:04:11.267 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.267 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.267 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.267 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1209600", 2024-03-18T15:04:11.267 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.267 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.267 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.267 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T15:04:11.267 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.268 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.268 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.268 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.268 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.268 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "dashboard", 2024-03-18T15:04:11.268 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.268 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.268 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.269 INFO:teuthology.orchestra.run.smithi076.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T15:04:11.269 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T15:04:11.269 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.269 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.269 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.269 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:11.269 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.269 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.270 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.270 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.270 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.270 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.270 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.270 INFO:teuthology.orchestra.run.smithi076.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T15:04:11.270 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T15:04:11.270 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.271 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.271 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.271 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.271 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.271 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.271 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.271 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.271 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.271 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.271 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T15:04:11.272 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T15:04:11.272 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.272 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.272 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.272 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.272 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.273 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.273 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.273 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.273 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.273 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.273 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.273 INFO:teuthology.orchestra.run.smithi076.stdout: "AUDIT_API_ENABLED": { 2024-03-18T15:04:11.273 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T15:04:11.273 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.274 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.274 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.274 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.274 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.274 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.274 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.274 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.274 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.275 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.275 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.275 INFO:teuthology.orchestra.run.smithi076.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T15:04:11.275 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T15:04:11.275 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.275 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.275 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.275 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.275 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.276 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.276 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.276 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.276 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.276 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.276 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.276 INFO:teuthology.orchestra.run.smithi076.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T15:04:11.277 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T15:04:11.277 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.277 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.277 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.277 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.277 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.277 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.277 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.277 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.277 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.278 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.278 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.278 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T15:04:11.278 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T15:04:11.278 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.278 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.278 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.279 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.279 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.279 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.279 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.279 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.279 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.279 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.279 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.279 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T15:04:11.280 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T15:04:11.280 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.280 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.280 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.280 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.280 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.280 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.280 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.280 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.281 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.281 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.281 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.281 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T15:04:11.281 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T15:04:11.281 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.281 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.282 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.282 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.282 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.282 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.282 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.282 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.282 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.282 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.283 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.283 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T15:04:11.283 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T15:04:11.283 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.283 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.283 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.283 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.283 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.284 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.284 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.284 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.284 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.284 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.284 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.284 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T15:04:11.285 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T15:04:11.285 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.285 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.285 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.285 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.285 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.285 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.285 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.286 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.286 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.286 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.286 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.286 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T15:04:11.286 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T15:04:11.286 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.287 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.287 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.287 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.287 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.287 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.287 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.287 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.287 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.288 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.288 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.288 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T15:04:11.288 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T15:04:11.288 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.288 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.288 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.288 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.289 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.289 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.289 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.289 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.289 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.289 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.289 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.289 INFO:teuthology.orchestra.run.smithi076.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T15:04:11.290 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T15:04:11.290 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.290 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.290 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.290 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.290 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.290 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.290 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.290 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.291 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.291 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.291 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.291 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T15:04:11.291 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T15:04:11.291 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.291 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.291 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.292 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-18T15:04:11.292 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.292 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.292 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.292 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.292 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.292 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.292 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.293 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T15:04:11.293 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T15:04:11.293 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.293 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.293 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.293 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.293 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.293 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.293 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.294 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.294 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.294 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.294 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.294 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_URL": { 2024-03-18T15:04:11.294 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_URL", 2024-03-18T15:04:11.294 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.295 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.295 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.295 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.295 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.295 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.295 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.295 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.295 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.295 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.296 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.296 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T15:04:11.296 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T15:04:11.296 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.296 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.296 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.296 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-18T15:04:11.297 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.297 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.297 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.297 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.297 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.297 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.297 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.298 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T15:04:11.298 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T15:04:11.298 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.298 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.298 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.298 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.298 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.298 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.298 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.299 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.299 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.299 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.299 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.299 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T15:04:11.299 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T15:04:11.299 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.300 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.300 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.300 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.300 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.300 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.300 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.300 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.300 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.301 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.301 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.301 INFO:teuthology.orchestra.run.smithi076.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T15:04:11.301 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T15:04:11.301 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.301 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.301 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.301 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.302 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.302 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.302 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.302 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.302 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.302 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.302 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.303 INFO:teuthology.orchestra.run.smithi076.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T15:04:11.303 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T15:04:11.303 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.303 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.303 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.303 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.303 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.303 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.304 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.304 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.304 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.304 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.304 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.304 INFO:teuthology.orchestra.run.smithi076.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T15:04:11.304 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T15:04:11.304 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.305 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.305 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.305 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "{}", 2024-03-18T15:04:11.305 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.305 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.305 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.305 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.305 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.305 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.305 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.306 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.306 INFO:teuthology.orchestra.run.smithi076.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T15:04:11.306 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T15:04:11.306 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.306 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.306 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.306 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.306 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.307 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.307 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.307 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.307 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.307 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.307 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.307 INFO:teuthology.orchestra.run.smithi076.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T15:04:11.307 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T15:04:11.308 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.308 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.308 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.308 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.308 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.308 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.308 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.308 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.308 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.309 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.309 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.309 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T15:04:11.309 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T15:04:11.309 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.309 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.309 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.309 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.310 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.310 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.310 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.310 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.310 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.310 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.310 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.311 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T15:04:11.311 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T15:04:11.311 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.311 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.311 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.311 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.311 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.311 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.311 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.312 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.312 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.312 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.312 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.312 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T15:04:11.312 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T15:04:11.312 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.312 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.313 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.313 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.313 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.313 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.313 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.313 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.313 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.313 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.314 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.314 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T15:04:11.314 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T15:04:11.314 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.314 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.314 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.314 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.314 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.315 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.315 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.315 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.315 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.315 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.315 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.315 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T15:04:11.315 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T15:04:11.316 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.316 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.316 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.316 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.316 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.316 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.316 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.316 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.317 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.317 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.317 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.317 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T15:04:11.317 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T15:04:11.317 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.317 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.317 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.318 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.318 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.318 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.318 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.318 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.318 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.318 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.318 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.319 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T15:04:11.319 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T15:04:11.319 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.319 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.319 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.319 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.319 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.319 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.319 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.320 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.320 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.320 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.320 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.320 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T15:04:11.320 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T15:04:11.320 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.320 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.321 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.321 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.321 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.321 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.321 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.321 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.321 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.321 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.321 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.322 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T15:04:11.322 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T15:04:11.322 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.322 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.322 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.322 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T15:04:11.322 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.322 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.323 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.323 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.323 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.323 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.323 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.323 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T15:04:11.323 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T15:04:11.324 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.324 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.324 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.324 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:11.324 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.324 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.324 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.324 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.324 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.325 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.325 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.325 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T15:04:11.325 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T15:04:11.325 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.325 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.325 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.326 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8", 2024-03-18T15:04:11.326 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.326 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.326 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.326 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.326 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.326 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.326 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.327 INFO:teuthology.orchestra.run.smithi076.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T15:04:11.327 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T15:04:11.327 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.327 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.327 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.327 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "45", 2024-03-18T15:04:11.327 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.327 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.327 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.328 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.328 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.328 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.328 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.328 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T15:04:11.328 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T15:04:11.328 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.329 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.329 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.329 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.329 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.329 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.329 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.329 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.329 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.330 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.330 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.330 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T15:04:11.330 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T15:04:11.330 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.330 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.330 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.331 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-18T15:04:11.331 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.331 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.331 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.331 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.331 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.331 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.331 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.332 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T15:04:11.332 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T15:04:11.332 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.332 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.332 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.332 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.332 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.332 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.333 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.333 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.333 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.333 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.333 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.333 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T15:04:11.333 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T15:04:11.333 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.334 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.334 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.334 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.334 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.334 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.334 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.334 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.334 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.335 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.335 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.335 INFO:teuthology.orchestra.run.smithi076.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T15:04:11.335 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T15:04:11.335 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.335 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.335 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.336 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.336 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.336 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.336 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.336 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.336 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.336 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.336 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T15:04:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T15:04:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-18T15:04:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T15:04:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T15:04:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T15:04:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T15:04:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-18T15:04:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.341 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.341 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.341 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.341 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.341 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.341 INFO:teuthology.orchestra.run.smithi076.stdout: "cross_origin_url": { 2024-03-18T15:04:11.341 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cross_origin_url", 2024-03-18T15:04:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: "crt_file": { 2024-03-18T15:04:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crt_file", 2024-03-18T15:04:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: "debug": { 2024-03-18T15:04:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "debug", 2024-03-18T15:04:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable/disable debug options", 2024-03-18T15:04:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: "jwt_token_ttl": { 2024-03-18T15:04:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "jwt_token_ttl", 2024-03-18T15:04:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "28800", 2024-03-18T15:04:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: "key_file": { 2024-03-18T15:04:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "key_file", 2024-03-18T15:04:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.349 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.349 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.349 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.349 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.349 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.349 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.349 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.349 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.350 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.350 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.350 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.350 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.350 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.350 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.350 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.350 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.351 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.351 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.351 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.351 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.351 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.351 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.351 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.351 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.351 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.352 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.352 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.352 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.352 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.352 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.352 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.352 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.352 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.353 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.353 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.353 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.353 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.353 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.353 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.353 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.353 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.354 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.354 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.354 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.354 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.354 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.354 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.354 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.354 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.355 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.355 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.355 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.355 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.355 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.355 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.355 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.355 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.356 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.356 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.356 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.356 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.356 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.356 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.356 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.356 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.357 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.357 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.357 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.357 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.357 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.357 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.357 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.357 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.358 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.358 INFO:teuthology.orchestra.run.smithi076.stdout: "motd": { 2024-03-18T15:04:11.358 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "motd", 2024-03-18T15:04:11.358 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.358 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.358 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.358 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.359 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.359 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.359 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.359 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "The message of the day", 2024-03-18T15:04:11.359 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.359 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.359 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.359 INFO:teuthology.orchestra.run.smithi076.stdout: "redirect_resolve_ip_addr": { 2024-03-18T15:04:11.360 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T15:04:11.360 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.360 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.360 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.360 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.360 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.360 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.360 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.361 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.361 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.361 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.361 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.361 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-18T15:04:11.361 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-18T15:04:11.361 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.362 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.362 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.362 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "::", 2024-03-18T15:04:11.362 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.362 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.362 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.362 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.362 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.363 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.363 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.363 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-18T15:04:11.363 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-18T15:04:11.363 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.363 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.363 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.363 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8080", 2024-03-18T15:04:11.364 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.364 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.364 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.364 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.364 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.364 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.364 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.364 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl": { 2024-03-18T15:04:11.365 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl", 2024-03-18T15:04:11.365 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.365 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.365 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.365 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.365 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.365 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.365 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.365 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.366 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.366 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.366 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.366 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl_server_port": { 2024-03-18T15:04:11.366 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl_server_port", 2024-03-18T15:04:11.366 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.366 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.367 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.367 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8443", 2024-03-18T15:04:11.367 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.367 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.367 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.367 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.367 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.367 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.368 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.368 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_behaviour": { 2024-03-18T15:04:11.368 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_behaviour", 2024-03-18T15:04:11.368 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.368 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.368 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.368 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "redirect", 2024-03-18T15:04:11.368 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.369 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.369 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.369 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.369 INFO:teuthology.orchestra.run.smithi076.stdout: "redirect" 2024-03-18T15:04:11.369 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.369 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.369 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.369 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.369 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.370 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_error_status_code": { 2024-03-18T15:04:11.370 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_error_status_code", 2024-03-18T15:04:11.370 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.370 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.370 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.370 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "500", 2024-03-18T15:04:11.370 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "400", 2024-03-18T15:04:11.371 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "599", 2024-03-18T15:04:11.371 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.371 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.371 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.371 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.371 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.371 INFO:teuthology.orchestra.run.smithi076.stdout: "url_prefix": { 2024-03-18T15:04:11.371 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "url_prefix", 2024-03-18T15:04:11.372 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.372 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.372 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.372 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.372 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.372 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.372 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.372 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.373 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.373 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.373 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.373 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.373 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.373 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "devicehealth", 2024-03-18T15:04:11.373 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.374 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.374 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.374 INFO:teuthology.orchestra.run.smithi076.stdout: "enable_monitoring": { 2024-03-18T15:04:11.374 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enable_monitoring", 2024-03-18T15:04:11.374 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.374 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.374 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.374 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.375 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.375 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.375 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.375 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "monitor device health metrics", 2024-03-18T15:04:11.375 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.375 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.375 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.376 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.376 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.376 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.376 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.376 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.376 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.376 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.376 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.376 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.377 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.377 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.377 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.377 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.377 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.377 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.377 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.377 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.377 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.378 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.378 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.378 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.378 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.378 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.378 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.378 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.378 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.379 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.379 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.379 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.379 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.379 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.379 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.379 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.380 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.380 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.380 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.380 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.380 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.380 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.380 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.380 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.380 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.381 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.381 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.381 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.381 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.381 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.381 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.381 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.381 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.381 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.382 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.382 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.382 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.382 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.382 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.382 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.382 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.383 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.383 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.383 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.383 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.383 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.383 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.383 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.383 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.384 INFO:teuthology.orchestra.run.smithi076.stdout: "mark_out_threshold": { 2024-03-18T15:04:11.384 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mark_out_threshold", 2024-03-18T15:04:11.384 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.384 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.384 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.384 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "2419200", 2024-03-18T15:04:11.384 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.384 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.384 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.385 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T15:04:11.385 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.385 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.385 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.385 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": { 2024-03-18T15:04:11.385 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pool_name", 2024-03-18T15:04:11.385 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.385 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.386 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.386 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "device_health_metrics", 2024-03-18T15:04:11.386 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.386 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.386 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.386 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T15:04:11.386 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.386 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.387 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.387 INFO:teuthology.orchestra.run.smithi076.stdout: "retention_period": { 2024-03-18T15:04:11.387 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "retention_period", 2024-03-18T15:04:11.387 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.387 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.387 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.387 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15552000", 2024-03-18T15:04:11.387 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.388 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.388 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.388 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long to retain device health metrics", 2024-03-18T15:04:11.388 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.388 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.388 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.388 INFO:teuthology.orchestra.run.smithi076.stdout: "scrape_frequency": { 2024-03-18T15:04:11.389 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "scrape_frequency", 2024-03-18T15:04:11.389 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.389 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.389 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.389 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "86400", 2024-03-18T15:04:11.389 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.389 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.389 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.389 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T15:04:11.390 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.390 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.390 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.390 INFO:teuthology.orchestra.run.smithi076.stdout: "self_heal": { 2024-03-18T15:04:11.390 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "self_heal", 2024-03-18T15:04:11.390 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.390 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.391 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.391 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.391 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.391 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.391 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.391 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T15:04:11.391 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.391 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.392 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.392 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-18T15:04:11.392 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-18T15:04:11.392 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.392 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.392 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.392 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-18T15:04:11.392 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T15:04:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_threshold": { 2024-03-18T15:04:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_threshold", 2024-03-18T15:04:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "7257600", 2024-03-18T15:04:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T15:04:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "diskprediction_local", 2024-03-18T15:04:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.399 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.401 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.401 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.401 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.401 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.405 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.405 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.405 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.405 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.405 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.405 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.405 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.405 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.405 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.406 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.406 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.406 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.406 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.406 INFO:teuthology.orchestra.run.smithi076.stdout: "predict_interval": { 2024-03-18T15:04:11.406 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "predict_interval", 2024-03-18T15:04:11.406 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.406 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.407 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.407 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "86400", 2024-03-18T15:04:11.407 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.407 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.407 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.407 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.407 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.407 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.407 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.408 INFO:teuthology.orchestra.run.smithi076.stdout: "predictor_model": { 2024-03-18T15:04:11.408 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "predictor_model", 2024-03-18T15:04:11.408 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.408 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.408 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.408 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "prophetstor", 2024-03-18T15:04:11.409 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.409 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.409 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.409 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.409 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.409 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.409 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.410 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-18T15:04:11.410 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-18T15:04:11.410 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.410 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.410 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.410 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-18T15:04:11.410 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.410 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.410 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.411 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.411 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.411 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.411 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.411 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.411 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.411 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "influx", 2024-03-18T15:04:11.412 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": false, 2024-03-18T15:04:11.412 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "influxdb python module not found", 2024-03-18T15:04:11.412 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.412 INFO:teuthology.orchestra.run.smithi076.stdout: "batch_size": { 2024-03-18T15:04:11.412 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "batch_size", 2024-03-18T15:04:11.412 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.412 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.412 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.413 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5000", 2024-03-18T15:04:11.413 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.413 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.413 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.413 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T15:04:11.413 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.413 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.413 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.414 INFO:teuthology.orchestra.run.smithi076.stdout: "database": { 2024-03-18T15:04:11.414 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "database", 2024-03-18T15:04:11.414 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.414 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.414 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.414 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "ceph", 2024-03-18T15:04:11.414 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.414 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.415 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.415 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T15:04:11.415 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.415 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.415 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.415 INFO:teuthology.orchestra.run.smithi076.stdout: "hostname": { 2024-03-18T15:04:11.415 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "hostname", 2024-03-18T15:04:11.416 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.416 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.416 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.416 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.416 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.416 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.416 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.416 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB server hostname", 2024-03-18T15:04:11.416 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.417 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.417 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.417 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-18T15:04:11.417 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-18T15:04:11.417 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.417 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.417 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.417 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "30", 2024-03-18T15:04:11.418 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "5", 2024-03-18T15:04:11.418 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.418 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.418 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T15:04:11.418 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.418 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.418 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.419 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.419 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.419 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.419 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.419 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.419 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.419 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.419 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.419 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.420 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.420 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.420 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.420 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.420 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.420 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.420 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.420 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.420 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.421 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.421 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.421 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.421 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.421 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.421 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.421 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.421 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.422 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.422 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.422 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.422 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.422 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.422 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.422 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.423 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.423 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.423 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.423 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.423 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.423 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.423 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.423 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.423 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.424 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.424 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.424 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.424 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.424 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.424 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.424 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.424 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.425 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.425 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.425 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.425 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.425 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.425 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.425 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.425 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.426 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.426 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.426 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.426 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.426 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.426 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.426 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.426 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.427 INFO:teuthology.orchestra.run.smithi076.stdout: "password": { 2024-03-18T15:04:11.427 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "password", 2024-03-18T15:04:11.427 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.427 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.427 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.427 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.427 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.427 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.428 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.428 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "password of InfluxDB server user", 2024-03-18T15:04:11.428 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.428 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.428 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.428 INFO:teuthology.orchestra.run.smithi076.stdout: "port": { 2024-03-18T15:04:11.428 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "port", 2024-03-18T15:04:11.428 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.428 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.429 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.429 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8086", 2024-03-18T15:04:11.429 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.429 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.429 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.429 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB server port", 2024-03-18T15:04:11.429 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.429 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.429 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.430 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl": { 2024-03-18T15:04:11.430 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl", 2024-03-18T15:04:11.430 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.430 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.430 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.430 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "false", 2024-03-18T15:04:11.430 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.430 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.431 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.431 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T15:04:11.431 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.431 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.431 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.431 INFO:teuthology.orchestra.run.smithi076.stdout: "threads": { 2024-03-18T15:04:11.431 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "threads", 2024-03-18T15:04:11.431 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.432 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.432 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.432 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-18T15:04:11.432 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-18T15:04:11.432 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "32", 2024-03-18T15:04:11.432 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.432 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T15:04:11.432 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.432 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.433 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.433 INFO:teuthology.orchestra.run.smithi076.stdout: "username": { 2024-03-18T15:04:11.433 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "username", 2024-03-18T15:04:11.433 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.433 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.433 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.433 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.433 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.434 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.434 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.434 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "username of InfluxDB server user", 2024-03-18T15:04:11.434 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.434 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.434 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.434 INFO:teuthology.orchestra.run.smithi076.stdout: "verify_ssl": { 2024-03-18T15:04:11.435 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "verify_ssl", 2024-03-18T15:04:11.435 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.435 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.435 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.435 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "true", 2024-03-18T15:04:11.435 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.435 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.435 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.435 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T15:04:11.436 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.436 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.436 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.436 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.436 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.436 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "insights", 2024-03-18T15:04:11.436 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.437 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.437 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.437 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.437 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.437 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.437 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.437 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.437 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.437 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.438 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.438 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.438 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.438 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.438 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.438 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.438 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.438 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.438 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.439 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.439 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.439 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.439 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.439 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.439 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.439 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.440 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.440 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.440 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.440 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.440 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.440 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.440 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.440 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.441 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.441 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.441 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.441 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.441 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.441 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.441 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.442 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.442 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.442 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.442 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.442 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.442 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.442 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.442 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.443 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.443 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.443 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.443 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.443 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.443 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.443 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.443 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.444 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.444 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.444 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.444 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.444 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.444 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.444 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.444 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.444 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.445 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.445 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.445 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.445 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.445 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.445 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "iostat", 2024-03-18T15:04:11.445 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.445 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.446 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.446 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.446 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.446 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.446 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.446 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.446 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.446 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.446 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.447 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.447 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.447 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.447 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.447 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.447 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.447 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.447 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.447 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.448 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.448 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.448 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.448 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.448 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.448 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.448 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.449 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.449 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.449 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.449 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.449 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.449 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.449 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.449 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.449 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.450 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.450 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.450 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.450 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.450 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.450 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.450 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.450 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.451 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.451 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.451 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.451 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.451 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.451 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.451 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.451 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.451 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.451 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.452 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.452 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.452 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.452 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.452 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.452 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.452 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.452 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.452 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.453 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.453 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.453 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.453 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.453 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.453 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.453 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.453 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.454 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "k8sevents", 2024-03-18T15:04:11.454 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.454 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.454 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.454 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_event_retention_days": { 2024-03-18T15:04:11.454 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ceph_event_retention_days", 2024-03-18T15:04:11.454 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.455 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.455 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.455 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "7", 2024-03-18T15:04:11.455 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.455 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.455 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.455 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T15:04:11.455 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.455 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.456 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.456 INFO:teuthology.orchestra.run.smithi076.stdout: "config_check_secs": { 2024-03-18T15:04:11.456 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_check_secs", 2024-03-18T15:04:11.456 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.456 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.456 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.456 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:11.457 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "10", 2024-03-18T15:04:11.457 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.457 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.457 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T15:04:11.457 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.457 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.457 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.457 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.458 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.458 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.458 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.458 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.458 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.458 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.458 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.458 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.459 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.459 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.459 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.459 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.459 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.459 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.459 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.459 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.459 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.460 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.460 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.460 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.460 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.460 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.460 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.460 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.461 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.461 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.461 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.461 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.461 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.461 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.461 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.461 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.462 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.462 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.462 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.462 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.462 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.462 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.462 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.462 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.463 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.463 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.463 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.463 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.463 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.463 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.463 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.463 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.463 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.464 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.464 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.464 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.464 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.464 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.464 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.464 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.464 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.465 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.465 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.465 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.465 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.465 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.465 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.465 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.465 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.465 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.465 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.466 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "localpool", 2024-03-18T15:04:11.466 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.466 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.466 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.466 INFO:teuthology.orchestra.run.smithi076.stdout: "failure_domain": { 2024-03-18T15:04:11.466 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "failure_domain", 2024-03-18T15:04:11.466 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.467 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.467 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.467 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "host", 2024-03-18T15:04:11.467 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.467 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.467 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.467 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "failure domain for any created local pool", 2024-03-18T15:04:11.467 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T15:04:11.468 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.468 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.468 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.468 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.468 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.468 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.468 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.468 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.468 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.469 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.469 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.469 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.469 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.469 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.469 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.469 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.469 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.469 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.470 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.470 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.470 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.470 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.470 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.470 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.470 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.470 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.471 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.471 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.471 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.471 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.471 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.471 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.471 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.471 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.471 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.472 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.472 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.472 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.472 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.472 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.472 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.472 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.472 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.473 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.473 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.473 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.473 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.473 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.473 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.473 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.473 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.473 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.474 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.474 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.474 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.474 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.474 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.474 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.474 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.474 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.475 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.475 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.475 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.475 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.475 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.475 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.475 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.475 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.475 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": { 2024-03-18T15:04:11.476 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "min_size", 2024-03-18T15:04:11.476 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.476 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.476 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.476 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.476 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.476 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.476 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.476 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default min_size for any created local pool", 2024-03-18T15:04:11.477 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T15:04:11.477 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.477 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.477 INFO:teuthology.orchestra.run.smithi076.stdout: "num_rep": { 2024-03-18T15:04:11.477 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "num_rep", 2024-03-18T15:04:11.477 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.477 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.477 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.478 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3", 2024-03-18T15:04:11.478 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.478 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.478 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.478 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default replica count for any created local pool", 2024-03-18T15:04:11.478 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.478 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.478 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.479 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": { 2024-03-18T15:04:11.479 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pg_num", 2024-03-18T15:04:11.479 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.479 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.479 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.479 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "128", 2024-03-18T15:04:11.479 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.480 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.480 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.480 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T15:04:11.480 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.480 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.480 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.480 INFO:teuthology.orchestra.run.smithi076.stdout: "prefix": { 2024-03-18T15:04:11.480 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prefix", 2024-03-18T15:04:11.481 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.481 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.481 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.481 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.481 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.481 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.481 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.481 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "name prefix for any created local pool", 2024-03-18T15:04:11.481 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.482 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.482 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.482 INFO:teuthology.orchestra.run.smithi076.stdout: "subtree": { 2024-03-18T15:04:11.482 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "subtree", 2024-03-18T15:04:11.482 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.482 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.482 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.482 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "rack", 2024-03-18T15:04:11.483 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.483 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.483 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.483 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T15:04:11.483 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T15:04:11.483 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.483 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.483 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.484 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.484 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mds_autoscaler", 2024-03-18T15:04:11.484 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.484 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.484 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.484 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.484 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.484 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.485 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.485 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.485 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.485 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.485 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.485 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.485 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.485 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.485 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.486 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.486 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.486 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.486 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.486 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.486 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.486 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.486 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.487 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.487 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.487 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.487 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.487 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.487 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.487 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.487 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.488 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.488 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.488 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.488 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.488 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.488 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.488 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.489 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.489 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.489 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.489 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.489 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.489 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.489 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.489 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.489 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.490 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.490 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.490 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.490 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.490 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.490 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.490 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.490 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.491 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.491 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.491 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.491 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.491 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.491 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.491 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.491 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.492 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.492 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.492 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.492 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.492 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.492 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.492 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.492 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.493 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mirroring", 2024-03-18T15:04:11.493 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.493 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.493 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.493 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.493 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.493 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.494 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.494 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.494 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.494 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.494 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.494 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.494 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.494 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.494 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.495 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.495 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.495 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.495 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.495 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.495 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.495 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.495 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.496 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.496 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.496 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.496 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.496 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.496 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.496 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.496 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.496 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.497 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.497 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.497 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.497 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.497 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.497 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.497 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.497 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.498 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.498 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.498 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.498 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.498 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.498 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.498 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.498 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.498 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.499 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.499 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.499 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.499 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.499 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.499 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.499 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.499 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.500 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.500 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.500 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.500 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.500 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.500 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.500 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.500 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.501 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.501 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.501 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.501 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.501 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.501 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.502 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "nfs", 2024-03-18T15:04:11.502 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.502 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.502 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.502 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.502 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.502 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.502 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.503 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.503 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.503 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.503 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.503 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.503 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.503 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.503 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.503 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.504 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.504 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.504 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.504 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.504 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.504 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.504 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.505 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.505 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.505 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.505 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.505 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.505 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.505 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.505 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.505 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.505 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.506 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.506 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.506 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.506 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.506 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.506 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.506 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.507 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.507 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.507 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.507 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.507 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.507 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.507 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.507 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.507 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.508 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.508 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.508 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.508 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.508 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.508 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.508 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.508 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.509 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.509 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.509 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.509 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.509 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.509 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.509 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.509 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.509 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.510 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.510 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.510 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.510 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.510 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.510 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "orchestrator", 2024-03-18T15:04:11.511 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.511 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.511 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.511 INFO:teuthology.orchestra.run.smithi076.stdout: "fail_fs": { 2024-03-18T15:04:11.511 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "fail_fs", 2024-03-18T15:04:11.511 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.511 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.511 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.511 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.512 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.512 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.512 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.512 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T15:04:11.512 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.512 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.512 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.512 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.513 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.513 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.513 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.513 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.513 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.513 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.513 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.513 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.513 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.514 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.514 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.514 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.514 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.514 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.514 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.514 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.514 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.514 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.515 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.515 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.515 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.515 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.515 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.515 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.515 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.515 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.516 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.516 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.516 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.516 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.516 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.516 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.516 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.516 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.518 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.518 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.518 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.518 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.518 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.518 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.518 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.518 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: "orchestrator": { 2024-03-18T15:04:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "orchestrator", 2024-03-18T15:04:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm", 2024-03-18T15:04:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: "rook", 2024-03-18T15:04:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: "test_orchestrator" 2024-03-18T15:04:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Orchestrator backend", 2024-03-18T15:04:11.522 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.522 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.522 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.522 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.522 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.522 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "osd_perf_query", 2024-03-18T15:04:11.523 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.523 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.523 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.523 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.523 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.523 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.523 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.523 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.523 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.524 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.524 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.524 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.524 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.524 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.524 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.524 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.524 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.524 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.525 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.525 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.525 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.525 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.525 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.525 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.525 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.525 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.526 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.526 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.526 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.526 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.526 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.526 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.526 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.526 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.528 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.528 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.528 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.528 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.528 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.528 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.528 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.528 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "osd_support", 2024-03-18T15:04:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.533 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.533 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.533 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.533 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.533 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.533 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.533 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.533 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.534 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.534 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.534 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.534 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.534 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.534 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.534 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.534 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.535 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.535 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.535 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.535 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.535 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.535 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.535 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.535 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.535 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.536 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.536 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.536 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.536 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.536 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.536 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.536 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.536 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.536 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.537 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.537 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.537 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.537 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.537 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.537 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.537 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.537 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.537 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.537 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.538 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.538 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.538 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.538 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.538 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.538 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.538 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.539 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.539 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.539 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.539 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.539 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.539 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.539 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.539 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.539 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.540 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pg_autoscaler", 2024-03-18T15:04:11.540 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.540 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.540 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.540 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.540 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.540 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.540 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.541 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.541 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.541 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.541 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.541 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.541 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.541 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.541 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.541 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.542 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.542 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.542 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.542 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.542 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.542 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.542 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.543 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.543 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.543 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.543 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.543 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.543 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.543 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.543 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.543 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.544 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.544 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.544 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.544 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.544 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.544 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.544 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.545 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.545 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.545 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.545 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.545 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.545 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.545 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.545 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.545 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.545 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.546 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.546 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.546 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.546 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.546 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.546 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.546 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.547 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.547 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.547 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.547 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.547 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.547 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.547 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.547 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.547 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.548 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.548 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.548 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.548 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.548 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-18T15:04:11.548 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-18T15:04:11.548 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.548 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.549 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.549 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-18T15:04:11.549 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.549 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.549 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.549 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.549 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.549 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.549 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.550 INFO:teuthology.orchestra.run.smithi076.stdout: "threshold": { 2024-03-18T15:04:11.550 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "threshold", 2024-03-18T15:04:11.550 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:11.550 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.550 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.550 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3.0", 2024-03-18T15:04:11.550 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1.0", 2024-03-18T15:04:11.551 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.551 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.551 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "scaling threshold", 2024-03-18T15:04:11.551 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T15:04:11.551 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.551 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.551 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.551 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.552 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "progress", 2024-03-18T15:04:11.552 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.552 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.552 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.552 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_pg_recovery_event": { 2024-03-18T15:04:11.552 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_pg_recovery_event", 2024-03-18T15:04:11.552 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.552 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.553 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.553 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.553 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.553 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.553 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.553 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T15:04:11.553 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.553 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "enabled": { 2024-03-18T15:04:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enabled", 2024-03-18T15:04:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: "max_completed_events": { 2024-03-18T15:04:11.564 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_completed_events", 2024-03-18T15:04:11.564 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.564 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.564 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.564 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "50", 2024-03-18T15:04:11.564 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.564 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.564 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.564 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "number of past completed events to remember", 2024-03-18T15:04:11.565 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.565 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.565 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.565 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-18T15:04:11.565 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-18T15:04:11.565 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.565 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.565 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.566 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-18T15:04:11.566 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.566 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.566 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.566 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long the module is going to sleep", 2024-03-18T15:04:11.566 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.566 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.566 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.567 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.567 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.567 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prometheus", 2024-03-18T15:04:11.567 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.567 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.567 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.568 INFO:teuthology.orchestra.run.smithi076.stdout: "cache": { 2024-03-18T15:04:11.568 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cache", 2024-03-18T15:04:11.568 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.568 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.568 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.568 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.568 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.568 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.568 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.569 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.569 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.569 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.569 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.569 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.569 INFO:teuthology.orchestra.run.smithi076.stdout: "exclude_perf_counters": { 2024-03-18T15:04:11.569 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "exclude_perf_counters", 2024-03-18T15:04:11.569 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.569 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.570 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.570 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.570 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.570 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.570 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.570 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T15:04:11.570 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T15:04:11.570 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.571 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.571 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.571 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.571 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.571 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.571 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.571 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.571 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.572 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.572 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.572 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.572 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.572 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.572 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.572 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.572 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.573 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.573 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.573 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.573 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.573 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.573 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.573 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.573 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.574 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.574 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.574 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.574 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.574 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.574 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.574 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.574 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.575 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.575 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.575 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.575 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.575 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.575 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.575 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.575 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.575 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.576 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.576 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.576 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.576 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.576 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.576 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.576 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.576 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.576 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.577 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.577 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.577 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.577 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.577 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.577 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.577 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.577 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.577 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.578 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.578 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.578 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.578 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.578 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.578 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.578 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.578 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.578 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_stats_pools": { 2024-03-18T15:04:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_stats_pools", 2024-03-18T15:04:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T15:04:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T15:04:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "300", 2024-03-18T15:04:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "scrape_interval": { 2024-03-18T15:04:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "scrape_interval", 2024-03-18T15:04:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15.0", 2024-03-18T15:04:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-18T15:04:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-18T15:04:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "::", 2024-03-18T15:04:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T15:04:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-18T15:04:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-18T15:04:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "9283", 2024-03-18T15:04:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T15:04:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "stale_cache_strategy": { 2024-03-18T15:04:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "stale_cache_strategy", 2024-03-18T15:04:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "log", 2024-03-18T15:04:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_behaviour": { 2024-03-18T15:04:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_behaviour", 2024-03-18T15:04:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "default", 2024-03-18T15:04:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "default", 2024-03-18T15:04:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "error" 2024-03-18T15:04:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_error_status_code": { 2024-03-18T15:04:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_error_status_code", 2024-03-18T15:04:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "500", 2024-03-18T15:04:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "400", 2024-03-18T15:04:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "599", 2024-03-18T15:04:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_support", 2024-03-18T15:04:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "max_concurrent_snap_create": { 2024-03-18T15:04:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_concurrent_snap_create", 2024-03-18T15:04:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-18T15:04:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "mirror_snapshot_schedule": { 2024-03-18T15:04:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T15:04:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: "trash_purge_schedule": { 2024-03-18T15:04:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "trash_purge_schedule", 2024-03-18T15:04:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "restful", 2024-03-18T15:04:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: "enable_auth": { 2024-03-18T15:04:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enable_auth", 2024-03-18T15:04:11.609 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.609 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.609 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: "key_file": { 2024-03-18T15:04:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "key_file", 2024-03-18T15:04:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-18T15:04:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-18T15:04:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-18T15:04:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-18T15:04:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rgw", 2024-03-18T15:04:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rook", 2024-03-18T15:04:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "storage_class": { 2024-03-18T15:04:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "storage_class", 2024-03-18T15:04:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "local", 2024-03-18T15:04:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T15:04:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "selftest", 2024-03-18T15:04:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "roption1": { 2024-03-18T15:04:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "roption1", 2024-03-18T15:04:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "roption2": { 2024-03-18T15:04:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "roption2", 2024-03-18T15:04:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "xyz", 2024-03-18T15:04:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption1": { 2024-03-18T15:04:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption1", 2024-03-18T15:04:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption2": { 2024-03-18T15:04:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption2", 2024-03-18T15:04:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption3": { 2024-03-18T15:04:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption3", 2024-03-18T15:04:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-18T15:04:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.658 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.658 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.658 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.658 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.658 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.658 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption4": { 2024-03-18T15:04:11.658 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption4", 2024-03-18T15:04:11.658 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.658 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.659 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.659 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.659 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.659 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.659 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.659 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.659 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.659 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.659 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.660 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption5": { 2024-03-18T15:04:11.660 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption5", 2024-03-18T15:04:11.660 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.660 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.660 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.660 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.660 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.660 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.661 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.661 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.661 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.661 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.661 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.661 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption6": { 2024-03-18T15:04:11.661 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption6", 2024-03-18T15:04:11.661 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.662 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.662 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.662 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.662 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.662 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.662 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.662 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.662 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.663 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.663 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.663 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption7": { 2024-03-18T15:04:11.663 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption7", 2024-03-18T15:04:11.663 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.663 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.663 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.663 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.664 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-18T15:04:11.664 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "42", 2024-03-18T15:04:11.664 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.664 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.664 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.664 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.664 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.665 INFO:teuthology.orchestra.run.smithi076.stdout: "testkey": { 2024-03-18T15:04:11.665 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testkey", 2024-03-18T15:04:11.665 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.665 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.665 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.665 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.665 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.665 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.666 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.666 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.666 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.666 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.666 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.666 INFO:teuthology.orchestra.run.smithi076.stdout: "testlkey": { 2024-03-18T15:04:11.666 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testlkey", 2024-03-18T15:04:11.667 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.667 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.667 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.667 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.667 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.667 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.667 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.667 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.668 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.668 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.668 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.668 INFO:teuthology.orchestra.run.smithi076.stdout: "testnewline": { 2024-03-18T15:04:11.668 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testnewline", 2024-03-18T15:04:11.668 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.668 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.669 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.669 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.669 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.669 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.669 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.669 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.669 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.669 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.669 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.670 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.670 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.670 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snap_schedule", 2024-03-18T15:04:11.670 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.670 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.671 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.671 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_m_granularity": { 2024-03-18T15:04:11.671 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_m_granularity", 2024-03-18T15:04:11.671 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.671 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.671 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.671 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.672 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.672 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.672 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.672 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T15:04:11.672 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.672 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.672 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.672 INFO:teuthology.orchestra.run.smithi076.stdout: "dump_on_update": { 2024-03-18T15:04:11.673 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "dump_on_update", 2024-03-18T15:04:11.673 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.673 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.673 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.673 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.673 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.673 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.673 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.673 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "dump database to debug log on update", 2024-03-18T15:04:11.674 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.674 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.674 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.674 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.674 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.674 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.674 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.674 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.675 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.675 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.675 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.675 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.675 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.675 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.675 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.675 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.675 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.676 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.676 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.676 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.676 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.676 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.676 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.676 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.676 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.676 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.677 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.677 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.677 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.677 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.677 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.677 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.677 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.677 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.678 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.678 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.678 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.678 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.678 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.678 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.678 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.678 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.679 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.679 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.679 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.679 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.679 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.679 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.679 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.679 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.679 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.679 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.680 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.680 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.680 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.680 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.680 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.680 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.680 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.680 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.681 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.681 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.681 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.681 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.681 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.681 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.681 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.681 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.681 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.681 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.682 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.682 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "stats", 2024-03-18T15:04:11.682 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.682 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.682 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.682 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.683 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.683 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.683 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.683 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.683 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.683 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.683 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.683 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.683 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.684 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.684 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.684 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.684 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.684 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.684 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.684 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.684 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.684 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.685 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.685 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.685 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.685 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.685 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.685 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.685 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.685 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.686 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.686 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.686 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.686 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.686 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.686 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.686 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.686 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.687 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.687 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.687 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.687 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.687 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.687 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.687 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.687 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.687 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.688 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.688 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.688 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.688 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.688 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.688 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.688 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.688 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.689 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.689 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.689 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.689 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.689 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.689 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.689 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.689 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.690 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.690 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.690 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.690 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.690 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.690 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.690 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.690 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "status", 2024-03-18T15:04:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.693 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.693 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.693 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.693 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.693 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.693 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.693 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "telegraf", 2024-03-18T15:04:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "address": { 2024-03-18T15:04:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "address", 2024-03-18T15:04:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T15:04:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-18T15:04:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-18T15:04:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15", 2024-03-18T15:04:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.703 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.703 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.703 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.703 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.703 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.703 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.703 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.703 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.704 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.704 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.704 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.704 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.704 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.704 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.704 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.704 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.704 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.705 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.705 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.705 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.705 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.705 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.705 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.705 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.705 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.706 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.706 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.706 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.706 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.706 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.706 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.706 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.706 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.707 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.707 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.707 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.707 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.707 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.707 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.707 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.707 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.708 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.708 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.708 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.708 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.708 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.708 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.708 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.708 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.708 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.709 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.709 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.709 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.709 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.709 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.709 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.709 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.709 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.710 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.710 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.710 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.710 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.710 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.710 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.710 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.710 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.711 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.711 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.711 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "telemetry", 2024-03-18T15:04:11.711 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.711 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.711 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.712 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_basic": { 2024-03-18T15:04:11.712 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_basic", 2024-03-18T15:04:11.712 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.712 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.712 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.712 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.712 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.712 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.712 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.713 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T15:04:11.713 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.713 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.713 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.713 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_crash": { 2024-03-18T15:04:11.713 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_crash", 2024-03-18T15:04:11.713 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.713 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.714 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.714 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.714 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.714 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.714 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.714 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T15:04:11.714 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.714 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.715 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.715 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_device": { 2024-03-18T15:04:11.715 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_device", 2024-03-18T15:04:11.715 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.715 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.715 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.715 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.715 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.715 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.716 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.716 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T15:04:11.716 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.716 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.716 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.716 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_ident": { 2024-03-18T15:04:11.716 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_ident", 2024-03-18T15:04:11.717 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.717 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.717 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.717 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.717 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.717 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.717 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.717 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T15:04:11.717 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.717 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.718 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.718 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_perf": { 2024-03-18T15:04:11.718 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_perf", 2024-03-18T15:04:11.718 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.718 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.718 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.718 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.719 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.719 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.719 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.719 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T15:04:11.719 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.719 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.719 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.719 INFO:teuthology.orchestra.run.smithi076.stdout: "contact": { 2024-03-18T15:04:11.719 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "contact", 2024-03-18T15:04:11.720 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.720 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.720 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.720 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.720 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.720 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.720 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.720 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.720 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.720 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.721 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.721 INFO:teuthology.orchestra.run.smithi076.stdout: "description": { 2024-03-18T15:04:11.721 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "description", 2024-03-18T15:04:11.721 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.721 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.721 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.721 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.721 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.721 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.722 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.722 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.722 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.722 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.722 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.722 INFO:teuthology.orchestra.run.smithi076.stdout: "device_url": { 2024-03-18T15:04:11.722 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_url", 2024-03-18T15:04:11.723 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.723 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.723 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.723 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T15:04:11.723 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.723 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.723 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.723 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.723 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.723 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.724 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.724 INFO:teuthology.orchestra.run.smithi076.stdout: "enabled": { 2024-03-18T15:04:11.724 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enabled", 2024-03-18T15:04:11.724 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.724 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.724 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.724 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.724 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.725 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.725 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.725 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.725 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.725 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.725 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.725 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-18T15:04:11.725 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-18T15:04:11.725 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.726 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.726 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.726 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "24", 2024-03-18T15:04:11.726 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "8", 2024-03-18T15:04:11.726 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.726 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.726 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.726 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.726 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.726 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.727 INFO:teuthology.orchestra.run.smithi076.stdout: "last_opt_revision": { 2024-03-18T15:04:11.727 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "last_opt_revision", 2024-03-18T15:04:11.727 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.727 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.727 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.727 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1", 2024-03-18T15:04:11.727 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.727 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.728 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.728 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.728 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.728 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.728 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.728 INFO:teuthology.orchestra.run.smithi076.stdout: "leaderboard": { 2024-03-18T15:04:11.728 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "leaderboard", 2024-03-18T15:04:11.728 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.729 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.729 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.729 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.729 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.729 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.729 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.729 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.729 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.729 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.730 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.730 INFO:teuthology.orchestra.run.smithi076.stdout: "leaderboard_description": { 2024-03-18T15:04:11.730 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "leaderboard_description", 2024-03-18T15:04:11.730 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.730 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.730 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.730 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.730 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.731 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.731 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.731 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.731 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.731 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.731 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.731 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.731 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.731 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.732 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.732 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.732 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.732 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.732 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.732 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.732 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.732 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.732 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.733 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.733 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.733 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.733 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.733 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.733 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.733 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.733 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.734 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.734 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.734 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.734 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.734 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.734 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.734 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.734 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.734 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.735 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.735 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.735 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.735 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.735 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.735 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.735 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.735 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.735 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.736 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.736 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.736 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.736 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.736 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.736 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.736 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.736 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.737 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.737 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.737 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.737 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.737 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.737 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.737 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.738 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.738 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.738 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.738 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.738 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.738 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.738 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.738 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.738 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.739 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.739 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.739 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.739 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.739 INFO:teuthology.orchestra.run.smithi076.stdout: "organization": { 2024-03-18T15:04:11.739 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "organization", 2024-03-18T15:04:11.739 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.739 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.740 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.740 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.740 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.740 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.740 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.740 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.740 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.740 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.740 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.741 INFO:teuthology.orchestra.run.smithi076.stdout: "proxy": { 2024-03-18T15:04:11.741 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "proxy", 2024-03-18T15:04:11.741 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.741 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.741 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.741 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.741 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.742 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.742 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.742 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.742 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.742 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.742 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.742 INFO:teuthology.orchestra.run.smithi076.stdout: "url": { 2024-03-18T15:04:11.742 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "url", 2024-03-18T15:04:11.743 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.743 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.743 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.743 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T15:04:11.743 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.743 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.743 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.743 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.744 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.744 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.744 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.744 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.744 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.744 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "test_orchestrator", 2024-03-18T15:04:11.744 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.744 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.745 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.745 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.745 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.745 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.745 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.745 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.745 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.745 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.745 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.746 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.746 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.746 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.746 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.746 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.746 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.746 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.746 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.747 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.747 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.747 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.747 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.747 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.747 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.747 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.747 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.747 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.748 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.748 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.748 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.748 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.748 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.748 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.748 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.748 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.749 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.749 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.749 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.749 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.749 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.749 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.749 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.749 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.750 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.750 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.750 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.750 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.750 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.750 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.750 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.750 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.750 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.751 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.751 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.751 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.751 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.751 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.751 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.751 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.751 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.752 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.752 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.752 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.752 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.752 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.752 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.752 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.752 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.753 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.753 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.753 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "volumes", 2024-03-18T15:04:11.753 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.753 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.753 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.753 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.754 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.754 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.754 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.754 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.754 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.754 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.754 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.754 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.755 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.755 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.755 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.755 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.755 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.755 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.755 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.755 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.755 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.756 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.756 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.756 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.756 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.756 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.756 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.756 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.757 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.757 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.757 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.757 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.757 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.757 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.757 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.757 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.758 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.758 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.758 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.758 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.758 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.758 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.758 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.758 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.758 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.759 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.759 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.759 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.759 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.759 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.759 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.759 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.759 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.760 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.760 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.760 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.760 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.760 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.760 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.760 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.761 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.761 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.761 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.761 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.761 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.761 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.761 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.761 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.761 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.762 INFO:teuthology.orchestra.run.smithi076.stdout: "max_concurrent_clones": { 2024-03-18T15:04:11.762 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_concurrent_clones", 2024-03-18T15:04:11.762 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.762 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.762 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.762 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "4", 2024-03-18T15:04:11.762 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.763 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.763 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.763 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T15:04:11.763 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.763 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.763 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.763 INFO:teuthology.orchestra.run.smithi076.stdout: "periodic_async_work": { 2024-03-18T15:04:11.764 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "periodic_async_work", 2024-03-18T15:04:11.764 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.764 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.764 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.764 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.764 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.764 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.764 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.765 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Periodically check for async work", 2024-03-18T15:04:11.765 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.765 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.765 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.765 INFO:teuthology.orchestra.run.smithi076.stdout: "snapshot_clone_delay": { 2024-03-18T15:04:11.765 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snapshot_clone_delay", 2024-03-18T15:04:11.765 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.765 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.766 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.766 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-18T15:04:11.766 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.766 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.766 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.766 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T15:04:11.766 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.766 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.767 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.767 INFO:teuthology.orchestra.run.smithi076.stdout: "snapshot_clone_no_wait": { 2024-03-18T15:04:11.767 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T15:04:11.767 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.767 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.767 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.767 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-18T15:04:11.768 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.768 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.768 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.768 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T15:04:11.768 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.768 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.768 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.768 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.769 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.769 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix", 2024-03-18T15:04:11.769 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-18T15:04:11.769 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-18T15:04:11.769 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-18T15:04:11.769 INFO:teuthology.orchestra.run.smithi076.stdout: "discovery_interval": { 2024-03-18T15:04:11.769 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "discovery_interval", 2024-03-18T15:04:11.770 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-18T15:04:11.770 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.770 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.770 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "100", 2024-03-18T15:04:11.770 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.770 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.770 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.770 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.770 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.771 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.771 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.771 INFO:teuthology.orchestra.run.smithi076.stdout: "identifier": { 2024-03-18T15:04:11.771 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "identifier", 2024-03-18T15:04:11.771 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.771 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.771 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.771 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.772 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.772 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.772 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.772 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.772 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.772 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.772 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.773 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-18T15:04:11.773 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-18T15:04:11.773 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-18T15:04:11.773 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.773 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.773 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-18T15:04:11.773 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.773 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.773 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.774 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.774 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.774 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.774 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.774 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-18T15:04:11.774 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-18T15:04:11.774 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.774 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.775 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.775 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.775 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.775 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.775 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.775 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.775 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.775 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.775 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.776 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.776 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.776 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.776 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.776 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.776 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.776 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.777 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-18T15:04:11.777 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-18T15:04:11.777 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.777 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.777 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.777 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.777 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.777 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.778 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.778 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.778 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.778 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.778 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.778 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-18T15:04:11.778 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-18T15:04:11.779 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.779 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.779 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.779 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-18T15:04:11.779 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.779 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.779 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-18T15:04:11.779 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-18T15:04:11.780 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-18T15:04:11.780 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-18T15:04:11.780 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-18T15:04:11.780 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-18T15:04:11.780 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-18T15:04:11.780 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.780 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.780 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.780 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.780 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.781 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-18T15:04:11.781 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-18T15:04:11.781 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-18T15:04:11.781 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.781 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-18T15:04:11.781 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-18T15:04:11.781 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.781 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.782 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.782 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.782 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.782 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.782 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.782 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_host": { 2024-03-18T15:04:11.782 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_host", 2024-03-18T15:04:11.782 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.782 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.783 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.783 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-18T15:04:11.783 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.783 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.783 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.783 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.783 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.783 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.784 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.784 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_port": { 2024-03-18T15:04:11.784 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_port", 2024-03-18T15:04:11.784 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-18T15:04:11.784 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.784 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.784 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10051", 2024-03-18T15:04:11.784 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.785 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.785 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.785 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.785 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.785 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.785 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.785 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_sender": { 2024-03-18T15:04:11.785 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_sender", 2024-03-18T15:04:11.785 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-18T15:04:11.786 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-18T15:04:11.786 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-18T15:04:11.786 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T15:04:11.786 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-18T15:04:11.786 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-18T15:04:11.786 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-18T15:04:11.786 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-18T15:04:11.786 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-18T15:04:11.787 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-18T15:04:11.787 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-18T15:04:11.787 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.787 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.787 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.787 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.787 INFO:teuthology.orchestra.run.smithi076.stdout: "services": { 2024-03-18T15:04:11.787 INFO:teuthology.orchestra.run.smithi076.stdout: "dashboard": "https://172.21.15.76:8443/" 2024-03-18T15:04:11.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.788 INFO:teuthology.orchestra.run.smithi076.stdout: "always_on_modules": { 2024-03-18T15:04:11.788 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus": [ 2024-03-18T15:04:11.788 INFO:teuthology.orchestra.run.smithi076.stdout: "balancer", 2024-03-18T15:04:11.788 INFO:teuthology.orchestra.run.smithi076.stdout: "crash", 2024-03-18T15:04:11.788 INFO:teuthology.orchestra.run.smithi076.stdout: "devicehealth", 2024-03-18T15:04:11.788 INFO:teuthology.orchestra.run.smithi076.stdout: "orchestrator", 2024-03-18T15:04:11.788 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscaler", 2024-03-18T15:04:11.788 INFO:teuthology.orchestra.run.smithi076.stdout: "progress", 2024-03-18T15:04:11.788 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_support", 2024-03-18T15:04:11.788 INFO:teuthology.orchestra.run.smithi076.stdout: "status", 2024-03-18T15:04:11.789 INFO:teuthology.orchestra.run.smithi076.stdout: "telemetry", 2024-03-18T15:04:11.789 INFO:teuthology.orchestra.run.smithi076.stdout: "volumes" 2024-03-18T15:04:11.789 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.789 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific": [ 2024-03-18T15:04:11.789 INFO:teuthology.orchestra.run.smithi076.stdout: "balancer", 2024-03-18T15:04:11.789 INFO:teuthology.orchestra.run.smithi076.stdout: "crash", 2024-03-18T15:04:11.789 INFO:teuthology.orchestra.run.smithi076.stdout: "devicehealth", 2024-03-18T15:04:11.789 INFO:teuthology.orchestra.run.smithi076.stdout: "orchestrator", 2024-03-18T15:04:11.789 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscaler", 2024-03-18T15:04:11.790 INFO:teuthology.orchestra.run.smithi076.stdout: "progress", 2024-03-18T15:04:11.790 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_support", 2024-03-18T15:04:11.790 INFO:teuthology.orchestra.run.smithi076.stdout: "status", 2024-03-18T15:04:11.790 INFO:teuthology.orchestra.run.smithi076.stdout: "telemetry", 2024-03-18T15:04:11.790 INFO:teuthology.orchestra.run.smithi076.stdout: "volumes" 2024-03-18T15:04:11.790 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.790 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy": [ 2024-03-18T15:04:11.790 INFO:teuthology.orchestra.run.smithi076.stdout: "balancer", 2024-03-18T15:04:11.790 INFO:teuthology.orchestra.run.smithi076.stdout: "crash", 2024-03-18T15:04:11.790 INFO:teuthology.orchestra.run.smithi076.stdout: "devicehealth", 2024-03-18T15:04:11.791 INFO:teuthology.orchestra.run.smithi076.stdout: "orchestrator", 2024-03-18T15:04:11.791 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscaler", 2024-03-18T15:04:11.791 INFO:teuthology.orchestra.run.smithi076.stdout: "progress", 2024-03-18T15:04:11.791 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_support", 2024-03-18T15:04:11.791 INFO:teuthology.orchestra.run.smithi076.stdout: "status", 2024-03-18T15:04:11.791 INFO:teuthology.orchestra.run.smithi076.stdout: "telemetry", 2024-03-18T15:04:11.791 INFO:teuthology.orchestra.run.smithi076.stdout: "volumes" 2024-03-18T15:04:11.791 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.791 INFO:teuthology.orchestra.run.smithi076.stdout: "reef": [ 2024-03-18T15:04:11.792 INFO:teuthology.orchestra.run.smithi076.stdout: "balancer", 2024-03-18T15:04:11.792 INFO:teuthology.orchestra.run.smithi076.stdout: "crash", 2024-03-18T15:04:11.792 INFO:teuthology.orchestra.run.smithi076.stdout: "devicehealth", 2024-03-18T15:04:11.792 INFO:teuthology.orchestra.run.smithi076.stdout: "orchestrator", 2024-03-18T15:04:11.792 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscaler", 2024-03-18T15:04:11.792 INFO:teuthology.orchestra.run.smithi076.stdout: "progress", 2024-03-18T15:04:11.792 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_support", 2024-03-18T15:04:11.792 INFO:teuthology.orchestra.run.smithi076.stdout: "status", 2024-03-18T15:04:11.792 INFO:teuthology.orchestra.run.smithi076.stdout: "telemetry", 2024-03-18T15:04:11.792 INFO:teuthology.orchestra.run.smithi076.stdout: "volumes" 2024-03-18T15:04:11.793 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-18T15:04:11.793 INFO:teuthology.orchestra.run.smithi076.stdout: "squid": [ 2024-03-18T15:04:11.793 INFO:teuthology.orchestra.run.smithi076.stdout: "balancer", 2024-03-18T15:04:11.793 INFO:teuthology.orchestra.run.smithi076.stdout: "crash", 2024-03-18T15:04:11.793 INFO:teuthology.orchestra.run.smithi076.stdout: "devicehealth", 2024-03-18T15:04:11.793 INFO:teuthology.orchestra.run.smithi076.stdout: "orchestrator", 2024-03-18T15:04:11.793 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscaler", 2024-03-18T15:04:11.793 INFO:teuthology.orchestra.run.smithi076.stdout: "progress", 2024-03-18T15:04:11.793 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_support", 2024-03-18T15:04:11.794 INFO:teuthology.orchestra.run.smithi076.stdout: "status", 2024-03-18T15:04:11.794 INFO:teuthology.orchestra.run.smithi076.stdout: "telemetry", 2024-03-18T15:04:11.794 INFO:teuthology.orchestra.run.smithi076.stdout: "volumes" 2024-03-18T15:04:11.794 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-18T15:04:11.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.794 INFO:teuthology.orchestra.run.smithi076.stdout: "last_failure_osd_epoch": 89, 2024-03-18T15:04:11.794 INFO:teuthology.orchestra.run.smithi076.stdout: "active_clients": [ 2024-03-18T15:04:11.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.795 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "devicehealth", 2024-03-18T15:04:11.795 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-03-18T15:04:11.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.795 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-03-18T15:04:11.795 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:0", 2024-03-18T15:04:11.795 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1279977901 2024-03-18T15:04:11.795 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.795 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-18T15:04:11.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.796 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "libcephsqlite", 2024-03-18T15:04:11.796 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-03-18T15:04:11.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.796 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-03-18T15:04:11.796 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:0", 2024-03-18T15:04:11.796 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3066176762 2024-03-18T15:04:11.796 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.796 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-18T15:04:11.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.797 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_support", 2024-03-18T15:04:11.797 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-03-18T15:04:11.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.797 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-03-18T15:04:11.797 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:0", 2024-03-18T15:04:11.797 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3570635277 2024-03-18T15:04:11.797 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.797 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-18T15:04:11.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.798 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snap_schedule", 2024-03-18T15:04:11.798 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-03-18T15:04:11.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.798 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-03-18T15:04:11.798 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:0", 2024-03-18T15:04:11.798 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1857055697 2024-03-18T15:04:11.798 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-18T15:04:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snap_schedule", 2024-03-18T15:04:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-03-18T15:04:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-03-18T15:04:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:0", 2024-03-18T15:04:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1113422282 2024-03-18T15:04:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-18T15:04:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "volumes", 2024-03-18T15:04:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-03-18T15:04:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-18T15:04:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-03-18T15:04:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:0", 2024-03-18T15:04:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3969302248 2024-03-18T15:04:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-18T15:04:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-18T15:04:11.801 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T15:04:11.803 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:09 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/773906698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:04:11.803 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:09 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/218072147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:04:11.803 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:09 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3244418721' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T15:04:11.803 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:10 smithi077 ceph-mon[35865]: pgmap v346: 97 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 64 active+clean; 308 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 2.3 MiB/s wr, 169 op/s 2024-03-18T15:04:11.803 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:09 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/773906698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:04:11.803 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:09 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/218072147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:04:11.803 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:09 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3244418721' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T15:04:11.804 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:10 smithi174 ceph-mon[36787]: pgmap v346: 97 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 64 active+clean; 308 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 2.3 MiB/s wr, 169 op/s 2024-03-18T15:04:11.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.692+0000 7fe6128e8640 1 -- 172.21.15.76:0/3244418721 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe5f40a2b10 msgr2=0x7fe5f40a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:11.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.692+0000 7fe6128e8640 1 --2- 172.21.15.76:0/3244418721 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe5f40a2b10 0x7fe5f40a4fd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009b50 tx=0x7fe5fc005f90 comp rx=0 tx=0).stop 2024-03-18T15:04:11.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.692+0000 7fe6128e8640 1 -- 172.21.15.76:0/3244418721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe60c101120 msgr2=0x7fe60c199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:11.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.692+0000 7fe6128e8640 1 --2- 172.21.15.76:0/3244418721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe60c101120 0x7fe60c199470 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe600009230 tx=0x7fe60002cb30 comp rx=0 tx=0).stop 2024-03-18T15:04:11.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.692+0000 7fe6128e8640 1 -- 172.21.15.76:0/3244418721 shutdown_connections 2024-03-18T15:04:11.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.692+0000 7fe6128e8640 1 --2- 172.21.15.76:0/3244418721 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe5f40a2b10 0x7fe5f40a4fd0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:11.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.692+0000 7fe6128e8640 1 --2- 172.21.15.76:0/3244418721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60c106720 0x7fe60c1a3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:11.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.692+0000 7fe6128e8640 1 --2- 172.21.15.76:0/3244418721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe60c101120 0x7fe60c199470 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:11.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.692+0000 7fe6128e8640 1 --2- 172.21.15.76:0/3244418721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe60c100760 0x7fe60c198f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:11.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.692+0000 7fe6128e8640 1 -- 172.21.15.76:0/3244418721 >> 172.21.15.76:0/3244418721 conn(0x7fe60c077840 msgr2=0x7fe60c102ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:11.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.692+0000 7fe6128e8640 1 -- 172.21.15.76:0/3244418721 shutdown_connections 2024-03-18T15:04:11.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:09.692+0000 7fe6128e8640 1 -- 172.21.15.76:0/3244418721 wait complete. 2024-03-18T15:04:11.816 DEBUG:tasks.check_counter:Getting stats from a 2024-03-18T15:04:11.816 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-18T15:04:11.817 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:09 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/773906698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:04:11.817 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:09 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/218072147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:04:11.817 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:09 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3244418721' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T15:04:11.817 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:10 smithi076 ceph-mon[32205]: pgmap v346: 97 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 64 active+clean; 308 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 2.3 MiB/s wr, 169 op/s 2024-03-18T15:04:12.067 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:04:13.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:12 smithi076 ceph-mon[32205]: pgmap v347: 97 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 64 active+clean; 308 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 2.3 MiB/s wr, 169 op/s 2024-03-18T15:04:13.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:12 smithi174 ceph-mon[36787]: pgmap v347: 97 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 64 active+clean; 308 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 2.3 MiB/s wr, 169 op/s 2024-03-18T15:04:13.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:12 smithi077 ceph-mon[35865]: pgmap v347: 97 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 64 active+clean; 308 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 2.3 MiB/s wr, 169 op/s 2024-03-18T15:04:13.868 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T15:04:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:04:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 0, 2024-03-18T15:04:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 2, 2024-03-18T15:04:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 0, 2024-03-18T15:04:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 0, 2024-03-18T15:04:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 4, 2024-03-18T15:04:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 0, 2024-03-18T15:04:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.001199287, 2024-03-18T15:04:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T15:04:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.000201695, 2024-03-18T15:04:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T15:04:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T15:04:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T15:04:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:04:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 225, 2024-03-18T15:04:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 215, 2024-03-18T15:04:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 1438480, 2024-03-18T15:04:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 131904, 2024-03-18T15:04:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 2, 2024-03-18T15:04:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 1, 2024-03-18T15:04:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.037295166, 2024-03-18T15:04:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.011479527, 2024-03-18T15:04:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.030285565, 2024-03-18T15:04:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.003830415, 2024-03-18T15:04:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 214, 2024-03-18T15:04:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.008069827, 2024-03-18T15:04:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000037709 2024-03-18T15:04:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 1438480, 2024-03-18T15:04:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 131904 2024-03-18T15:04:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:04:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 6, 2024-03-18T15:04:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 167, 2024-03-18T15:04:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 3312, 2024-03-18T15:04:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 184960, 2024-03-18T15:04:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 5, 2024-03-18T15:04:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 3, 2024-03-18T15:04:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.012224786, 2024-03-18T15:04:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.007461745, 2024-03-18T15:04:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.002480753, 2024-03-18T15:04:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T15:04:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 165, 2024-03-18T15:04:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.007445216, 2024-03-18T15:04:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000045122 2024-03-18T15:04:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-18T15:04:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 184960 2024-03-18T15:04:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "cct": { 2024-03-18T15:04:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "total_workers": 0, 2024-03-18T15:04:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "unhealthy_workers": 0 2024-03-18T15:04:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "mempool": { 2024-03-18T15:04:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:04:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_items": 0, 2024-03-18T15:04:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:04:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:04:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:04:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:04:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:04:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:04:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:04:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:04:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:04:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:04:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:04:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:04:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:04:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_items": 0, 2024-03-18T15:04:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:04:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_items": 0, 2024-03-18T15:04:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:04:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:04:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:04:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:04:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:04:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:04:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:04:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_items": 0, 2024-03-18T15:04:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:04:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:04:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:04:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_items": 0, 2024-03-18T15:04:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_bytes": 0, 2024-03-18T15:04:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_items": 0, 2024-03-18T15:04:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:04:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:04:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:04:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:04:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_bytes": 149017, 2024-03-18T15:04:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_items": 31, 2024-03-18T15:04:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:04:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_items": 0, 2024-03-18T15:04:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bytes": 0, 2024-03-18T15:04:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_items": 0, 2024-03-18T15:04:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:04:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_items": 0, 2024-03-18T15:04:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:04:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_items": 0, 2024-03-18T15:04:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_bytes": 0, 2024-03-18T15:04:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_items": 0, 2024-03-18T15:04:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:04:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:04:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_bytes": 0, 2024-03-18T15:04:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_items": 0, 2024-03-18T15:04:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_bytes": 928, 2024-03-18T15:04:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_items": 2, 2024-03-18T15:04:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_bytes": 0, 2024-03-18T15:04:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_items": 0, 2024-03-18T15:04:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_bytes": 0, 2024-03-18T15:04:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_items": 0 2024-03-18T15:04:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:04:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-03-18T15:04:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 104857600, 2024-03-18T15:04:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-03-18T15:04:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 227, 2024-03-18T15:04:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 1406760, 2024-03-18T15:04:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 227, 2024-03-18T15:04:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-03-18T15:04:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-03-18T15:04:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 227, 2024-03-18T15:04:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 1406760, 2024-03-18T15:04:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-03-18T15:04:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:13.885 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T15:04:14.599 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-18T15:04:14.599 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-18T15:04:14.599 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-03-18T15:04:14.599 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-03-18T15:04:14.599 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-03-18T15:04:14.599 DEBUG:tasks.check_counter:Getting stats from d 2024-03-18T15:04:14.600 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-18T15:04:14.849 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:04:14.879 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:14 smithi076 ceph-mon[32205]: pgmap v348: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-03-18T15:04:15.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:14 smithi174 ceph-mon[36787]: pgmap v348: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-03-18T15:04:15.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:14 smithi077 ceph-mon[35865]: pgmap v348: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-03-18T15:04:16.757 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T15:04:16.757 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:04:16.757 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 0, 2024-03-18T15:04:16.757 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 2, 2024-03-18T15:04:16.757 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 0, 2024-03-18T15:04:16.758 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 0, 2024-03-18T15:04:16.758 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 4, 2024-03-18T15:04:16.758 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 0, 2024-03-18T15:04:16.758 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.001180907, 2024-03-18T15:04:16.758 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T15:04:16.758 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.000283108, 2024-03-18T15:04:16.758 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T15:04:16.758 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:16.759 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:16.759 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:16.759 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:16.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:16.759 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:16.759 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:16.759 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:16.759 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:16.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:16.760 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T15:04:16.760 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T15:04:16.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:16.760 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:04:16.760 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 235, 2024-03-18T15:04:16.760 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 223, 2024-03-18T15:04:16.760 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 1443104, 2024-03-18T15:04:16.760 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 136192, 2024-03-18T15:04:16.760 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 2, 2024-03-18T15:04:16.761 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 1, 2024-03-18T15:04:16.761 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.038816318, 2024-03-18T15:04:16.761 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.012419760, 2024-03-18T15:04:16.761 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.032321436, 2024-03-18T15:04:16.761 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.003707513, 2024-03-18T15:04:16.761 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:16.761 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 223, 2024-03-18T15:04:16.761 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.009396384, 2024-03-18T15:04:16.761 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000042136 2024-03-18T15:04:16.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:16.762 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:16.762 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:16.762 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:16.762 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:16.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:16.762 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 1443104, 2024-03-18T15:04:16.762 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 136192 2024-03-18T15:04:16.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:16.763 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:04:16.763 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 2, 2024-03-18T15:04:16.763 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 169, 2024-03-18T15:04:16.763 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 320, 2024-03-18T15:04:16.763 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 185216, 2024-03-18T15:04:16.763 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 5, 2024-03-18T15:04:16.763 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 3, 2024-03-18T15:04:16.764 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.012075462, 2024-03-18T15:04:16.764 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.008116129, 2024-03-18T15:04:16.764 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.001908277, 2024-03-18T15:04:16.764 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T15:04:16.764 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:16.764 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 166, 2024-03-18T15:04:16.764 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.012214825, 2024-03-18T15:04:16.764 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000073583 2024-03-18T15:04:16.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:16.764 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:16.765 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:16.765 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:16.765 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:16.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:16.765 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T15:04:16.765 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 185216 2024-03-18T15:04:16.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:16.765 INFO:teuthology.orchestra.run.smithi076.stdout: "cct": { 2024-03-18T15:04:16.765 INFO:teuthology.orchestra.run.smithi076.stdout: "total_workers": 0, 2024-03-18T15:04:16.766 INFO:teuthology.orchestra.run.smithi076.stdout: "unhealthy_workers": 0 2024-03-18T15:04:16.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:16.766 INFO:teuthology.orchestra.run.smithi076.stdout: "mempool": { 2024-03-18T15:04:16.766 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:04:16.766 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_items": 0, 2024-03-18T15:04:16.766 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:04:16.766 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:04:16.766 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:04:16.767 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:04:16.767 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:04:16.767 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:04:16.767 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:04:16.767 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:04:16.767 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:04:16.767 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:04:16.767 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:04:16.767 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:04:16.768 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:04:16.768 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_items": 0, 2024-03-18T15:04:16.768 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:04:16.768 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_items": 0, 2024-03-18T15:04:16.768 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:04:16.768 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:04:16.768 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:04:16.768 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:04:16.768 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:04:16.769 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:04:16.769 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:04:16.769 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_items": 0, 2024-03-18T15:04:16.769 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:04:16.769 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:04:16.769 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:04:16.769 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_items": 0, 2024-03-18T15:04:16.769 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_bytes": 0, 2024-03-18T15:04:16.769 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_items": 0, 2024-03-18T15:04:16.769 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:04:16.770 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:04:16.770 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:04:16.770 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:04:16.770 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_bytes": 149057, 2024-03-18T15:04:16.770 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_items": 32, 2024-03-18T15:04:16.770 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:04:16.770 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_items": 0, 2024-03-18T15:04:16.770 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bytes": 0, 2024-03-18T15:04:16.771 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_items": 0, 2024-03-18T15:04:16.771 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:04:16.771 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_items": 0, 2024-03-18T15:04:16.771 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:04:16.771 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_items": 0, 2024-03-18T15:04:16.771 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_bytes": 0, 2024-03-18T15:04:16.771 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_items": 0, 2024-03-18T15:04:16.771 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:04:16.772 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:04:16.772 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_bytes": 0, 2024-03-18T15:04:16.772 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_items": 0, 2024-03-18T15:04:16.772 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_bytes": 928, 2024-03-18T15:04:16.772 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_items": 2, 2024-03-18T15:04:16.772 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_bytes": 0, 2024-03-18T15:04:16.772 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_items": 0, 2024-03-18T15:04:16.772 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_bytes": 0, 2024-03-18T15:04:16.773 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_items": 0 2024-03-18T15:04:16.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:16.773 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:04:16.773 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-03-18T15:04:16.773 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 104857600, 2024-03-18T15:04:16.773 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-03-18T15:04:16.773 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 233, 2024-03-18T15:04:16.773 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 1407540, 2024-03-18T15:04:16.773 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:16.774 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 233, 2024-03-18T15:04:16.774 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-03-18T15:04:16.774 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-03-18T15:04:16.774 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 233, 2024-03-18T15:04:16.774 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 1407540, 2024-03-18T15:04:16.774 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-03-18T15:04:16.774 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:16.774 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:16.775 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:16.775 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:16.775 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:16.775 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T15:04:17.076 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:16 smithi076 ceph-mon[32205]: pgmap v349: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-03-18T15:04:17.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:16 smithi174 ceph-mon[36787]: pgmap v349: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-03-18T15:04:17.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:16 smithi077 ceph-mon[35865]: pgmap v349: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-03-18T15:04:17.462 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-18T15:04:17.463 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-18T15:04:17.463 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-18T15:04:17.463 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-18T15:04:17.463 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-18T15:04:17.463 DEBUG:tasks.check_counter:Getting stats from g 2024-03-18T15:04:17.464 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-18T15:04:17.704 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:04:19.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:19 smithi076 ceph-mon[32205]: pgmap v350: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-03-18T15:04:19.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:19 smithi076 ceph-mon[32205]: osdmap e107: 12 total, 12 up, 12 in 2024-03-18T15:04:19.443 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T15:04:19.444 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:04:19.444 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 0, 2024-03-18T15:04:19.444 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 2, 2024-03-18T15:04:19.444 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 0, 2024-03-18T15:04:19.444 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 0, 2024-03-18T15:04:19.444 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 4, 2024-03-18T15:04:19.445 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 0, 2024-03-18T15:04:19.445 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.001182731, 2024-03-18T15:04:19.445 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T15:04:19.445 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.000275563, 2024-03-18T15:04:19.445 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T15:04:19.445 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:19.445 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:19.445 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:19.445 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:19.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:19.446 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:19.446 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:19.446 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:19.446 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:19.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:19.446 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T15:04:19.446 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T15:04:19.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:19.447 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:04:19.447 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 227, 2024-03-18T15:04:19.447 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 217, 2024-03-18T15:04:19.447 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 1439024, 2024-03-18T15:04:19.447 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 133184, 2024-03-18T15:04:19.447 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 2, 2024-03-18T15:04:19.447 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 1, 2024-03-18T15:04:19.447 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.035194893, 2024-03-18T15:04:19.447 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.010863906, 2024-03-18T15:04:19.448 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.028895823, 2024-03-18T15:04:19.448 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.003709821, 2024-03-18T15:04:19.448 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:19.448 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 216, 2024-03-18T15:04:19.448 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.007234835, 2024-03-18T15:04:19.448 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000033494 2024-03-18T15:04:19.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:19.448 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:19.448 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:19.448 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:19.449 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:19.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:19.449 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 1439024, 2024-03-18T15:04:19.449 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 133184 2024-03-18T15:04:19.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:19.449 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:04:19.449 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 6, 2024-03-18T15:04:19.449 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 168, 2024-03-18T15:04:19.449 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 3312, 2024-03-18T15:04:19.449 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 185136, 2024-03-18T15:04:19.450 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 5, 2024-03-18T15:04:19.450 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 3, 2024-03-18T15:04:19.450 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.011551342, 2024-03-18T15:04:19.450 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.007312521, 2024-03-18T15:04:19.450 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.002191113, 2024-03-18T15:04:19.450 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T15:04:19.450 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:19.450 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 166, 2024-03-18T15:04:19.450 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.012040996, 2024-03-18T15:04:19.451 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000072536 2024-03-18T15:04:19.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:19.451 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:19.451 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:19.451 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:19.451 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:19.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:19.451 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-18T15:04:19.452 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 185136 2024-03-18T15:04:19.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:19.452 INFO:teuthology.orchestra.run.smithi076.stdout: "cct": { 2024-03-18T15:04:19.452 INFO:teuthology.orchestra.run.smithi076.stdout: "total_workers": 0, 2024-03-18T15:04:19.452 INFO:teuthology.orchestra.run.smithi076.stdout: "unhealthy_workers": 0 2024-03-18T15:04:19.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:19.452 INFO:teuthology.orchestra.run.smithi076.stdout: "mempool": { 2024-03-18T15:04:19.452 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:04:19.452 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_items": 0, 2024-03-18T15:04:19.452 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:04:19.453 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:04:19.453 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:04:19.453 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:04:19.453 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:04:19.453 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:04:19.453 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:04:19.453 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:04:19.453 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:04:19.453 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:04:19.453 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:04:19.454 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:04:19.454 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:04:19.454 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_items": 0, 2024-03-18T15:04:19.454 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:04:19.454 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_items": 0, 2024-03-18T15:04:19.454 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:04:19.454 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:04:19.454 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:04:19.455 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:04:19.455 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:04:19.455 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:04:19.455 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:04:19.455 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_items": 0, 2024-03-18T15:04:19.455 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:04:19.455 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:04:19.455 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:04:19.455 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_items": 0, 2024-03-18T15:04:19.456 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_bytes": 0, 2024-03-18T15:04:19.456 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_items": 0, 2024-03-18T15:04:19.456 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:04:19.456 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:04:19.456 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:04:19.456 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:04:19.456 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_bytes": 149017, 2024-03-18T15:04:19.456 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_items": 31, 2024-03-18T15:04:19.456 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:04:19.456 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_items": 0, 2024-03-18T15:04:19.457 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bytes": 0, 2024-03-18T15:04:19.457 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_items": 0, 2024-03-18T15:04:19.457 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:04:19.457 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_items": 0, 2024-03-18T15:04:19.457 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:04:19.457 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_items": 0, 2024-03-18T15:04:19.457 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_bytes": 0, 2024-03-18T15:04:19.457 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_items": 0, 2024-03-18T15:04:19.457 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:04:19.458 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:04:19.458 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_bytes": 0, 2024-03-18T15:04:19.458 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_items": 0, 2024-03-18T15:04:19.458 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_bytes": 928, 2024-03-18T15:04:19.458 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_items": 2, 2024-03-18T15:04:19.458 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_bytes": 0, 2024-03-18T15:04:19.458 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_items": 0, 2024-03-18T15:04:19.458 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_bytes": 0, 2024-03-18T15:04:19.458 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_items": 0 2024-03-18T15:04:19.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:19.459 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:04:19.459 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-03-18T15:04:19.459 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 104857600, 2024-03-18T15:04:19.459 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-03-18T15:04:19.459 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 229, 2024-03-18T15:04:19.459 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 1407020, 2024-03-18T15:04:19.459 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:19.459 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 229, 2024-03-18T15:04:19.460 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-03-18T15:04:19.460 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-03-18T15:04:19.460 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 229, 2024-03-18T15:04:19.460 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 1407020, 2024-03-18T15:04:19.460 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-03-18T15:04:19.460 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:19.460 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:19.460 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:19.460 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:19.461 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:19.461 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T15:04:19.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:19 smithi174 ceph-mon[36787]: pgmap v350: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-03-18T15:04:19.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:19 smithi174 ceph-mon[36787]: osdmap e107: 12 total, 12 up, 12 in 2024-03-18T15:04:19.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:19 smithi077 ceph-mon[35865]: pgmap v350: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-03-18T15:04:19.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:19 smithi077 ceph-mon[35865]: osdmap e107: 12 total, 12 up, 12 in 2024-03-18T15:04:20.178 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-18T15:04:20.178 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-18T15:04:20.178 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-18T15:04:20.179 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-18T15:04:20.179 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-18T15:04:20.179 DEBUG:tasks.check_counter:Getting stats from j 2024-03-18T15:04:20.179 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-18T15:04:20.421 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:04:21.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:21 smithi076 ceph-mon[32205]: pgmap v352: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 1.6 MiB/s wr, 62 op/s 2024-03-18T15:04:21.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:21 smithi174 ceph-mon[36787]: pgmap v352: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 1.6 MiB/s wr, 62 op/s 2024-03-18T15:04:21.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:21 smithi077 ceph-mon[35865]: pgmap v352: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 1.6 MiB/s wr, 62 op/s 2024-03-18T15:04:22.658 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-18T15:04:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:04:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 4, 2024-03-18T15:04:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 4, 2024-03-18T15:04:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 2992, 2024-03-18T15:04:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 448, 2024-03-18T15:04:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 4, 2024-03-18T15:04:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 0, 2024-03-18T15:04:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.001507940, 2024-03-18T15:04:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.000026858, 2024-03-18T15:04:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.000673848, 2024-03-18T15:04:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T15:04:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 3, 2024-03-18T15:04:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000817915, 2024-03-18T15:04:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000272638 2024-03-18T15:04:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-18T15:04:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T15:04:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:04:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 233, 2024-03-18T15:04:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 223, 2024-03-18T15:04:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 1440656, 2024-03-18T15:04:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 137024, 2024-03-18T15:04:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 2, 2024-03-18T15:04:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 1, 2024-03-18T15:04:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.037956839, 2024-03-18T15:04:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.011570714, 2024-03-18T15:04:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.032182470, 2024-03-18T15:04:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.003783049, 2024-03-18T15:04:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 222, 2024-03-18T15:04:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.009017152, 2024-03-18T15:04:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000040617 2024-03-18T15:04:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 1440656, 2024-03-18T15:04:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 137024 2024-03-18T15:04:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:04:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 2, 2024-03-18T15:04:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 170, 2024-03-18T15:04:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 320, 2024-03-18T15:04:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 185392, 2024-03-18T15:04:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 5, 2024-03-18T15:04:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 3, 2024-03-18T15:04:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.012485460, 2024-03-18T15:04:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.007738318, 2024-03-18T15:04:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.002594775, 2024-03-18T15:04:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T15:04:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 167, 2024-03-18T15:04:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.013046612, 2024-03-18T15:04:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000078123 2024-03-18T15:04:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T15:04:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 185392 2024-03-18T15:04:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: "cct": { 2024-03-18T15:04:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: "total_workers": 0, 2024-03-18T15:04:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: "unhealthy_workers": 0 2024-03-18T15:04:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: "mempool": { 2024-03-18T15:04:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:04:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_items": 0, 2024-03-18T15:04:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:04:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:04:22.668 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:04:22.668 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:04:22.668 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:04:22.668 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:04:22.668 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:04:22.668 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:04:22.668 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:04:22.668 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:04:22.668 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:04:22.669 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:04:22.669 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:04:22.669 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_items": 0, 2024-03-18T15:04:22.669 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:04:22.669 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_items": 0, 2024-03-18T15:04:22.669 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:04:22.669 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:04:22.669 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:04:22.669 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:04:22.670 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:04:22.670 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:04:22.670 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:04:22.670 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_items": 0, 2024-03-18T15:04:22.670 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:04:22.670 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:04:22.670 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:04:22.670 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_items": 0, 2024-03-18T15:04:22.671 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_bytes": 0, 2024-03-18T15:04:22.671 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_items": 0, 2024-03-18T15:04:22.671 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:04:22.671 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:04:22.671 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:04:22.671 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:04:22.671 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_bytes": 148977, 2024-03-18T15:04:22.672 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_items": 30, 2024-03-18T15:04:22.672 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:04:22.672 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_items": 0, 2024-03-18T15:04:22.672 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bytes": 0, 2024-03-18T15:04:22.672 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_items": 0, 2024-03-18T15:04:22.672 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:04:22.672 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_items": 0, 2024-03-18T15:04:22.672 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:04:22.672 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_items": 0, 2024-03-18T15:04:22.673 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_bytes": 0, 2024-03-18T15:04:22.673 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_items": 0, 2024-03-18T15:04:22.673 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:04:22.673 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:04:22.673 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_bytes": 0, 2024-03-18T15:04:22.673 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_items": 0, 2024-03-18T15:04:22.673 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_bytes": 928, 2024-03-18T15:04:22.673 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_items": 2, 2024-03-18T15:04:22.674 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_bytes": 0, 2024-03-18T15:04:22.674 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_items": 0, 2024-03-18T15:04:22.674 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_bytes": 0, 2024-03-18T15:04:22.674 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_items": 0 2024-03-18T15:04:22.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-18T15:04:22.674 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:04:22.674 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-03-18T15:04:22.674 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 104857600, 2024-03-18T15:04:22.675 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-03-18T15:04:22.675 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 235, 2024-03-18T15:04:22.675 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 1407800, 2024-03-18T15:04:22.675 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:22.675 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 235, 2024-03-18T15:04:22.675 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-03-18T15:04:22.675 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-03-18T15:04:22.675 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 235, 2024-03-18T15:04:22.675 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 1407800, 2024-03-18T15:04:22.676 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-03-18T15:04:22.676 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-18T15:04:22.676 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-18T15:04:22.676 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-18T15:04:22.676 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:22.676 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-18T15:04:22.676 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-18T15:04:23.440 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-18T15:04:23.441 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-18T15:04:23.441 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-18T15:04:23.441 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-18T15:04:23.441 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-18T15:04:23.441 DEBUG:tasks.check_counter:Getting stats from b 2024-03-18T15:04:23.442 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-18T15:04:23.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:23 smithi076 ceph-mon[32205]: pgmap v353: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 1.6 MiB/s wr, 62 op/s 2024-03-18T15:04:23.713 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:04:23.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:23 smithi174 ceph-mon[36787]: pgmap v353: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 1.6 MiB/s wr, 62 op/s 2024-03-18T15:04:23.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:23 smithi077 ceph-mon[35865]: pgmap v353: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 1.6 MiB/s wr, 62 op/s 2024-03-18T15:04:25.569 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:25 smithi077 ceph-mon[35865]: pgmap v354: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 1.4 MiB/s wr, 61 op/s 2024-03-18T15:04:25.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:25 smithi076 ceph-mon[32205]: pgmap v354: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 1.4 MiB/s wr, 61 op/s 2024-03-18T15:04:25.698 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-18T15:04:25.698 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:04:25.698 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 230125, 2024-03-18T15:04:25.698 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 301957, 2024-03-18T15:04:25.699 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 63363873, 2024-03-18T15:04:25.699 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 421966779, 2024-03-18T15:04:25.699 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 122, 2024-03-18T15:04:25.699 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 7, 2024-03-18T15:04:25.699 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 10.895665555, 2024-03-18T15:04:25.699 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 5.462697058, 2024-03-18T15:04:25.699 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 5.583608816, 2024-03-18T15:04:25.699 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.946817926, 2024-03-18T15:04:25.700 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:25.700 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 301952, 2024-03-18T15:04:25.700 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 11.843734001, 2024-03-18T15:04:25.700 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000039223 2024-03-18T15:04:25.700 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.700 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:25.700 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 252991, 2024-03-18T15:04:25.700 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.043107435, 2024-03-18T15:04:25.701 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000170 2024-03-18T15:04:25.701 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.701 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 100384, 2024-03-18T15:04:25.701 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 32640 2024-03-18T15:04:25.701 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.701 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:04:25.701 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 189545, 2024-03-18T15:04:25.702 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 195000, 2024-03-18T15:04:25.702 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 47650487, 2024-03-18T15:04:25.702 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 205659733, 2024-03-18T15:04:25.702 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 8, 2024-03-18T15:04:25.702 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 6, 2024-03-18T15:04:25.702 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 9.090843414, 2024-03-18T15:04:25.702 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 3.681503419, 2024-03-18T15:04:25.702 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 5.490508738, 2024-03-18T15:04:25.703 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.715057993, 2024-03-18T15:04:25.703 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:25.703 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 195002, 2024-03-18T15:04:25.703 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 10.253973453, 2024-03-18T15:04:25.703 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000052583 2024-03-18T15:04:25.703 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.703 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:25.703 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 331350, 2024-03-18T15:04:25.704 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.042538945, 2024-03-18T15:04:25.704 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000128 2024-03-18T15:04:25.704 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.704 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 1573664, 2024-03-18T15:04:25.704 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 478448 2024-03-18T15:04:25.704 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.704 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:04:25.704 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 103843, 2024-03-18T15:04:25.705 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 139216, 2024-03-18T15:04:25.705 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 26967856, 2024-03-18T15:04:25.705 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 150202542, 2024-03-18T15:04:25.705 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 27, 2024-03-18T15:04:25.705 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 6, 2024-03-18T15:04:25.705 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 6.271135984, 2024-03-18T15:04:25.705 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 2.755400863, 2024-03-18T15:04:25.705 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 3.181221176, 2024-03-18T15:04:25.705 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.531220336, 2024-03-18T15:04:25.706 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:25.706 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 139214, 2024-03-18T15:04:25.706 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 8.219039210, 2024-03-18T15:04:25.706 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000059038 2024-03-18T15:04:25.706 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.706 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:25.706 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 205652, 2024-03-18T15:04:25.706 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.029913619, 2024-03-18T15:04:25.707 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000145 2024-03-18T15:04:25.707 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.707 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 376320, 2024-03-18T15:04:25.707 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 235264 2024-03-18T15:04:25.707 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.707 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2024-03-18T15:04:25.707 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 1, 2024-03-18T15:04:25.707 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2024-03-18T15:04:25.708 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.708 INFO:teuthology.orchestra.run.smithi077.stdout: "finisher-MDSRank": { 2024-03-18T15:04:25.708 INFO:teuthology.orchestra.run.smithi077.stdout: "queue_len": 0, 2024-03-18T15:04:25.708 INFO:teuthology.orchestra.run.smithi077.stdout: "complete_latency": { 2024-03-18T15:04:25.708 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 103394, 2024-03-18T15:04:25.708 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 71.027335501, 2024-03-18T15:04:25.708 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000686958 2024-03-18T15:04:25.708 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:25.708 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.709 INFO:teuthology.orchestra.run.smithi077.stdout: "finisher-PurgeQueue": { 2024-03-18T15:04:25.709 INFO:teuthology.orchestra.run.smithi077.stdout: "queue_len": 0, 2024-03-18T15:04:25.709 INFO:teuthology.orchestra.run.smithi077.stdout: "complete_latency": { 2024-03-18T15:04:25.709 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 23538, 2024-03-18T15:04:25.709 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 2.125759601, 2024-03-18T15:04:25.709 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000090311 2024-03-18T15:04:25.709 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:25.709 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.709 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": { 2024-03-18T15:04:25.710 INFO:teuthology.orchestra.run.smithi077.stdout: "request": 134123, 2024-03-18T15:04:25.710 INFO:teuthology.orchestra.run.smithi077.stdout: "reply": 121235, 2024-03-18T15:04:25.710 INFO:teuthology.orchestra.run.smithi077.stdout: "reply_latency": { 2024-03-18T15:04:25.710 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 121235, 2024-03-18T15:04:25.710 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 320.141375193, 2024-03-18T15:04:25.710 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.002640667 2024-03-18T15:04:25.710 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.710 INFO:teuthology.orchestra.run.smithi077.stdout: "slow_reply": 0, 2024-03-18T15:04:25.711 INFO:teuthology.orchestra.run.smithi077.stdout: "forward": 12888, 2024-03-18T15:04:25.711 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_fetch_complete": 6, 2024-03-18T15:04:25.711 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_fetch_keys": 2, 2024-03-18T15:04:25.711 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_commit": 1053, 2024-03-18T15:04:25.711 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_split": 1, 2024-03-18T15:04:25.711 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_merge": 2, 2024-03-18T15:04:25.711 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes": 804, 2024-03-18T15:04:25.711 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_top": 162, 2024-03-18T15:04:25.712 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_bottom": 616, 2024-03-18T15:04:25.712 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_pin_tail": 26, 2024-03-18T15:04:25.712 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_pinned": 573, 2024-03-18T15:04:25.712 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_expired": 211, 2024-03-18T15:04:25.712 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_with_caps": 173, 2024-03-18T15:04:25.713 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": 173, 2024-03-18T15:04:25.713 INFO:teuthology.orchestra.run.smithi077.stdout: "subtrees": 15, 2024-03-18T15:04:25.713 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse": 206477, 2024-03-18T15:04:25.713 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_hit": 139651, 2024-03-18T15:04:25.713 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_forward": 12879, 2024-03-18T15:04:25.713 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_discover": 2, 2024-03-18T15:04:25.713 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_dir_fetch": 1, 2024-03-18T15:04:25.713 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_remote_ino": 0, 2024-03-18T15:04:25.713 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_lock": 583, 2024-03-18T15:04:25.714 INFO:teuthology.orchestra.run.smithi077.stdout: "load_cent": 90596, 2024-03-18T15:04:25.714 INFO:teuthology.orchestra.run.smithi077.stdout: "q": 13, 2024-03-18T15:04:25.714 INFO:teuthology.orchestra.run.smithi077.stdout: "exported": 4, 2024-03-18T15:04:25.714 INFO:teuthology.orchestra.run.smithi077.stdout: "exported_inodes": 127, 2024-03-18T15:04:25.715 INFO:teuthology.orchestra.run.smithi077.stdout: "imported": 0, 2024-03-18T15:04:25.715 INFO:teuthology.orchestra.run.smithi077.stdout: "imported_inodes": 0, 2024-03-18T15:04:25.715 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_dir_fetch": 0, 2024-03-18T15:04:25.715 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_backtrace_fetch": 82, 2024-03-18T15:04:25.715 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_peer_discover": 0, 2024-03-18T15:04:25.715 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rfiles": 67, 2024-03-18T15:04:25.716 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rbytes": 21137952, 2024-03-18T15:04:25.716 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rsnaps": 6, 2024-03-18T15:04:25.716 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_backtrace_fetch": 82, 2024-03-18T15:04:25.716 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_set_tag": 0, 2024-03-18T15:04:25.716 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T15:04:25.716 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_inotable_repaired": 0, 2024-03-18T15:04:25.716 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dir_inodes": 36, 2024-03-18T15:04:25.716 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dir_base_inodes": 3, 2024-03-18T15:04:25.717 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dirfrag_rstats": 33, 2024-03-18T15:04:25.717 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_file_inodes": 46, 2024-03-18T15:04:25.717 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_inode_file_caps": 5, 2024-03-18T15:04:25.717 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_revoke": 28648, 2024-03-18T15:04:25.717 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_grant": 56217, 2024-03-18T15:04:25.717 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T15:04:25.717 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T15:04:25.717 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T15:04:25.718 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_caps": 49067, 2024-03-18T15:04:25.718 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_caps_dirty": 11979, 2024-03-18T15:04:25.718 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_cap_release": 7236, 2024-03-18T15:04:25.718 INFO:teuthology.orchestra.run.smithi077.stdout: "process_request_cap_release": 60981 2024-03-18T15:04:25.718 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.718 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_cache": { 2024-03-18T15:04:25.718 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_update": 6, 2024-03-18T15:04:25.718 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_update_receipt": 0, 2024-03-18T15:04:25.719 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_try_discover": 0, 2024-03-18T15:04:25.719 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_send_discover": 2, 2024-03-18T15:04:25.719 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_handle_discover": 184, 2024-03-18T15:04:25.719 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays": 140, 2024-03-18T15:04:25.719 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays_delayed": 0, 2024-03-18T15:04:25.719 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays_enqueuing": 0, 2024-03-18T15:04:25.719 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_created": 14795, 2024-03-18T15:04:25.719 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_enqueued": 14655, 2024-03-18T15:04:25.720 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_reintegrated": 0, 2024-03-18T15:04:25.720 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_migrated": 0, 2024-03-18T15:04:25.720 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_processing": 0, 2024-03-18T15:04:25.720 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_enqueued": 0, 2024-03-18T15:04:25.720 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_prioritized": 0, 2024-03-18T15:04:25.720 INFO:teuthology.orchestra.run.smithi077.stdout: "recovery_started": 0, 2024-03-18T15:04:25.720 INFO:teuthology.orchestra.run.smithi077.stdout: "recovery_completed": 0, 2024-03-18T15:04:25.720 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_quiesce_path": 0, 2024-03-18T15:04:25.720 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_quiesce_inode": 0, 2024-03-18T15:04:25.721 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_enqueue_scrub": 3, 2024-03-18T15:04:25.721 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_exportdir": 4, 2024-03-18T15:04:25.721 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_flush": 0, 2024-03-18T15:04:25.721 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_fragmentdir": 3, 2024-03-18T15:04:25.721 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_fragstats": 0, 2024-03-18T15:04:25.721 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_inodestats": 0 2024-03-18T15:04:25.721 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.721 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_log": { 2024-03-18T15:04:25.722 INFO:teuthology.orchestra.run.smithi077.stdout: "evlrg": 0, 2024-03-18T15:04:25.722 INFO:teuthology.orchestra.run.smithi077.stdout: "evadd": 98574, 2024-03-18T15:04:25.722 INFO:teuthology.orchestra.run.smithi077.stdout: "evex": 96639, 2024-03-18T15:04:25.722 INFO:teuthology.orchestra.run.smithi077.stdout: "evtrm": 96639, 2024-03-18T15:04:25.722 INFO:teuthology.orchestra.run.smithi077.stdout: "ev": 1935, 2024-03-18T15:04:25.722 INFO:teuthology.orchestra.run.smithi077.stdout: "evexg": 0, 2024-03-18T15:04:25.722 INFO:teuthology.orchestra.run.smithi077.stdout: "evexd": 1879, 2024-03-18T15:04:25.722 INFO:teuthology.orchestra.run.smithi077.stdout: "segadd": 182, 2024-03-18T15:04:25.723 INFO:teuthology.orchestra.run.smithi077.stdout: "segex": 179, 2024-03-18T15:04:25.723 INFO:teuthology.orchestra.run.smithi077.stdout: "segtrm": 179, 2024-03-18T15:04:25.723 INFO:teuthology.orchestra.run.smithi077.stdout: "seg": 3, 2024-03-18T15:04:25.723 INFO:teuthology.orchestra.run.smithi077.stdout: "segmjr": 1, 2024-03-18T15:04:25.723 INFO:teuthology.orchestra.run.smithi077.stdout: "segexg": 0, 2024-03-18T15:04:25.723 INFO:teuthology.orchestra.run.smithi077.stdout: "segexd": 4, 2024-03-18T15:04:25.723 INFO:teuthology.orchestra.run.smithi077.stdout: "expos": 469926512, 2024-03-18T15:04:25.723 INFO:teuthology.orchestra.run.smithi077.stdout: "wrpos": 479333631, 2024-03-18T15:04:25.723 INFO:teuthology.orchestra.run.smithi077.stdout: "rdpos": 0, 2024-03-18T15:04:25.724 INFO:teuthology.orchestra.run.smithi077.stdout: "jlat": { 2024-03-18T15:04:25.724 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 51590, 2024-03-18T15:04:25.724 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 117.840436155, 2024-03-18T15:04:25.724 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.002284172 2024-03-18T15:04:25.724 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.724 INFO:teuthology.orchestra.run.smithi077.stdout: "replayed": 0 2024-03-18T15:04:25.724 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.724 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_mem": { 2024-03-18T15:04:25.725 INFO:teuthology.orchestra.run.smithi077.stdout: "ino": 421, 2024-03-18T15:04:25.725 INFO:teuthology.orchestra.run.smithi077.stdout: "ino+": 15214, 2024-03-18T15:04:25.725 INFO:teuthology.orchestra.run.smithi077.stdout: "ino-": 14793, 2024-03-18T15:04:25.725 INFO:teuthology.orchestra.run.smithi077.stdout: "dir": 44, 2024-03-18T15:04:25.725 INFO:teuthology.orchestra.run.smithi077.stdout: "dir+": 57, 2024-03-18T15:04:25.725 INFO:teuthology.orchestra.run.smithi077.stdout: "dir-": 13, 2024-03-18T15:04:25.725 INFO:teuthology.orchestra.run.smithi077.stdout: "dn": 827, 2024-03-18T15:04:25.725 INFO:teuthology.orchestra.run.smithi077.stdout: "dn+": 29310, 2024-03-18T15:04:25.725 INFO:teuthology.orchestra.run.smithi077.stdout: "dn-": 28483, 2024-03-18T15:04:25.726 INFO:teuthology.orchestra.run.smithi077.stdout: "cap": 111, 2024-03-18T15:04:25.726 INFO:teuthology.orchestra.run.smithi077.stdout: "cap+": 15049, 2024-03-18T15:04:25.726 INFO:teuthology.orchestra.run.smithi077.stdout: "cap-": 14938, 2024-03-18T15:04:25.726 INFO:teuthology.orchestra.run.smithi077.stdout: "rss": 75132, 2024-03-18T15:04:25.726 INFO:teuthology.orchestra.run.smithi077.stdout: "heap": 231804 2024-03-18T15:04:25.726 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.726 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_server": { 2024-03-18T15:04:25.726 INFO:teuthology.orchestra.run.smithi077.stdout: "dispatch_client_request": 185850, 2024-03-18T15:04:25.727 INFO:teuthology.orchestra.run.smithi077.stdout: "dispatch_server_request": 46968, 2024-03-18T15:04:25.727 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_request": 134123, 2024-03-18T15:04:25.727 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_session": 2798, 2024-03-18T15:04:25.727 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_peer_request": 93930, 2024-03-18T15:04:25.727 INFO:teuthology.orchestra.run.smithi077.stdout: "req_create_latency": { 2024-03-18T15:04:25.727 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 14863, 2024-03-18T15:04:25.727 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 20.507990169, 2024-03-18T15:04:25.727 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.001379801 2024-03-18T15:04:25.728 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.728 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getattr_latency": { 2024-03-18T15:04:25.728 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 28295, 2024-03-18T15:04:25.728 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 177.316947361, 2024-03-18T15:04:25.728 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.006266723 2024-03-18T15:04:25.728 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.728 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getfilelock_latency": { 2024-03-18T15:04:25.728 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.728 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.729 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.729 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.729 INFO:teuthology.orchestra.run.smithi077.stdout: "req_link_latency": { 2024-03-18T15:04:25.729 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.729 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.729 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.729 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.729 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookup_latency": { 2024-03-18T15:04:25.729 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 33044, 2024-03-18T15:04:25.730 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 26.696599344, 2024-03-18T15:04:25.730 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000807910 2024-03-18T15:04:25.730 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.730 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookuphash_latency": { 2024-03-18T15:04:25.730 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.730 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.730 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.730 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.731 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupino_latency": { 2024-03-18T15:04:25.731 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 1, 2024-03-18T15:04:25.731 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000170394, 2024-03-18T15:04:25.731 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000170394 2024-03-18T15:04:25.731 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.731 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupname_latency": { 2024-03-18T15:04:25.731 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.731 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.731 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.732 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.732 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupparent_latency": { 2024-03-18T15:04:25.732 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.732 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.732 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.732 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.732 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupsnap_latency": { 2024-03-18T15:04:25.732 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.733 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:25 smithi174 ceph-mon[36787]: pgmap v354: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 1.4 MiB/s wr, 61 op/s 2024-03-18T15:04:25.733 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.734 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.734 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.734 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lssnap_latency": { 2024-03-18T15:04:25.734 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 11, 2024-03-18T15:04:25.734 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.001982012, 2024-03-18T15:04:25.734 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000180182 2024-03-18T15:04:25.734 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.734 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mkdir_latency": { 2024-03-18T15:04:25.735 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 20, 2024-03-18T15:04:25.735 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.019847889, 2024-03-18T15:04:25.735 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000992394 2024-03-18T15:04:25.735 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.735 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mknod_latency": { 2024-03-18T15:04:25.735 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.735 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.735 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.736 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.736 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mksnap_latency": { 2024-03-18T15:04:25.736 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 11, 2024-03-18T15:04:25.736 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 1.655588606, 2024-03-18T15:04:25.736 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.150508055 2024-03-18T15:04:25.736 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.736 INFO:teuthology.orchestra.run.smithi077.stdout: "req_open_latency": { 2024-03-18T15:04:25.736 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 1, 2024-03-18T15:04:25.737 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000151435, 2024-03-18T15:04:25.737 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000151435 2024-03-18T15:04:25.737 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.737 INFO:teuthology.orchestra.run.smithi077.stdout: "req_readdir_latency": { 2024-03-18T15:04:25.737 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 23841, 2024-03-18T15:04:25.737 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 59.194469577, 2024-03-18T15:04:25.737 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.002482885 2024-03-18T15:04:25.738 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.738 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rename_latency": { 2024-03-18T15:04:25.738 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 5152, 2024-03-18T15:04:25.738 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 8.782973020, 2024-03-18T15:04:25.738 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.001704769 2024-03-18T15:04:25.738 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.738 INFO:teuthology.orchestra.run.smithi077.stdout: "req_renamesnap_latency": { 2024-03-18T15:04:25.738 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.739 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.739 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.739 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.739 INFO:teuthology.orchestra.run.smithi077.stdout: "req_snapdiff_latency": { 2024-03-18T15:04:25.739 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.739 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.739 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.739 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.740 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmdir_latency": { 2024-03-18T15:04:25.740 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.740 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.740 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.740 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.740 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmsnap_latency": { 2024-03-18T15:04:25.740 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 5, 2024-03-18T15:04:25.740 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 3.360079584, 2024-03-18T15:04:25.741 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.672015916 2024-03-18T15:04:25.741 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.741 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmxattr_latency": { 2024-03-18T15:04:25.741 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.741 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.741 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.741 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.741 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setattr_latency": { 2024-03-18T15:04:25.741 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 1, 2024-03-18T15:04:25.742 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.001047145, 2024-03-18T15:04:25.742 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.001047145 2024-03-18T15:04:25.742 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.742 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setdirlayout_latency": { 2024-03-18T15:04:25.742 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.742 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.742 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.742 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.743 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setfilelock_latency": { 2024-03-18T15:04:25.743 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 1188, 2024-03-18T15:04:25.743 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.718171391, 2024-03-18T15:04:25.743 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000604521 2024-03-18T15:04:25.743 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.743 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setlayout_latency": { 2024-03-18T15:04:25.743 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.743 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.743 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.744 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.744 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setxattr_latency": { 2024-03-18T15:04:25.744 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 7, 2024-03-18T15:04:25.744 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.015864793, 2024-03-18T15:04:25.744 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.002266399 2024-03-18T15:04:25.744 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.744 INFO:teuthology.orchestra.run.smithi077.stdout: "req_symlink_latency": { 2024-03-18T15:04:25.744 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.745 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.745 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.745 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.745 INFO:teuthology.orchestra.run.smithi077.stdout: "req_unlink_latency": { 2024-03-18T15:04:25.745 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 14795, 2024-03-18T15:04:25.745 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 21.869492473, 2024-03-18T15:04:25.745 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.001478167 2024-03-18T15:04:25.745 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.745 INFO:teuthology.orchestra.run.smithi077.stdout: "cap_revoke_eviction": 0, 2024-03-18T15:04:25.746 INFO:teuthology.orchestra.run.smithi077.stdout: "cap_acquisition_throttle": 0, 2024-03-18T15:04:25.746 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getvxattr_latency": { 2024-03-18T15:04:25.746 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.746 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.746 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.746 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:25.746 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.746 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_sessions": { 2024-03-18T15:04:25.747 INFO:teuthology.orchestra.run.smithi077.stdout: "session_count": 2, 2024-03-18T15:04:25.747 INFO:teuthology.orchestra.run.smithi077.stdout: "session_add": 5, 2024-03-18T15:04:25.747 INFO:teuthology.orchestra.run.smithi077.stdout: "session_remove": 3, 2024-03-18T15:04:25.747 INFO:teuthology.orchestra.run.smithi077.stdout: "sessions_open": 2, 2024-03-18T15:04:25.747 INFO:teuthology.orchestra.run.smithi077.stdout: "sessions_stale": 0, 2024-03-18T15:04:25.747 INFO:teuthology.orchestra.run.smithi077.stdout: "total_load": 15399, 2024-03-18T15:04:25.747 INFO:teuthology.orchestra.run.smithi077.stdout: "average_load": 7699, 2024-03-18T15:04:25.747 INFO:teuthology.orchestra.run.smithi077.stdout: "avg_session_uptime": 651, 2024-03-18T15:04:25.747 INFO:teuthology.orchestra.run.smithi077.stdout: "mdthresh_evicted": 0 2024-03-18T15:04:25.748 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.748 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2024-03-18T15:04:25.748 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:04:25.748 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2024-03-18T15:04:25.748 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:04:25.748 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:04:25.748 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:04:25.748 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:04:25.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:04:25.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:04:25.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:04:25.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:04:25.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:04:25.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:04:25.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:04:25.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:04:25.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:04:25.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2024-03-18T15:04:25.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:04:25.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2024-03-18T15:04:25.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:04:25.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:04:25.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:04:25.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:04:25.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:04:25.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:04:25.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:04:25.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2024-03-18T15:04:25.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:04:25.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:04:25.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:04:25.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2024-03-18T15:04:25.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2024-03-18T15:04:25.752 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2024-03-18T15:04:25.752 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:04:25.752 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:04:25.752 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:04:25.752 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:04:25.752 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 1615003, 2024-03-18T15:04:25.752 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 840, 2024-03-18T15:04:25.752 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:04:25.752 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2024-03-18T15:04:25.753 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2024-03-18T15:04:25.753 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2024-03-18T15:04:25.753 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:04:25.753 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2024-03-18T15:04:25.753 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:04:25.753 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2024-03-18T15:04:25.753 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 7936, 2024-03-18T15:04:25.753 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 165, 2024-03-18T15:04:25.753 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:04:25.754 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:04:25.754 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2024-03-18T15:04:25.754 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2024-03-18T15:04:25.754 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 2262306, 2024-03-18T15:04:25.754 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 54635, 2024-03-18T15:04:25.754 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2024-03-18T15:04:25.754 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2024-03-18T15:04:25.754 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2024-03-18T15:04:25.755 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2024-03-18T15:04:25.755 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.755 INFO:teuthology.orchestra.run.smithi077.stdout: "objecter": { 2024-03-18T15:04:25.755 INFO:teuthology.orchestra.run.smithi077.stdout: "op_active": 0, 2024-03-18T15:04:25.755 INFO:teuthology.orchestra.run.smithi077.stdout: "op_laggy": 0, 2024-03-18T15:04:25.755 INFO:teuthology.orchestra.run.smithi077.stdout: "op_send": 79349, 2024-03-18T15:04:25.755 INFO:teuthology.orchestra.run.smithi077.stdout: "op_send_bytes": 489965149, 2024-03-18T15:04:25.755 INFO:teuthology.orchestra.run.smithi077.stdout: "op_resend": 31, 2024-03-18T15:04:25.755 INFO:teuthology.orchestra.run.smithi077.stdout: "op_reply": 79349, 2024-03-18T15:04:25.756 INFO:teuthology.orchestra.run.smithi077.stdout: "op_latency": { 2024-03-18T15:04:25.756 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 79349, 2024-03-18T15:04:25.756 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 446.403301457, 2024-03-18T15:04:25.756 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.005625821 2024-03-18T15:04:25.756 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.757 INFO:teuthology.orchestra.run.smithi077.stdout: "op_inflight": 0, 2024-03-18T15:04:25.757 INFO:teuthology.orchestra.run.smithi077.stdout: "oplen_avg": { 2024-03-18T15:04:25.757 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 79349, 2024-03-18T15:04:25.757 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 100728 2024-03-18T15:04:25.757 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.757 INFO:teuthology.orchestra.run.smithi077.stdout: "op": 79349, 2024-03-18T15:04:25.757 INFO:teuthology.orchestra.run.smithi077.stdout: "op_r": 608, 2024-03-18T15:04:25.757 INFO:teuthology.orchestra.run.smithi077.stdout: "op_w": 78741, 2024-03-18T15:04:25.758 INFO:teuthology.orchestra.run.smithi077.stdout: "op_rmw": 0, 2024-03-18T15:04:25.758 INFO:teuthology.orchestra.run.smithi077.stdout: "op_pg": 0, 2024-03-18T15:04:25.758 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_stat": 1011, 2024-03-18T15:04:25.758 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_create": 9031, 2024-03-18T15:04:25.758 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_read": 515, 2024-03-18T15:04:25.758 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_write": 52102, 2024-03-18T15:04:25.758 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_writefull": 461, 2024-03-18T15:04:25.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_writesame": 0, 2024-03-18T15:04:25.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_append": 0, 2024-03-18T15:04:25.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_zero": 0, 2024-03-18T15:04:25.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_truncate": 1, 2024-03-18T15:04:25.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_delete": 15400, 2024-03-18T15:04:25.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_mapext": 0, 2024-03-18T15:04:25.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_sparse_read": 0, 2024-03-18T15:04:25.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_clonerange": 0, 2024-03-18T15:04:25.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_getxattr": 93, 2024-03-18T15:04:25.760 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_setxattr": 18062, 2024-03-18T15:04:25.760 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_cmpxattr": 0, 2024-03-18T15:04:25.760 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_rmxattr": 0, 2024-03-18T15:04:25.760 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_resetxattrs": 0, 2024-03-18T15:04:25.760 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_call": 0, 2024-03-18T15:04:25.760 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_watch": 0, 2024-03-18T15:04:25.760 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_notify": 0, 2024-03-18T15:04:25.760 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T15:04:25.761 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_pgls": 0, 2024-03-18T15:04:25.761 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_pgls_filter": 0, 2024-03-18T15:04:25.761 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_other": 0, 2024-03-18T15:04:25.761 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_active": 0, 2024-03-18T15:04:25.761 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_send": 0, 2024-03-18T15:04:25.761 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_resend": 0, 2024-03-18T15:04:25.761 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_ping": 0, 2024-03-18T15:04:25.761 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_active": 0, 2024-03-18T15:04:25.762 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_send": 0, 2024-03-18T15:04:25.762 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_resend": 0, 2024-03-18T15:04:25.762 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_active": 0, 2024-03-18T15:04:25.762 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_send": 0, 2024-03-18T15:04:25.762 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_resend": 0, 2024-03-18T15:04:25.762 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_active": 0, 2024-03-18T15:04:25.762 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_send": 0, 2024-03-18T15:04:25.762 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_resend": 0, 2024-03-18T15:04:25.763 INFO:teuthology.orchestra.run.smithi077.stdout: "command_active": 0, 2024-03-18T15:04:25.763 INFO:teuthology.orchestra.run.smithi077.stdout: "command_send": 0, 2024-03-18T15:04:25.763 INFO:teuthology.orchestra.run.smithi077.stdout: "command_resend": 0, 2024-03-18T15:04:25.763 INFO:teuthology.orchestra.run.smithi077.stdout: "map_epoch": 107, 2024-03-18T15:04:25.763 INFO:teuthology.orchestra.run.smithi077.stdout: "map_full": 0, 2024-03-18T15:04:25.763 INFO:teuthology.orchestra.run.smithi077.stdout: "map_inc": 22, 2024-03-18T15:04:25.763 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_sessions": 12, 2024-03-18T15:04:25.763 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_session_open": 12, 2024-03-18T15:04:25.763 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_session_close": 0, 2024-03-18T15:04:25.764 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_laggy": 0, 2024-03-18T15:04:25.764 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_wr": 3292, 2024-03-18T15:04:25.764 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_rd": 16, 2024-03-18T15:04:25.764 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_del": 744 2024-03-18T15:04:25.764 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.764 INFO:teuthology.orchestra.run.smithi077.stdout: "oft": { 2024-03-18T15:04:25.764 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_objs": 1, 2024-03-18T15:04:25.764 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_kv_pairs": 0, 2024-03-18T15:04:25.765 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_updates": 0, 2024-03-18T15:04:25.765 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_removes": 0 2024-03-18T15:04:25.765 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.765 INFO:teuthology.orchestra.run.smithi077.stdout: "osd-slow-ops": { 2024-03-18T15:04:25.765 INFO:teuthology.orchestra.run.smithi077.stdout: "slow_ops_count": 0 2024-03-18T15:04:25.765 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.765 INFO:teuthology.orchestra.run.smithi077.stdout: "purge_queue": { 2024-03-18T15:04:25.765 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_ops": 0, 2024-03-18T15:04:25.765 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_ops_high_water": 10, 2024-03-18T15:04:25.766 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing": 0, 2024-03-18T15:04:25.766 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_high_water": 10, 2024-03-18T15:04:25.766 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executed_ops": 14655, 2024-03-18T15:04:25.766 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executed": 14655, 2024-03-18T15:04:25.766 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_item_in_journal": 0 2024-03-18T15:04:25.766 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.766 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:04:25.766 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-18T15:04:25.767 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-03-18T15:04:25.767 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-18T15:04:25.767 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 523509, 2024-03-18T15:04:25.767 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 93916759, 2024-03-18T15:04:25.767 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:25.767 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 523509, 2024-03-18T15:04:25.767 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-18T15:04:25.767 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-18T15:04:25.767 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 523509, 2024-03-18T15:04:25.768 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 93916759, 2024-03-18T15:04:25.768 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-18T15:04:25.768 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.768 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.768 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.768 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:25.768 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.768 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-objecter_bytes": { 2024-03-18T15:04:25.768 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-18T15:04:25.769 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-03-18T15:04:25.769 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-18T15:04:25.769 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-03-18T15:04:25.769 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-03-18T15:04:25.769 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:25.769 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:25.769 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 79349, 2024-03-18T15:04:25.769 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 491447200, 2024-03-18T15:04:25.769 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 79349, 2024-03-18T15:04:25.770 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 491447200, 2024-03-18T15:04:25.770 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-18T15:04:25.770 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.770 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.770 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.770 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:25.770 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.771 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-objecter_ops": { 2024-03-18T15:04:25.771 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-18T15:04:25.771 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 1024, 2024-03-18T15:04:25.771 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-18T15:04:25.771 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-03-18T15:04:25.771 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-03-18T15:04:25.771 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:25.771 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:25.771 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 79349, 2024-03-18T15:04:25.772 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 79349, 2024-03-18T15:04:25.772 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 79349, 2024-03-18T15:04:25.772 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 79349, 2024-03-18T15:04:25.772 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-18T15:04:25.772 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.772 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.772 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.772 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:25.773 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.773 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-write_buf_throttle": { 2024-03-18T15:04:25.773 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-18T15:04:25.773 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 3758096384, 2024-03-18T15:04:25.773 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-18T15:04:25.773 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 14655, 2024-03-18T15:04:25.773 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 1482155, 2024-03-18T15:04:25.773 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:25.774 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 14655, 2024-03-18T15:04:25.774 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-18T15:04:25.774 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-18T15:04:25.774 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 512, 2024-03-18T15:04:25.774 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 1482155, 2024-03-18T15:04:25.774 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-18T15:04:25.774 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.774 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.775 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.775 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:25.775 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:25.775 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-write_buf_throttle-0x55a743fa4500": { 2024-03-18T15:04:25.775 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 154666, 2024-03-18T15:04:25.775 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 3758096384, 2024-03-18T15:04:25.775 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-18T15:04:25.775 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 98574, 2024-03-18T15:04:25.775 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 475139327, 2024-03-18T15:04:25.776 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:25.776 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 98574, 2024-03-18T15:04:25.776 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-18T15:04:25.776 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-18T15:04:25.776 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 51590, 2024-03-18T15:04:25.776 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 474984661, 2024-03-18T15:04:25.776 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-18T15:04:25.776 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:25.776 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:25.777 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:25.777 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:25.777 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:25.777 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-18T15:04:27.437 INFO:tasks.check_counter:Daemon mds.b mds.exported=4 2024-03-18T15:04:27.437 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-18T15:04:27.437 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-18T15:04:27.438 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2024-03-18T15:04:27.438 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2024-03-18T15:04:27.438 DEBUG:tasks.check_counter:Getting stats from e 2024-03-18T15:04:27.438 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-18T15:04:27.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:27 smithi076 ceph-mon[32205]: pgmap v355: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 1.4 MiB/s wr, 61 op/s 2024-03-18T15:04:27.683 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:04:27.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:27 smithi174 ceph-mon[36787]: pgmap v355: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 1.4 MiB/s wr, 61 op/s 2024-03-18T15:04:27.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:27 smithi077 ceph-mon[35865]: pgmap v355: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 1.4 MiB/s wr, 61 op/s 2024-03-18T15:04:29.535 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:29 smithi077 ceph-mon[35865]: pgmap v356: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 1.4 MiB/s wr, 61 op/s 2024-03-18T15:04:29.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:29 smithi076 ceph-mon[32205]: pgmap v356: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 1.4 MiB/s wr, 61 op/s 2024-03-18T15:04:29.703 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-18T15:04:29.703 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:04:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 92, 2024-03-18T15:04:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 94, 2024-03-18T15:04:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 5347352, 2024-03-18T15:04:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 29266, 2024-03-18T15:04:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 6, 2024-03-18T15:04:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 2, 2024-03-18T15:04:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.009634946, 2024-03-18T15:04:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.001799365, 2024-03-18T15:04:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.004765219, 2024-03-18T15:04:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.001801544, 2024-03-18T15:04:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 92, 2024-03-18T15:04:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.005765070, 2024-03-18T15:04:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000062663 2024-03-18T15:04:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T15:04:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T15:04:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:04:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 26, 2024-03-18T15:04:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 33, 2024-03-18T15:04:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 9997, 2024-03-18T15:04:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 107192, 2024-03-18T15:04:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 5, 2024-03-18T15:04:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 4, 2024-03-18T15:04:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.003456075, 2024-03-18T15:04:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.000749283, 2024-03-18T15:04:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.002078752, 2024-03-18T15:04:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000496331, 2024-03-18T15:04:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 32, 2024-03-18T15:04:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.003886199, 2024-03-18T15:04:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000121443 2024-03-18T15:04:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-18T15:04:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 100448 2024-03-18T15:04:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:04:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 52, 2024-03-18T15:04:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 51, 2024-03-18T15:04:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 4361201, 2024-03-18T15:04:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 17625, 2024-03-18T15:04:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 3, 2024-03-18T15:04:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 2, 2024-03-18T15:04:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.007654344, 2024-03-18T15:04:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.001049409, 2024-03-18T15:04:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.008103988, 2024-03-18T15:04:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000734894, 2024-03-18T15:04:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 50, 2024-03-18T15:04:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.002619540, 2024-03-18T15:04:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000052390 2024-03-18T15:04:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 159216, 2024-03-18T15:04:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 8720 2024-03-18T15:04:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2024-03-18T15:04:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 1, 2024-03-18T15:04:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2024-03-18T15:04:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "finisher-MDSRank": { 2024-03-18T15:04:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "queue_len": 0, 2024-03-18T15:04:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "complete_latency": { 2024-03-18T15:04:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 224, 2024-03-18T15:04:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.004348238, 2024-03-18T15:04:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000019411 2024-03-18T15:04:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: "finisher-PurgeQueue": { 2024-03-18T15:04:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: "queue_len": 0, 2024-03-18T15:04:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: "complete_latency": { 2024-03-18T15:04:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": { 2024-03-18T15:04:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "request": 0, 2024-03-18T15:04:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "reply": 0, 2024-03-18T15:04:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "reply_latency": { 2024-03-18T15:04:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "slow_reply": 0, 2024-03-18T15:04:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "forward": 0, 2024-03-18T15:04:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_fetch_complete": 0, 2024-03-18T15:04:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_fetch_keys": 0, 2024-03-18T15:04:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_commit": 0, 2024-03-18T15:04:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_split": 0, 2024-03-18T15:04:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_merge": 0, 2024-03-18T15:04:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes": 660, 2024-03-18T15:04:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_top": 0, 2024-03-18T15:04:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_bottom": 660, 2024-03-18T15:04:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_pin_tail": 0, 2024-03-18T15:04:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_pinned": 660, 2024-03-18T15:04:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_expired": 0, 2024-03-18T15:04:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_with_caps": 0, 2024-03-18T15:04:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": 0, 2024-03-18T15:04:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "subtrees": 13, 2024-03-18T15:04:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse": 0, 2024-03-18T15:04:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_hit": 0, 2024-03-18T15:04:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_forward": 0, 2024-03-18T15:04:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_discover": 0, 2024-03-18T15:04:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_dir_fetch": 0, 2024-03-18T15:04:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_remote_ino": 0, 2024-03-18T15:04:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_lock": 0, 2024-03-18T15:04:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "load_cent": 0, 2024-03-18T15:04:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "q": 0, 2024-03-18T15:04:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "exported": 0, 2024-03-18T15:04:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "exported_inodes": 0, 2024-03-18T15:04:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "imported": 0, 2024-03-18T15:04:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "imported_inodes": 0, 2024-03-18T15:04:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_dir_fetch": 0, 2024-03-18T15:04:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_backtrace_fetch": 0, 2024-03-18T15:04:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_peer_discover": 0, 2024-03-18T15:04:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rfiles": 67, 2024-03-18T15:04:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rbytes": 21137952, 2024-03-18T15:04:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rsnaps": 6, 2024-03-18T15:04:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T15:04:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_set_tag": 0, 2024-03-18T15:04:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T15:04:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_inotable_repaired": 0, 2024-03-18T15:04:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dir_inodes": 0, 2024-03-18T15:04:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T15:04:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T15:04:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_file_inodes": 0, 2024-03-18T15:04:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_inode_file_caps": 0, 2024-03-18T15:04:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T15:04:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_grant": 0, 2024-03-18T15:04:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T15:04:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T15:04:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T15:04:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_caps": 0, 2024-03-18T15:04:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_caps_dirty": 0, 2024-03-18T15:04:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_cap_release": 0, 2024-03-18T15:04:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "process_request_cap_release": 0 2024-03-18T15:04:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_cache": { 2024-03-18T15:04:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_update": 0, 2024-03-18T15:04:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_update_receipt": 0, 2024-03-18T15:04:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_try_discover": 0, 2024-03-18T15:04:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_send_discover": 0, 2024-03-18T15:04:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_handle_discover": 0, 2024-03-18T15:04:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays": 0, 2024-03-18T15:04:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays_delayed": 0, 2024-03-18T15:04:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays_enqueuing": 0, 2024-03-18T15:04:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_created": 0, 2024-03-18T15:04:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_enqueued": 0, 2024-03-18T15:04:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_reintegrated": 0, 2024-03-18T15:04:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_migrated": 0, 2024-03-18T15:04:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_processing": 0, 2024-03-18T15:04:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_enqueued": 0, 2024-03-18T15:04:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_prioritized": 0, 2024-03-18T15:04:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "recovery_started": 0, 2024-03-18T15:04:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "recovery_completed": 0, 2024-03-18T15:04:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_quiesce_path": 0, 2024-03-18T15:04:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_quiesce_inode": 0, 2024-03-18T15:04:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T15:04:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_exportdir": 0, 2024-03-18T15:04:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_flush": 0, 2024-03-18T15:04:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_fragmentdir": 0, 2024-03-18T15:04:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_fragstats": 0, 2024-03-18T15:04:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_inodestats": 0 2024-03-18T15:04:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_log": { 2024-03-18T15:04:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "evlrg": 0, 2024-03-18T15:04:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "evadd": 0, 2024-03-18T15:04:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "evex": 0, 2024-03-18T15:04:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "evtrm": 0, 2024-03-18T15:04:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "ev": 1997, 2024-03-18T15:04:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "evexg": 0, 2024-03-18T15:04:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "evexd": 0, 2024-03-18T15:04:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "segadd": 0, 2024-03-18T15:04:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "segex": 0, 2024-03-18T15:04:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "segtrm": 0, 2024-03-18T15:04:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "seg": 3, 2024-03-18T15:04:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "segmjr": 1, 2024-03-18T15:04:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "segexg": 0, 2024-03-18T15:04:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "segexd": 0, 2024-03-18T15:04:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "expos": 469926512, 2024-03-18T15:04:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "wrpos": 479437789, 2024-03-18T15:04:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: "rdpos": 479427734, 2024-03-18T15:04:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: "jlat": { 2024-03-18T15:04:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: "replayed": 1997 2024-03-18T15:04:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_mem": { 2024-03-18T15:04:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "ino": 190, 2024-03-18T15:04:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "ino+": 627, 2024-03-18T15:04:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "ino-": 437, 2024-03-18T15:04:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "dir": 37, 2024-03-18T15:04:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "dir+": 37, 2024-03-18T15:04:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "dir-": 0, 2024-03-18T15:04:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "dn": 660, 2024-03-18T15:04:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "dn+": 660, 2024-03-18T15:04:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "dn-": 0, 2024-03-18T15:04:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "cap": 0, 2024-03-18T15:04:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "cap+": 0, 2024-03-18T15:04:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "cap-": 0, 2024-03-18T15:04:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "rss": 48224, 2024-03-18T15:04:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "heap": 215396 2024-03-18T15:04:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_server": { 2024-03-18T15:04:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "dispatch_client_request": 0, 2024-03-18T15:04:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "dispatch_server_request": 0, 2024-03-18T15:04:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_request": 0, 2024-03-18T15:04:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_session": 0, 2024-03-18T15:04:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_peer_request": 0, 2024-03-18T15:04:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "req_create_latency": { 2024-03-18T15:04:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getattr_latency": { 2024-03-18T15:04:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getfilelock_latency": { 2024-03-18T15:04:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "req_link_latency": { 2024-03-18T15:04:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookup_latency": { 2024-03-18T15:04:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookuphash_latency": { 2024-03-18T15:04:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupino_latency": { 2024-03-18T15:04:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupname_latency": { 2024-03-18T15:04:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupparent_latency": { 2024-03-18T15:04:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupsnap_latency": { 2024-03-18T15:04:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lssnap_latency": { 2024-03-18T15:04:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mkdir_latency": { 2024-03-18T15:04:29.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:29 smithi174 ceph-mon[36787]: pgmap v356: 97 pgs: 97 active+clean; 304 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 1.4 MiB/s wr, 61 op/s 2024-03-18T15:04:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mknod_latency": { 2024-03-18T15:04:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mksnap_latency": { 2024-03-18T15:04:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "req_open_latency": { 2024-03-18T15:04:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "req_readdir_latency": { 2024-03-18T15:04:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rename_latency": { 2024-03-18T15:04:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "req_renamesnap_latency": { 2024-03-18T15:04:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "req_snapdiff_latency": { 2024-03-18T15:04:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmdir_latency": { 2024-03-18T15:04:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmsnap_latency": { 2024-03-18T15:04:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmxattr_latency": { 2024-03-18T15:04:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setattr_latency": { 2024-03-18T15:04:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setdirlayout_latency": { 2024-03-18T15:04:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setfilelock_latency": { 2024-03-18T15:04:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setlayout_latency": { 2024-03-18T15:04:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setxattr_latency": { 2024-03-18T15:04:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "req_symlink_latency": { 2024-03-18T15:04:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "req_unlink_latency": { 2024-03-18T15:04:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "cap_revoke_eviction": 0, 2024-03-18T15:04:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "cap_acquisition_throttle": 0, 2024-03-18T15:04:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getvxattr_latency": { 2024-03-18T15:04:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_sessions": { 2024-03-18T15:04:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "session_count": 2, 2024-03-18T15:04:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "session_add": 2, 2024-03-18T15:04:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "session_remove": 0, 2024-03-18T15:04:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "sessions_open": 0, 2024-03-18T15:04:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "sessions_stale": 0, 2024-03-18T15:04:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "total_load": 0, 2024-03-18T15:04:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "average_load": 0, 2024-03-18T15:04:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "avg_session_uptime": 2354, 2024-03-18T15:04:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "mdthresh_evicted": 0 2024-03-18T15:04:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2024-03-18T15:04:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:04:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2024-03-18T15:04:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:04:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:04:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:04:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:04:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:04:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:04:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:04:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:04:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:04:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:04:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:04:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:04:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:04:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2024-03-18T15:04:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:04:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2024-03-18T15:04:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:04:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:04:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:04:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:04:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:04:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:04:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:04:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2024-03-18T15:04:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:04:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:04:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:04:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2024-03-18T15:04:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2024-03-18T15:04:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2024-03-18T15:04:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:04:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:04:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:04:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 200824, 2024-03-18T15:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 326, 2024-03-18T15:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2024-03-18T15:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2024-03-18T15:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2024-03-18T15:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2024-03-18T15:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2024-03-18T15:04:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 7936, 2024-03-18T15:04:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 165, 2024-03-18T15:04:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:04:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:04:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2024-03-18T15:04:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2024-03-18T15:04:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 1096088, 2024-03-18T15:04:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 24180, 2024-03-18T15:04:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2024-03-18T15:04:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2024-03-18T15:04:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2024-03-18T15:04:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2024-03-18T15:04:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "objecter": { 2024-03-18T15:04:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "op_active": 0, 2024-03-18T15:04:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "op_laggy": 0, 2024-03-18T15:04:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "op_send": 141, 2024-03-18T15:04:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "op_send_bytes": 26, 2024-03-18T15:04:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "op_resend": 4, 2024-03-18T15:04:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "op_reply": 141, 2024-03-18T15:04:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "op_latency": { 2024-03-18T15:04:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 141, 2024-03-18T15:04:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.128997293, 2024-03-18T15:04:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000914874 2024-03-18T15:04:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "op_inflight": 0, 2024-03-18T15:04:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "oplen_avg": { 2024-03-18T15:04:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 141, 2024-03-18T15:04:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 142 2024-03-18T15:04:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "op": 141, 2024-03-18T15:04:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "op_r": 141, 2024-03-18T15:04:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "op_w": 0, 2024-03-18T15:04:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "op_rmw": 0, 2024-03-18T15:04:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "op_pg": 0, 2024-03-18T15:04:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_stat": 75, 2024-03-18T15:04:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_create": 0, 2024-03-18T15:04:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_read": 63, 2024-03-18T15:04:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_write": 0, 2024-03-18T15:04:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_writefull": 0, 2024-03-18T15:04:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_writesame": 0, 2024-03-18T15:04:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_append": 0, 2024-03-18T15:04:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_zero": 0, 2024-03-18T15:04:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_truncate": 0, 2024-03-18T15:04:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_delete": 0, 2024-03-18T15:04:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_mapext": 0, 2024-03-18T15:04:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_sparse_read": 0, 2024-03-18T15:04:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_clonerange": 0, 2024-03-18T15:04:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_getxattr": 2, 2024-03-18T15:04:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_setxattr": 0, 2024-03-18T15:04:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_cmpxattr": 0, 2024-03-18T15:04:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_rmxattr": 0, 2024-03-18T15:04:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_resetxattrs": 0, 2024-03-18T15:04:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_call": 0, 2024-03-18T15:04:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_watch": 0, 2024-03-18T15:04:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_notify": 0, 2024-03-18T15:04:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T15:04:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_pgls": 0, 2024-03-18T15:04:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_pgls_filter": 0, 2024-03-18T15:04:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_other": 0, 2024-03-18T15:04:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_active": 0, 2024-03-18T15:04:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_send": 0, 2024-03-18T15:04:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_resend": 0, 2024-03-18T15:04:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_ping": 0, 2024-03-18T15:04:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_active": 0, 2024-03-18T15:04:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_send": 0, 2024-03-18T15:04:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_resend": 0, 2024-03-18T15:04:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_active": 0, 2024-03-18T15:04:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_send": 0, 2024-03-18T15:04:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_resend": 0, 2024-03-18T15:04:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_active": 0, 2024-03-18T15:04:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_send": 0, 2024-03-18T15:04:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_resend": 0, 2024-03-18T15:04:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "command_active": 0, 2024-03-18T15:04:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "command_send": 0, 2024-03-18T15:04:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "command_resend": 0, 2024-03-18T15:04:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "map_epoch": 107, 2024-03-18T15:04:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "map_full": 0, 2024-03-18T15:04:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "map_inc": 2, 2024-03-18T15:04:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_sessions": 5, 2024-03-18T15:04:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_session_open": 5, 2024-03-18T15:04:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_session_close": 0, 2024-03-18T15:04:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_laggy": 0, 2024-03-18T15:04:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_wr": 0, 2024-03-18T15:04:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_rd": 2, 2024-03-18T15:04:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_del": 0 2024-03-18T15:04:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "oft": { 2024-03-18T15:04:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_objs": 0, 2024-03-18T15:04:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_kv_pairs": 0, 2024-03-18T15:04:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_updates": 0, 2024-03-18T15:04:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_removes": 0 2024-03-18T15:04:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "osd-slow-ops": { 2024-03-18T15:04:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "slow_ops_count": 0 2024-03-18T15:04:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "purge_queue": { 2024-03-18T15:04:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_ops": 0, 2024-03-18T15:04:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T15:04:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing": 0, 2024-03-18T15:04:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_high_water": 0, 2024-03-18T15:04:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executed_ops": 0, 2024-03-18T15:04:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executed": 0, 2024-03-18T15:04:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_item_in_journal": 0 2024-03-18T15:04:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:04:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-18T15:04:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-03-18T15:04:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-18T15:04:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 166, 2024-03-18T15:04:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 9699414, 2024-03-18T15:04:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 166, 2024-03-18T15:04:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-18T15:04:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-18T15:04:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 166, 2024-03-18T15:04:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 9699414, 2024-03-18T15:04:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-18T15:04:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-objecter_bytes": { 2024-03-18T15:04:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-18T15:04:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-03-18T15:04:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-18T15:04:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-03-18T15:04:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-03-18T15:04:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 141, 2024-03-18T15:04:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 9511287, 2024-03-18T15:04:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 141, 2024-03-18T15:04:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 9511287, 2024-03-18T15:04:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-18T15:04:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-objecter_ops": { 2024-03-18T15:04:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-18T15:04:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 1024, 2024-03-18T15:04:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-18T15:04:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-03-18T15:04:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-03-18T15:04:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 141, 2024-03-18T15:04:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 141, 2024-03-18T15:04:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 141, 2024-03-18T15:04:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 141, 2024-03-18T15:04:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-18T15:04:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-write_buf_throttle": { 2024-03-18T15:04:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-18T15:04:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 3758096384, 2024-03-18T15:04:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-18T15:04:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-03-18T15:04:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-03-18T15:04:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-18T15:04:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-18T15:04:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 0, 2024-03-18T15:04:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 0, 2024-03-18T15:04:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-18T15:04:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-write_buf_throttle-0x55f6471366e0": { 2024-03-18T15:04:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-18T15:04:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 3758096384, 2024-03-18T15:04:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-18T15:04:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-03-18T15:04:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-03-18T15:04:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-18T15:04:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-18T15:04:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 0, 2024-03-18T15:04:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 0, 2024-03-18T15:04:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-18T15:04:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:29.776 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-18T15:04:30.660 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-18T15:04:30.660 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-18T15:04:30.660 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-03-18T15:04:30.660 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-18T15:04:30.660 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T15:04:30.660 DEBUG:tasks.check_counter:Getting stats from h 2024-03-18T15:04:30.661 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-18T15:04:30.908 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:04:31.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:31 smithi076 ceph-mon[32205]: pgmap v357: 97 pgs: 97 active+clean; 303 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 1.9 MiB/s wr, 81 op/s 2024-03-18T15:04:31.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:31 smithi174 ceph-mon[36787]: pgmap v357: 97 pgs: 97 active+clean; 303 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 1.9 MiB/s wr, 81 op/s 2024-03-18T15:04:31.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:31 smithi077 ceph-mon[35865]: pgmap v357: 97 pgs: 97 active+clean; 303 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 1.9 MiB/s wr, 81 op/s 2024-03-18T15:04:32.873 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-18T15:04:32.873 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:04:32.873 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 0, 2024-03-18T15:04:32.874 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 2, 2024-03-18T15:04:32.874 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 0, 2024-03-18T15:04:32.874 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 0, 2024-03-18T15:04:32.874 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 4, 2024-03-18T15:04:32.874 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 0, 2024-03-18T15:04:32.874 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.001336918, 2024-03-18T15:04:32.874 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T15:04:32.874 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.000448338, 2024-03-18T15:04:32.874 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T15:04:32.875 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:32.875 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:32.875 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:32.875 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:32.875 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:32.875 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:32.875 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:32.875 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:32.876 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:32.876 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:32.876 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T15:04:32.876 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T15:04:32.876 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:32.876 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:04:32.876 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 236, 2024-03-18T15:04:32.876 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 224, 2024-03-18T15:04:32.876 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 1443376, 2024-03-18T15:04:32.877 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 136832, 2024-03-18T15:04:32.877 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 2, 2024-03-18T15:04:32.877 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 1, 2024-03-18T15:04:32.877 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.031882280, 2024-03-18T15:04:32.877 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.006990228, 2024-03-18T15:04:32.877 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.028906746, 2024-03-18T15:04:32.877 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.003949998, 2024-03-18T15:04:32.877 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:32.877 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 224, 2024-03-18T15:04:32.878 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.006844561, 2024-03-18T15:04:32.878 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000030556 2024-03-18T15:04:32.878 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:32.878 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:32.878 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:32.878 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:32.878 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:32.878 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:32.878 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 1443376, 2024-03-18T15:04:32.879 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 136832 2024-03-18T15:04:32.879 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:32.879 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:04:32.879 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 2, 2024-03-18T15:04:32.879 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 169, 2024-03-18T15:04:32.879 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 320, 2024-03-18T15:04:32.879 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 185216, 2024-03-18T15:04:32.879 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 5, 2024-03-18T15:04:32.879 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 3, 2024-03-18T15:04:32.879 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.007670235, 2024-03-18T15:04:32.880 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.004620861, 2024-03-18T15:04:32.880 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.001535754, 2024-03-18T15:04:32.880 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T15:04:32.880 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:32.880 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 166, 2024-03-18T15:04:32.880 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.011166802, 2024-03-18T15:04:32.880 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000067269 2024-03-18T15:04:32.880 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:32.881 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:32.881 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:32.881 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:32.881 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:32.881 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:32.881 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T15:04:32.881 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 185216 2024-03-18T15:04:32.881 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:32.881 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2024-03-18T15:04:32.882 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 0, 2024-03-18T15:04:32.882 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2024-03-18T15:04:32.882 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:32.882 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2024-03-18T15:04:32.882 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:04:32.882 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2024-03-18T15:04:32.882 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:04:32.882 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:04:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:04:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:04:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:04:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:04:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:04:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:04:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:04:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:04:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:04:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:04:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:04:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2024-03-18T15:04:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:04:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2024-03-18T15:04:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:04:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:04:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:04:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:04:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:04:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:04:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:04:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2024-03-18T15:04:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:04:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:04:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:04:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2024-03-18T15:04:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2024-03-18T15:04:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2024-03-18T15:04:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:04:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:04:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:04:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:04:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 150008, 2024-03-18T15:04:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 31, 2024-03-18T15:04:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:04:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2024-03-18T15:04:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2024-03-18T15:04:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2024-03-18T15:04:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:04:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2024-03-18T15:04:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:04:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2024-03-18T15:04:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 0, 2024-03-18T15:04:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 0, 2024-03-18T15:04:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:04:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:04:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2024-03-18T15:04:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2024-03-18T15:04:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 928, 2024-03-18T15:04:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 2, 2024-03-18T15:04:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2024-03-18T15:04:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2024-03-18T15:04:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2024-03-18T15:04:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2024-03-18T15:04:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:04:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-18T15:04:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-03-18T15:04:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-18T15:04:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 234, 2024-03-18T15:04:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 1407670, 2024-03-18T15:04:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 234, 2024-03-18T15:04:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-18T15:04:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-18T15:04:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 234, 2024-03-18T15:04:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 1407670, 2024-03-18T15:04:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-18T15:04:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:32.891 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:32.891 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-18T15:04:33.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:33 smithi077 ceph-mon[35865]: pgmap v358: 97 pgs: 97 active+clean; 303 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 1.2 MiB/s wr, 52 op/s 2024-03-18T15:04:33.635 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.h 2024-03-18T15:04:33.635 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.h 2024-03-18T15:04:33.635 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.h 2024-03-18T15:04:33.636 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.h 2024-03-18T15:04:33.636 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.h 2024-03-18T15:04:33.636 DEBUG:tasks.check_counter:Getting stats from k 2024-03-18T15:04:33.636 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-18T15:04:33.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:33 smithi076 ceph-mon[32205]: pgmap v358: 97 pgs: 97 active+clean; 303 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 1.2 MiB/s wr, 52 op/s 2024-03-18T15:04:33.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:33 smithi174 ceph-mon[36787]: pgmap v358: 97 pgs: 97 active+clean; 303 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 1.2 MiB/s wr, 52 op/s 2024-03-18T15:04:33.882 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:04:35.586 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:35 smithi077 ceph-mon[35865]: pgmap v359: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-03-18T15:04:35.587 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-18T15:04:35.587 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:04:35.587 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 667, 2024-03-18T15:04:35.587 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 669, 2024-03-18T15:04:35.587 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 42114593, 2024-03-18T15:04:35.587 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 212170, 2024-03-18T15:04:35.587 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 8, 2024-03-18T15:04:35.588 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 4, 2024-03-18T15:04:35.588 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.063949638, 2024-03-18T15:04:35.588 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.012383659, 2024-03-18T15:04:35.588 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.034788049, 2024-03-18T15:04:35.588 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.012480764, 2024-03-18T15:04:35.588 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:35.588 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 667, 2024-03-18T15:04:35.588 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.015469819, 2024-03-18T15:04:35.589 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000023193 2024-03-18T15:04:35.589 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.589 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:35.589 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.589 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.589 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.589 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.589 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T15:04:35.589 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T15:04:35.590 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.590 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:04:35.590 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 1031, 2024-03-18T15:04:35.590 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 998, 2024-03-18T15:04:35.590 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 56274103, 2024-03-18T15:04:35.590 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 380839, 2024-03-18T15:04:35.590 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 5, 2024-03-18T15:04:35.591 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 4, 2024-03-18T15:04:35.591 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.110645427, 2024-03-18T15:04:35.591 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.021580231, 2024-03-18T15:04:35.591 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.076574310, 2024-03-18T15:04:35.591 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.018718272, 2024-03-18T15:04:35.591 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:35.591 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 998, 2024-03-18T15:04:35.591 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.023378241, 2024-03-18T15:04:35.592 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000023425 2024-03-18T15:04:35.592 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.592 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:35.592 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.592 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.592 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.592 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.592 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 1557312, 2024-03-18T15:04:35.592 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 144560 2024-03-18T15:04:35.593 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.593 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:04:35.593 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 1520, 2024-03-18T15:04:35.593 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 1692, 2024-03-18T15:04:35.593 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 63910555, 2024-03-18T15:04:35.593 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 947556, 2024-03-18T15:04:35.593 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 8, 2024-03-18T15:04:35.593 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 6, 2024-03-18T15:04:35.594 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.134940775, 2024-03-18T15:04:35.594 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.031451201, 2024-03-18T15:04:35.594 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.063042856, 2024-03-18T15:04:35.594 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.030881621, 2024-03-18T15:04:35.594 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:35.594 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 1689, 2024-03-18T15:04:35.594 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.042350181, 2024-03-18T15:04:35.594 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000025074 2024-03-18T15:04:35.594 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.595 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:35.595 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.595 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.595 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.595 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.595 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T15:04:35.595 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 464832 2024-03-18T15:04:35.595 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.595 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2024-03-18T15:04:35.596 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 1, 2024-03-18T15:04:35.596 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2024-03-18T15:04:35.596 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.596 INFO:teuthology.orchestra.run.smithi077.stdout: "finisher-MDSRank": { 2024-03-18T15:04:35.596 INFO:teuthology.orchestra.run.smithi077.stdout: "queue_len": 0, 2024-03-18T15:04:35.596 INFO:teuthology.orchestra.run.smithi077.stdout: "complete_latency": { 2024-03-18T15:04:35.596 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 5061, 2024-03-18T15:04:35.596 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.087713669, 2024-03-18T15:04:35.596 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000017331 2024-03-18T15:04:35.597 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:35.597 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.597 INFO:teuthology.orchestra.run.smithi077.stdout: "finisher-PurgeQueue": { 2024-03-18T15:04:35.597 INFO:teuthology.orchestra.run.smithi077.stdout: "queue_len": 0, 2024-03-18T15:04:35.597 INFO:teuthology.orchestra.run.smithi077.stdout: "complete_latency": { 2024-03-18T15:04:35.597 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.597 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.597 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.597 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:35.598 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.598 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": { 2024-03-18T15:04:35.598 INFO:teuthology.orchestra.run.smithi077.stdout: "request": 0, 2024-03-18T15:04:35.598 INFO:teuthology.orchestra.run.smithi077.stdout: "reply": 0, 2024-03-18T15:04:35.598 INFO:teuthology.orchestra.run.smithi077.stdout: "reply_latency": { 2024-03-18T15:04:35.598 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.598 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.598 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.599 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.599 INFO:teuthology.orchestra.run.smithi077.stdout: "slow_reply": 0, 2024-03-18T15:04:35.599 INFO:teuthology.orchestra.run.smithi077.stdout: "forward": 0, 2024-03-18T15:04:35.599 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_fetch_complete": 0, 2024-03-18T15:04:35.599 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_fetch_keys": 0, 2024-03-18T15:04:35.599 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_commit": 0, 2024-03-18T15:04:35.599 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_split": 0, 2024-03-18T15:04:35.599 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_merge": 0, 2024-03-18T15:04:35.599 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes": 17521, 2024-03-18T15:04:35.600 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_top": 24, 2024-03-18T15:04:35.600 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_bottom": 17497, 2024-03-18T15:04:35.600 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_pin_tail": 0, 2024-03-18T15:04:35.600 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_pinned": 17487, 2024-03-18T15:04:35.600 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_expired": 0, 2024-03-18T15:04:35.600 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_with_caps": 0, 2024-03-18T15:04:35.600 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": 0, 2024-03-18T15:04:35.600 INFO:teuthology.orchestra.run.smithi077.stdout: "subtrees": 5, 2024-03-18T15:04:35.601 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse": 0, 2024-03-18T15:04:35.601 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_hit": 0, 2024-03-18T15:04:35.601 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_forward": 0, 2024-03-18T15:04:35.601 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_discover": 0, 2024-03-18T15:04:35.601 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_dir_fetch": 0, 2024-03-18T15:04:35.601 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_remote_ino": 0, 2024-03-18T15:04:35.601 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_lock": 0, 2024-03-18T15:04:35.601 INFO:teuthology.orchestra.run.smithi077.stdout: "load_cent": 0, 2024-03-18T15:04:35.601 INFO:teuthology.orchestra.run.smithi077.stdout: "q": 0, 2024-03-18T15:04:35.602 INFO:teuthology.orchestra.run.smithi077.stdout: "exported": 0, 2024-03-18T15:04:35.602 INFO:teuthology.orchestra.run.smithi077.stdout: "exported_inodes": 0, 2024-03-18T15:04:35.602 INFO:teuthology.orchestra.run.smithi077.stdout: "imported": 0, 2024-03-18T15:04:35.602 INFO:teuthology.orchestra.run.smithi077.stdout: "imported_inodes": 0, 2024-03-18T15:04:35.602 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_dir_fetch": 0, 2024-03-18T15:04:35.602 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_backtrace_fetch": 0, 2024-03-18T15:04:35.602 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_peer_discover": 0, 2024-03-18T15:04:35.603 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rfiles": 0, 2024-03-18T15:04:35.603 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rbytes": 0, 2024-03-18T15:04:35.603 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rsnaps": 0, 2024-03-18T15:04:35.603 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T15:04:35.603 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_set_tag": 0, 2024-03-18T15:04:35.603 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T15:04:35.603 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_inotable_repaired": 0, 2024-03-18T15:04:35.603 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dir_inodes": 0, 2024-03-18T15:04:35.604 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T15:04:35.604 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T15:04:35.604 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_file_inodes": 0, 2024-03-18T15:04:35.604 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_inode_file_caps": 0, 2024-03-18T15:04:35.604 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T15:04:35.604 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_grant": 0, 2024-03-18T15:04:35.604 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T15:04:35.604 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T15:04:35.604 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T15:04:35.605 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_caps": 0, 2024-03-18T15:04:35.605 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_caps_dirty": 0, 2024-03-18T15:04:35.605 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_cap_release": 0, 2024-03-18T15:04:35.605 INFO:teuthology.orchestra.run.smithi077.stdout: "process_request_cap_release": 0 2024-03-18T15:04:35.605 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.605 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_cache": { 2024-03-18T15:04:35.605 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_update": 0, 2024-03-18T15:04:35.605 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_update_receipt": 0, 2024-03-18T15:04:35.605 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_try_discover": 0, 2024-03-18T15:04:35.606 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_send_discover": 0, 2024-03-18T15:04:35.606 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_handle_discover": 0, 2024-03-18T15:04:35.606 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays": 0, 2024-03-18T15:04:35.606 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays_delayed": 0, 2024-03-18T15:04:35.606 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays_enqueuing": 0, 2024-03-18T15:04:35.606 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_created": 0, 2024-03-18T15:04:35.606 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_enqueued": 0, 2024-03-18T15:04:35.606 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_reintegrated": 0, 2024-03-18T15:04:35.607 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_migrated": 0, 2024-03-18T15:04:35.607 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_processing": 0, 2024-03-18T15:04:35.607 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_enqueued": 0, 2024-03-18T15:04:35.607 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_prioritized": 0, 2024-03-18T15:04:35.607 INFO:teuthology.orchestra.run.smithi077.stdout: "recovery_started": 0, 2024-03-18T15:04:35.607 INFO:teuthology.orchestra.run.smithi077.stdout: "recovery_completed": 0, 2024-03-18T15:04:35.607 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_quiesce_path": 0, 2024-03-18T15:04:35.607 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_quiesce_inode": 0, 2024-03-18T15:04:35.607 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T15:04:35.608 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_exportdir": 0, 2024-03-18T15:04:35.608 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_flush": 0, 2024-03-18T15:04:35.608 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_fragmentdir": 0, 2024-03-18T15:04:35.608 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_fragstats": 0, 2024-03-18T15:04:35.608 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_inodestats": 0 2024-03-18T15:04:35.608 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.608 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_log": { 2024-03-18T15:04:35.608 INFO:teuthology.orchestra.run.smithi077.stdout: "evlrg": 0, 2024-03-18T15:04:35.609 INFO:teuthology.orchestra.run.smithi077.stdout: "evadd": 0, 2024-03-18T15:04:35.609 INFO:teuthology.orchestra.run.smithi077.stdout: "evex": 0, 2024-03-18T15:04:35.609 INFO:teuthology.orchestra.run.smithi077.stdout: "evtrm": 0, 2024-03-18T15:04:35.609 INFO:teuthology.orchestra.run.smithi077.stdout: "ev": 88385, 2024-03-18T15:04:35.609 INFO:teuthology.orchestra.run.smithi077.stdout: "evexg": 0, 2024-03-18T15:04:35.609 INFO:teuthology.orchestra.run.smithi077.stdout: "evexd": 0, 2024-03-18T15:04:35.609 INFO:teuthology.orchestra.run.smithi077.stdout: "segadd": 0, 2024-03-18T15:04:35.609 INFO:teuthology.orchestra.run.smithi077.stdout: "segex": 0, 2024-03-18T15:04:35.609 INFO:teuthology.orchestra.run.smithi077.stdout: "segtrm": 0, 2024-03-18T15:04:35.610 INFO:teuthology.orchestra.run.smithi077.stdout: "seg": 111, 2024-03-18T15:04:35.610 INFO:teuthology.orchestra.run.smithi077.stdout: "segmjr": 9, 2024-03-18T15:04:35.610 INFO:teuthology.orchestra.run.smithi077.stdout: "segexg": 0, 2024-03-18T15:04:35.610 INFO:teuthology.orchestra.run.smithi077.stdout: "segexd": 0, 2024-03-18T15:04:35.610 INFO:teuthology.orchestra.run.smithi077.stdout: "expos": 4194304, 2024-03-18T15:04:35.610 INFO:teuthology.orchestra.run.smithi077.stdout: "wrpos": 164119580, 2024-03-18T15:04:35.610 INFO:teuthology.orchestra.run.smithi077.stdout: "rdpos": 164112289, 2024-03-18T15:04:35.610 INFO:teuthology.orchestra.run.smithi077.stdout: "jlat": { 2024-03-18T15:04:35.610 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.611 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.611 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.611 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.611 INFO:teuthology.orchestra.run.smithi077.stdout: "replayed": 88385 2024-03-18T15:04:35.611 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.611 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_mem": { 2024-03-18T15:04:35.611 INFO:teuthology.orchestra.run.smithi077.stdout: "ino": 234, 2024-03-18T15:04:35.611 INFO:teuthology.orchestra.run.smithi077.stdout: "ino+": 17464, 2024-03-18T15:04:35.612 INFO:teuthology.orchestra.run.smithi077.stdout: "ino-": 17230, 2024-03-18T15:04:35.612 INFO:teuthology.orchestra.run.smithi077.stdout: "dir": 24, 2024-03-18T15:04:35.612 INFO:teuthology.orchestra.run.smithi077.stdout: "dir+": 24, 2024-03-18T15:04:35.612 INFO:teuthology.orchestra.run.smithi077.stdout: "dir-": 0, 2024-03-18T15:04:35.612 INFO:teuthology.orchestra.run.smithi077.stdout: "dn": 17521, 2024-03-18T15:04:35.612 INFO:teuthology.orchestra.run.smithi077.stdout: "dn+": 17521, 2024-03-18T15:04:35.612 INFO:teuthology.orchestra.run.smithi077.stdout: "dn-": 0, 2024-03-18T15:04:35.612 INFO:teuthology.orchestra.run.smithi077.stdout: "cap": 0, 2024-03-18T15:04:35.612 INFO:teuthology.orchestra.run.smithi077.stdout: "cap+": 0, 2024-03-18T15:04:35.613 INFO:teuthology.orchestra.run.smithi077.stdout: "cap-": 0, 2024-03-18T15:04:35.613 INFO:teuthology.orchestra.run.smithi077.stdout: "rss": 65492, 2024-03-18T15:04:35.613 INFO:teuthology.orchestra.run.smithi077.stdout: "heap": 215420 2024-03-18T15:04:35.613 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.613 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_server": { 2024-03-18T15:04:35.613 INFO:teuthology.orchestra.run.smithi077.stdout: "dispatch_client_request": 0, 2024-03-18T15:04:35.613 INFO:teuthology.orchestra.run.smithi077.stdout: "dispatch_server_request": 0, 2024-03-18T15:04:35.613 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_request": 0, 2024-03-18T15:04:35.614 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_session": 0, 2024-03-18T15:04:35.614 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_peer_request": 0, 2024-03-18T15:04:35.614 INFO:teuthology.orchestra.run.smithi077.stdout: "req_create_latency": { 2024-03-18T15:04:35.614 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.614 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.614 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.614 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.614 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getattr_latency": { 2024-03-18T15:04:35.614 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.615 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.615 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.615 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.615 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getfilelock_latency": { 2024-03-18T15:04:35.615 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.615 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.615 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.615 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.616 INFO:teuthology.orchestra.run.smithi077.stdout: "req_link_latency": { 2024-03-18T15:04:35.616 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.616 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.616 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.616 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.616 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookup_latency": { 2024-03-18T15:04:35.616 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.616 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.616 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.617 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.617 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookuphash_latency": { 2024-03-18T15:04:35.617 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.617 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.617 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.617 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.617 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupino_latency": { 2024-03-18T15:04:35.617 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.617 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.618 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.618 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.618 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupname_latency": { 2024-03-18T15:04:35.618 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.618 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.618 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.618 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.618 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupparent_latency": { 2024-03-18T15:04:35.618 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.619 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.619 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.619 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.619 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupsnap_latency": { 2024-03-18T15:04:35.619 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.619 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.619 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.619 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.619 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lssnap_latency": { 2024-03-18T15:04:35.620 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.620 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.620 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.620 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.620 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mkdir_latency": { 2024-03-18T15:04:35.621 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.621 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.621 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.621 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.621 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mknod_latency": { 2024-03-18T15:04:35.621 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.621 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.622 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.622 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.622 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mksnap_latency": { 2024-03-18T15:04:35.622 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.622 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.622 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.622 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.622 INFO:teuthology.orchestra.run.smithi077.stdout: "req_open_latency": { 2024-03-18T15:04:35.622 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.623 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.623 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.623 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.623 INFO:teuthology.orchestra.run.smithi077.stdout: "req_readdir_latency": { 2024-03-18T15:04:35.623 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.623 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.623 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.623 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.623 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rename_latency": { 2024-03-18T15:04:35.624 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.624 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.624 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.624 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.624 INFO:teuthology.orchestra.run.smithi077.stdout: "req_renamesnap_latency": { 2024-03-18T15:04:35.624 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.624 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.624 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.624 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.625 INFO:teuthology.orchestra.run.smithi077.stdout: "req_snapdiff_latency": { 2024-03-18T15:04:35.625 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.625 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.625 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.625 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.625 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmdir_latency": { 2024-03-18T15:04:35.625 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.625 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.625 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.626 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.626 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmsnap_latency": { 2024-03-18T15:04:35.626 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.626 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.626 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.626 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.626 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmxattr_latency": { 2024-03-18T15:04:35.626 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.627 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.627 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.627 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.627 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setattr_latency": { 2024-03-18T15:04:35.627 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.627 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.627 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.627 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.627 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setdirlayout_latency": { 2024-03-18T15:04:35.628 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.628 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.628 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.628 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.628 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setfilelock_latency": { 2024-03-18T15:04:35.628 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.628 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.628 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.629 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.629 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setlayout_latency": { 2024-03-18T15:04:35.629 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.629 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.629 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.629 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.629 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setxattr_latency": { 2024-03-18T15:04:35.629 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.629 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.630 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.630 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.630 INFO:teuthology.orchestra.run.smithi077.stdout: "req_symlink_latency": { 2024-03-18T15:04:35.630 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.630 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.630 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.630 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.630 INFO:teuthology.orchestra.run.smithi077.stdout: "req_unlink_latency": { 2024-03-18T15:04:35.631 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.631 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.631 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.631 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.631 INFO:teuthology.orchestra.run.smithi077.stdout: "cap_revoke_eviction": 0, 2024-03-18T15:04:35.631 INFO:teuthology.orchestra.run.smithi077.stdout: "cap_acquisition_throttle": 0, 2024-03-18T15:04:35.631 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getvxattr_latency": { 2024-03-18T15:04:35.631 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.631 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.632 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.632 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:35.632 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.632 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_sessions": { 2024-03-18T15:04:35.632 INFO:teuthology.orchestra.run.smithi077.stdout: "session_count": 1, 2024-03-18T15:04:35.632 INFO:teuthology.orchestra.run.smithi077.stdout: "session_add": 4, 2024-03-18T15:04:35.632 INFO:teuthology.orchestra.run.smithi077.stdout: "session_remove": 3, 2024-03-18T15:04:35.632 INFO:teuthology.orchestra.run.smithi077.stdout: "sessions_open": 3, 2024-03-18T15:04:35.633 INFO:teuthology.orchestra.run.smithi077.stdout: "sessions_stale": 0, 2024-03-18T15:04:35.633 INFO:teuthology.orchestra.run.smithi077.stdout: "total_load": 0, 2024-03-18T15:04:35.633 INFO:teuthology.orchestra.run.smithi077.stdout: "average_load": 0, 2024-03-18T15:04:35.633 INFO:teuthology.orchestra.run.smithi077.stdout: "avg_session_uptime": 5726, 2024-03-18T15:04:35.633 INFO:teuthology.orchestra.run.smithi077.stdout: "mdthresh_evicted": 0 2024-03-18T15:04:35.633 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.633 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2024-03-18T15:04:35.633 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:04:35.633 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2024-03-18T15:04:35.634 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:04:35.634 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:04:35.634 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:04:35.634 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:04:35.634 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:04:35.634 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:04:35.634 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:04:35.634 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:04:35.635 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:04:35.635 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:04:35.635 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:04:35.635 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:04:35.635 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:04:35.635 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2024-03-18T15:04:35.635 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:04:35.635 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2024-03-18T15:04:35.635 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:04:35.636 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:04:35.636 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:04:35.636 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:04:35.636 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:04:35.636 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:04:35.636 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:04:35.636 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2024-03-18T15:04:35.636 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:04:35.636 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:04:35.636 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:04:35.637 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2024-03-18T15:04:35.637 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2024-03-18T15:04:35.637 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2024-03-18T15:04:35.637 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:04:35.637 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:04:35.637 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:04:35.637 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:04:35.637 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 239766, 2024-03-18T15:04:35.637 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 302, 2024-03-18T15:04:35.638 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:04:35.638 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2024-03-18T15:04:35.638 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2024-03-18T15:04:35.638 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2024-03-18T15:04:35.638 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:04:35.638 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2024-03-18T15:04:35.638 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:04:35.638 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2024-03-18T15:04:35.639 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 7936, 2024-03-18T15:04:35.639 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 165, 2024-03-18T15:04:35.639 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:04:35.639 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:04:35.639 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2024-03-18T15:04:35.639 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2024-03-18T15:04:35.639 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 13001970, 2024-03-18T15:04:35.639 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 79347, 2024-03-18T15:04:35.639 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2024-03-18T15:04:35.640 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2024-03-18T15:04:35.640 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2024-03-18T15:04:35.640 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2024-03-18T15:04:35.640 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.640 INFO:teuthology.orchestra.run.smithi077.stdout: "objecter": { 2024-03-18T15:04:35.640 INFO:teuthology.orchestra.run.smithi077.stdout: "op_active": 0, 2024-03-18T15:04:35.640 INFO:teuthology.orchestra.run.smithi077.stdout: "op_laggy": 0, 2024-03-18T15:04:35.640 INFO:teuthology.orchestra.run.smithi077.stdout: "op_send": 2928, 2024-03-18T15:04:35.640 INFO:teuthology.orchestra.run.smithi077.stdout: "op_send_bytes": 21, 2024-03-18T15:04:35.641 INFO:teuthology.orchestra.run.smithi077.stdout: "op_resend": 3, 2024-03-18T15:04:35.641 INFO:teuthology.orchestra.run.smithi077.stdout: "op_reply": 2928, 2024-03-18T15:04:35.641 INFO:teuthology.orchestra.run.smithi077.stdout: "op_latency": { 2024-03-18T15:04:35.641 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 2928, 2024-03-18T15:04:35.641 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 2.134953624, 2024-03-18T15:04:35.641 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000729150 2024-03-18T15:04:35.641 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.641 INFO:teuthology.orchestra.run.smithi077.stdout: "op_inflight": 0, 2024-03-18T15:04:35.642 INFO:teuthology.orchestra.run.smithi077.stdout: "oplen_avg": { 2024-03-18T15:04:35.642 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 2928, 2024-03-18T15:04:35.642 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 2929 2024-03-18T15:04:35.642 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.642 INFO:teuthology.orchestra.run.smithi077.stdout: "op": 2928, 2024-03-18T15:04:35.642 INFO:teuthology.orchestra.run.smithi077.stdout: "op_r": 2928, 2024-03-18T15:04:35.642 INFO:teuthology.orchestra.run.smithi077.stdout: "op_w": 0, 2024-03-18T15:04:35.642 INFO:teuthology.orchestra.run.smithi077.stdout: "op_rmw": 0, 2024-03-18T15:04:35.642 INFO:teuthology.orchestra.run.smithi077.stdout: "op_pg": 0, 2024-03-18T15:04:35.643 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_stat": 1514, 2024-03-18T15:04:35.643 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_create": 0, 2024-03-18T15:04:35.643 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_read": 1412, 2024-03-18T15:04:35.643 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_write": 0, 2024-03-18T15:04:35.643 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_writefull": 0, 2024-03-18T15:04:35.643 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_writesame": 0, 2024-03-18T15:04:35.643 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_append": 0, 2024-03-18T15:04:35.643 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_zero": 0, 2024-03-18T15:04:35.643 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_truncate": 0, 2024-03-18T15:04:35.644 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_delete": 0, 2024-03-18T15:04:35.644 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_mapext": 0, 2024-03-18T15:04:35.644 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_sparse_read": 0, 2024-03-18T15:04:35.644 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_clonerange": 0, 2024-03-18T15:04:35.644 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_getxattr": 1, 2024-03-18T15:04:35.644 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_setxattr": 0, 2024-03-18T15:04:35.644 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_cmpxattr": 0, 2024-03-18T15:04:35.644 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_rmxattr": 0, 2024-03-18T15:04:35.644 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_resetxattrs": 0, 2024-03-18T15:04:35.645 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_call": 0, 2024-03-18T15:04:35.645 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_watch": 0, 2024-03-18T15:04:35.645 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_notify": 0, 2024-03-18T15:04:35.645 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T15:04:35.645 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_pgls": 0, 2024-03-18T15:04:35.645 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_pgls_filter": 0, 2024-03-18T15:04:35.645 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_other": 0, 2024-03-18T15:04:35.645 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_active": 0, 2024-03-18T15:04:35.646 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_send": 0, 2024-03-18T15:04:35.646 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_resend": 0, 2024-03-18T15:04:35.646 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_ping": 0, 2024-03-18T15:04:35.646 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_active": 0, 2024-03-18T15:04:35.646 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_send": 0, 2024-03-18T15:04:35.646 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_resend": 0, 2024-03-18T15:04:35.646 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_active": 0, 2024-03-18T15:04:35.646 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_send": 0, 2024-03-18T15:04:35.646 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_resend": 0, 2024-03-18T15:04:35.647 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_active": 0, 2024-03-18T15:04:35.647 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_send": 0, 2024-03-18T15:04:35.647 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_resend": 0, 2024-03-18T15:04:35.647 INFO:teuthology.orchestra.run.smithi077.stdout: "command_active": 0, 2024-03-18T15:04:35.647 INFO:teuthology.orchestra.run.smithi077.stdout: "command_send": 0, 2024-03-18T15:04:35.647 INFO:teuthology.orchestra.run.smithi077.stdout: "command_resend": 0, 2024-03-18T15:04:35.647 INFO:teuthology.orchestra.run.smithi077.stdout: "map_epoch": 107, 2024-03-18T15:04:35.647 INFO:teuthology.orchestra.run.smithi077.stdout: "map_full": 0, 2024-03-18T15:04:35.647 INFO:teuthology.orchestra.run.smithi077.stdout: "map_inc": 21, 2024-03-18T15:04:35.648 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_sessions": 10, 2024-03-18T15:04:35.648 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_session_open": 10, 2024-03-18T15:04:35.648 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_session_close": 0, 2024-03-18T15:04:35.648 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_laggy": 0, 2024-03-18T15:04:35.648 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_wr": 0, 2024-03-18T15:04:35.648 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_rd": 2, 2024-03-18T15:04:35.648 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_del": 0 2024-03-18T15:04:35.648 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.648 INFO:teuthology.orchestra.run.smithi077.stdout: "oft": { 2024-03-18T15:04:35.649 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_objs": 0, 2024-03-18T15:04:35.649 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_kv_pairs": 0, 2024-03-18T15:04:35.649 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_updates": 0, 2024-03-18T15:04:35.649 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_removes": 0 2024-03-18T15:04:35.649 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.649 INFO:teuthology.orchestra.run.smithi077.stdout: "osd-slow-ops": { 2024-03-18T15:04:35.649 INFO:teuthology.orchestra.run.smithi077.stdout: "slow_ops_count": 0 2024-03-18T15:04:35.649 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.649 INFO:teuthology.orchestra.run.smithi077.stdout: "purge_queue": { 2024-03-18T15:04:35.650 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_ops": 0, 2024-03-18T15:04:35.650 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T15:04:35.650 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing": 0, 2024-03-18T15:04:35.650 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_high_water": 0, 2024-03-18T15:04:35.650 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executed_ops": 0, 2024-03-18T15:04:35.650 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executed": 0, 2024-03-18T15:04:35.650 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_item_in_journal": 0 2024-03-18T15:04:35.650 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.651 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:04:35.651 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-18T15:04:35.651 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-03-18T15:04:35.651 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-18T15:04:35.651 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 3214, 2024-03-18T15:04:35.651 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 161992614, 2024-03-18T15:04:35.651 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:35.651 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 3214, 2024-03-18T15:04:35.651 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-18T15:04:35.652 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-18T15:04:35.652 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 3214, 2024-03-18T15:04:35.652 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 161992614, 2024-03-18T15:04:35.652 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-18T15:04:35.652 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.652 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.652 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.652 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:35.652 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.652 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-objecter_bytes": { 2024-03-18T15:04:35.653 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-18T15:04:35.653 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-03-18T15:04:35.653 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-18T15:04:35.653 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-03-18T15:04:35.653 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-03-18T15:04:35.653 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:35.653 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:35.653 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 2928, 2024-03-18T15:04:35.654 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 159931491, 2024-03-18T15:04:35.654 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 2928, 2024-03-18T15:04:35.654 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 159931491, 2024-03-18T15:04:35.654 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-18T15:04:35.654 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.654 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.654 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.654 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:35.654 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.655 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-objecter_ops": { 2024-03-18T15:04:35.655 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-18T15:04:35.655 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 1024, 2024-03-18T15:04:35.655 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-18T15:04:35.655 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-03-18T15:04:35.655 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-03-18T15:04:35.655 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:35.655 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:35.655 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 2928, 2024-03-18T15:04:35.656 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 2928, 2024-03-18T15:04:35.656 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 2928, 2024-03-18T15:04:35.656 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 2928, 2024-03-18T15:04:35.656 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-18T15:04:35.656 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.656 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.656 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.656 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:35.656 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.657 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-write_buf_throttle": { 2024-03-18T15:04:35.657 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-18T15:04:35.657 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 3758096384, 2024-03-18T15:04:35.657 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-18T15:04:35.657 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-03-18T15:04:35.657 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-03-18T15:04:35.657 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:35.657 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:35.657 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-18T15:04:35.657 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-18T15:04:35.658 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 0, 2024-03-18T15:04:35.658 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 0, 2024-03-18T15:04:35.658 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-18T15:04:35.658 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.658 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.658 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.658 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:35.658 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-18T15:04:35.659 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-write_buf_throttle-0x562eb203c6e0": { 2024-03-18T15:04:35.659 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-18T15:04:35.659 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 3758096384, 2024-03-18T15:04:35.659 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-18T15:04:35.659 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-03-18T15:04:35.659 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-03-18T15:04:35.659 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:35.659 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:35.659 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-18T15:04:35.660 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-18T15:04:35.660 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 0, 2024-03-18T15:04:35.660 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 0, 2024-03-18T15:04:35.660 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-18T15:04:35.660 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-18T15:04:35.660 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-18T15:04:35.660 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-18T15:04:35.660 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:35.660 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-18T15:04:35.661 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-18T15:04:35.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:35 smithi076 ceph-mon[32205]: pgmap v359: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-03-18T15:04:35.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:35 smithi174 ceph-mon[36787]: pgmap v359: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-03-18T15:04:36.343 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:36 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:04:36.343 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:36 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:04:36.343 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:36 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:04:36.505 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-18T15:04:36.505 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-18T15:04:36.505 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-03-18T15:04:36.506 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-03-18T15:04:36.506 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T15:04:36.506 DEBUG:tasks.check_counter:Getting stats from c 2024-03-18T15:04:36.509 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-18T15:04:36.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:36 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:04:36.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:36 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:04:36.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:36 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:04:36.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:36 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:04:36.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:36 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:04:36.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:36 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:04:36.828 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:04:37.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:37 smithi076 ceph-mon[32205]: pgmap v360: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-03-18T15:04:37.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:37 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:04:37.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:37 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:04:37.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:37 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:04:37.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:37 smithi174 ceph-mon[36787]: pgmap v360: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-03-18T15:04:37.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:37 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:04:37.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:37 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:04:37.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:37 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:04:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:37 smithi077 ceph-mon[35865]: pgmap v360: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-03-18T15:04:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:37 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:04:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:37 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:04:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:37 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' 2024-03-18T15:04:38.739 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-18T15:04:38.739 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:04:38.739 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 129593, 2024-03-18T15:04:38.739 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 216735, 2024-03-18T15:04:38.739 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 37383952, 2024-03-18T15:04:38.739 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 201012486, 2024-03-18T15:04:38.740 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 14, 2024-03-18T15:04:38.740 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 4, 2024-03-18T15:04:38.740 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 6.378656254, 2024-03-18T15:04:38.740 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 3.194467196, 2024-03-18T15:04:38.740 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 3.449829067, 2024-03-18T15:04:38.740 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.351482796, 2024-03-18T15:04:38.740 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:38.740 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 216733, 2024-03-18T15:04:38.740 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 4.138439589, 2024-03-18T15:04:38.741 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000019094 2024-03-18T15:04:38.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.741 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:38.741 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 187093, 2024-03-18T15:04:38.741 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.033574662, 2024-03-18T15:04:38.741 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000179 2024-03-18T15:04:38.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.741 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1744, 2024-03-18T15:04:38.741 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 334464 2024-03-18T15:04:38.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.742 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:04:38.742 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 21974, 2024-03-18T15:04:38.742 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 21936, 2024-03-18T15:04:38.742 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 7038413, 2024-03-18T15:04:38.742 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 93228779, 2024-03-18T15:04:38.742 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 14, 2024-03-18T15:04:38.742 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 7, 2024-03-18T15:04:38.742 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 1.883683760, 2024-03-18T15:04:38.742 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.781245965, 2024-03-18T15:04:38.743 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.422873859, 2024-03-18T15:04:38.743 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.576615854, 2024-03-18T15:04:38.743 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:38.743 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 21935, 2024-03-18T15:04:38.743 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.806567076, 2024-03-18T15:04:38.743 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000036770 2024-03-18T15:04:38.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.743 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:38.743 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 735, 2024-03-18T15:04:38.744 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000124846, 2024-03-18T15:04:38.744 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000169 2024-03-18T15:04:38.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.744 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1564096, 2024-03-18T15:04:38.744 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 162048 2024-03-18T15:04:38.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.744 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:04:38.744 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 179523, 2024-03-18T15:04:38.744 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 174186, 2024-03-18T15:04:38.744 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 55950954, 2024-03-18T15:04:38.745 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 68307300, 2024-03-18T15:04:38.745 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 13, 2024-03-18T15:04:38.745 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 7, 2024-03-18T15:04:38.745 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 8.122887004, 2024-03-18T15:04:38.745 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 3.334371725, 2024-03-18T15:04:38.745 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 5.423652059, 2024-03-18T15:04:38.745 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.295761754, 2024-03-18T15:04:38.745 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:38.745 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 174184, 2024-03-18T15:04:38.746 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 2.952519836, 2024-03-18T15:04:38.746 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000016950 2024-03-18T15:04:38.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.746 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:38.746 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 331203, 2024-03-18T15:04:38.746 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.040439341, 2024-03-18T15:04:38.746 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000122 2024-03-18T15:04:38.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.747 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 5008, 2024-03-18T15:04:38.747 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 135536 2024-03-18T15:04:38.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.747 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-03-18T15:04:38.747 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-03-18T15:04:38.747 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-03-18T15:04:38.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.747 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-03-18T15:04:38.747 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-03-18T15:04:38.747 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-03-18T15:04:38.748 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 51643, 2024-03-18T15:04:38.748 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 32.677854099, 2024-03-18T15:04:38.748 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000632764 2024-03-18T15:04:38.748 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:38.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.748 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-03-18T15:04:38.748 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-03-18T15:04:38.748 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-03-18T15:04:38.748 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 26274, 2024-03-18T15:04:38.749 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 2.417843540, 2024-03-18T15:04:38.749 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000092024 2024-03-18T15:04:38.749 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:38.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.749 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-03-18T15:04:38.749 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 87460, 2024-03-18T15:04:38.749 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 85073, 2024-03-18T15:04:38.749 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-03-18T15:04:38.749 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 85073, 2024-03-18T15:04:38.749 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 105.822020672, 2024-03-18T15:04:38.750 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.001243896 2024-03-18T15:04:38.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.750 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-03-18T15:04:38.750 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 2387, 2024-03-18T15:04:38.750 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-03-18T15:04:38.750 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-03-18T15:04:38.750 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 11, 2024-03-18T15:04:38.750 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 0, 2024-03-18T15:04:38.750 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-03-18T15:04:38.751 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 301, 2024-03-18T15:04:38.751 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 30, 2024-03-18T15:04:38.751 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 271, 2024-03-18T15:04:38.751 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 0, 2024-03-18T15:04:38.751 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 257, 2024-03-18T15:04:38.751 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 0, 2024-03-18T15:04:38.751 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 34, 2024-03-18T15:04:38.751 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 34, 2024-03-18T15:04:38.751 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 5, 2024-03-18T15:04:38.752 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 206082, 2024-03-18T15:04:38.752 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 84733, 2024-03-18T15:04:38.752 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 2385, 2024-03-18T15:04:38.752 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 8, 2024-03-18T15:04:38.752 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-03-18T15:04:38.752 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-03-18T15:04:38.752 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 2298, 2024-03-18T15:04:38.752 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 9629, 2024-03-18T15:04:38.752 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-03-18T15:04:38.753 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 0, 2024-03-18T15:04:38.753 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 0, 2024-03-18T15:04:38.753 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 3, 2024-03-18T15:04:38.753 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 77, 2024-03-18T15:04:38.753 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-03-18T15:04:38.753 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 25, 2024-03-18T15:04:38.753 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-03-18T15:04:38.753 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 71, 2024-03-18T15:04:38.753 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 21133856, 2024-03-18T15:04:38.754 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 6, 2024-03-18T15:04:38.754 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 25, 2024-03-18T15:04:38.754 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-03-18T15:04:38.754 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T15:04:38.754 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-03-18T15:04:38.754 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 0, 2024-03-18T15:04:38.754 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T15:04:38.754 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T15:04:38.754 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 25, 2024-03-18T15:04:38.755 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-03-18T15:04:38.755 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 341, 2024-03-18T15:04:38.755 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 58312, 2024-03-18T15:04:38.755 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T15:04:38.755 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T15:04:38.755 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T15:04:38.755 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 14500, 2024-03-18T15:04:38.755 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 10309, 2024-03-18T15:04:38.755 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 2617, 2024-03-18T15:04:38.756 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 85927 2024-03-18T15:04:38.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.756 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-03-18T15:04:38.756 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 92, 2024-03-18T15:04:38.756 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 95, 2024-03-18T15:04:38.756 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 49, 2024-03-18T15:04:38.756 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 100, 2024-03-18T15:04:38.756 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 121, 2024-03-18T15:04:38.756 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 9, 2024-03-18T15:04:38.756 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-03-18T15:04:38.757 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-03-18T15:04:38.757 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 17266, 2024-03-18T15:04:38.757 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 17257, 2024-03-18T15:04:38.757 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-03-18T15:04:38.757 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-03-18T15:04:38.757 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-03-18T15:04:38.757 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-03-18T15:04:38.757 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-03-18T15:04:38.757 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-03-18T15:04:38.758 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-03-18T15:04:38.758 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_path": 0, 2024-03-18T15:04:38.758 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_inode": 0, 2024-03-18T15:04:38.758 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T15:04:38.758 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 0, 2024-03-18T15:04:38.758 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-03-18T15:04:38.758 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 0, 2024-03-18T15:04:38.758 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-03-18T15:04:38.758 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-03-18T15:04:38.759 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.759 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-03-18T15:04:38.759 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-03-18T15:04:38.759 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 88415, 2024-03-18T15:04:38.759 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-03-18T15:04:38.759 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-03-18T15:04:38.759 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 88415, 2024-03-18T15:04:38.759 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-03-18T15:04:38.759 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-03-18T15:04:38.759 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 111, 2024-03-18T15:04:38.760 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-03-18T15:04:38.760 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-03-18T15:04:38.760 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 111, 2024-03-18T15:04:38.760 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 9, 2024-03-18T15:04:38.760 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-03-18T15:04:38.760 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-03-18T15:04:38.760 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 4194304, 2024-03-18T15:04:38.760 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 164162921, 2024-03-18T15:04:38.760 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 0, 2024-03-18T15:04:38.761 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-03-18T15:04:38.761 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 26065, 2024-03-18T15:04:38.761 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 55.582096291, 2024-03-18T15:04:38.761 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002132441 2024-03-18T15:04:38.761 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.761 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 0 2024-03-18T15:04:38.761 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.761 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-03-18T15:04:38.761 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 212, 2024-03-18T15:04:38.761 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 17466, 2024-03-18T15:04:38.762 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 17254, 2024-03-18T15:04:38.762 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 25, 2024-03-18T15:04:38.762 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 72, 2024-03-18T15:04:38.762 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 47, 2024-03-18T15:04:38.762 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 277, 2024-03-18T15:04:38.762 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 23392, 2024-03-18T15:04:38.762 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 23115, 2024-03-18T15:04:38.762 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 34, 2024-03-18T15:04:38.762 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 17523, 2024-03-18T15:04:38.763 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 17489, 2024-03-18T15:04:38.763 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 63624, 2024-03-18T15:04:38.763 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 223612 2024-03-18T15:04:38.763 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.763 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-03-18T15:04:38.763 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 178161, 2024-03-18T15:04:38.763 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 0, 2024-03-18T15:04:38.763 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 87460, 2024-03-18T15:04:38.763 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 1746, 2024-03-18T15:04:38.763 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 36856, 2024-03-18T15:04:38.764 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-03-18T15:04:38.764 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 17266, 2024-03-18T15:04:38.764 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 40.285869725, 2024-03-18T15:04:38.764 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002333248 2024-03-18T15:04:38.764 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.764 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-03-18T15:04:38.764 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 230, 2024-03-18T15:04:38.764 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.177736016, 2024-03-18T15:04:38.764 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000772765 2024-03-18T15:04:38.765 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.765 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-03-18T15:04:38.765 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.765 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.765 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.765 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.765 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-03-18T15:04:38.765 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.765 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.765 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.766 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.766 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-03-18T15:04:38.766 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 20205, 2024-03-18T15:04:38.766 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 11.000525462, 2024-03-18T15:04:38.766 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000544445 2024-03-18T15:04:38.766 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.766 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-03-18T15:04:38.766 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.766 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.767 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.767 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.767 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-03-18T15:04:38.767 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.767 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.767 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.767 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.767 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-03-18T15:04:38.767 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.768 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.768 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.768 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.768 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-03-18T15:04:38.768 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.768 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.768 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.768 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.768 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-03-18T15:04:38.768 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.769 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.769 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.769 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.769 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-03-18T15:04:38.770 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.770 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.770 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.770 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.770 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-03-18T15:04:38.770 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.770 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.770 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.771 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.771 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-03-18T15:04:38.771 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.771 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.771 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.771 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.771 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-03-18T15:04:38.771 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.772 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.772 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.772 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.772 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-03-18T15:04:38.772 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.772 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.772 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.772 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.772 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-03-18T15:04:38.772 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 27808, 2024-03-18T15:04:38.773 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 14.990008973, 2024-03-18T15:04:38.773 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000539053 2024-03-18T15:04:38.773 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.773 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-03-18T15:04:38.773 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2298, 2024-03-18T15:04:38.773 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 5.599385805, 2024-03-18T15:04:38.773 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002436634 2024-03-18T15:04:38.773 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.774 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-03-18T15:04:38.774 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.774 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.774 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.774 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.774 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-03-18T15:04:38.774 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.774 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.774 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.774 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.775 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-03-18T15:04:38.775 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.775 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.775 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.775 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.775 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-03-18T15:04:38.775 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.775 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.775 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.776 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.776 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-03-18T15:04:38.776 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.776 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.776 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.776 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.776 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-03-18T15:04:38.776 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.776 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.777 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.777 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.777 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-03-18T15:04:38.777 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.777 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.777 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.777 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.777 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-03-18T15:04:38.777 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.778 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.778 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.778 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.778 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-03-18T15:04:38.778 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.778 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.778 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.778 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.778 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-03-18T15:04:38.779 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.779 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.779 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.779 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.779 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-03-18T15:04:38.779 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.779 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.779 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.779 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.779 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-03-18T15:04:38.780 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 17266, 2024-03-18T15:04:38.780 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 33.768494691, 2024-03-18T15:04:38.780 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.001955779 2024-03-18T15:04:38.780 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.780 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-03-18T15:04:38.780 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-03-18T15:04:38.780 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-03-18T15:04:38.780 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.780 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.780 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.781 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:38.781 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.781 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-03-18T15:04:38.781 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 1, 2024-03-18T15:04:38.781 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 4, 2024-03-18T15:04:38.781 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 3, 2024-03-18T15:04:38.781 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 1, 2024-03-18T15:04:38.781 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-03-18T15:04:38.781 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 10364, 2024-03-18T15:04:38.782 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 10364, 2024-03-18T15:04:38.782 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 696, 2024-03-18T15:04:38.782 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-03-18T15:04:38.782 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.782 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-03-18T15:04:38.782 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:04:38.782 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-03-18T15:04:38.782 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:04:38.782 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:04:38.783 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:04:38.783 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:04:38.783 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:04:38.783 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:04:38.783 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:04:38.783 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:04:38.783 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:04:38.783 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:04:38.783 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:04:38.783 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:04:38.784 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:04:38.784 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-03-18T15:04:38.784 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:04:38.784 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-03-18T15:04:38.784 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:04:38.784 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:04:38.784 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:04:38.784 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:04:38.784 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:04:38.785 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:04:38.785 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:04:38.785 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-03-18T15:04:38.785 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:04:38.785 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:04:38.785 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:04:38.785 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-03-18T15:04:38.785 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-03-18T15:04:38.785 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-03-18T15:04:38.785 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:04:38.786 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:04:38.786 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:04:38.786 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:04:38.786 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 576015, 2024-03-18T15:04:38.786 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 345, 2024-03-18T15:04:38.786 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 352, 2024-03-18T15:04:38.786 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 4, 2024-03-18T15:04:38.786 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-03-18T15:04:38.786 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-03-18T15:04:38.787 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:04:38.787 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-03-18T15:04:38.787 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:04:38.787 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-03-18T15:04:38.787 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 7936, 2024-03-18T15:04:38.787 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 165, 2024-03-18T15:04:38.787 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:04:38.787 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:04:38.788 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-03-18T15:04:38.788 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-03-18T15:04:38.788 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 913496, 2024-03-18T15:04:38.788 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 11289, 2024-03-18T15:04:38.788 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-03-18T15:04:38.788 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-03-18T15:04:38.788 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-03-18T15:04:38.788 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-03-18T15:04:38.788 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.789 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-03-18T15:04:38.789 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-03-18T15:04:38.789 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-03-18T15:04:38.789 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 45681, 2024-03-18T15:04:38.789 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 163234610, 2024-03-18T15:04:38.789 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 33, 2024-03-18T15:04:38.789 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 45681, 2024-03-18T15:04:38.789 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-03-18T15:04:38.789 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 45681, 2024-03-18T15:04:38.790 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 109.768615986, 2024-03-18T15:04:38.790 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002402938 2024-03-18T15:04:38.790 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.790 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-03-18T15:04:38.790 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-03-18T15:04:38.790 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 45681, 2024-03-18T15:04:38.790 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 46604 2024-03-18T15:04:38.790 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.790 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 45681, 2024-03-18T15:04:38.791 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 701, 2024-03-18T15:04:38.791 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 44980, 2024-03-18T15:04:38.791 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-03-18T15:04:38.791 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-03-18T15:04:38.791 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 0, 2024-03-18T15:04:38.791 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 10, 2024-03-18T15:04:38.791 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 676, 2024-03-18T15:04:38.791 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 26741, 2024-03-18T15:04:38.791 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 327, 2024-03-18T15:04:38.791 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-03-18T15:04:38.792 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-03-18T15:04:38.792 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-03-18T15:04:38.792 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 0, 2024-03-18T15:04:38.792 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 17307, 2024-03-18T15:04:38.792 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-03-18T15:04:38.792 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-03-18T15:04:38.792 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-03-18T15:04:38.792 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 25, 2024-03-18T15:04:38.792 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 20, 2024-03-18T15:04:38.793 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-03-18T15:04:38.793 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-03-18T15:04:38.793 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-03-18T15:04:38.793 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-03-18T15:04:38.793 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-03-18T15:04:38.793 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-03-18T15:04:38.793 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T15:04:38.793 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-03-18T15:04:38.793 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-03-18T15:04:38.794 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-03-18T15:04:38.794 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-03-18T15:04:38.794 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-03-18T15:04:38.794 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-03-18T15:04:38.794 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-03-18T15:04:38.794 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-03-18T15:04:38.794 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-03-18T15:04:38.794 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-03-18T15:04:38.794 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-03-18T15:04:38.795 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-03-18T15:04:38.795 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-03-18T15:04:38.795 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-03-18T15:04:38.795 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-03-18T15:04:38.795 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-03-18T15:04:38.795 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-03-18T15:04:38.795 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-03-18T15:04:38.795 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-03-18T15:04:38.795 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 107, 2024-03-18T15:04:38.796 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-03-18T15:04:38.796 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 21, 2024-03-18T15:04:38.796 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 12, 2024-03-18T15:04:38.796 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 12, 2024-03-18T15:04:38.796 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 0, 2024-03-18T15:04:38.796 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-03-18T15:04:38.796 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 1163, 2024-03-18T15:04:38.796 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 0, 2024-03-18T15:04:38.796 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 335 2024-03-18T15:04:38.796 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.797 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-03-18T15:04:38.797 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 1, 2024-03-18T15:04:38.797 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-03-18T15:04:38.797 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-03-18T15:04:38.797 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-03-18T15:04:38.797 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.797 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-03-18T15:04:38.797 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 0 2024-03-18T15:04:38.797 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.798 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-03-18T15:04:38.798 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-03-18T15:04:38.798 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 10, 2024-03-18T15:04:38.798 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-03-18T15:04:38.798 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 10, 2024-03-18T15:04:38.798 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 17257, 2024-03-18T15:04:38.798 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 17257, 2024-03-18T15:04:38.798 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-03-18T15:04:38.798 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.798 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:04:38.799 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-18T15:04:38.799 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-03-18T15:04:38.799 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-18T15:04:38.799 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 331086, 2024-03-18T15:04:38.799 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 72413885, 2024-03-18T15:04:38.799 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:38.799 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 331086, 2024-03-18T15:04:38.799 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-18T15:04:38.799 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-18T15:04:38.800 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 331086, 2024-03-18T15:04:38.800 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 72413885, 2024-03-18T15:04:38.800 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-18T15:04:38.800 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.800 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.800 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.800 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:38.800 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.800 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-03-18T15:04:38.801 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-18T15:04:38.801 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-03-18T15:04:38.801 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-18T15:04:38.801 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-18T15:04:38.801 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-18T15:04:38.801 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:38.801 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:38.801 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 45681, 2024-03-18T15:04:38.801 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 164979215, 2024-03-18T15:04:38.802 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 45681, 2024-03-18T15:04:38.802 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 164979215, 2024-03-18T15:04:38.802 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-18T15:04:38.802 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.802 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.802 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.802 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:38.802 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.803 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-03-18T15:04:38.803 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-18T15:04:38.803 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-03-18T15:04:38.803 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-18T15:04:38.803 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-18T15:04:38.803 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-18T15:04:38.803 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:38.803 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:38.803 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 45681, 2024-03-18T15:04:38.804 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 45681, 2024-03-18T15:04:38.804 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 45681, 2024-03-18T15:04:38.804 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 45681, 2024-03-18T15:04:38.804 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-18T15:04:38.804 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.804 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.804 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.804 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:38.804 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.805 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-03-18T15:04:38.805 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-18T15:04:38.805 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-03-18T15:04:38.805 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-18T15:04:38.805 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 17257, 2024-03-18T15:04:38.805 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 1744605, 2024-03-18T15:04:38.805 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:38.805 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 17257, 2024-03-18T15:04:38.806 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-18T15:04:38.806 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-18T15:04:38.806 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 676, 2024-03-18T15:04:38.806 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 1744605, 2024-03-18T15:04:38.806 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-18T15:04:38.806 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.806 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.806 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.807 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:38.807 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:38.807 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x5587dc1d8500": { 2024-03-18T15:04:38.807 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 1797, 2024-03-18T15:04:38.807 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-03-18T15:04:38.807 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-18T15:04:38.807 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 88415, 2024-03-18T15:04:38.807 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 159968617, 2024-03-18T15:04:38.807 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:38.808 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 88415, 2024-03-18T15:04:38.808 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-18T15:04:38.808 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-18T15:04:38.808 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 26065, 2024-03-18T15:04:38.808 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 159966820, 2024-03-18T15:04:38.808 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-18T15:04:38.808 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:38.808 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:38.808 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:38.809 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:38.809 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:38.809 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-18T15:04:39.642 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:39 smithi174 ceph-mon[36787]: pgmap v361: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-03-18T15:04:39.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:39 smithi076 ceph-mon[32205]: pgmap v361: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-03-18T15:04:39.671 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-18T15:04:39.671 INFO:tasks.check_counter:Daemon mds.c mds.imported=3 2024-03-18T15:04:39.671 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-03-18T15:04:39.672 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-18T15:04:39.672 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T15:04:39.672 DEBUG:tasks.check_counter:Getting stats from f 2024-03-18T15:04:39.672 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-18T15:04:39.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:39 smithi077 ceph-mon[35865]: pgmap v361: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-03-18T15:04:39.958 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:04:41.637 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:41 smithi174 ceph-mon[36787]: pgmap v362: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 1.5 MiB/s wr, 64 op/s 2024-03-18T15:04:41.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:41 smithi076 ceph-mon[32205]: pgmap v362: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 1.5 MiB/s wr, 64 op/s 2024-03-18T15:04:41.666 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-18T15:04:41.666 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:04:41.666 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 1786, 2024-03-18T15:04:41.667 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 1788, 2024-03-18T15:04:41.667 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 34077096, 2024-03-18T15:04:41.667 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 567955, 2024-03-18T15:04:41.667 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 8, 2024-03-18T15:04:41.667 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 4, 2024-03-18T15:04:41.667 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.130444705, 2024-03-18T15:04:41.667 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.034613291, 2024-03-18T15:04:41.667 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.047650943, 2024-03-18T15:04:41.668 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.038665317, 2024-03-18T15:04:41.668 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:41.668 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1786, 2024-03-18T15:04:41.668 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.036902532, 2024-03-18T15:04:41.668 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000020662 2024-03-18T15:04:41.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.668 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:41.668 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.668 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.669 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.669 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T15:04:41.669 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T15:04:41.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.669 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:04:41.669 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 840, 2024-03-18T15:04:41.669 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 807, 2024-03-18T15:04:41.670 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 30946141, 2024-03-18T15:04:41.670 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 319836, 2024-03-18T15:04:41.670 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 5, 2024-03-18T15:04:41.670 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 4, 2024-03-18T15:04:41.670 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.091074238, 2024-03-18T15:04:41.670 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.019368897, 2024-03-18T15:04:41.670 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.063897290, 2024-03-18T15:04:41.670 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.015676202, 2024-03-18T15:04:41.670 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:41.671 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 807, 2024-03-18T15:04:41.671 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.019619662, 2024-03-18T15:04:41.671 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000024311 2024-03-18T15:04:41.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.671 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:41.671 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.672 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.672 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.672 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1557040, 2024-03-18T15:04:41.672 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 143920 2024-03-18T15:04:41.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.672 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:04:41.673 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 511, 2024-03-18T15:04:41.673 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 681, 2024-03-18T15:04:41.673 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 25303178, 2024-03-18T15:04:41.673 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 626342, 2024-03-18T15:04:41.673 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 8, 2024-03-18T15:04:41.673 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 6, 2024-03-18T15:04:41.673 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.056853693, 2024-03-18T15:04:41.673 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.013560126, 2024-03-18T15:04:41.673 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.027930496, 2024-03-18T15:04:41.674 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.011086230, 2024-03-18T15:04:41.674 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:41.674 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 678, 2024-03-18T15:04:41.674 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.020257956, 2024-03-18T15:04:41.674 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000029878 2024-03-18T15:04:41.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.674 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:41.674 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.675 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.675 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.675 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T15:04:41.675 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 464480 2024-03-18T15:04:41.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.675 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-03-18T15:04:41.675 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-03-18T15:04:41.675 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-03-18T15:04:41.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.676 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-03-18T15:04:41.676 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-03-18T15:04:41.676 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-03-18T15:04:41.676 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 4955, 2024-03-18T15:04:41.676 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.093242940, 2024-03-18T15:04:41.676 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000018817 2024-03-18T15:04:41.676 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:41.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.676 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-03-18T15:04:41.677 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-03-18T15:04:41.677 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-03-18T15:04:41.677 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.677 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.677 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.677 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:41.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.677 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-03-18T15:04:41.677 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 0, 2024-03-18T15:04:41.678 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 0, 2024-03-18T15:04:41.678 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-03-18T15:04:41.678 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.678 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.678 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.678 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-03-18T15:04:41.678 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 0, 2024-03-18T15:04:41.678 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-03-18T15:04:41.679 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-03-18T15:04:41.679 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 0, 2024-03-18T15:04:41.679 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 0, 2024-03-18T15:04:41.679 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-03-18T15:04:41.679 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 5158, 2024-03-18T15:04:41.679 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 9, 2024-03-18T15:04:41.679 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 5149, 2024-03-18T15:04:41.680 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 0, 2024-03-18T15:04:41.680 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 5146, 2024-03-18T15:04:41.681 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 0, 2024-03-18T15:04:41.681 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 0, 2024-03-18T15:04:41.681 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 0, 2024-03-18T15:04:41.681 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 10, 2024-03-18T15:04:41.681 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 0, 2024-03-18T15:04:41.681 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 0, 2024-03-18T15:04:41.681 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 0, 2024-03-18T15:04:41.681 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 0, 2024-03-18T15:04:41.682 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-03-18T15:04:41.682 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-03-18T15:04:41.682 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 0, 2024-03-18T15:04:41.682 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-03-18T15:04:41.682 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-03-18T15:04:41.682 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 0, 2024-03-18T15:04:41.682 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 0, 2024-03-18T15:04:41.682 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 0, 2024-03-18T15:04:41.683 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 0, 2024-03-18T15:04:41.683 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-03-18T15:04:41.683 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 0, 2024-03-18T15:04:41.683 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-03-18T15:04:41.683 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 0, 2024-03-18T15:04:41.683 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 0, 2024-03-18T15:04:41.683 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 0, 2024-03-18T15:04:41.683 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T15:04:41.684 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-03-18T15:04:41.684 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T15:04:41.684 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-03-18T15:04:41.684 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 0, 2024-03-18T15:04:41.684 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T15:04:41.684 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T15:04:41.684 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 0, 2024-03-18T15:04:41.684 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-03-18T15:04:41.684 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T15:04:41.685 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 0, 2024-03-18T15:04:41.685 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T15:04:41.685 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T15:04:41.685 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T15:04:41.685 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 0, 2024-03-18T15:04:41.685 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 0, 2024-03-18T15:04:41.685 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 0, 2024-03-18T15:04:41.685 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 0 2024-03-18T15:04:41.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.686 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-03-18T15:04:41.686 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 0, 2024-03-18T15:04:41.686 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 0, 2024-03-18T15:04:41.686 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 0, 2024-03-18T15:04:41.686 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 0, 2024-03-18T15:04:41.686 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 0, 2024-03-18T15:04:41.686 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 0, 2024-03-18T15:04:41.686 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-03-18T15:04:41.687 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-03-18T15:04:41.687 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 0, 2024-03-18T15:04:41.687 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 0, 2024-03-18T15:04:41.687 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-03-18T15:04:41.687 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-03-18T15:04:41.687 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-03-18T15:04:41.687 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-03-18T15:04:41.687 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-03-18T15:04:41.688 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-03-18T15:04:41.688 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-03-18T15:04:41.688 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_path": 0, 2024-03-18T15:04:41.688 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_inode": 0, 2024-03-18T15:04:41.688 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T15:04:41.688 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 0, 2024-03-18T15:04:41.688 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-03-18T15:04:41.688 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 0, 2024-03-18T15:04:41.688 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-03-18T15:04:41.689 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-03-18T15:04:41.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.689 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-03-18T15:04:41.689 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-03-18T15:04:41.689 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 0, 2024-03-18T15:04:41.689 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-03-18T15:04:41.689 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-03-18T15:04:41.689 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 66060, 2024-03-18T15:04:41.689 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-03-18T15:04:41.690 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-03-18T15:04:41.690 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 0, 2024-03-18T15:04:41.690 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-03-18T15:04:41.690 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-03-18T15:04:41.690 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 81, 2024-03-18T15:04:41.690 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 7, 2024-03-18T15:04:41.690 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-03-18T15:04:41.690 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-03-18T15:04:41.691 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 4194304, 2024-03-18T15:04:41.691 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 92160693, 2024-03-18T15:04:41.691 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 92154356, 2024-03-18T15:04:41.691 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-03-18T15:04:41.691 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.691 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.691 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.691 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 66060 2024-03-18T15:04:41.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.692 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-03-18T15:04:41.692 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 239, 2024-03-18T15:04:41.692 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 5134, 2024-03-18T15:04:41.692 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 4895, 2024-03-18T15:04:41.692 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 29, 2024-03-18T15:04:41.692 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 30, 2024-03-18T15:04:41.692 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 1, 2024-03-18T15:04:41.692 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 5158, 2024-03-18T15:04:41.693 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 5158, 2024-03-18T15:04:41.693 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 0, 2024-03-18T15:04:41.693 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 0, 2024-03-18T15:04:41.693 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 0, 2024-03-18T15:04:41.693 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 0, 2024-03-18T15:04:41.693 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 58188, 2024-03-18T15:04:41.693 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 215420 2024-03-18T15:04:41.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.694 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-03-18T15:04:41.694 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 0, 2024-03-18T15:04:41.694 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 0, 2024-03-18T15:04:41.694 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 0, 2024-03-18T15:04:41.694 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 0, 2024-03-18T15:04:41.694 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 0, 2024-03-18T15:04:41.694 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-03-18T15:04:41.694 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.695 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.695 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.695 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-03-18T15:04:41.695 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.695 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.695 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.695 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-03-18T15:04:41.696 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.696 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.696 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.696 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-03-18T15:04:41.696 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.696 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.696 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.697 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-03-18T15:04:41.697 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.697 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.697 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.697 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-03-18T15:04:41.697 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.697 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.698 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.698 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-03-18T15:04:41.698 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.698 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.698 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.698 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-03-18T15:04:41.699 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.699 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.699 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.699 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-03-18T15:04:41.699 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.699 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.699 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.700 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-03-18T15:04:41.700 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.700 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.700 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.700 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-03-18T15:04:41.700 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.700 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.700 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.701 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-03-18T15:04:41.701 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.701 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.702 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.702 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-03-18T15:04:41.702 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.702 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.702 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.702 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-03-18T15:04:41.703 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.703 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.703 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.703 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-03-18T15:04:41.703 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.703 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.703 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.704 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-03-18T15:04:41.704 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.704 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.704 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.704 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-03-18T15:04:41.704 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.704 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.704 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.705 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-03-18T15:04:41.705 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.705 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.705 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.705 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-03-18T15:04:41.705 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.705 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.705 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.706 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-03-18T15:04:41.706 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.706 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.706 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.706 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-03-18T15:04:41.706 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.706 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.707 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.707 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-03-18T15:04:41.707 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.707 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.707 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.707 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-03-18T15:04:41.707 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.708 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.708 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.708 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-03-18T15:04:41.708 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.708 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.708 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.708 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-03-18T15:04:41.709 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.709 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.709 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.709 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-03-18T15:04:41.709 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.709 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.709 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.709 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-03-18T15:04:41.710 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.710 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.710 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.710 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-03-18T15:04:41.710 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.710 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.710 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.711 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-03-18T15:04:41.711 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.711 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.711 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.711 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-03-18T15:04:41.711 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-03-18T15:04:41.711 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-03-18T15:04:41.711 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.712 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.712 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.712 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:41.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.712 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-03-18T15:04:41.712 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 2, 2024-03-18T15:04:41.712 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 3, 2024-03-18T15:04:41.712 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 1, 2024-03-18T15:04:41.713 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 3, 2024-03-18T15:04:41.713 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-03-18T15:04:41.713 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 0, 2024-03-18T15:04:41.713 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 0, 2024-03-18T15:04:41.713 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 3154, 2024-03-18T15:04:41.713 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-03-18T15:04:41.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.713 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-03-18T15:04:41.713 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:04:41.714 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-03-18T15:04:41.714 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:04:41.714 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:04:41.714 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:04:41.714 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:04:41.714 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:04:41.714 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:04:41.714 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:04:41.714 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:04:41.715 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:04:41.715 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:04:41.715 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:04:41.715 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:04:41.715 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:04:41.715 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-03-18T15:04:41.715 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:04:41.715 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-03-18T15:04:41.715 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:04:41.716 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:04:41.716 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:04:41.716 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:04:41.716 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:04:41.716 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:04:41.716 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:04:41.716 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-03-18T15:04:41.716 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:04:41.716 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:04:41.716 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:04:41.717 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-03-18T15:04:41.717 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-03-18T15:04:41.717 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-03-18T15:04:41.717 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:04:41.717 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:04:41.717 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:04:41.717 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:04:41.717 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 240661, 2024-03-18T15:04:41.717 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 307, 2024-03-18T15:04:41.718 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:04:41.718 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-03-18T15:04:41.718 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-03-18T15:04:41.718 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-03-18T15:04:41.718 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:04:41.718 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-03-18T15:04:41.718 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:04:41.718 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-03-18T15:04:41.719 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 7936, 2024-03-18T15:04:41.719 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 165, 2024-03-18T15:04:41.719 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:04:41.719 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:04:41.719 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-03-18T15:04:41.719 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-03-18T15:04:41.719 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 4308441, 2024-03-18T15:04:41.719 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 27704, 2024-03-18T15:04:41.719 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-03-18T15:04:41.720 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-03-18T15:04:41.720 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-03-18T15:04:41.720 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-03-18T15:04:41.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.720 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-03-18T15:04:41.720 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-03-18T15:04:41.720 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-03-18T15:04:41.720 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 2848, 2024-03-18T15:04:41.721 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 21, 2024-03-18T15:04:41.721 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 3, 2024-03-18T15:04:41.721 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 2848, 2024-03-18T15:04:41.721 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-03-18T15:04:41.721 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2848, 2024-03-18T15:04:41.721 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 1.739977911, 2024-03-18T15:04:41.721 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000610947 2024-03-18T15:04:41.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.721 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-03-18T15:04:41.722 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-03-18T15:04:41.722 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2848, 2024-03-18T15:04:41.722 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 2849 2024-03-18T15:04:41.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.722 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 2848, 2024-03-18T15:04:41.722 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 2848, 2024-03-18T15:04:41.722 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 0, 2024-03-18T15:04:41.722 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-03-18T15:04:41.723 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-03-18T15:04:41.723 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 1487, 2024-03-18T15:04:41.723 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 0, 2024-03-18T15:04:41.723 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 1359, 2024-03-18T15:04:41.723 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 0, 2024-03-18T15:04:41.723 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 0, 2024-03-18T15:04:41.723 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-03-18T15:04:41.723 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-03-18T15:04:41.723 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-03-18T15:04:41.724 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 0, 2024-03-18T15:04:41.724 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 0, 2024-03-18T15:04:41.724 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-03-18T15:04:41.724 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-03-18T15:04:41.724 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-03-18T15:04:41.724 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 1, 2024-03-18T15:04:41.724 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 0, 2024-03-18T15:04:41.724 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-03-18T15:04:41.724 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-03-18T15:04:41.725 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-03-18T15:04:41.725 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-03-18T15:04:41.725 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-03-18T15:04:41.725 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-03-18T15:04:41.725 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T15:04:41.725 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-03-18T15:04:41.725 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-03-18T15:04:41.725 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-03-18T15:04:41.725 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-03-18T15:04:41.726 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-03-18T15:04:41.726 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-03-18T15:04:41.726 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-03-18T15:04:41.726 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-03-18T15:04:41.726 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-03-18T15:04:41.726 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-03-18T15:04:41.726 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-03-18T15:04:41.726 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-03-18T15:04:41.727 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-03-18T15:04:41.727 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-03-18T15:04:41.727 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-03-18T15:04:41.727 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-03-18T15:04:41.727 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-03-18T15:04:41.727 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-03-18T15:04:41.727 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-03-18T15:04:41.727 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 107, 2024-03-18T15:04:41.728 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-03-18T15:04:41.728 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 21, 2024-03-18T15:04:41.728 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 10, 2024-03-18T15:04:41.728 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 10, 2024-03-18T15:04:41.728 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 0, 2024-03-18T15:04:41.728 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-03-18T15:04:41.728 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 0, 2024-03-18T15:04:41.728 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 2, 2024-03-18T15:04:41.728 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 0 2024-03-18T15:04:41.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.729 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-03-18T15:04:41.729 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 0, 2024-03-18T15:04:41.729 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-03-18T15:04:41.729 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-03-18T15:04:41.729 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-03-18T15:04:41.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.729 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-03-18T15:04:41.730 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 0 2024-03-18T15:04:41.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.730 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-03-18T15:04:41.730 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-03-18T15:04:41.730 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T15:04:41.730 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-03-18T15:04:41.730 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 0, 2024-03-18T15:04:41.730 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 0, 2024-03-18T15:04:41.731 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 0, 2024-03-18T15:04:41.731 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-03-18T15:04:41.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.731 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:04:41.731 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-18T15:04:41.731 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-03-18T15:04:41.731 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-18T15:04:41.731 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 3133, 2024-03-18T15:04:41.731 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 90027120, 2024-03-18T15:04:41.732 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:41.732 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 3133, 2024-03-18T15:04:41.732 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-18T15:04:41.732 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-18T15:04:41.732 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 3133, 2024-03-18T15:04:41.732 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 90027120, 2024-03-18T15:04:41.732 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-18T15:04:41.732 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.733 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.733 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.733 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:41.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.733 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-03-18T15:04:41.733 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-18T15:04:41.733 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-03-18T15:04:41.733 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-18T15:04:41.734 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-18T15:04:41.734 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-18T15:04:41.734 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:41.734 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:41.734 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 2848, 2024-03-18T15:04:41.734 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 87979509, 2024-03-18T15:04:41.734 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2848, 2024-03-18T15:04:41.734 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 87979509, 2024-03-18T15:04:41.735 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-18T15:04:41.735 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.735 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.735 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.735 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:41.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.735 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-03-18T15:04:41.735 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-18T15:04:41.735 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-03-18T15:04:41.736 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-18T15:04:41.736 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-18T15:04:41.736 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-18T15:04:41.736 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:41.736 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:41.736 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 2848, 2024-03-18T15:04:41.736 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 2848, 2024-03-18T15:04:41.736 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2848, 2024-03-18T15:04:41.736 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 2848, 2024-03-18T15:04:41.737 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-18T15:04:41.737 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.737 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.737 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.737 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:41.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.737 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-03-18T15:04:41.737 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-18T15:04:41.737 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-03-18T15:04:41.738 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-18T15:04:41.738 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-18T15:04:41.738 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-18T15:04:41.738 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:41.738 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:41.738 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-18T15:04:41.738 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-18T15:04:41.738 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-03-18T15:04:41.738 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-03-18T15:04:41.739 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-18T15:04:41.739 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.739 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.739 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.739 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:41.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:41.739 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x56275a9366e0": { 2024-03-18T15:04:41.739 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-18T15:04:41.739 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-03-18T15:04:41.740 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-18T15:04:41.740 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-18T15:04:41.740 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-18T15:04:41.740 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:41.740 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:41.740 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-18T15:04:41.740 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-18T15:04:41.740 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-03-18T15:04:41.740 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-03-18T15:04:41.740 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-18T15:04:41.741 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:41.741 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:41.741 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:41.741 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:41.741 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:41.741 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-18T15:04:41.741 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:41 smithi077 ceph-mon[35865]: pgmap v362: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 1.5 MiB/s wr, 64 op/s 2024-03-18T15:04:42.569 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-18T15:04:42.570 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-18T15:04:42.570 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-03-18T15:04:42.570 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-18T15:04:42.570 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T15:04:42.570 DEBUG:tasks.check_counter:Getting stats from i 2024-03-18T15:04:42.570 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-18T15:04:42.849 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:04:43.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:43 smithi076 ceph-mon[32205]: pgmap v363: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 820 KiB/s wr, 38 op/s 2024-03-18T15:04:43.684 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:43 smithi174 ceph-mon[36787]: pgmap v363: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 820 KiB/s wr, 38 op/s 2024-03-18T15:04:43.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:43 smithi077 ceph-mon[35865]: pgmap v363: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 820 KiB/s wr, 38 op/s 2024-03-18T15:04:44.669 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-18T15:04:44.670 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:04:44.670 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 211676, 2024-03-18T15:04:44.670 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 198464, 2024-03-18T15:04:44.670 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 65852952, 2024-03-18T15:04:44.670 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 180812328, 2024-03-18T15:04:44.670 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 9, 2024-03-18T15:04:44.671 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 4, 2024-03-18T15:04:44.671 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 8.446048342, 2024-03-18T15:04:44.671 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 3.726777467, 2024-03-18T15:04:44.671 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 6.133595330, 2024-03-18T15:04:44.671 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.143037705, 2024-03-18T15:04:44.671 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:44.672 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 198466, 2024-03-18T15:04:44.672 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 3.887201285, 2024-03-18T15:04:44.672 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000019586 2024-03-18T15:04:44.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.672 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:44.672 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 308175, 2024-03-18T15:04:44.672 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.044758830, 2024-03-18T15:04:44.672 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000145 2024-03-18T15:04:44.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.673 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1570576, 2024-03-18T15:04:44.673 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 162784 2024-03-18T15:04:44.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.673 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:04:44.673 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 4451, 2024-03-18T15:04:44.673 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 4658, 2024-03-18T15:04:44.673 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 1110277, 2024-03-18T15:04:44.673 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 35014266, 2024-03-18T15:04:44.673 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 21, 2024-03-18T15:04:44.674 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 9, 2024-03-18T15:04:44.674 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.360170099, 2024-03-18T15:04:44.674 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.146126551, 2024-03-18T15:04:44.674 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.076765780, 2024-03-18T15:04:44.674 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.112074182, 2024-03-18T15:04:44.674 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:44.674 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 4651, 2024-03-18T15:04:44.675 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.178204772, 2024-03-18T15:04:44.675 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000038315 2024-03-18T15:04:44.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.675 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:44.675 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 75, 2024-03-18T15:04:44.675 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000011193, 2024-03-18T15:04:44.675 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000149 2024-03-18T15:04:44.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.676 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 4896, 2024-03-18T15:04:44.676 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 481056 2024-03-18T15:04:44.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.676 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:04:44.676 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 6226, 2024-03-18T15:04:44.677 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 6231, 2024-03-18T15:04:44.677 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 2149192, 2024-03-18T15:04:44.677 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 36530882, 2024-03-18T15:04:44.677 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 10, 2024-03-18T15:04:44.677 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 5, 2024-03-18T15:04:44.677 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.498405603, 2024-03-18T15:04:44.677 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.212937307, 2024-03-18T15:04:44.677 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.113274812, 2024-03-18T15:04:44.678 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.145296804, 2024-03-18T15:04:44.678 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:44.678 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 6231, 2024-03-18T15:04:44.678 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.246997863, 2024-03-18T15:04:44.678 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000039640 2024-03-18T15:04:44.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.678 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:44.678 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 736, 2024-03-18T15:04:44.678 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000128734, 2024-03-18T15:04:44.679 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000174 2024-03-18T15:04:44.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.679 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T15:04:44.679 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T15:04:44.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.679 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-03-18T15:04:44.679 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-03-18T15:04:44.679 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-03-18T15:04:44.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.680 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-03-18T15:04:44.680 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-03-18T15:04:44.680 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-03-18T15:04:44.680 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 18314, 2024-03-18T15:04:44.680 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 12.355849708, 2024-03-18T15:04:44.680 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000674666 2024-03-18T15:04:44.680 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:44.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.680 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-03-18T15:04:44.681 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-03-18T15:04:44.681 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-03-18T15:04:44.681 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 7874, 2024-03-18T15:04:44.681 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.600311205, 2024-03-18T15:04:44.681 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000076239 2024-03-18T15:04:44.681 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:44.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.681 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-03-18T15:04:44.682 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 62098, 2024-03-18T15:04:44.682 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 59393, 2024-03-18T15:04:44.682 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-03-18T15:04:44.682 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 59393, 2024-03-18T15:04:44.682 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 67.741182986, 2024-03-18T15:04:44.682 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.001140558 2024-03-18T15:04:44.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.682 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-03-18T15:04:44.683 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 2705, 2024-03-18T15:04:44.683 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-03-18T15:04:44.683 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-03-18T15:04:44.683 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 19, 2024-03-18T15:04:44.683 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 1, 2024-03-18T15:04:44.683 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-03-18T15:04:44.683 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 279, 2024-03-18T15:04:44.683 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 76, 2024-03-18T15:04:44.683 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 203, 2024-03-18T15:04:44.684 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 0, 2024-03-18T15:04:44.684 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 171, 2024-03-18T15:04:44.684 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 0, 2024-03-18T15:04:44.684 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 6, 2024-03-18T15:04:44.684 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 7, 2024-03-18T15:04:44.684 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 10, 2024-03-18T15:04:44.684 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 101017, 2024-03-18T15:04:44.684 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 53405, 2024-03-18T15:04:44.684 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 2701, 2024-03-18T15:04:44.685 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 33, 2024-03-18T15:04:44.685 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-03-18T15:04:44.685 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-03-18T15:04:44.685 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 286, 2024-03-18T15:04:44.685 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 1089, 2024-03-18T15:04:44.685 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-03-18T15:04:44.685 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 0, 2024-03-18T15:04:44.686 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 0, 2024-03-18T15:04:44.686 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 1, 2024-03-18T15:04:44.686 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 50, 2024-03-18T15:04:44.686 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-03-18T15:04:44.686 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 24, 2024-03-18T15:04:44.686 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 1, 2024-03-18T15:04:44.686 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 71, 2024-03-18T15:04:44.686 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 21133856, 2024-03-18T15:04:44.687 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 6, 2024-03-18T15:04:44.687 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 24, 2024-03-18T15:04:44.687 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-03-18T15:04:44.687 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T15:04:44.687 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-03-18T15:04:44.687 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 0, 2024-03-18T15:04:44.688 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T15:04:44.688 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T15:04:44.688 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 24, 2024-03-18T15:04:44.688 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-03-18T15:04:44.688 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 1079, 2024-03-18T15:04:44.688 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 24211, 2024-03-18T15:04:44.688 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T15:04:44.689 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T15:04:44.689 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T15:04:44.689 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 13960, 2024-03-18T15:04:44.689 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 6906, 2024-03-18T15:04:44.689 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 4766, 2024-03-18T15:04:44.689 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 56089 2024-03-18T15:04:44.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.689 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-03-18T15:04:44.689 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 43, 2024-03-18T15:04:44.690 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 194, 2024-03-18T15:04:44.690 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 99, 2024-03-18T15:04:44.690 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 250, 2024-03-18T15:04:44.690 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 47, 2024-03-18T15:04:44.690 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 0, 2024-03-18T15:04:44.690 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-03-18T15:04:44.690 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-03-18T15:04:44.691 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 4909, 2024-03-18T15:04:44.691 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 4909, 2024-03-18T15:04:44.691 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-03-18T15:04:44.691 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-03-18T15:04:44.691 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-03-18T15:04:44.691 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-03-18T15:04:44.691 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-03-18T15:04:44.691 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-03-18T15:04:44.692 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-03-18T15:04:44.692 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_path": 0, 2024-03-18T15:04:44.692 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_inode": 0, 2024-03-18T15:04:44.692 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T15:04:44.692 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 1, 2024-03-18T15:04:44.692 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-03-18T15:04:44.692 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 1, 2024-03-18T15:04:44.692 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-03-18T15:04:44.692 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-03-18T15:04:44.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.693 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-03-18T15:04:44.693 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-03-18T15:04:44.693 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 66095, 2024-03-18T15:04:44.693 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-03-18T15:04:44.693 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-03-18T15:04:44.693 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 66095, 2024-03-18T15:04:44.693 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-03-18T15:04:44.694 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-03-18T15:04:44.694 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 81, 2024-03-18T15:04:44.694 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-03-18T15:04:44.694 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-03-18T15:04:44.694 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 81, 2024-03-18T15:04:44.694 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 7, 2024-03-18T15:04:44.695 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-03-18T15:04:44.695 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-03-18T15:04:44.695 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 4194304, 2024-03-18T15:04:44.695 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 92237036, 2024-03-18T15:04:44.695 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 0, 2024-03-18T15:04:44.695 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-03-18T15:04:44.695 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 9158, 2024-03-18T15:04:44.695 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 19.783151544, 2024-03-18T15:04:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002160204 2024-03-18T15:04:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 0 2024-03-18T15:04:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-03-18T15:04:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 245, 2024-03-18T15:04:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 5151, 2024-03-18T15:04:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 4906, 2024-03-18T15:04:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 30, 2024-03-18T15:04:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 154, 2024-03-18T15:04:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 124, 2024-03-18T15:04:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 268, 2024-03-18T15:04:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 11472, 2024-03-18T15:04:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 11204, 2024-03-18T15:04:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 7, 2024-03-18T15:04:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 5576, 2024-03-18T15:04:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 5569, 2024-03-18T15:04:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 61620, 2024-03-18T15:04:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 223612 2024-03-18T15:04:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-03-18T15:04:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 91717, 2024-03-18T15:04:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 0, 2024-03-18T15:04:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 62098, 2024-03-18T15:04:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 781, 2024-03-18T15:04:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 10112, 2024-03-18T15:04:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-03-18T15:04:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 4908, 2024-03-18T15:04:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 12.791935671, 2024-03-18T15:04:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002606343 2024-03-18T15:04:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-03-18T15:04:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 8, 2024-03-18T15:04:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.008032189, 2024-03-18T15:04:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.001004023 2024-03-18T15:04:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-03-18T15:04:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-03-18T15:04:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-03-18T15:04:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 8772, 2024-03-18T15:04:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 6.281757373, 2024-03-18T15:04:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000716114 2024-03-18T15:04:44.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.702 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-03-18T15:04:44.702 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.702 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.702 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.702 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-03-18T15:04:44.702 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1, 2024-03-18T15:04:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000111394, 2024-03-18T15:04:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000111394 2024-03-18T15:04:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-03-18T15:04:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-03-18T15:04:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-03-18T15:04:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.705 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-03-18T15:04:44.705 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.705 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-03-18T15:04:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-03-18T15:04:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-03-18T15:04:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-03-18T15:04:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-03-18T15:04:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 40509, 2024-03-18T15:04:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 21.492037800, 2024-03-18T15:04:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000530549 2024-03-18T15:04:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-03-18T15:04:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 286, 2024-03-18T15:04:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.716093208, 2024-03-18T15:04:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002503822 2024-03-18T15:04:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-03-18T15:04:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.710 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.710 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.710 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-03-18T15:04:44.710 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.710 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.710 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.711 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-03-18T15:04:44.711 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.711 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.711 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.711 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-03-18T15:04:44.711 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.711 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.711 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.712 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-03-18T15:04:44.712 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.712 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.712 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.712 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-03-18T15:04:44.712 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.712 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.713 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.713 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-03-18T15:04:44.713 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.713 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.713 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.713 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-03-18T15:04:44.713 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.714 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.714 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.714 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-03-18T15:04:44.714 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.714 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.714 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.714 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-03-18T15:04:44.715 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.715 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.715 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.715 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-03-18T15:04:44.715 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.715 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.715 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.716 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-03-18T15:04:44.716 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 4909, 2024-03-18T15:04:44.716 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 26.451215351, 2024-03-18T15:04:44.716 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.005388310 2024-03-18T15:04:44.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.716 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-03-18T15:04:44.716 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-03-18T15:04:44.716 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-03-18T15:04:44.717 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.717 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.717 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.717 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:44.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.717 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-03-18T15:04:44.717 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 2, 2024-03-18T15:04:44.717 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 3, 2024-03-18T15:04:44.717 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 1, 2024-03-18T15:04:44.718 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 2, 2024-03-18T15:04:44.718 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-03-18T15:04:44.718 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 10502, 2024-03-18T15:04:44.718 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 5251, 2024-03-18T15:04:44.718 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 663, 2024-03-18T15:04:44.718 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-03-18T15:04:44.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.718 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-03-18T15:04:44.719 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:04:44.719 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-03-18T15:04:44.719 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:04:44.719 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:04:44.719 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:04:44.719 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:04:44.719 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:04:44.719 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:04:44.719 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:04:44.720 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:04:44.720 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:04:44.720 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:04:44.720 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:04:44.720 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:04:44.720 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:04:44.720 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-03-18T15:04:44.720 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:04:44.720 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-03-18T15:04:44.721 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:04:44.721 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:04:44.721 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:04:44.721 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:04:44.721 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:04:44.721 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:04:44.721 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:04:44.721 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-03-18T15:04:44.722 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:04:44.722 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:04:44.722 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:04:44.722 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-03-18T15:04:44.722 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-03-18T15:04:44.722 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-03-18T15:04:44.722 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:04:44.722 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:04:44.723 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:04:44.723 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:04:44.723 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 590711, 2024-03-18T15:04:44.723 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 327, 2024-03-18T15:04:44.723 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 528, 2024-03-18T15:04:44.723 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 6, 2024-03-18T15:04:44.723 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-03-18T15:04:44.723 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-03-18T15:04:44.723 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:04:44.724 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-03-18T15:04:44.724 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:04:44.724 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-03-18T15:04:44.724 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 7936, 2024-03-18T15:04:44.724 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 165, 2024-03-18T15:04:44.724 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:04:44.724 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:04:44.724 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-03-18T15:04:44.724 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-03-18T15:04:44.725 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 938560, 2024-03-18T15:04:44.725 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 8565, 2024-03-18T15:04:44.725 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-03-18T15:04:44.725 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-03-18T15:04:44.725 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-03-18T15:04:44.725 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-03-18T15:04:44.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.725 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-03-18T15:04:44.725 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-03-18T15:04:44.726 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-03-18T15:04:44.726 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 15467, 2024-03-18T15:04:44.726 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 89218178, 2024-03-18T15:04:44.726 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 30, 2024-03-18T15:04:44.726 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 15467, 2024-03-18T15:04:44.726 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-03-18T15:04:44.726 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 15467, 2024-03-18T15:04:44.726 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 37.284528900, 2024-03-18T15:04:44.727 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002410585 2024-03-18T15:04:44.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.727 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-03-18T15:04:44.727 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-03-18T15:04:44.727 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 15467, 2024-03-18T15:04:44.727 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 16185 2024-03-18T15:04:44.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.727 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 15467, 2024-03-18T15:04:44.727 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 315, 2024-03-18T15:04:44.728 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 15152, 2024-03-18T15:04:44.728 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-03-18T15:04:44.728 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-03-18T15:04:44.728 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 0, 2024-03-18T15:04:44.728 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 10, 2024-03-18T15:04:44.728 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 291, 2024-03-18T15:04:44.728 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 9449, 2024-03-18T15:04:44.728 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 178, 2024-03-18T15:04:44.728 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-03-18T15:04:44.729 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-03-18T15:04:44.729 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-03-18T15:04:44.729 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 1, 2024-03-18T15:04:44.729 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 4941, 2024-03-18T15:04:44.729 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-03-18T15:04:44.729 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-03-18T15:04:44.729 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-03-18T15:04:44.729 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 24, 2024-03-18T15:04:44.729 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 20, 2024-03-18T15:04:44.730 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-03-18T15:04:44.730 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-03-18T15:04:44.730 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-03-18T15:04:44.730 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-03-18T15:04:44.730 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-03-18T15:04:44.730 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-03-18T15:04:44.730 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T15:04:44.730 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-03-18T15:04:44.731 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-03-18T15:04:44.731 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-03-18T15:04:44.731 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-03-18T15:04:44.731 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-03-18T15:04:44.731 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-03-18T15:04:44.731 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-03-18T15:04:44.731 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-03-18T15:04:44.731 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-03-18T15:04:44.731 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-03-18T15:04:44.732 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-03-18T15:04:44.732 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-03-18T15:04:44.732 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-03-18T15:04:44.732 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-03-18T15:04:44.732 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-03-18T15:04:44.732 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-03-18T15:04:44.732 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-03-18T15:04:44.732 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-03-18T15:04:44.732 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-03-18T15:04:44.733 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 107, 2024-03-18T15:04:44.733 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-03-18T15:04:44.733 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 21, 2024-03-18T15:04:44.733 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 12, 2024-03-18T15:04:44.733 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 12, 2024-03-18T15:04:44.733 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 0, 2024-03-18T15:04:44.733 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-03-18T15:04:44.733 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 1110, 2024-03-18T15:04:44.733 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 0, 2024-03-18T15:04:44.733 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 161 2024-03-18T15:04:44.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.734 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-03-18T15:04:44.734 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 1, 2024-03-18T15:04:44.734 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-03-18T15:04:44.734 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-03-18T15:04:44.734 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-03-18T15:04:44.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.734 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-03-18T15:04:44.735 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 1 2024-03-18T15:04:44.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.735 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-03-18T15:04:44.735 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-03-18T15:04:44.735 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 10, 2024-03-18T15:04:44.735 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-03-18T15:04:44.735 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 10, 2024-03-18T15:04:44.735 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 4909, 2024-03-18T15:04:44.735 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 4909, 2024-03-18T15:04:44.736 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-03-18T15:04:44.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.736 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:04:44.736 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-18T15:04:44.736 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-03-18T15:04:44.736 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-18T15:04:44.736 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 222349, 2024-03-18T15:04:44.736 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 50450716, 2024-03-18T15:04:44.736 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:44.737 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 222349, 2024-03-18T15:04:44.737 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-18T15:04:44.737 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-18T15:04:44.737 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 222349, 2024-03-18T15:04:44.737 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 50450716, 2024-03-18T15:04:44.737 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-18T15:04:44.737 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.737 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.737 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.738 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:44.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.738 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-03-18T15:04:44.738 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-18T15:04:44.738 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-03-18T15:04:44.738 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-18T15:04:44.738 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-18T15:04:44.738 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-18T15:04:44.739 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:44.739 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:44.739 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 15467, 2024-03-18T15:04:44.739 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 89715731, 2024-03-18T15:04:44.739 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 15467, 2024-03-18T15:04:44.739 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 89715731, 2024-03-18T15:04:44.739 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-18T15:04:44.739 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.739 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.740 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.740 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:44.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.740 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-03-18T15:04:44.740 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-18T15:04:44.740 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-03-18T15:04:44.740 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-18T15:04:44.740 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-18T15:04:44.741 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-18T15:04:44.741 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:44.741 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-18T15:04:44.741 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 15467, 2024-03-18T15:04:44.741 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 15467, 2024-03-18T15:04:44.741 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 15467, 2024-03-18T15:04:44.741 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 15467, 2024-03-18T15:04:44.741 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-18T15:04:44.741 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.742 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.742 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.742 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:44.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.742 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-03-18T15:04:44.742 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-18T15:04:44.742 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-03-18T15:04:44.742 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-18T15:04:44.743 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 4909, 2024-03-18T15:04:44.743 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 497553, 2024-03-18T15:04:44.743 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:44.743 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 4909, 2024-03-18T15:04:44.743 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-18T15:04:44.743 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-18T15:04:44.743 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 291, 2024-03-18T15:04:44.743 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 497553, 2024-03-18T15:04:44.743 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-18T15:04:44.744 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.744 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.744 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.744 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:44.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:44.744 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x562d037cc500": { 2024-03-18T15:04:44.744 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 1797, 2024-03-18T15:04:44.744 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-03-18T15:04:44.744 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-18T15:04:44.745 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 66095, 2024-03-18T15:04:44.745 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 88042732, 2024-03-18T15:04:44.745 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:44.745 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 66095, 2024-03-18T15:04:44.745 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-18T15:04:44.745 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-18T15:04:44.745 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 9158, 2024-03-18T15:04:44.745 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 88040935, 2024-03-18T15:04:44.745 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-18T15:04:44.746 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:44.746 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:44.746 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:44.746 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:44.746 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:44.746 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-18T15:04:45.487 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-18T15:04:45.488 INFO:tasks.check_counter:Daemon mds.i mds.imported=1 2024-03-18T15:04:45.488 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-18T15:04:45.488 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-18T15:04:45.488 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T15:04:45.488 DEBUG:tasks.check_counter:Getting stats from l 2024-03-18T15:04:45.488 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-18T15:04:45.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:45 smithi076 ceph-mon[32205]: pgmap v364: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 889 KiB/s wr, 47 op/s 2024-03-18T15:04:45.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:45 smithi174 ceph-mon[36787]: pgmap v364: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 889 KiB/s wr, 47 op/s 2024-03-18T15:04:45.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:45 smithi077 ceph-mon[35865]: pgmap v364: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 889 KiB/s wr, 47 op/s 2024-03-18T15:04:45.773 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:04:47.594 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:47 smithi174 ceph-mon[36787]: pgmap v365: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 299 KiB/s wr, 22 op/s 2024-03-18T15:04:47.595 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-18T15:04:47.595 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:04:47.595 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 241, 2024-03-18T15:04:47.595 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 231, 2024-03-18T15:04:47.595 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 1442832, 2024-03-18T15:04:47.595 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 142144, 2024-03-18T15:04:47.595 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 4, 2024-03-18T15:04:47.595 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 0, 2024-03-18T15:04:47.596 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.039470884, 2024-03-18T15:04:47.596 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.009882512, 2024-03-18T15:04:47.596 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.033999139, 2024-03-18T15:04:47.596 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.004239021, 2024-03-18T15:04:47.596 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:47.596 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 230, 2024-03-18T15:04:47.596 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.006487789, 2024-03-18T15:04:47.596 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000028207 2024-03-18T15:04:47.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:47.597 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:47.597 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:47.597 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:47.597 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:47.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:47.597 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1442832, 2024-03-18T15:04:47.597 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 142144 2024-03-18T15:04:47.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:47.597 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:04:47.597 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 2, 2024-03-18T15:04:47.598 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 176, 2024-03-18T15:04:47.598 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 320, 2024-03-18T15:04:47.598 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 186448, 2024-03-18T15:04:47.598 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 5, 2024-03-18T15:04:47.598 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 3, 2024-03-18T15:04:47.598 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.008248603, 2024-03-18T15:04:47.598 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.005087266, 2024-03-18T15:04:47.598 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.001724304, 2024-03-18T15:04:47.599 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T15:04:47.599 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:47.599 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 173, 2024-03-18T15:04:47.599 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.017902597, 2024-03-18T15:04:47.599 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000103483 2024-03-18T15:04:47.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:47.599 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:47.599 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:47.599 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:47.600 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:47.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:47.600 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T15:04:47.600 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 186448 2024-03-18T15:04:47.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:47.600 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:04:47.600 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 4, 2024-03-18T15:04:47.600 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 4, 2024-03-18T15:04:47.600 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 2992, 2024-03-18T15:04:47.601 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 448, 2024-03-18T15:04:47.601 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 2, 2024-03-18T15:04:47.601 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 1, 2024-03-18T15:04:47.601 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.000553985, 2024-03-18T15:04:47.601 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.000027855, 2024-03-18T15:04:47.601 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.000495228, 2024-03-18T15:04:47.601 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T15:04:47.601 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:04:47.602 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3, 2024-03-18T15:04:47.602 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000673296, 2024-03-18T15:04:47.602 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000224432 2024-03-18T15:04:47.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:47.602 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:04:47.602 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:47.602 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:47.602 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:47.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:47.603 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-18T15:04:47.603 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T15:04:47.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:47.603 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-03-18T15:04:47.603 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 0, 2024-03-18T15:04:47.603 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-03-18T15:04:47.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:47.603 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-03-18T15:04:47.604 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:04:47.604 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-03-18T15:04:47.604 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:04:47.604 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:04:47.604 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:04:47.604 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:04:47.604 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:04:47.604 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:04:47.604 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:04:47.605 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:04:47.605 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:04:47.605 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:04:47.605 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:04:47.605 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:04:47.605 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:04:47.605 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-03-18T15:04:47.605 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:04:47.605 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-03-18T15:04:47.606 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:04:47.606 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:04:47.606 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:04:47.606 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:04:47.606 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:04:47.606 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:04:47.606 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:04:47.606 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-03-18T15:04:47.607 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:04:47.607 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:04:47.607 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:04:47.607 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-03-18T15:04:47.607 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-03-18T15:04:47.607 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-03-18T15:04:47.607 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:04:47.607 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:04:47.607 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:04:47.607 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:04:47.608 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 149017, 2024-03-18T15:04:47.608 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 31, 2024-03-18T15:04:47.608 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:04:47.608 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-03-18T15:04:47.608 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-03-18T15:04:47.608 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-03-18T15:04:47.608 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:04:47.608 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-03-18T15:04:47.609 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:04:47.609 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-03-18T15:04:47.609 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 0, 2024-03-18T15:04:47.609 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 0, 2024-03-18T15:04:47.609 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:04:47.609 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:04:47.609 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-03-18T15:04:47.609 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-03-18T15:04:47.609 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 928, 2024-03-18T15:04:47.610 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 2, 2024-03-18T15:04:47.610 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-03-18T15:04:47.610 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-03-18T15:04:47.610 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-03-18T15:04:47.610 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-03-18T15:04:47.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-18T15:04:47.610 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:04:47.610 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-18T15:04:47.611 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-03-18T15:04:47.611 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-18T15:04:47.611 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 243, 2024-03-18T15:04:47.611 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 1408840, 2024-03-18T15:04:47.611 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-18T15:04:47.611 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 243, 2024-03-18T15:04:47.611 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-18T15:04:47.611 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-18T15:04:47.611 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 243, 2024-03-18T15:04:47.612 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 1408840, 2024-03-18T15:04:47.612 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-18T15:04:47.612 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-18T15:04:47.612 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-18T15:04:47.612 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-18T15:04:47.612 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:47.612 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-18T15:04:47.612 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-18T15:04:47.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:47 smithi076 ceph-mon[32205]: pgmap v365: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 299 KiB/s wr, 22 op/s 2024-03-18T15:04:47.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:47 smithi077 ceph-mon[35865]: pgmap v365: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 299 KiB/s wr, 22 op/s 2024-03-18T15:04:48.326 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-18T15:04:48.326 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-18T15:04:48.326 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-03-18T15:04:48.327 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-03-18T15:04:48.327 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-03-18T15:04:48.327 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-18T15:04:48.343 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-18T15:04:48.343 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T15:04:48.343 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T15:04:48.364 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T15:04:48.364 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T15:04:48.424 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:04:48.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.578+0000 7ff3933c4640 1 -- 172.21.15.76:0/2247885751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff38c100680 msgr2=0x7ff38c100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:48.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.578+0000 7ff3933c4640 1 --2- 172.21.15.76:0/2247885751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff38c100680 0x7ff38c100b00 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009930 tx=0x7ff37c02bc30 comp rx=0 tx=0).stop 2024-03-18T15:04:48.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.578+0000 7ff3933c4640 1 -- 172.21.15.76:0/2247885751 shutdown_connections 2024-03-18T15:04:48.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.578+0000 7ff3933c4640 1 --2- 172.21.15.76:0/2247885751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff38c101040 0x7ff38c10d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:48.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.578+0000 7ff3933c4640 1 --2- 172.21.15.76:0/2247885751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff38c100680 0x7ff38c100b00 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:48.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.578+0000 7ff3933c4640 1 --2- 172.21.15.76:0/2247885751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c106690 0x7ff38c106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:48.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.578+0000 7ff3933c4640 1 -- 172.21.15.76:0/2247885751 >> 172.21.15.76:0/2247885751 conn(0x7ff38c076a60 msgr2=0x7ff38c076e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:48.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.578+0000 7ff3933c4640 1 -- 172.21.15.76:0/2247885751 shutdown_connections 2024-03-18T15:04:48.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.578+0000 7ff3933c4640 1 -- 172.21.15.76:0/2247885751 wait complete. 2024-03-18T15:04:48.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.578+0000 7ff3933c4640 1 Processor -- start 2024-03-18T15:04:48.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.578+0000 7ff3933c4640 1 -- start start 2024-03-18T15:04:48.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.578+0000 7ff3933c4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff38c100680 0x7ff38c1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:48.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff3933c4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c101040 0x7ff38c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:48.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff3933c4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff38c106690 0x7ff38c19c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:48.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff3933c4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff38c110aa0 con 0x7ff38c106690 2024-03-18T15:04:48.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff3933c4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff38c110920 con 0x7ff38c101040 2024-03-18T15:04:48.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff3933c4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff38c110c20 con 0x7ff38c100680 2024-03-18T15:04:48.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff391961640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff38c106690 0x7ff38c19c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:48.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff391961640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff38c106690 0x7ff38c19c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56544/0 (socket says 172.21.15.76:56544) 2024-03-18T15:04:48.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff391961640 1 -- 172.21.15.76:0/3930583755 learned_addr learned my addr 172.21.15.76:0/3930583755 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:48.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff39095f640 1 --2- 172.21.15.76:0/3930583755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c101040 0x7ff38c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:48.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff391160640 1 --2- 172.21.15.76:0/3930583755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff38c100680 0x7ff38c1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff391961640 1 -- 172.21.15.76:0/3930583755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff38c100680 msgr2=0x7ff38c1980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff391961640 1 --2- 172.21.15.76:0/3930583755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff38c100680 0x7ff38c1980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff391961640 1 -- 172.21.15.76:0/3930583755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c101040 msgr2=0x7ff38c198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff391961640 1 --2- 172.21.15.76:0/3930583755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c101040 0x7ff38c198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff391961640 1 -- 172.21.15.76:0/3930583755 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff38c19d0e0 con 0x7ff38c106690 2024-03-18T15:04:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff391961640 1 --2- 172.21.15.76:0/3930583755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff38c106690 0x7ff38c19c9b0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff38400eb10 tx=0x7ff38400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff3827fc640 1 -- 172.21.15.76:0/3930583755 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384004b20 con 0x7ff38c106690 2024-03-18T15:04:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff3827fc640 1 -- 172.21.15.76:0/3930583755 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff384004cc0 con 0x7ff38c106690 2024-03-18T15:04:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff3933c4640 1 -- 172.21.15.76:0/3930583755 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff38c19d3d0 con 0x7ff38c106690 2024-03-18T15:04:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff3827fc640 1 -- 172.21.15.76:0/3930583755 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384010760 con 0x7ff38c106690 2024-03-18T15:04:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.579+0000 7ff3933c4640 1 -- 172.21.15.76:0/3930583755 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff38c1a4c60 con 0x7ff38c106690 2024-03-18T15:04:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.580+0000 7ff3933c4640 1 -- 172.21.15.76:0/3930583755 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff38c101dc0 con 0x7ff38c106690 2024-03-18T15:04:48.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.582+0000 7ff3827fc640 1 -- 172.21.15.76:0/3930583755 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7ff384005000 con 0x7ff38c106690 2024-03-18T15:04:48.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.582+0000 7ff3827fc640 1 --2- 172.21.15.76:0/3930583755 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff3780a2970 0x7ff3780a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:48.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.582+0000 7ff391160640 1 --2- 172.21.15.76:0/3930583755 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff3780a2970 0x7ff3780a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:48.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.582+0000 7ff3827fc640 1 -- 172.21.15.76:0/3930583755 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7ff3840cf880 con 0x7ff38c106690 2024-03-18T15:04:48.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.583+0000 7ff391160640 1 --2- 172.21.15.76:0/3930583755 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff3780a2970 0x7ff3780a4e30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff388009b50 tx=0x7ff388005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:48.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.586+0000 7ff3827fc640 1 -- 172.21.15.76:0/3930583755 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff38409c600 con 0x7ff38c106690 2024-03-18T15:04:48.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.765+0000 7ff3933c4640 1 -- 172.21.15.76:0/3930583755 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff38c199440 con 0x7ff38c106690 2024-03-18T15:04:48.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.767+0000 7ff3827fc640 1 -- 172.21.15.76:0/3930583755 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) ==== 74+0+23049 (secure 0 0 0) 0x7ff38409bf00 con 0x7ff38c106690 2024-03-18T15:04:48.768 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T15:04:48.768 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":107,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T15:04:18.312749+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":106,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3010549595":"2024-03-19T14:53:17.537054+0000","172.21.15.76:6800/3010549595":"2024-03-19T14:53:17.537054+0000","172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/880938501":"2024-03-19T14:53:17.537054+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/2082819586":"2024-03-19T14:53:17.537054+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/3349561522":"2024-03-19T14:53:17.537054+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/788250061":"2024-03-19T14:53:17.537054+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:04:48.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.769+0000 7ff3933c4640 1 -- 172.21.15.76:0/3930583755 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff3780a2970 msgr2=0x7ff3780a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:48.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.769+0000 7ff3933c4640 1 --2- 172.21.15.76:0/3930583755 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff3780a2970 0x7ff3780a4e30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff388009b50 tx=0x7ff388005f90 comp rx=0 tx=0).stop 2024-03-18T15:04:48.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.769+0000 7ff3933c4640 1 -- 172.21.15.76:0/3930583755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff38c106690 msgr2=0x7ff38c19c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:48.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.769+0000 7ff3933c4640 1 --2- 172.21.15.76:0/3930583755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff38c106690 0x7ff38c19c9b0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff38400eb10 tx=0x7ff38400c2d0 comp rx=0 tx=0).stop 2024-03-18T15:04:48.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.769+0000 7ff3933c4640 1 -- 172.21.15.76:0/3930583755 shutdown_connections 2024-03-18T15:04:48.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.769+0000 7ff3933c4640 1 --2- 172.21.15.76:0/3930583755 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7ff3780a2970 0x7ff3780a4e30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:48.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.769+0000 7ff3933c4640 1 --2- 172.21.15.76:0/3930583755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff38c106690 0x7ff38c19c9b0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:48.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.769+0000 7ff3933c4640 1 --2- 172.21.15.76:0/3930583755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c101040 0x7ff38c198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:48.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.769+0000 7ff3933c4640 1 --2- 172.21.15.76:0/3930583755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff38c100680 0x7ff38c1980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:48.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.769+0000 7ff3933c4640 1 -- 172.21.15.76:0/3930583755 >> 172.21.15.76:0/3930583755 conn(0x7ff38c076a60 msgr2=0x7ff38c1044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:48.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.769+0000 7ff3933c4640 1 -- 172.21.15.76:0/3930583755 shutdown_connections 2024-03-18T15:04:48.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.769+0000 7ff3933c4640 1 -- 172.21.15.76:0/3930583755 wait complete. 2024-03-18T15:04:48.781 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:46:26.588384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:50:59.606852+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:51:00.465325+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 106, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T15:04:48.782 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:04:48.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.885+0000 7f9c016e8640 1 -- 172.21.15.76:0/3210118553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9bfc103340 msgr2=0x7f9bfc10b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:48.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.885+0000 7f9c016e8640 1 --2- 172.21.15.76:0/3210118553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9bfc103340 0x7f9bfc10b820 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0009900 tx=0x7f9bf002bcd0 comp rx=0 tx=0).stop 2024-03-18T15:04:48.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.885+0000 7f9c016e8640 1 -- 172.21.15.76:0/3210118553 shutdown_connections 2024-03-18T15:04:48.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.885+0000 7f9c016e8640 1 --2- 172.21.15.76:0/3210118553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bfc10bdf0 0x7f9bfc10e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:48.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.885+0000 7f9c016e8640 1 --2- 172.21.15.76:0/3210118553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9bfc103340 0x7f9bfc10b820 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:48.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.885+0000 7f9c016e8640 1 --2- 172.21.15.76:0/3210118553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bfc102a20 0x7f9bfc102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:48.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.885+0000 7f9c016e8640 1 -- 172.21.15.76:0/3210118553 >> 172.21.15.76:0/3210118553 conn(0x7f9bfc076a60 msgr2=0x7f9bfc076e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:48.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.885+0000 7f9c016e8640 1 -- 172.21.15.76:0/3210118553 shutdown_connections 2024-03-18T15:04:48.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.885+0000 7f9c016e8640 1 -- 172.21.15.76:0/3210118553 wait complete. 2024-03-18T15:04:48.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.886+0000 7f9c016e8640 1 Processor -- start 2024-03-18T15:04:48.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.886+0000 7f9c016e8640 1 -- start start 2024-03-18T15:04:48.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.886+0000 7f9c016e8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bfc102a20 0x7f9bfc198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:48.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.886+0000 7f9c016e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bfc103340 0x7f9bfc1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:48.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.886+0000 7f9c016e8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9bfc10bdf0 0x7f9bfc19c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:48.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.886+0000 7f9c016e8640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9bfc111b80 con 0x7f9bfc10bdf0 2024-03-18T15:04:48.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.886+0000 7f9c016e8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9bfc111a00 con 0x7f9bfc103340 2024-03-18T15:04:48.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.886+0000 7f9c016e8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9bfc111d00 con 0x7f9bfc102a20 2024-03-18T15:04:48.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.886+0000 7f9bfb7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9bfc10bdf0 0x7f9bfc19c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:48.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.886+0000 7f9bfa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bfc103340 0x7f9bfc1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:48.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.886+0000 7f9bfaffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bfc102a20 0x7f9bfc198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:48.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.886+0000 7f9bfb7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9bfc10bdf0 0x7f9bfc19c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56566/0 (socket says 172.21.15.76:56566) 2024-03-18T15:04:48.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.886+0000 7f9bfaffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bfc102a20 0x7f9bfc198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:54904/0 (socket says 172.21.15.76:54904) 2024-03-18T15:04:48.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.886+0000 7f9bfa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bfc103340 0x7f9bfc1985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.76:49414/0 (socket says 172.21.15.76:49414) 2024-03-18T15:04:48.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.886+0000 7f9bfb7fe640 1 -- 172.21.15.76:0/407461439 learned_addr learned my addr 172.21.15.76:0/407461439 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:48.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.887+0000 7f9bfb7fe640 1 -- 172.21.15.76:0/407461439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bfc102a20 msgr2=0x7f9bfc198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:48.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.887+0000 7f9bfb7fe640 1 --2- 172.21.15.76:0/407461439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bfc102a20 0x7f9bfc198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:48.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.887+0000 7f9bfb7fe640 1 -- 172.21.15.76:0/407461439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bfc103340 msgr2=0x7f9bfc1985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:48.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.887+0000 7f9bfb7fe640 1 --2- 172.21.15.76:0/407461439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bfc103340 0x7f9bfc1985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:48.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.887+0000 7f9bfb7fe640 1 -- 172.21.15.76:0/407461439 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bfc19d070 con 0x7f9bfc10bdf0 2024-03-18T15:04:48.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.887+0000 7f9bfb7fe640 1 --2- 172.21.15.76:0/407461439 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9bfc10bdf0 0x7f9bfc19c940 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f9bf400eb40 tx=0x7f9bf400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:48.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.887+0000 7f9be3fff640 1 -- 172.21.15.76:0/407461439 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4004b20 con 0x7f9bfc10bdf0 2024-03-18T15:04:48.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.887+0000 7f9c016e8640 1 -- 172.21.15.76:0/407461439 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bfc19d360 con 0x7f9bfc10bdf0 2024-03-18T15:04:48.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.887+0000 7f9be3fff640 1 -- 172.21.15.76:0/407461439 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bf4004cc0 con 0x7f9bfc10bdf0 2024-03-18T15:04:48.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.887+0000 7f9be3fff640 1 -- 172.21.15.76:0/407461439 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4010850 con 0x7f9bfc10bdf0 2024-03-18T15:04:48.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.887+0000 7f9c016e8640 1 -- 172.21.15.76:0/407461439 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bfc1a4bf0 con 0x7f9bfc10bdf0 2024-03-18T15:04:48.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.888+0000 7f9c016e8640 1 -- 172.21.15.76:0/407461439 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bfc06a5d0 con 0x7f9bfc10bdf0 2024-03-18T15:04:48.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.889+0000 7f9be3fff640 1 -- 172.21.15.76:0/407461439 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f9bf4005000 con 0x7f9bfc10bdf0 2024-03-18T15:04:48.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.890+0000 7f9be3fff640 1 --2- 172.21.15.76:0/407461439 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f9be40a2970 0x7f9be40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:48.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.890+0000 7f9bfaffd640 1 --2- 172.21.15.76:0/407461439 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f9be40a2970 0x7f9be40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:48.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.890+0000 7f9be3fff640 1 -- 172.21.15.76:0/407461439 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7f9bf40cf540 con 0x7f9bfc10bdf0 2024-03-18T15:04:48.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.890+0000 7f9bfaffd640 1 --2- 172.21.15.76:0/407461439 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f9be40a2970 0x7f9be40a4e30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0095f0 tx=0x7f9bec005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:48.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:48.893+0000 7f9be3fff640 1 -- 172.21.15.76:0/407461439 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9bf409c2c0 con 0x7f9bfc10bdf0 2024-03-18T15:04:49.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.067+0000 7f9c016e8640 1 -- 172.21.15.76:0/407461439 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9bfc1993d0 con 0x7f9bfc10bdf0 2024-03-18T15:04:49.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.067+0000 7f9be3fff640 1 -- 172.21.15.76:0/407461439 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) ==== 93+0+10 (secure 0 0 0) 0x7f9bf409bbc0 con 0x7f9bfc10bdf0 2024-03-18T15:04:49.068 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-18T15:04:49.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.069+0000 7f9c016e8640 1 -- 172.21.15.76:0/407461439 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f9be40a2970 msgr2=0x7f9be40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:49.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.069+0000 7f9c016e8640 1 --2- 172.21.15.76:0/407461439 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f9be40a2970 0x7f9be40a4e30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0095f0 tx=0x7f9bec005f90 comp rx=0 tx=0).stop 2024-03-18T15:04:49.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.069+0000 7f9c016e8640 1 -- 172.21.15.76:0/407461439 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9bfc10bdf0 msgr2=0x7f9bfc19c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:49.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.069+0000 7f9c016e8640 1 --2- 172.21.15.76:0/407461439 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9bfc10bdf0 0x7f9bfc19c940 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f9bf400eb40 tx=0x7f9bf400c2d0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.069+0000 7f9c016e8640 1 -- 172.21.15.76:0/407461439 shutdown_connections 2024-03-18T15:04:49.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.069+0000 7f9c016e8640 1 --2- 172.21.15.76:0/407461439 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f9be40a2970 0x7f9be40a4e30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.069+0000 7f9c016e8640 1 --2- 172.21.15.76:0/407461439 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9bfc10bdf0 0x7f9bfc19c940 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.069+0000 7f9c016e8640 1 --2- 172.21.15.76:0/407461439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bfc103340 0x7f9bfc1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.069+0000 7f9c016e8640 1 --2- 172.21.15.76:0/407461439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bfc102a20 0x7f9bfc198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.069+0000 7f9c016e8640 1 -- 172.21.15.76:0/407461439 >> 172.21.15.76:0/407461439 conn(0x7f9bfc076a60 msgr2=0x7f9bfc0fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:49.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.069+0000 7f9c016e8640 1 -- 172.21.15.76:0/407461439 shutdown_connections 2024-03-18T15:04:49.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.069+0000 7f9c016e8640 1 -- 172.21.15.76:0/407461439 wait complete. 2024-03-18T15:04:49.081 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T15:04:49.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.185+0000 7f54242a1640 1 -- 172.21.15.76:0/1276091196 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f541c107830 msgr2=0x7f541c107c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:49.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.185+0000 7f54242a1640 1 --2- 172.21.15.76:0/1276091196 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f541c107830 0x7f541c107c10 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f540c009930 tx=0x7f540c02bc30 comp rx=0 tx=0).stop 2024-03-18T15:04:49.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.185+0000 7f54242a1640 1 -- 172.21.15.76:0/1276091196 shutdown_connections 2024-03-18T15:04:49.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.185+0000 7f54242a1640 1 --2- 172.21.15.76:0/1276091196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f541c0693a0 0x7f541c10e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.185+0000 7f54242a1640 1 --2- 172.21.15.76:0/1276091196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f541c0689e0 0x7f541c068e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.185+0000 7f54242a1640 1 --2- 172.21.15.76:0/1276091196 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f541c107830 0x7f541c107c10 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.185+0000 7f54242a1640 1 -- 172.21.15.76:0/1276091196 >> 172.21.15.76:0/1276091196 conn(0x7f541c0777e0 msgr2=0x7f541c077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:49.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.185+0000 7f54242a1640 1 -- 172.21.15.76:0/1276091196 shutdown_connections 2024-03-18T15:04:49.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.185+0000 7f54242a1640 1 -- 172.21.15.76:0/1276091196 wait complete. 2024-03-18T15:04:49.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.185+0000 7f54242a1640 1 Processor -- start 2024-03-18T15:04:49.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.185+0000 7f54242a1640 1 -- start start 2024-03-18T15:04:49.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f54242a1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f541c0689e0 0x7f541c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:49.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f54242a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f541c0693a0 0x7f541c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:49.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f54242a1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f541c107830 0x7f541c19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:49.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f54242a1640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f541c111b40 con 0x7f541c0689e0 2024-03-18T15:04:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f54242a1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f541c1119c0 con 0x7f541c0693a0 2024-03-18T15:04:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f54242a1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f541c111cc0 con 0x7f541c107830 2024-03-18T15:04:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f542203d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f541c0689e0 0x7f541c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f542203d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f541c0689e0 0x7f541c198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56572/0 (socket says 172.21.15.76:56572) 2024-03-18T15:04:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f542203d640 1 -- 172.21.15.76:0/4229996178 learned_addr learned my addr 172.21.15.76:0/4229996178 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f542183c640 1 --2- 172.21.15.76:0/4229996178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f541c0693a0 0x7f541c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f542283e640 1 --2- 172.21.15.76:0/4229996178 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f541c107830 0x7f541c19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f542203d640 1 -- 172.21.15.76:0/4229996178 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f541c107830 msgr2=0x7f541c19c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f542203d640 1 --2- 172.21.15.76:0/4229996178 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f541c107830 0x7f541c19c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f542203d640 1 -- 172.21.15.76:0/4229996178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f541c0693a0 msgr2=0x7f541c198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f542203d640 1 --2- 172.21.15.76:0/4229996178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f541c0693a0 0x7f541c198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f542203d640 1 -- 172.21.15.76:0/4229996178 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541c19d010 con 0x7f541c0689e0 2024-03-18T15:04:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f542203d640 1 --2- 172.21.15.76:0/4229996178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f541c0689e0 0x7f541c198020 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f540c009230 tx=0x7f540c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f54137fe640 1 -- 172.21.15.76:0/4229996178 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c041070 con 0x7f541c0689e0 2024-03-18T15:04:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f54242a1640 1 -- 172.21.15.76:0/4229996178 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f541c19d2a0 con 0x7f541c0689e0 2024-03-18T15:04:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f54137fe640 1 -- 172.21.15.76:0/4229996178 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f540c0054e0 con 0x7f541c0689e0 2024-03-18T15:04:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f54137fe640 1 -- 172.21.15.76:0/4229996178 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c003c90 con 0x7f541c0689e0 2024-03-18T15:04:49.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.186+0000 7f54242a1640 1 -- 172.21.15.76:0/4229996178 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f541c1a4b90 con 0x7f541c0689e0 2024-03-18T15:04:49.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.187+0000 7f54242a1640 1 -- 172.21.15.76:0/4229996178 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f541c199370 con 0x7f541c0689e0 2024-03-18T15:04:49.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.189+0000 7f54137fe640 1 -- 172.21.15.76:0/4229996178 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f540c044070 con 0x7f541c0689e0 2024-03-18T15:04:49.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.189+0000 7f54137fe640 1 --2- 172.21.15.76:0/4229996178 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f54080a2970 0x7f54080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:49.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.189+0000 7f542183c640 1 --2- 172.21.15.76:0/4229996178 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f54080a2970 0x7f54080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:49.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.189+0000 7f54137fe640 1 -- 172.21.15.76:0/4229996178 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7f540c0f0cb0 con 0x7f541c0689e0 2024-03-18T15:04:49.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.190+0000 7f542183c640 1 --2- 172.21.15.76:0/4229996178 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f54080a2970 0x7f54080a4e30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f541c1000f0 tx=0x7f5418005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:49.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.193+0000 7f54137fe640 1 -- 172.21.15.76:0/4229996178 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f540c0bd4f0 con 0x7f541c0689e0 2024-03-18T15:04:49.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.367+0000 7f54242a1640 1 -- 172.21.15.76:0/4229996178 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f541c02d1b0 con 0x7f541c0689e0 2024-03-18T15:04:49.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.367+0000 7f54137fe640 1 -- 172.21.15.76:0/4229996178 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) ==== 104+0+11 (secure 0 0 0) 0x7f540c0c1590 con 0x7f541c0689e0 2024-03-18T15:04:49.368 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-18T15:04:49.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.369+0000 7f54242a1640 1 -- 172.21.15.76:0/4229996178 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f54080a2970 msgr2=0x7f54080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:49.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.369+0000 7f54242a1640 1 --2- 172.21.15.76:0/4229996178 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f54080a2970 0x7f54080a4e30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f541c1000f0 tx=0x7f5418005f90 comp rx=0 tx=0).stop 2024-03-18T15:04:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.369+0000 7f54242a1640 1 -- 172.21.15.76:0/4229996178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f541c0689e0 msgr2=0x7f541c198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.369+0000 7f54242a1640 1 --2- 172.21.15.76:0/4229996178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f541c0689e0 0x7f541c198020 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f540c009230 tx=0x7f540c004270 comp rx=0 tx=0).stop 2024-03-18T15:04:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.369+0000 7f54242a1640 1 -- 172.21.15.76:0/4229996178 shutdown_connections 2024-03-18T15:04:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.369+0000 7f54242a1640 1 --2- 172.21.15.76:0/4229996178 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f54080a2970 0x7f54080a4e30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.369+0000 7f54242a1640 1 --2- 172.21.15.76:0/4229996178 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f541c107830 0x7f541c19c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.369+0000 7f54242a1640 1 --2- 172.21.15.76:0/4229996178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f541c0693a0 0x7f541c198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.369+0000 7f54242a1640 1 --2- 172.21.15.76:0/4229996178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f541c0689e0 0x7f541c198020 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.369+0000 7f54242a1640 1 -- 172.21.15.76:0/4229996178 >> 172.21.15.76:0/4229996178 conn(0x7f541c0777e0 msgr2=0x7f541c0fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.369+0000 7f54242a1640 1 -- 172.21.15.76:0/4229996178 shutdown_connections 2024-03-18T15:04:49.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.369+0000 7f54242a1640 1 -- 172.21.15.76:0/4229996178 wait complete. 2024-03-18T15:04:49.382 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T15:04:49.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.486+0000 7f259b753640 1 -- 172.21.15.76:0/550115670 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2594101750 msgr2=0x7f2594101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:49.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.486+0000 7f259b753640 1 --2- 172.21.15.76:0/550115670 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2594101750 0x7f2594101bd0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f2590009930 tx=0x7f259002bc30 comp rx=0 tx=0).stop 2024-03-18T15:04:49.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.486+0000 7f259b753640 1 -- 172.21.15.76:0/550115670 shutdown_connections 2024-03-18T15:04:49.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.486+0000 7f259b753640 1 --2- 172.21.15.76:0/550115670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2594102110 0x7f259410e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.486+0000 7f259b753640 1 --2- 172.21.15.76:0/550115670 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2594101750 0x7f2594101bd0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.486+0000 7f259b753640 1 --2- 172.21.15.76:0/550115670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2594107760 0x7f2594107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.486+0000 7f259b753640 1 -- 172.21.15.76:0/550115670 >> 172.21.15.76:0/550115670 conn(0x7f25940fd240 msgr2=0x7f25940ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:49.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.486+0000 7f259b753640 1 -- 172.21.15.76:0/550115670 shutdown_connections 2024-03-18T15:04:49.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.486+0000 7f259b753640 1 -- 172.21.15.76:0/550115670 wait complete. 2024-03-18T15:04:49.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f259b753640 1 Processor -- start 2024-03-18T15:04:49.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f259b753640 1 -- start start 2024-03-18T15:04:49.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f259b753640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2594101750 0x7f2594198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:49.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f259b753640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2594102110 0x7f25941985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:49.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f259b753640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2594107760 0x7f259419c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:49.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f259b753640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2594111b30 con 0x7f2594107760 2024-03-18T15:04:49.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f259b753640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f25941119b0 con 0x7f2594102110 2024-03-18T15:04:49.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f259b753640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2594111cb0 con 0x7f2594101750 2024-03-18T15:04:49.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f2599cf0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2594107760 0x7f259419c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:49.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f2599cf0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2594107760 0x7f259419c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56602/0 (socket says 172.21.15.76:56602) 2024-03-18T15:04:49.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f2599cf0640 1 -- 172.21.15.76:0/3786503373 learned_addr learned my addr 172.21.15.76:0/3786503373 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:49.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f25994ef640 1 --2- 172.21.15.76:0/3786503373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2594101750 0x7f2594198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:49.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f2598cee640 1 --2- 172.21.15.76:0/3786503373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2594102110 0x7f25941985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:49.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f2599cf0640 1 -- 172.21.15.76:0/3786503373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2594101750 msgr2=0x7f2594198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:49.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f2599cf0640 1 --2- 172.21.15.76:0/3786503373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2594101750 0x7f2594198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f2599cf0640 1 -- 172.21.15.76:0/3786503373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2594102110 msgr2=0x7f25941985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:49.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f2599cf0640 1 --2- 172.21.15.76:0/3786503373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2594102110 0x7f25941985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f2599cf0640 1 -- 172.21.15.76:0/3786503373 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f259419d080 con 0x7f2594107760 2024-03-18T15:04:49.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f25994ef640 1 --2- 172.21.15.76:0/3786503373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2594101750 0x7f2594198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:04:49.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f2599cf0640 1 --2- 172.21.15.76:0/3786503373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2594107760 0x7f259419c950 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f258c00eb10 tx=0x7f258c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:49.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.487+0000 7f258a7fc640 1 -- 172.21.15.76:0/3786503373 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c004b20 con 0x7f2594107760 2024-03-18T15:04:49.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.488+0000 7f259b753640 1 -- 172.21.15.76:0/3786503373 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f259419d370 con 0x7f2594107760 2024-03-18T15:04:49.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.488+0000 7f258a7fc640 1 -- 172.21.15.76:0/3786503373 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f258c004cc0 con 0x7f2594107760 2024-03-18T15:04:49.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.488+0000 7f258a7fc640 1 -- 172.21.15.76:0/3786503373 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c010760 con 0x7f2594107760 2024-03-18T15:04:49.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.488+0000 7f259b753640 1 -- 172.21.15.76:0/3786503373 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25941a4c00 con 0x7f2594107760 2024-03-18T15:04:49.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.489+0000 7f259b753640 1 -- 172.21.15.76:0/3786503373 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2594068a20 con 0x7f2594107760 2024-03-18T15:04:49.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.490+0000 7f258a7fc640 1 -- 172.21.15.76:0/3786503373 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f258c010900 con 0x7f2594107760 2024-03-18T15:04:49.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.490+0000 7f258a7fc640 1 --2- 172.21.15.76:0/3786503373 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f25800a2970 0x7f25800a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:49.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.490+0000 7f25994ef640 1 --2- 172.21.15.76:0/3786503373 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f25800a2970 0x7f25800a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:49.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.490+0000 7f258a7fc640 1 -- 172.21.15.76:0/3786503373 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7f258c0cf5f0 con 0x7f2594107760 2024-03-18T15:04:49.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.491+0000 7f25994ef640 1 --2- 172.21.15.76:0/3786503373 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f25800a2970 0x7f25800a4e30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2584009f40 tx=0x7f2584005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:49.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.494+0000 7f258a7fc640 1 -- 172.21.15.76:0/3786503373 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f258c053740 con 0x7f2594107760 2024-03-18T15:04:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:49 smithi076 ceph-mon[32205]: pgmap v366: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 299 KiB/s wr, 22 op/s 2024-03-18T15:04:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:49 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3930583755' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:04:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:49 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/407461439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:04:49.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:49 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/4229996178' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:04:49.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.669+0000 7f259b753640 1 -- 172.21.15.76:0/3786503373 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f259402d1b0 con 0x7f2594107760 2024-03-18T15:04:49.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.669+0000 7f258a7fc640 1 -- 172.21.15.76:0/3786503373 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) ==== 100+0+11 (secure 0 0 0) 0x7f258c086020 con 0x7f2594107760 2024-03-18T15:04:49.670 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-18T15:04:49.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.672+0000 7f259b753640 1 -- 172.21.15.76:0/3786503373 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f25800a2970 msgr2=0x7f25800a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:49.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.672+0000 7f259b753640 1 --2- 172.21.15.76:0/3786503373 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f25800a2970 0x7f25800a4e30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2584009f40 tx=0x7f2584005f90 comp rx=0 tx=0).stop 2024-03-18T15:04:49.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.672+0000 7f259b753640 1 -- 172.21.15.76:0/3786503373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2594107760 msgr2=0x7f259419c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:49.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.672+0000 7f259b753640 1 --2- 172.21.15.76:0/3786503373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2594107760 0x7f259419c950 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f258c00eb10 tx=0x7f258c00c2d0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.672+0000 7f259b753640 1 -- 172.21.15.76:0/3786503373 shutdown_connections 2024-03-18T15:04:49.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.672+0000 7f259b753640 1 --2- 172.21.15.76:0/3786503373 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f25800a2970 0x7f25800a4e30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.672+0000 7f259b753640 1 --2- 172.21.15.76:0/3786503373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2594107760 0x7f259419c950 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.672+0000 7f259b753640 1 --2- 172.21.15.76:0/3786503373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2594102110 0x7f25941985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.672+0000 7f259b753640 1 --2- 172.21.15.76:0/3786503373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2594101750 0x7f2594198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.672+0000 7f259b753640 1 -- 172.21.15.76:0/3786503373 >> 172.21.15.76:0/3786503373 conn(0x7f25940fd240 msgr2=0x7f25940fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:49.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.672+0000 7f259b753640 1 -- 172.21.15.76:0/3786503373 shutdown_connections 2024-03-18T15:04:49.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.672+0000 7f259b753640 1 -- 172.21.15.76:0/3786503373 wait complete. 2024-03-18T15:04:49.683 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T15:04:49.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:49 smithi174 ceph-mon[36787]: pgmap v366: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 299 KiB/s wr, 22 op/s 2024-03-18T15:04:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:49 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3930583755' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:04:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:49 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/407461439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:04:49.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:49 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/4229996178' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:04:49.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:49 smithi077 ceph-mon[35865]: pgmap v366: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 299 KiB/s wr, 22 op/s 2024-03-18T15:04:49.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:49 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3930583755' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:04:49.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:49 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/407461439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:04:49.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:49 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/4229996178' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:04:49.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.786+0000 7febccf89640 1 -- 172.21.15.76:0/1423918528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febc8100670 msgr2=0x7febc8100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:49.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.786+0000 7febccf89640 1 --2- 172.21.15.76:0/1423918528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febc8100670 0x7febc8100af0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7febb0009860 tx=0x7febb002bb70 comp rx=0 tx=0).stop 2024-03-18T15:04:49.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.786+0000 7febccf89640 1 -- 172.21.15.76:0/1423918528 shutdown_connections 2024-03-18T15:04:49.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.786+0000 7febccf89640 1 --2- 172.21.15.76:0/1423918528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febc8101030 0x7febc810d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.786+0000 7febccf89640 1 --2- 172.21.15.76:0/1423918528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febc8100670 0x7febc8100af0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.786+0000 7febccf89640 1 --2- 172.21.15.76:0/1423918528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febc8106680 0x7febc8106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.786+0000 7febccf89640 1 -- 172.21.15.76:0/1423918528 >> 172.21.15.76:0/1423918528 conn(0x7febc80777b0 msgr2=0x7febc8077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:49.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.786+0000 7febccf89640 1 -- 172.21.15.76:0/1423918528 shutdown_connections 2024-03-18T15:04:49.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.786+0000 7febccf89640 1 -- 172.21.15.76:0/1423918528 wait complete. 2024-03-18T15:04:49.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febccf89640 1 Processor -- start 2024-03-18T15:04:49.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febccf89640 1 -- start start 2024-03-18T15:04:49.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febccf89640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febc8100670 0x7febc8198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:49.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febccf89640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febc8101030 0x7febc8198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:49.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febccf89640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febc8106680 0x7febc819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:49.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febccf89640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7febc8111b90 con 0x7febc8101030 2024-03-18T15:04:49.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febccf89640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7febc8111a10 con 0x7febc8106680 2024-03-18T15:04:49.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febccf89640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7febc8111d10 con 0x7febc8100670 2024-03-18T15:04:49.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febbffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febc8101030 0x7febc8198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:49.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febbffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febc8101030 0x7febc8198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56610/0 (socket says 172.21.15.76:56610) 2024-03-18T15:04:49.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febbffff640 1 -- 172.21.15.76:0/3473998909 learned_addr learned my addr 172.21.15.76:0/3473998909 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:49.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febc659c640 1 --2- 172.21.15.76:0/3473998909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febc8100670 0x7febc8198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:49.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febc6d9d640 1 --2- 172.21.15.76:0/3473998909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febc8106680 0x7febc819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:49.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febbffff640 1 -- 172.21.15.76:0/3473998909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febc8100670 msgr2=0x7febc8198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:49.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febbffff640 1 --2- 172.21.15.76:0/3473998909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febc8100670 0x7febc8198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febbffff640 1 -- 172.21.15.76:0/3473998909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febc8106680 msgr2=0x7febc819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:49.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febbffff640 1 --2- 172.21.15.76:0/3473998909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febc8106680 0x7febc819c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febbffff640 1 -- 172.21.15.76:0/3473998909 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febc819d020 con 0x7febc8101030 2024-03-18T15:04:49.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.787+0000 7febc6d9d640 1 --2- 172.21.15.76:0/3473998909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febc8106680 0x7febc819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:04:49.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.788+0000 7febbffff640 1 --2- 172.21.15.76:0/3473998909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febc8101030 0x7febc8198570 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7febb0037040 tx=0x7febb00045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:49.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.788+0000 7febbf7fe640 1 -- 172.21.15.76:0/3473998909 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb0041020 con 0x7febc8101030 2024-03-18T15:04:49.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.788+0000 7febccf89640 1 -- 172.21.15.76:0/3473998909 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febc819d2b0 con 0x7febc8101030 2024-03-18T15:04:49.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.788+0000 7febbf7fe640 1 -- 172.21.15.76:0/3473998909 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febb002c9f0 con 0x7febc8101030 2024-03-18T15:04:49.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.788+0000 7febccf89640 1 -- 172.21.15.76:0/3473998909 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febc81a4ba0 con 0x7febc8101030 2024-03-18T15:04:49.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.788+0000 7febbf7fe640 1 -- 172.21.15.76:0/3473998909 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb0039040 con 0x7febc8101030 2024-03-18T15:04:49.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.789+0000 7febccf89640 1 -- 172.21.15.76:0/3473998909 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febc8101db0 con 0x7febc8101030 2024-03-18T15:04:49.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.790+0000 7febbf7fe640 1 -- 172.21.15.76:0/3473998909 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7febb0044020 con 0x7febc8101030 2024-03-18T15:04:49.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.791+0000 7febbf7fe640 1 --2- 172.21.15.76:0/3473998909 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7febb40a2970 0x7febb40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:49.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.791+0000 7febc659c640 1 --2- 172.21.15.76:0/3473998909 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7febb40a2970 0x7febb40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:49.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.791+0000 7febbf7fe640 1 -- 172.21.15.76:0/3473998909 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7febb00f0990 con 0x7febc8101030 2024-03-18T15:04:49.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.791+0000 7febc659c640 1 --2- 172.21.15.76:0/3473998909 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7febb40a2970 0x7febb40a4e30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7febb80096c0 tx=0x7febb8009260 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:49.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.794+0000 7febbf7fe640 1 -- 172.21.15.76:0/3473998909 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7febb004b020 con 0x7febc8101030 2024-03-18T15:04:49.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.982+0000 7febccf89640 1 -- 172.21.15.76:0/3473998909 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7febc8199380 con 0x7febc8101030 2024-03-18T15:04:49.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.983+0000 7febbf7fe640 1 -- 172.21.15.76:0/3473998909 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v107) ==== 81+0+939 (secure 0 0 0) 0x7febb00bd820 con 0x7febc8101030 2024-03-18T15:04:49.984 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6801/3010549595 2024-03-19T14:53:17.537054+0000 2024-03-18T15:04:49.984 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6800/3010549595 2024-03-19T14:53:17.537054+0000 2024-03-18T15:04:49.984 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/3122154733 2024-03-19T14:44:06.251768+0000 2024-03-18T15:04:49.984 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/880938501 2024-03-19T14:53:17.537054+0000 2024-03-18T15:04:49.984 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/752916178 2024-03-19T14:44:06.251768+0000 2024-03-18T15:04:49.985 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/3261047930 2024-03-19T14:44:06.251768+0000 2024-03-18T15:04:49.985 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6801/3054039863 2024-03-19T14:44:06.251768+0000 2024-03-18T15:04:49.985 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/2082819586 2024-03-19T14:53:17.537054+0000 2024-03-18T15:04:49.985 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/600153683 2024-03-19T14:43:39.730762+0000 2024-03-18T15:04:49.985 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6801/216126402 2024-03-19T14:43:39.730762+0000 2024-03-18T15:04:49.985 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/1895213347 2024-03-19T14:43:39.730762+0000 2024-03-18T15:04:49.985 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/3349561522 2024-03-19T14:53:17.537054+0000 2024-03-18T15:04:49.985 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/918193033 2024-03-19T14:43:39.730762+0000 2024-03-18T15:04:49.985 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/788250061 2024-03-19T14:53:17.537054+0000 2024-03-18T15:04:49.986 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6800/216126402 2024-03-19T14:43:39.730762+0000 2024-03-18T15:04:49.986 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6800/3054039863 2024-03-19T14:44:06.251768+0000 2024-03-18T15:04:49.987 INFO:teuthology.orchestra.run.smithi076.stderr:listed 16 entries 2024-03-18T15:04:49.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.985+0000 7febccf89640 1 -- 172.21.15.76:0/3473998909 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7febb40a2970 msgr2=0x7febb40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:49.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.985+0000 7febccf89640 1 --2- 172.21.15.76:0/3473998909 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7febb40a2970 0x7febb40a4e30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7febb80096c0 tx=0x7febb8009260 comp rx=0 tx=0).stop 2024-03-18T15:04:49.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.985+0000 7febccf89640 1 -- 172.21.15.76:0/3473998909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febc8101030 msgr2=0x7febc8198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:49.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.985+0000 7febccf89640 1 --2- 172.21.15.76:0/3473998909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febc8101030 0x7febc8198570 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7febb0037040 tx=0x7febb00045b0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.985+0000 7febccf89640 1 -- 172.21.15.76:0/3473998909 shutdown_connections 2024-03-18T15:04:49.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.985+0000 7febccf89640 1 --2- 172.21.15.76:0/3473998909 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7febb40a2970 0x7febb40a4e30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.985+0000 7febccf89640 1 --2- 172.21.15.76:0/3473998909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febc8106680 0x7febc819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.985+0000 7febccf89640 1 --2- 172.21.15.76:0/3473998909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febc8101030 0x7febc8198570 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.985+0000 7febccf89640 1 --2- 172.21.15.76:0/3473998909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febc8100670 0x7febc8198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:49.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.985+0000 7febccf89640 1 -- 172.21.15.76:0/3473998909 >> 172.21.15.76:0/3473998909 conn(0x7febc80777b0 msgr2=0x7febc8102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:49.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.985+0000 7febccf89640 1 -- 172.21.15.76:0/3473998909 shutdown_connections 2024-03-18T15:04:49.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:49.985+0000 7febccf89640 1 -- 172.21.15.76:0/3473998909 wait complete. 2024-03-18T15:04:49.996 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-18T15:04:49.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:04:49.997 DEBUG:teuthology.orchestra.run.smithi076:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-18T15:04:50.162 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi076.front.sepia.ceph.com 2024-03-18T15:04:50.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:04:50.163 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T15:04:50.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:04:50.213 DEBUG:teuthology.orchestra.run.smithi076:> ip netns list 2024-03-18T15:04:50.242 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T15:04:50.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:04:50.243 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T15:04:50.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:04:50.316 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link delete ceph-brx 2024-03-18T15:04:50.420 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-18T15:04:50.420 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-18T15:04:50.420 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T15:04:50.436 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=304 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/6cd870b55aa2b70df50459a689762736c3b4f2750bac2753f65db1a190f336c1/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3JTECNWC757AUEZZHLENSOXWZ:/var/lib/containers/storage/overlay/l/PUWLUYZR5CQOAMCTA33M7CFN5G,upperdir=/var/lib/containers/storage/overlay/6cd870b55aa2b70df50459a689762736c3b4f2750bac2753f65db1a190f336c1/diff,workdir=/var/lib/containers/storage/overlay/6cd870b55aa2b70df50459a689762736c3b4f2750bac2753f65db1a190f336c1/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/8ff4c585f8009b271cb954f8a4bd81238086976e59c06d9d20f600332b6a62f5/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3JTECNWC757AUEZZHLENSOXWZ:/var/lib/containers/storage/overlay/l/PUWLUYZR5CQOAMCTA33M7CFN5G,upperdir=/var/lib/containers/storage/overlay/8ff4c585f8009b271cb954f8a4bd81238086976e59c06d9d20f600332b6a62f5/diff,workdir=/var/lib/containers/storage/overlay/8ff4c585f8009b271cb954f8a4bd81238086976e59c06d9d20f600332b6a62f5/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/247e7c10d26db57974eeff15eef9dffc5cda52e9f64849524c6e34a3ae0f088f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3JTECNWC757AUEZZHLENSOXWZ:/var/lib/containers/storage/overlay/l/PUWLUYZR5CQOAMCTA33M7CFN5G,upperdir=/var/lib/containers/storage/overlay/247e7c10d26db57974eeff15eef9dffc5cda52e9f64849524c6e34a3ae0f088f/diff,workdir=/var/lib/containers/storage/overlay/247e7c10d26db57974eeff15eef9dffc5cda52e9f64849524c6e34a3ae0f088f/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/9fcded519b91d04397ba11bf1d375832e44fe832550d84796c1c59b4af2421c9/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3JTECNWC757AUEZZHLENSOXWZ:/var/lib/containers/storage/overlay/l/PUWLUYZR5CQOAMCTA33M7CFN5G,upperdir=/var/lib/containers/storage/overlay/9fcded519b91d04397ba11bf1d375832e44fe832550d84796c1c59b4af2421c9/diff,workdir=/var/lib/containers/storage/overlay/9fcded519b91d04397ba11bf1d375832e44fe832550d84796c1c59b4af2421c9/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/e05c89c6d48f206e434e3fd9c6b5048d44b30eb6a92e4acf74d491354135043f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3JTECNWC757AUEZZHLENSOXWZ:/var/lib/containers/storage/overlay/l/PUWLUYZR5CQOAMCTA33M7CFN5G,upperdir=/var/lib/containers/storage/overlay/e05c89c6d48f206e434e3fd9c6b5048d44b30eb6a92e4acf74d491354135043f/diff,workdir=/var/lib/containers/storage/overlay/e05c89c6d48f206e434e3fd9c6b5048d44b30eb6a92e4acf74d491354135043f/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d87a6c0acf6b3dd0284682c4fa6e61946addc33fac7e182b14a1fc056632f57c/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3JTECNWC757AUEZZHLENSOXWZ:/var/lib/containers/storage/overlay/l/PUWLUYZR5CQOAMCTA33M7CFN5G,upperdir=/var/lib/containers/storage/overlay/d87a6c0acf6b3dd0284682c4fa6e61946addc33fac7e182b14a1fc056632f57c/diff,workdir=/var/lib/containers/storage/overlay/d87a6c0acf6b3dd0284682c4fa6e61946addc33fac7e182b14a1fc056632f57c/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/353326057f30eaabedd1dd0f5978a59359399ed00716bccf0830807a8004d3be/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3JTECNWC757AUEZZHLENSOXWZ:/var/lib/containers/storage/overlay/l/PUWLUYZR5CQOAMCTA33M7CFN5G,upperdir=/var/lib/containers/storage/overlay/353326057f30eaabedd1dd0f5978a59359399ed00716bccf0830807a8004d3be/diff,workdir=/var/lib/containers/storage/overlay/353326057f30eaabedd1dd0f5978a59359399ed00716bccf0830807a8004d3be/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/dbd4cf389558edbbb4f0f4d8f2b9a483dac4ac4b03fa9f79b058a0ecb628e547/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3JTECNWC757AUEZZHLENSOXWZ:/var/lib/containers/storage/overlay/l/PUWLUYZR5CQOAMCTA33M7CFN5G,upperdir=/var/lib/containers/storage/overlay/dbd4cf389558edbbb4f0f4d8f2b9a483dac4ac4b03fa9f79b058a0ecb628e547/diff,workdir=/var/lib/containers/storage/overlay/dbd4cf389558edbbb4f0f4d8f2b9a483dac4ac4b03fa9f79b058a0ecb628e547/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/c10508f6d5bb183fefa44b86b7e440bd4eba9aff4e1fce41b87de9ccc4935952/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3JTECNWC757AUEZZHLENSOXWZ:/var/lib/containers/storage/overlay/l/PUWLUYZR5CQOAMCTA33M7CFN5G,upperdir=/var/lib/containers/storage/overlay/c10508f6d5bb183fefa44b86b7e440bd4eba9aff4e1fce41b87de9ccc4935952/diff,workdir=/var/lib/containers/storage/overlay/c10508f6d5bb183fefa44b86b7e440bd4eba9aff4e1fce41b87de9ccc4935952/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/5ff66536dd2d17343e0bc9fda82d09d40bff4ee64a08bb6020f0bb3bbcc78a0f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3JTECNWC757AUEZZHLENSOXWZ:/var/lib/containers/storage/overlay/l/PUWLUYZR5CQOAMCTA33M7CFN5G,upperdir=/var/lib/containers/storage/overlay/5ff66536dd2d17343e0bc9fda82d09d40bff4ee64a08bb6020f0bb3bbcc78a0f/diff,workdir=/var/lib/containers/storage/overlay/5ff66536dd2d17343e0bc9fda82d09d40bff4ee64a08bb6020f0bb3bbcc78a0f/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 2024-03-18T15:04:50.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:04:50.436 DEBUG:teuthology.orchestra.run.smithi076:> ip netns list 2024-03-18T15:04:50.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:04:50.492 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link delete ceph-brx 2024-03-18T15:04:50.560 INFO:teuthology.orchestra.run.smithi076.stderr:Cannot find device "ceph-brx" 2024-03-18T15:04:50.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:04:50.562 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-18T15:04:50.573 INFO:tasks.cephadm:Teardown begin 2024-03-18T15:04:50.574 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T15:04:50.574 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:04:50.625 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:50 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3786503373' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:04:50.626 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:50 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3473998909' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T15:04:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.704+0000 7fe691553640 1 -- 172.21.15.76:0/1415719532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe68c101080 msgr2=0x7fe68c10d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.704+0000 7fe691553640 1 --2- 172.21.15.76:0/1415719532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe68c101080 0x7fe68c10d540 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe684009be0 tx=0x7fe68402bf80 comp rx=0 tx=0).stop 2024-03-18T15:04:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.704+0000 7fe691553640 1 -- 172.21.15.76:0/1415719532 shutdown_connections 2024-03-18T15:04:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.704+0000 7fe691553640 1 --2- 172.21.15.76:0/1415719532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe68c101080 0x7fe68c10d540 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.704+0000 7fe691553640 1 --2- 172.21.15.76:0/1415719532 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe68c1006e0 0x7fe68c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.704+0000 7fe691553640 1 --2- 172.21.15.76:0/1415719532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe68c1055c0 0x7fe68c1059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.704+0000 7fe691553640 1 -- 172.21.15.76:0/1415719532 >> 172.21.15.76:0/1415719532 conn(0x7fe68c077810 msgr2=0x7fe68c077c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.704+0000 7fe691553640 1 -- 172.21.15.76:0/1415719532 shutdown_connections 2024-03-18T15:04:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.704+0000 7fe691553640 1 -- 172.21.15.76:0/1415719532 wait complete. 2024-03-18T15:04:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.704+0000 7fe691553640 1 Processor -- start 2024-03-18T15:04:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe691553640 1 -- start start 2024-03-18T15:04:50.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe691553640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe68c1006e0 0x7fe68c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:50.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe691553640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe68c101080 0x7fe68c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:50.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe691553640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe68c1055c0 0x7fe68c19c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:50.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe691553640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c111bd0 con 0x7fe68c1006e0 2024-03-18T15:04:50.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe691553640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c111a50 con 0x7fe68c101080 2024-03-18T15:04:50.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe691553640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c111d50 con 0x7fe68c1055c0 2024-03-18T15:04:50.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe68affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe68c1006e0 0x7fe68c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:50.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe68affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe68c1006e0 0x7fe68c197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56626/0 (socket says 172.21.15.76:56626) 2024-03-18T15:04:50.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe68affd640 1 -- 172.21.15.76:0/1199602476 learned_addr learned my addr 172.21.15.76:0/1199602476 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:50.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe68a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe68c101080 0x7fe68c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:50.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe68b7fe640 1 --2- 172.21.15.76:0/1199602476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe68c1055c0 0x7fe68c19c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:50.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe68affd640 1 -- 172.21.15.76:0/1199602476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe68c1055c0 msgr2=0x7fe68c19c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:50.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe68affd640 1 --2- 172.21.15.76:0/1199602476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe68c1055c0 0x7fe68c19c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:50.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe68affd640 1 -- 172.21.15.76:0/1199602476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe68c101080 msgr2=0x7fe68c198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:50.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe68affd640 1 --2- 172.21.15.76:0/1199602476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe68c101080 0x7fe68c198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:50.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe68affd640 1 -- 172.21.15.76:0/1199602476 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68c19cfe0 con 0x7fe68c1006e0 2024-03-18T15:04:50.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe68a7fc640 1 --2- 172.21.15.76:0/1199602476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe68c101080 0x7fe68c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:04:50.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe68affd640 1 --2- 172.21.15.76:0/1199602476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe68c1006e0 0x7fe68c197ff0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fe67c00e9f0 tx=0x7fe67c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:50.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe673fff640 1 -- 172.21.15.76:0/1199602476 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67c00cd80 con 0x7fe68c1006e0 2024-03-18T15:04:50.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe691553640 1 -- 172.21.15.76:0/1199602476 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe68c19d2d0 con 0x7fe68c1006e0 2024-03-18T15:04:50.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe673fff640 1 -- 172.21.15.76:0/1199602476 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe67c010910 con 0x7fe68c1006e0 2024-03-18T15:04:50.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.705+0000 7fe673fff640 1 -- 172.21.15.76:0/1199602476 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67c0188f0 con 0x7fe68c1006e0 2024-03-18T15:04:50.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.706+0000 7fe691553640 1 -- 172.21.15.76:0/1199602476 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe68c1a4b60 con 0x7fe68c1006e0 2024-03-18T15:04:50.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.707+0000 7fe691553640 1 -- 172.21.15.76:0/1199602476 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe68c067d40 con 0x7fe68c1006e0 2024-03-18T15:04:50.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.708+0000 7fe673fff640 1 -- 172.21.15.76:0/1199602476 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fe67c010ac0 con 0x7fe68c1006e0 2024-03-18T15:04:50.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.708+0000 7fe673fff640 1 --2- 172.21.15.76:0/1199602476 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe6740a2970 0x7fe6740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:50.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.708+0000 7fe68a7fc640 1 --2- 172.21.15.76:0/1199602476 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe6740a2970 0x7fe6740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:50.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.708+0000 7fe673fff640 1 -- 172.21.15.76:0/1199602476 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7fe67c0cf0b0 con 0x7fe68c1006e0 2024-03-18T15:04:50.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.709+0000 7fe68a7fc640 1 --2- 172.21.15.76:0/1199602476 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe6740a2970 0x7fe6740a4e30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe6800096c0 tx=0x7fe680009260 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:50.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.713+0000 7fe673fff640 1 -- 172.21.15.76:0/1199602476 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe67c09b880 con 0x7fe68c1006e0 2024-03-18T15:04:50.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:50 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3786503373' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:04:50.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:50 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3473998909' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T15:04:50.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:50 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3786503373' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:04:50.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:50 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3473998909' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T15:04:50.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.926+0000 7fe691553640 1 -- 172.21.15.76:0/1199602476 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe68c02d1b0 con 0x7fe68c1006e0 2024-03-18T15:04:50.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.927+0000 7fe673fff640 1 -- 172.21.15.76:0/1199602476 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11865 (secure 0 0 0) 0x7fe67c09f920 con 0x7fe68c1006e0 2024-03-18T15:04:50.927 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T15:04:50.928 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/2801069369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":2801069369},{"type":"v1","addr":"172.21.15.76:6839","nonce":2801069369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2645229232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2645229232},{"type":"v1","addr":"172.21.15.76:6841","nonce":2645229232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":44,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T14:51:01.293009+0000","modified":"2024-03-18T15:03:54.767975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24490,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":56,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":55,"addr":"172.21.15.174:6839/399810652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":399810652},{"type":"v1","addr":"172.21.15.174:6839","nonce":399810652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1685041985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1685041985},{"type":"v1","addr":"172.21.15.77:6839","nonce":1685041985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26585":{"gid":26585,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/2933679925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2933679925},{"type":"v1","addr":"172.21.15.77:6835","nonce":2933679925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24463,24490,24524]},"id":1}]} 2024-03-18T15:04:50.928 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 44 2024-03-18T15:04:50.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.929+0000 7fe691553640 1 -- 172.21.15.76:0/1199602476 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe6740a2970 msgr2=0x7fe6740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:50.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.929+0000 7fe691553640 1 --2- 172.21.15.76:0/1199602476 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe6740a2970 0x7fe6740a4e30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe6800096c0 tx=0x7fe680009260 comp rx=0 tx=0).stop 2024-03-18T15:04:50.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.929+0000 7fe691553640 1 -- 172.21.15.76:0/1199602476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe68c1006e0 msgr2=0x7fe68c197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:50.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.929+0000 7fe691553640 1 --2- 172.21.15.76:0/1199602476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe68c1006e0 0x7fe68c197ff0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fe67c00e9f0 tx=0x7fe67c00eeb0 comp rx=0 tx=0).stop 2024-03-18T15:04:50.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.929+0000 7fe691553640 1 -- 172.21.15.76:0/1199602476 shutdown_connections 2024-03-18T15:04:50.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.929+0000 7fe691553640 1 --2- 172.21.15.76:0/1199602476 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fe6740a2970 0x7fe6740a4e30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:50.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.929+0000 7fe691553640 1 --2- 172.21.15.76:0/1199602476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe68c1055c0 0x7fe68c19c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:50.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.929+0000 7fe691553640 1 --2- 172.21.15.76:0/1199602476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe68c101080 0x7fe68c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:50.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.929+0000 7fe691553640 1 --2- 172.21.15.76:0/1199602476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe68c1006e0 0x7fe68c197ff0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:50.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.929+0000 7fe691553640 1 -- 172.21.15.76:0/1199602476 >> 172.21.15.76:0/1199602476 conn(0x7fe68c077810 msgr2=0x7fe68c103260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:50.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.929+0000 7fe691553640 1 -- 172.21.15.76:0/1199602476 shutdown_connections 2024-03-18T15:04:50.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:50.929+0000 7fe691553640 1 -- 172.21.15.76:0/1199602476 wait complete. 2024-03-18T15:04:50.940 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:04:51.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.081+0000 7faca6fa9640 1 -- 172.21.15.76:0/405021139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faca0100f80 msgr2=0x7faca010dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:51.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.081+0000 7faca6fa9640 1 --2- 172.21.15.76:0/405021139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faca0100f80 0x7faca010dc40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fac94009860 tx=0x7fac9402bb70 comp rx=0 tx=0).stop 2024-03-18T15:04:51.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.081+0000 7faca6fa9640 1 -- 172.21.15.76:0/405021139 shutdown_connections 2024-03-18T15:04:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.081+0000 7faca6fa9640 1 --2- 172.21.15.76:0/405021139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faca010e180 0x7faca0110570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.081+0000 7faca6fa9640 1 --2- 172.21.15.76:0/405021139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faca0100f80 0x7faca010dc40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.081+0000 7faca6fa9640 1 --2- 172.21.15.76:0/405021139 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faca01005d0 0x7faca01009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.081+0000 7faca6fa9640 1 -- 172.21.15.76:0/405021139 >> 172.21.15.76:0/405021139 conn(0x7faca00777e0 msgr2=0x7faca0077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.082+0000 7faca6fa9640 1 -- 172.21.15.76:0/405021139 shutdown_connections 2024-03-18T15:04:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.082+0000 7faca6fa9640 1 -- 172.21.15.76:0/405021139 wait complete. 2024-03-18T15:04:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.082+0000 7faca6fa9640 1 Processor -- start 2024-03-18T15:04:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.082+0000 7faca6fa9640 1 -- start start 2024-03-18T15:04:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.082+0000 7faca6fa9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faca01005d0 0x7faca019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.082+0000 7faca6fa9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faca0100f80 0x7faca019a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.082+0000 7faca6fa9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faca010e180 0x7faca019eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.082+0000 7faca6fa9640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7faca0113dc0 con 0x7faca010e180 2024-03-18T15:04:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.082+0000 7faca6fa9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faca0113c40 con 0x7faca0100f80 2024-03-18T15:04:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.082+0000 7faca6fa9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faca0113f40 con 0x7faca01005d0 2024-03-18T15:04:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.082+0000 7faca5546640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faca010e180 0x7faca019eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.082+0000 7faca5546640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faca010e180 0x7faca019eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56644/0 (socket says 172.21.15.76:56644) 2024-03-18T15:04:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.082+0000 7fac9ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faca0100f80 0x7faca019a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.082+0000 7faca5546640 1 -- 172.21.15.76:0/3834961174 learned_addr learned my addr 172.21.15.76:0/3834961174 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.082+0000 7faca4d45640 1 --2- 172.21.15.76:0/3834961174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faca01005d0 0x7faca019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.083+0000 7faca5546640 1 -- 172.21.15.76:0/3834961174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faca01005d0 msgr2=0x7faca019a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.083+0000 7faca5546640 1 --2- 172.21.15.76:0/3834961174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faca01005d0 0x7faca019a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.083+0000 7faca5546640 1 -- 172.21.15.76:0/3834961174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faca0100f80 msgr2=0x7faca019a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.083+0000 7faca5546640 1 --2- 172.21.15.76:0/3834961174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faca0100f80 0x7faca019a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.083+0000 7faca5546640 1 -- 172.21.15.76:0/3834961174 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faca019f2d0 con 0x7faca010e180 2024-03-18T15:04:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.083+0000 7fac9ffff640 1 --2- 172.21.15.76:0/3834961174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faca0100f80 0x7faca019a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:04:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.083+0000 7faca5546640 1 --2- 172.21.15.76:0/3834961174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faca010e180 0x7faca019eba0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fac9800eb40 tx=0x7fac9800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.083+0000 7fac9dffb640 1 -- 172.21.15.76:0/3834961174 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac98004b20 con 0x7faca010e180 2024-03-18T15:04:51.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.083+0000 7faca6fa9640 1 -- 172.21.15.76:0/3834961174 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faca019f5c0 con 0x7faca010e180 2024-03-18T15:04:51.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.083+0000 7fac9dffb640 1 -- 172.21.15.76:0/3834961174 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac98004cc0 con 0x7faca010e180 2024-03-18T15:04:51.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.083+0000 7fac9dffb640 1 -- 172.21.15.76:0/3834961174 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac98010850 con 0x7faca010e180 2024-03-18T15:04:51.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.083+0000 7faca6fa9640 1 -- 172.21.15.76:0/3834961174 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faca01a6e50 con 0x7faca010e180 2024-03-18T15:04:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.084+0000 7faca6fa9640 1 -- 172.21.15.76:0/3834961174 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faca00685d0 con 0x7faca010e180 2024-03-18T15:04:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.085+0000 7fac9dffb640 1 -- 172.21.15.76:0/3834961174 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fac98005020 con 0x7faca010e180 2024-03-18T15:04:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.085+0000 7fac9dffb640 1 --2- 172.21.15.76:0/3834961174 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fac880a2970 0x7fac880a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.086+0000 7faca4d45640 1 --2- 172.21.15.76:0/3834961174 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fac880a2970 0x7fac880a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.086+0000 7fac9dffb640 1 -- 172.21.15.76:0/3834961174 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7fac980cf590 con 0x7faca010e180 2024-03-18T15:04:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.086+0000 7faca4d45640 1 --2- 172.21.15.76:0/3834961174 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fac880a2970 0x7fac880a4e30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fac90009530 tx=0x7fac90009260 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:51.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.090+0000 7fac9dffb640 1 -- 172.21.15.76:0/3834961174 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fac9809c310 con 0x7faca010e180 2024-03-18T15:04:51.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.300+0000 7faca6fa9640 1 -- 172.21.15.76:0/3834961174 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faca0109310 con 0x7faca010e180 2024-03-18T15:04:51.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.301+0000 7fac9dffb640 1 -- 172.21.15.76:0/3834961174 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11865 (secure 0 0 0) 0x7fac9809bc10 con 0x7faca010e180 2024-03-18T15:04:51.302 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T15:04:51.302 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3585848317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3585848317},{"type":"v1","addr":"172.21.15.76:6835","nonce":3585848317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/3302809366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":3302809366},{"type":"v1","addr":"172.21.15.76:6837","nonce":3302809366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2568866075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2568866075},{"type":"v1","addr":"172.21.15.77:6837","nonce":2568866075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/2801069369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":2801069369},{"type":"v1","addr":"172.21.15.76:6839","nonce":2801069369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2645229232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2645229232},{"type":"v1","addr":"172.21.15.76:6841","nonce":2645229232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3569060524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3569060524},{"type":"v1","addr":"172.21.15.174:6833","nonce":3569060524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":44,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T14:51:01.293009+0000","modified":"2024-03-18T15:03:54.767975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24490,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/2726911960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2726911960},{"type":"v1","addr":"172.21.15.77:6833","nonce":2726911960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/603893267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":603893267},{"type":"v1","addr":"172.21.15.174:6835","nonce":603893267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":56,"addr":"172.21.15.174:6837/239386414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":239386414},{"type":"v1","addr":"172.21.15.174:6837","nonce":239386414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":55,"addr":"172.21.15.174:6839/399810652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":399810652},{"type":"v1","addr":"172.21.15.174:6839","nonce":399810652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1685041985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1685041985},{"type":"v1","addr":"172.21.15.77:6839","nonce":1685041985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26585":{"gid":26585,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/2933679925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2933679925},{"type":"v1","addr":"172.21.15.77:6835","nonce":2933679925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24463,24490,24524]},"id":1}]} 2024-03-18T15:04:51.302 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 44 2024-03-18T15:04:51.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.303+0000 7faca6fa9640 1 -- 172.21.15.76:0/3834961174 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fac880a2970 msgr2=0x7fac880a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:51.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.303+0000 7faca6fa9640 1 --2- 172.21.15.76:0/3834961174 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fac880a2970 0x7fac880a4e30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fac90009530 tx=0x7fac90009260 comp rx=0 tx=0).stop 2024-03-18T15:04:51.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.303+0000 7faca6fa9640 1 -- 172.21.15.76:0/3834961174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faca010e180 msgr2=0x7faca019eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:51.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.303+0000 7faca6fa9640 1 --2- 172.21.15.76:0/3834961174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faca010e180 0x7faca019eba0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fac9800eb40 tx=0x7fac9800c2d0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.303+0000 7faca6fa9640 1 -- 172.21.15.76:0/3834961174 shutdown_connections 2024-03-18T15:04:51.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.303+0000 7faca6fa9640 1 --2- 172.21.15.76:0/3834961174 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fac880a2970 0x7fac880a4e30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.303+0000 7faca6fa9640 1 --2- 172.21.15.76:0/3834961174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faca010e180 0x7faca019eba0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.303+0000 7faca6fa9640 1 --2- 172.21.15.76:0/3834961174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faca0100f80 0x7faca019a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.303+0000 7faca6fa9640 1 --2- 172.21.15.76:0/3834961174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faca01005d0 0x7faca019a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.303+0000 7faca6fa9640 1 -- 172.21.15.76:0/3834961174 >> 172.21.15.76:0/3834961174 conn(0x7faca00777e0 msgr2=0x7faca010f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:51.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.303+0000 7faca6fa9640 1 -- 172.21.15.76:0/3834961174 shutdown_connections 2024-03-18T15:04:51.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.303+0000 7faca6fa9640 1 -- 172.21.15.76:0/3834961174 wait complete. 2024-03-18T15:04:51.315 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:04:51.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.417+0000 7fec7d027640 1 -- 172.21.15.76:0/170573018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec78101040 msgr2=0x7fec7810d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:51.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.417+0000 7fec7d027640 1 --2- 172.21.15.76:0/170573018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec78101040 0x7fec7810d550 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fec70009cd0 tx=0x7fec7002c0f0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.417+0000 7fec7d027640 1 -- 172.21.15.76:0/170573018 shutdown_connections 2024-03-18T15:04:51.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.417+0000 7fec7d027640 1 --2- 172.21.15.76:0/170573018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec78101040 0x7fec7810d550 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.417+0000 7fec7d027640 1 --2- 172.21.15.76:0/170573018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec78100680 0x7fec78100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.417+0000 7fec7d027640 1 --2- 172.21.15.76:0/170573018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec78106690 0x7fec78106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.417+0000 7fec7d027640 1 -- 172.21.15.76:0/170573018 >> 172.21.15.76:0/170573018 conn(0x7fec78076a60 msgr2=0x7fec78076e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:51.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.417+0000 7fec7d027640 1 -- 172.21.15.76:0/170573018 shutdown_connections 2024-03-18T15:04:51.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.417+0000 7fec7d027640 1 -- 172.21.15.76:0/170573018 wait complete. 2024-03-18T15:04:51.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.418+0000 7fec7d027640 1 Processor -- start 2024-03-18T15:04:51.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.418+0000 7fec7d027640 1 -- start start 2024-03-18T15:04:51.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.418+0000 7fec7d027640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec78100680 0x7fec78102250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:51.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.418+0000 7fec7d027640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec78101040 0x7fec78102790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:51.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.418+0000 7fec7d027640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec78106690 0x7fec78102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:51.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.418+0000 7fec7d027640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fec78110aa0 con 0x7fec78106690 2024-03-18T15:04:51.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.418+0000 7fec7d027640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fec78110920 con 0x7fec78100680 2024-03-18T15:04:51.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.418+0000 7fec7d027640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fec78110c20 con 0x7fec78101040 2024-03-18T15:04:51.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.418+0000 7fec7759e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec78106690 0x7fec78102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:51.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.418+0000 7fec7659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec78101040 0x7fec78102790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:51.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.418+0000 7fec7659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec78101040 0x7fec78102790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.76:55018/0 (socket says 172.21.15.76:55018) 2024-03-18T15:04:51.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.418+0000 7fec7659c640 1 -- 172.21.15.76:0/3826839650 learned_addr learned my addr 172.21.15.76:0/3826839650 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:51.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.418+0000 7fec76d9d640 1 --2- 172.21.15.76:0/3826839650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec78100680 0x7fec78102250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:51.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.419+0000 7fec7759e640 1 -- 172.21.15.76:0/3826839650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec78101040 msgr2=0x7fec78102790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:51.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.419+0000 7fec7759e640 1 --2- 172.21.15.76:0/3826839650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec78101040 0x7fec78102790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.419+0000 7fec7759e640 1 -- 172.21.15.76:0/3826839650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec78100680 msgr2=0x7fec78102250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:51.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.419+0000 7fec7759e640 1 --2- 172.21.15.76:0/3826839650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec78100680 0x7fec78102250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.419+0000 7fec7759e640 1 -- 172.21.15.76:0/3826839650 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec781a6540 con 0x7fec78106690 2024-03-18T15:04:51.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.419+0000 7fec7659c640 1 --2- 172.21.15.76:0/3826839650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec78101040 0x7fec78102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:04:51.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.419+0000 7fec76d9d640 1 --2- 172.21.15.76:0/3826839650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec78100680 0x7fec78102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:04:51.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.419+0000 7fec7759e640 1 --2- 172.21.15.76:0/3826839650 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec78106690 0x7fec78102cd0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fec7002c630 tx=0x7fec70032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:51.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.419+0000 7fec5ffff640 1 -- 172.21.15.76:0/3826839650 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70041020 con 0x7fec78106690 2024-03-18T15:04:51.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.419+0000 7fec7d027640 1 -- 172.21.15.76:0/3826839650 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec781a67d0 con 0x7fec78106690 2024-03-18T15:04:51.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.419+0000 7fec5ffff640 1 -- 172.21.15.76:0/3826839650 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec70034cd0 con 0x7fec78106690 2024-03-18T15:04:51.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.419+0000 7fec5ffff640 1 -- 172.21.15.76:0/3826839650 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70039040 con 0x7fec78106690 2024-03-18T15:04:51.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.419+0000 7fec7d027640 1 -- 172.21.15.76:0/3826839650 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec781a6c90 con 0x7fec78106690 2024-03-18T15:04:51.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.420+0000 7fec7d027640 1 -- 172.21.15.76:0/3826839650 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec78101dc0 con 0x7fec78106690 2024-03-18T15:04:51.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.421+0000 7fec5ffff640 1 -- 172.21.15.76:0/3826839650 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fec70044020 con 0x7fec78106690 2024-03-18T15:04:51.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.421+0000 7fec5ffff640 1 --2- 172.21.15.76:0/3826839650 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fec600a2970 0x7fec600a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:51.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.422+0000 7fec76d9d640 1 --2- 172.21.15.76:0/3826839650 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fec600a2970 0x7fec600a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:51.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.422+0000 7fec5ffff640 1 -- 172.21.15.76:0/3826839650 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7fec700f1000 con 0x7fec78106690 2024-03-18T15:04:51.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.422+0000 7fec76d9d640 1 --2- 172.21.15.76:0/3826839650 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fec600a2970 0x7fec600a4e30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fec68009f40 tx=0x7fec68005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:51.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.425+0000 7fec5ffff640 1 -- 172.21.15.76:0/3826839650 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fec7004b020 con 0x7fec78106690 2024-03-18T15:04:51.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.601+0000 7fec7d027640 1 -- 172.21.15.76:0/3826839650 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fec781a6fb0 con 0x7fec78106690 2024-03-18T15:04:51.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.603+0000 7fec5ffff640 1 -- 172.21.15.76:0/3826839650 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) ==== 74+0+23049 (secure 0 0 0) 0x7fec700bdfb0 con 0x7fec78106690 2024-03-18T15:04:51.604 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-18T15:04:51.604 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":107,"fsid":"9cc83b72-e535-11ee-95c9-87774f69a715","created":"2024-03-18T14:43:12.899171+0000","modified":"2024-03-18T15:04:18.312749+0000","last_up_change":"2024-03-18T14:50:13.504521+0000","last_in_change":"2024-03-18T14:49:53.380101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:46:26.588384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:50:59.606852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:51:00.465325+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":106,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b8ee0a0-29f7-4333-978a-08c4db54db0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6803","nonce":2682004861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6805","nonce":2682004861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6809","nonce":2682004861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2682004861},{"type":"v1","addr":"172.21.15.76:6807","nonce":2682004861}]},"public_addr":"172.21.15.76:6803/2682004861","cluster_addr":"172.21.15.76:6805/2682004861","heartbeat_back_addr":"172.21.15.76:6809/2682004861","heartbeat_front_addr":"172.21.15.76:6807/2682004861","state":["exists","up"]},{"osd":1,"uuid":"71422220-775b-4b49-a9a4-ae0a7a16c2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6801","nonce":3243872836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6803","nonce":3243872836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6807","nonce":3243872836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3243872836},{"type":"v1","addr":"172.21.15.77:6805","nonce":3243872836}]},"public_addr":"172.21.15.77:6801/3243872836","cluster_addr":"172.21.15.77:6803/3243872836","heartbeat_back_addr":"172.21.15.77:6807/3243872836","heartbeat_front_addr":"172.21.15.77:6805/3243872836","state":["exists","up"]},{"osd":2,"uuid":"3a5bf599-bc62-4986-9bf3-6486a08857a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6801","nonce":3812090370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6803","nonce":3812090370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6807","nonce":3812090370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3812090370},{"type":"v1","addr":"172.21.15.174:6805","nonce":3812090370}]},"public_addr":"172.21.15.174:6801/3812090370","cluster_addr":"172.21.15.174:6803/3812090370","heartbeat_back_addr":"172.21.15.174:6807/3812090370","heartbeat_front_addr":"172.21.15.174:6805/3812090370","state":["exists","up"]},{"osd":3,"uuid":"5060b19a-4b1f-4a15-bd07-a920a267bcfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6811","nonce":140675439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6813","nonce":140675439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6817","nonce":140675439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":140675439},{"type":"v1","addr":"172.21.15.76:6815","nonce":140675439}]},"public_addr":"172.21.15.76:6811/140675439","cluster_addr":"172.21.15.76:6813/140675439","heartbeat_back_addr":"172.21.15.76:6817/140675439","heartbeat_front_addr":"172.21.15.76:6815/140675439","state":["exists","up"]},{"osd":4,"uuid":"b88f6022-4690-4c54-9072-8a1f0febddd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6809","nonce":1533069359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6811","nonce":1533069359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6815","nonce":1533069359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1533069359},{"type":"v1","addr":"172.21.15.77:6813","nonce":1533069359}]},"public_addr":"172.21.15.77:6809/1533069359","cluster_addr":"172.21.15.77:6811/1533069359","heartbeat_back_addr":"172.21.15.77:6815/1533069359","heartbeat_front_addr":"172.21.15.77:6813/1533069359","state":["exists","up"]},{"osd":5,"uuid":"53063e2c-b521-41d8-859d-7a4bcf470fa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6809","nonce":1607563490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6811","nonce":1607563490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6815","nonce":1607563490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1607563490},{"type":"v1","addr":"172.21.15.174:6813","nonce":1607563490}]},"public_addr":"172.21.15.174:6809/1607563490","cluster_addr":"172.21.15.174:6811/1607563490","heartbeat_back_addr":"172.21.15.174:6815/1607563490","heartbeat_front_addr":"172.21.15.174:6813/1607563490","state":["exists","up"]},{"osd":6,"uuid":"f008e913-20ac-4df7-8bed-fb345d6d84bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6819","nonce":1891318352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6821","nonce":1891318352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6825","nonce":1891318352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1891318352},{"type":"v1","addr":"172.21.15.76:6823","nonce":1891318352}]},"public_addr":"172.21.15.76:6819/1891318352","cluster_addr":"172.21.15.76:6821/1891318352","heartbeat_back_addr":"172.21.15.76:6825/1891318352","heartbeat_front_addr":"172.21.15.76:6823/1891318352","state":["exists","up"]},{"osd":7,"uuid":"7354ce4c-94d9-4d70-9134-b01836fbfc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6817","nonce":3828133263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6819","nonce":3828133263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6823","nonce":3828133263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3828133263},{"type":"v1","addr":"172.21.15.77:6821","nonce":3828133263}]},"public_addr":"172.21.15.77:6817/3828133263","cluster_addr":"172.21.15.77:6819/3828133263","heartbeat_back_addr":"172.21.15.77:6823/3828133263","heartbeat_front_addr":"172.21.15.77:6821/3828133263","state":["exists","up"]},{"osd":8,"uuid":"955ffff4-1c26-4dbd-82f7-aaf6ceb7da21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6817","nonce":2665844766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6819","nonce":2665844766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6823","nonce":2665844766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2665844766},{"type":"v1","addr":"172.21.15.174:6821","nonce":2665844766}]},"public_addr":"172.21.15.174:6817/2665844766","cluster_addr":"172.21.15.174:6819/2665844766","heartbeat_back_addr":"172.21.15.174:6823/2665844766","heartbeat_front_addr":"172.21.15.174:6821/2665844766","state":["exists","up"]},{"osd":9,"uuid":"1a818122-b7a3-4f62-adaf-0ec35156cf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6827","nonce":3937900663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6829","nonce":3937900663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6833","nonce":3937900663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3937900663},{"type":"v1","addr":"172.21.15.76:6831","nonce":3937900663}]},"public_addr":"172.21.15.76:6827/3937900663","cluster_addr":"172.21.15.76:6829/3937900663","heartbeat_back_addr":"172.21.15.76:6833/3937900663","heartbeat_front_addr":"172.21.15.76:6831/3937900663","state":["exists","up"]},{"osd":10,"uuid":"ee1f73e8-94b3-4969-9944-da77e7f46bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6825","nonce":4105457096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6827","nonce":4105457096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6831","nonce":4105457096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4105457096},{"type":"v1","addr":"172.21.15.77:6829","nonce":4105457096}]},"public_addr":"172.21.15.77:6825/4105457096","cluster_addr":"172.21.15.77:6827/4105457096","heartbeat_back_addr":"172.21.15.77:6831/4105457096","heartbeat_front_addr":"172.21.15.77:6829/4105457096","state":["exists","up"]},{"osd":11,"uuid":"35839388-3d56-497b-a49c-0db8686e3a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6825","nonce":1489039553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6827","nonce":1489039553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6831","nonce":1489039553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1489039553},{"type":"v1","addr":"172.21.15.174:6829","nonce":1489039553}]},"public_addr":"172.21.15.174:6825/1489039553","cluster_addr":"172.21.15.174:6827/1489039553","heartbeat_back_addr":"172.21.15.174:6831/1489039553","heartbeat_front_addr":"172.21.15.174:6829/1489039553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:45:37.265590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:00.607718+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:23.761914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:46:48.464683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:12.681504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:47:37.822329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:02.804457+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:27.722608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:48:53.446024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:20.251785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:49:45.980882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:50:11.791897+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3010549595":"2024-03-19T14:53:17.537054+0000","172.21.15.76:6800/3010549595":"2024-03-19T14:53:17.537054+0000","172.21.15.76:0/3122154733":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/880938501":"2024-03-19T14:53:17.537054+0000","172.21.15.76:0/752916178":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/3261047930":"2024-03-19T14:44:06.251768+0000","172.21.15.76:6801/3054039863":"2024-03-19T14:44:06.251768+0000","172.21.15.76:0/2082819586":"2024-03-19T14:53:17.537054+0000","172.21.15.76:0/600153683":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6801/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/1895213347":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/3349561522":"2024-03-19T14:53:17.537054+0000","172.21.15.76:0/918193033":"2024-03-19T14:43:39.730762+0000","172.21.15.76:0/788250061":"2024-03-19T14:53:17.537054+0000","172.21.15.76:6800/216126402":"2024-03-19T14:43:39.730762+0000","172.21.15.76:6800/3054039863":"2024-03-19T14:44:06.251768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:04:51.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.605+0000 7fec7d027640 1 -- 172.21.15.76:0/3826839650 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fec600a2970 msgr2=0x7fec600a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.605+0000 7fec7d027640 1 --2- 172.21.15.76:0/3826839650 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fec600a2970 0x7fec600a4e30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fec68009f40 tx=0x7fec68005f90 comp rx=0 tx=0).stop 2024-03-18T15:04:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.605+0000 7fec7d027640 1 -- 172.21.15.76:0/3826839650 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec78106690 msgr2=0x7fec78102cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.605+0000 7fec7d027640 1 --2- 172.21.15.76:0/3826839650 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec78106690 0x7fec78102cd0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fec7002c630 tx=0x7fec70032f20 comp rx=0 tx=0).stop 2024-03-18T15:04:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.605+0000 7fec7d027640 1 -- 172.21.15.76:0/3826839650 shutdown_connections 2024-03-18T15:04:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.605+0000 7fec7d027640 1 --2- 172.21.15.76:0/3826839650 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fec600a2970 0x7fec600a4e30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.605+0000 7fec7d027640 1 --2- 172.21.15.76:0/3826839650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec78101040 0x7fec78102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.605+0000 7fec7d027640 1 --2- 172.21.15.76:0/3826839650 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec78106690 0x7fec78102cd0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.605+0000 7fec7d027640 1 --2- 172.21.15.76:0/3826839650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec78100680 0x7fec78102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.605+0000 7fec7d027640 1 -- 172.21.15.76:0/3826839650 >> 172.21.15.76:0/3826839650 conn(0x7fec78076a60 msgr2=0x7fec78104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:51.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.605+0000 7fec7d027640 1 -- 172.21.15.76:0/3826839650 shutdown_connections 2024-03-18T15:04:51.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.605+0000 7fec7d027640 1 -- 172.21.15.76:0/3826839650 wait complete. 2024-03-18T15:04:51.615 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:51 smithi076 ceph-mon[32205]: pgmap v367: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 313 KiB/s wr, 26 op/s 2024-03-18T15:04:51.615 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:51 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1199602476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:04:51.615 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:51 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3834961174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:04:51.618 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T15:04:51.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:51 smithi174 ceph-mon[36787]: pgmap v367: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 313 KiB/s wr, 26 op/s 2024-03-18T15:04:51.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:51 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1199602476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:04:51.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:51 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3834961174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:04:51.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.720+0000 7fc459153640 1 -- 172.21.15.76:0/2331012315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc454107830 msgr2=0x7fc454107c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:51.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.720+0000 7fc459153640 1 --2- 172.21.15.76:0/2331012315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc454107830 0x7fc454107c10 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fc44c009930 tx=0x7fc44c02bc30 comp rx=0 tx=0).stop 2024-03-18T15:04:51.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.720+0000 7fc459153640 1 -- 172.21.15.76:0/2331012315 shutdown_connections 2024-03-18T15:04:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.720+0000 7fc459153640 1 --2- 172.21.15.76:0/2331012315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4540693a0 0x7fc45410e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.720+0000 7fc459153640 1 --2- 172.21.15.76:0/2331012315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4540689e0 0x7fc454068e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.720+0000 7fc459153640 1 --2- 172.21.15.76:0/2331012315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc454107830 0x7fc454107c10 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.720+0000 7fc459153640 1 -- 172.21.15.76:0/2331012315 >> 172.21.15.76:0/2331012315 conn(0x7fc4540777e0 msgr2=0x7fc454077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.720+0000 7fc459153640 1 -- 172.21.15.76:0/2331012315 shutdown_connections 2024-03-18T15:04:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.720+0000 7fc459153640 1 -- 172.21.15.76:0/2331012315 wait complete. 2024-03-18T15:04:51.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.721+0000 7fc459153640 1 Processor -- start 2024-03-18T15:04:51.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.721+0000 7fc459153640 1 -- start start 2024-03-18T15:04:51.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.721+0000 7fc459153640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4540689e0 0x7fc4541006a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:51.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.721+0000 7fc459153640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4540693a0 0x7fc454100be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:51.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.721+0000 7fc459153640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc454107830 0x7fc454101120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:51.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.721+0000 7fc459153640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc454111b80 con 0x7fc4540689e0 2024-03-18T15:04:51.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.721+0000 7fc459153640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc454111a00 con 0x7fc4540693a0 2024-03-18T15:04:51.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.721+0000 7fc459153640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc454111d00 con 0x7fc454107830 2024-03-18T15:04:51.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.722+0000 7fc452d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4540689e0 0x7fc4541006a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:51.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.722+0000 7fc452d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4540689e0 0x7fc4541006a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56686/0 (socket says 172.21.15.76:56686) 2024-03-18T15:04:51.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.722+0000 7fc452d9d640 1 -- 172.21.15.76:0/105976866 learned_addr learned my addr 172.21.15.76:0/105976866 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:51.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.722+0000 7fc45259c640 1 --2- 172.21.15.76:0/105976866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4540693a0 0x7fc454100be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:51.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.722+0000 7fc45359e640 1 --2- 172.21.15.76:0/105976866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc454107830 0x7fc454101120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:51.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.722+0000 7fc452d9d640 1 -- 172.21.15.76:0/105976866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc454107830 msgr2=0x7fc454101120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:51.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.722+0000 7fc452d9d640 1 --2- 172.21.15.76:0/105976866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc454107830 0x7fc454101120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.722+0000 7fc452d9d640 1 -- 172.21.15.76:0/105976866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4540693a0 msgr2=0x7fc454100be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:51.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.722+0000 7fc452d9d640 1 --2- 172.21.15.76:0/105976866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4540693a0 0x7fc454100be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.722+0000 7fc452d9d640 1 -- 172.21.15.76:0/105976866 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4541a6500 con 0x7fc4540689e0 2024-03-18T15:04:51.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.722+0000 7fc452d9d640 1 --2- 172.21.15.76:0/105976866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4540689e0 0x7fc4541006a0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc44c009230 tx=0x7fc44c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:51.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.722+0000 7fc43bfff640 1 -- 172.21.15.76:0/105976866 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c041070 con 0x7fc4540689e0 2024-03-18T15:04:51.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.722+0000 7fc459153640 1 -- 172.21.15.76:0/105976866 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4541a6790 con 0x7fc4540689e0 2024-03-18T15:04:51.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.722+0000 7fc43bfff640 1 -- 172.21.15.76:0/105976866 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc44c0054e0 con 0x7fc4540689e0 2024-03-18T15:04:51.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.722+0000 7fc43bfff640 1 -- 172.21.15.76:0/105976866 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c003c90 con 0x7fc4540689e0 2024-03-18T15:04:51.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.722+0000 7fc459153640 1 -- 172.21.15.76:0/105976866 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4541a6cd0 con 0x7fc4540689e0 2024-03-18T15:04:51.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.723+0000 7fc459153640 1 -- 172.21.15.76:0/105976866 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc454101bd0 con 0x7fc4540689e0 2024-03-18T15:04:51.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:51 smithi077 ceph-mon[35865]: pgmap v367: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 313 KiB/s wr, 26 op/s 2024-03-18T15:04:51.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:51 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1199602476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:04:51.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:51 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3834961174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:04:51.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.724+0000 7fc43bfff640 1 -- 172.21.15.76:0/105976866 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fc44c044070 con 0x7fc4540689e0 2024-03-18T15:04:51.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.725+0000 7fc43bfff640 1 --2- 172.21.15.76:0/105976866 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc43c0a2970 0x7fc43c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:51.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.725+0000 7fc45259c640 1 --2- 172.21.15.76:0/105976866 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc43c0a2970 0x7fc43c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:51.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.725+0000 7fc43bfff640 1 -- 172.21.15.76:0/105976866 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7fc44c0f0d30 con 0x7fc4540689e0 2024-03-18T15:04:51.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.725+0000 7fc45259c640 1 --2- 172.21.15.76:0/105976866 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc43c0a2970 0x7fc43c0a4e30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc4480095f0 tx=0x7fc448005f90 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:51.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.728+0000 7fc43bfff640 1 -- 172.21.15.76:0/105976866 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc44c0bd570 con 0x7fc4540689e0 2024-03-18T15:04:51.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.941+0000 7fc459153640 1 -- 172.21.15.76:0/105976866 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc45402d1b0 con 0x7fc4540689e0 2024-03-18T15:04:51.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.943+0000 7fc43bfff640 1 -- 172.21.15.76:0/105976866 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v45) ==== 161+0+0 (secure 0 0 0) 0x7fc44c0c1610 con 0x7fc4540689e0 2024-03-18T15:04:51.944 INFO:teuthology.orchestra.run.smithi076.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T15:04:51.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.946+0000 7fc459153640 1 -- 172.21.15.76:0/105976866 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc43c0a2970 msgr2=0x7fc43c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:51.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.946+0000 7fc459153640 1 --2- 172.21.15.76:0/105976866 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc43c0a2970 0x7fc43c0a4e30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc4480095f0 tx=0x7fc448005f90 comp rx=0 tx=0).stop 2024-03-18T15:04:51.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.946+0000 7fc459153640 1 -- 172.21.15.76:0/105976866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4540689e0 msgr2=0x7fc4541006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:51.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.946+0000 7fc459153640 1 --2- 172.21.15.76:0/105976866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4540689e0 0x7fc4541006a0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc44c009230 tx=0x7fc44c004270 comp rx=0 tx=0).stop 2024-03-18T15:04:51.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.946+0000 7fc459153640 1 -- 172.21.15.76:0/105976866 shutdown_connections 2024-03-18T15:04:51.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.946+0000 7fc459153640 1 --2- 172.21.15.76:0/105976866 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc43c0a2970 0x7fc43c0a4e30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.946+0000 7fc459153640 1 --2- 172.21.15.76:0/105976866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc454107830 0x7fc454101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.946+0000 7fc459153640 1 --2- 172.21.15.76:0/105976866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4540693a0 0x7fc454100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.946+0000 7fc459153640 1 --2- 172.21.15.76:0/105976866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc4540689e0 0x7fc4541006a0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:51.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.946+0000 7fc459153640 1 -- 172.21.15.76:0/105976866 >> 172.21.15.76:0/105976866 conn(0x7fc4540777e0 msgr2=0x7fc4540fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:51.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.946+0000 7fc459153640 1 -- 172.21.15.76:0/105976866 shutdown_connections 2024-03-18T15:04:51.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:51.946+0000 7fc459153640 1 -- 172.21.15.76:0/105976866 wait complete. 2024-03-18T15:04:51.958 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T15:04:52.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.063+0000 7fc6cab90640 1 -- 172.21.15.76:0/804076064 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6c4101010 msgr2=0x7fc6c410d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.063+0000 7fc6cab90640 1 --2- 172.21.15.76:0/804076064 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6c4101010 0x7fc6c410d520 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc009cd0 tx=0x7fc6bc02c0f0 comp rx=0 tx=0).stop 2024-03-18T15:04:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6cab90640 1 -- 172.21.15.76:0/804076064 shutdown_connections 2024-03-18T15:04:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6cab90640 1 --2- 172.21.15.76:0/804076064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6c4100650 0x7fc6c4100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6cab90640 1 --2- 172.21.15.76:0/804076064 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6c4101010 0x7fc6c410d520 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6cab90640 1 --2- 172.21.15.76:0/804076064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6c4106660 0x7fc6c4106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6cab90640 1 -- 172.21.15.76:0/804076064 >> 172.21.15.76:0/804076064 conn(0x7fc6c40777e0 msgr2=0x7fc6c4077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6cab90640 1 -- 172.21.15.76:0/804076064 shutdown_connections 2024-03-18T15:04:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6cab90640 1 -- 172.21.15.76:0/804076064 wait complete. 2024-03-18T15:04:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6cab90640 1 Processor -- start 2024-03-18T15:04:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6cab90640 1 -- start start 2024-03-18T15:04:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6cab90640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6c4100650 0x7fc6c419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6cab90640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6c4101010 0x7fc6c419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6cab90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6c4106660 0x7fc6c419ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6cab90640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c4110a70 con 0x7fc6c4100650 2024-03-18T15:04:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6cab90640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c41108f0 con 0x7fc6c4106660 2024-03-18T15:04:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6cab90640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c4110bf0 con 0x7fc6c4101010 2024-03-18T15:04:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6c892c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6c4100650 0x7fc6c419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6c892c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6c4100650 0x7fc6c419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56712/0 (socket says 172.21.15.76:56712) 2024-03-18T15:04:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.064+0000 7fc6c892c640 1 -- 172.21.15.76:0/1829615130 learned_addr learned my addr 172.21.15.76:0/1829615130 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.065+0000 7fc6c3fff640 1 --2- 172.21.15.76:0/1829615130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6c4101010 0x7fc6c419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.065+0000 7fc6c892c640 1 -- 172.21.15.76:0/1829615130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6c4101010 msgr2=0x7fc6c419a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.065+0000 7fc6c892c640 1 --2- 172.21.15.76:0/1829615130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6c4101010 0x7fc6c419a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.065+0000 7fc6c892c640 1 -- 172.21.15.76:0/1829615130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6c4106660 msgr2=0x7fc6c419ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.065+0000 7fc6c892c640 1 --2- 172.21.15.76:0/1829615130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6c4106660 0x7fc6c419ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.065+0000 7fc6c892c640 1 -- 172.21.15.76:0/1829615130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c419f1c0 con 0x7fc6c4100650 2024-03-18T15:04:52.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.065+0000 7fc6c3fff640 1 --2- 172.21.15.76:0/1829615130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6c4101010 0x7fc6c419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:04:52.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.065+0000 7fc6c892c640 1 --2- 172.21.15.76:0/1829615130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6c4100650 0x7fc6c419a1d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc6b800c890 tx=0x7fc6b800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:52.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.065+0000 7fc6c1ffb640 1 -- 172.21.15.76:0/1829615130 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8004220 con 0x7fc6c4100650 2024-03-18T15:04:52.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.065+0000 7fc6c1ffb640 1 -- 172.21.15.76:0/1829615130 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6b80043c0 con 0x7fc6c4100650 2024-03-18T15:04:52.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.065+0000 7fc6cab90640 1 -- 172.21.15.76:0/1829615130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6c419f4b0 con 0x7fc6c4100650 2024-03-18T15:04:52.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.065+0000 7fc6c1ffb640 1 -- 172.21.15.76:0/1829615130 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b80049e0 con 0x7fc6c4100650 2024-03-18T15:04:52.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.065+0000 7fc6cab90640 1 -- 172.21.15.76:0/1829615130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6c41a6d40 con 0x7fc6c4100650 2024-03-18T15:04:52.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.066+0000 7fc6cab90640 1 -- 172.21.15.76:0/1829615130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6c40685d0 con 0x7fc6c4100650 2024-03-18T15:04:52.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.067+0000 7fc6c1ffb640 1 -- 172.21.15.76:0/1829615130 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fc6b80079b0 con 0x7fc6c4100650 2024-03-18T15:04:52.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.068+0000 7fc6c1ffb640 1 --2- 172.21.15.76:0/1829615130 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc6b40a2970 0x7fc6b40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:52.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.068+0000 7fc6c3fff640 1 --2- 172.21.15.76:0/1829615130 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc6b40a2970 0x7fc6b40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:52.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.068+0000 7fc6c1ffb640 1 -- 172.21.15.76:0/1829615130 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8698+0+0 (secure 0 0 0) 0x7fc6b80cf000 con 0x7fc6c4100650 2024-03-18T15:04:52.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.068+0000 7fc6c3fff640 1 --2- 172.21.15.76:0/1829615130 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc6b40a2970 0x7fc6b40a4e30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc6c410d390 tx=0x7fc6b0005d30 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:52.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.072+0000 7fc6c1ffb640 1 -- 172.21.15.76:0/1829615130 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc6b8014030 con 0x7fc6c4100650 2024-03-18T15:04:52.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.283+0000 7fc6cab90640 1 -- 172.21.15.76:0/1829615130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fc6c41a7080 con 0x7fc6c4100650 2024-03-18T15:04:52.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:51 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a[32201]: 2024-03-18T15:04:51.942+0000 7f4cc3099700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T15:04:52.384 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 15:04:51 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-k[65079]: ignoring --setuser ceph since I am not root 2024-03-18T15:04:52.384 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 15:04:51 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-k[65079]: ignoring --setgroup ceph since I am not root 2024-03-18T15:04:52.384 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 15:04:51 smithi077 ceph-mds[65083]: -- 172.21.15.77:0/1115456096 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557e833b5a40 con 0x557e83388c00 2024-03-18T15:04:52.384 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 15:04:51 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-k[65079]: starting mds.k at 2024-03-18T15:04:52.384 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 15:04:51 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-b[66028]: ignoring --setuser ceph since I am not root 2024-03-18T15:04:52.385 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 15:04:51 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-b[66028]: ignoring --setgroup ceph since I am not root 2024-03-18T15:04:52.385 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 15:04:51 smithi077 ceph-mds[66032]: -- 172.21.15.77:0/3904034381 <== mon.0 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d39ae2da40 con 0x55d39ae00c00 2024-03-18T15:04:52.385 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 15:04:51 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-b[66028]: starting mds.b at 2024-03-18T15:04:52.385 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:04:51 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setuser ceph since I am not root 2024-03-18T15:04:52.385 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:04:51 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: ignoring --setgroup ceph since I am not root 2024-03-18T15:04:52.385 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:04:51 smithi077 ceph-mds[65404]: -- 172.21.15.77:0/642803960 <== mon.0 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aa81eb3a40 con 0x55aa81e87000 2024-03-18T15:04:52.385 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:04:51 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: starting mds.e at 2024-03-18T15:04:52.386 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 15:04:51 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: ignoring --setuser ceph since I am not root 2024-03-18T15:04:52.386 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 15:04:51 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: ignoring --setgroup ceph since I am not root 2024-03-18T15:04:52.386 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 15:04:51 smithi174 ceph-mds[67622]: -- 172.21.15.174:0/3309274793 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55705ae15a40 con 0x55705ade8c00 2024-03-18T15:04:52.386 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 15:04:51 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: starting mds.i at 2024-03-18T15:04:52.387 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 15:04:52 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: ignoring --setuser ceph since I am not root 2024-03-18T15:04:52.387 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 15:04:52 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: ignoring --setgroup ceph since I am not root 2024-03-18T15:04:52.387 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 15:04:52 smithi174 ceph-mds[67314]: -- 172.21.15.174:0/105511716 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ecf215fa40 con 0x55ecf2133000 2024-03-18T15:04:52.387 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 15:04:52 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: starting mds.c at 2024-03-18T15:04:52.387 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 15:04:51 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-f[67001]: ignoring --setuser ceph since I am not root 2024-03-18T15:04:52.388 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 15:04:51 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-f[67001]: ignoring --setgroup ceph since I am not root 2024-03-18T15:04:52.388 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 15:04:51 smithi174 ceph-mds[67005]: -- 172.21.15.174:0/165325808 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5575caf4da40 con 0x5575caf20c00 2024-03-18T15:04:52.388 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 15:04:51 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-f[67001]: starting mds.f at 2024-03-18T15:04:52.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:52 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/3826839650' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:04:52.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:52 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/105976866' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T15:04:52.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:52 smithi076 ceph-mon[32205]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T15:04:52.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:52 smithi076 ceph-mon[32205]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T15:04:52.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:52 smithi076 ceph-mon[32205]: Health check cleared: MDS_DAMAGE (was: 2 MDSs report damaged metadata) 2024-03-18T15:04:52.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:52 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/105976866' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T15:04:52.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:52 smithi076 ceph-mon[32205]: osdmap e108: 12 total, 12 up, 12 in 2024-03-18T15:04:52.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:52 smithi076 ceph-mon[32205]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-18T15:04:52.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:52 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1829615130' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:04:52.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:52 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/3826839650' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:04:52.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:52 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/105976866' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T15:04:52.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:52 smithi174 ceph-mon[36787]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T15:04:52.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:52 smithi174 ceph-mon[36787]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T15:04:52.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:52 smithi174 ceph-mon[36787]: Health check cleared: MDS_DAMAGE (was: 2 MDSs report damaged metadata) 2024-03-18T15:04:52.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:52 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/105976866' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T15:04:52.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:52 smithi174 ceph-mon[36787]: osdmap e108: 12 total, 12 up, 12 in 2024-03-18T15:04:52.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:52 smithi174 ceph-mon[36787]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-18T15:04:52.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:52 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1829615130' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:04:52.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:52 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/3826839650' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:04:52.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:52 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/105976866' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T15:04:52.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:52 smithi077 ceph-mon[35865]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T15:04:52.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:52 smithi077 ceph-mon[35865]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T15:04:52.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:52 smithi077 ceph-mon[35865]: Health check cleared: MDS_DAMAGE (was: 2 MDSs report damaged metadata) 2024-03-18T15:04:52.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:52 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/105976866' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T15:04:52.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:52 smithi077 ceph-mon[35865]: osdmap e108: 12 total, 12 up, 12 in 2024-03-18T15:04:52.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:52 smithi077 ceph-mon[35865]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-18T15:04:52.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:52 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1829615130' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:04:52.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.945+0000 7fc6c1ffb640 1 -- 172.21.15.76:0/1829615130 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v46) ==== 104+0+0 (secure 0 0 0) 0x7fc6b809bcb0 con 0x7fc6c4100650 2024-03-18T15:04:52.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.948+0000 7fc6cab90640 1 -- 172.21.15.76:0/1829615130 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc6b40a2970 msgr2=0x7fc6b40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:52.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.948+0000 7fc6cab90640 1 --2- 172.21.15.76:0/1829615130 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc6b40a2970 0x7fc6b40a4e30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc6c410d390 tx=0x7fc6b0005d30 comp rx=0 tx=0).stop 2024-03-18T15:04:52.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.948+0000 7fc6cab90640 1 -- 172.21.15.76:0/1829615130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6c4100650 msgr2=0x7fc6c419a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:52.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.948+0000 7fc6cab90640 1 --2- 172.21.15.76:0/1829615130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6c4100650 0x7fc6c419a1d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc6b800c890 tx=0x7fc6b800cd50 comp rx=0 tx=0).stop 2024-03-18T15:04:52.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.948+0000 7fc6cab90640 1 -- 172.21.15.76:0/1829615130 shutdown_connections 2024-03-18T15:04:52.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.948+0000 7fc6cab90640 1 --2- 172.21.15.76:0/1829615130 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fc6b40a2970 0x7fc6b40a4e30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:52.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.948+0000 7fc6cab90640 1 --2- 172.21.15.76:0/1829615130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6c4101010 0x7fc6c419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:52.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.948+0000 7fc6cab90640 1 --2- 172.21.15.76:0/1829615130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6c4106660 0x7fc6c419ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:52.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.948+0000 7fc6cab90640 1 --2- 172.21.15.76:0/1829615130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6c4100650 0x7fc6c419a1d0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:52.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.948+0000 7fc6cab90640 1 -- 172.21.15.76:0/1829615130 >> 172.21.15.76:0/1829615130 conn(0x7fc6c40777e0 msgr2=0x7fc6c4068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:52.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.948+0000 7fc6cab90640 1 -- 172.21.15.76:0/1829615130 shutdown_connections 2024-03-18T15:04:52.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:52.948+0000 7fc6cab90640 1 -- 172.21.15.76:0/1829615130 wait complete. 2024-03-18T15:04:52.962 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T15:04:53.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.071+0000 7fa515753640 1 -- 172.21.15.76:0/3165127979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa510102110 msgr2=0x7fa51010e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:53.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.071+0000 7fa515753640 1 --2- 172.21.15.76:0/3165127979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa510102110 0x7fa51010e620 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fa500009930 tx=0x7fa50002bc30 comp rx=0 tx=0).stop 2024-03-18T15:04:53.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.071+0000 7fa515753640 1 -- 172.21.15.76:0/3165127979 shutdown_connections 2024-03-18T15:04:53.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.071+0000 7fa515753640 1 --2- 172.21.15.76:0/3165127979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa510102110 0x7fa51010e620 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:53.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.071+0000 7fa515753640 1 --2- 172.21.15.76:0/3165127979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa510101750 0x7fa510101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:53.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.071+0000 7fa515753640 1 --2- 172.21.15.76:0/3165127979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa510107760 0x7fa510107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:53.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.071+0000 7fa515753640 1 -- 172.21.15.76:0/3165127979 >> 172.21.15.76:0/3165127979 conn(0x7fa5100fd280 msgr2=0x7fa5100ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:53.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.071+0000 7fa515753640 1 -- 172.21.15.76:0/3165127979 shutdown_connections 2024-03-18T15:04:53.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.071+0000 7fa515753640 1 -- 172.21.15.76:0/3165127979 wait complete. 2024-03-18T15:04:53.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.072+0000 7fa515753640 1 Processor -- start 2024-03-18T15:04:53.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.072+0000 7fa515753640 1 -- start start 2024-03-18T15:04:53.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.072+0000 7fa515753640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa510101750 0x7fa510198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:53.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.072+0000 7fa515753640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa510102110 0x7fa510198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:53.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.072+0000 7fa515753640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa510107760 0x7fa51019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:53.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.072+0000 7fa515753640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa510111b30 con 0x7fa510102110 2024-03-18T15:04:53.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.072+0000 7fa515753640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa5101119b0 con 0x7fa510107760 2024-03-18T15:04:53.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.072+0000 7fa515753640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa510111cb0 con 0x7fa510101750 2024-03-18T15:04:53.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.072+0000 7fa50e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa510102110 0x7fa510198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:53.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.072+0000 7fa50e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa510102110 0x7fa510198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56736/0 (socket says 172.21.15.76:56736) 2024-03-18T15:04:53.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.072+0000 7fa50e7fc640 1 -- 172.21.15.76:0/1728200658 learned_addr learned my addr 172.21.15.76:0/1728200658 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:53.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.073+0000 7fa50effd640 1 --2- 172.21.15.76:0/1728200658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa510101750 0x7fa510198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:53.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.073+0000 7fa50f7fe640 1 --2- 172.21.15.76:0/1728200658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa510107760 0x7fa51019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:53.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.073+0000 7fa50e7fc640 1 -- 172.21.15.76:0/1728200658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa510101750 msgr2=0x7fa510198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:53.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.073+0000 7fa50e7fc640 1 --2- 172.21.15.76:0/1728200658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa510101750 0x7fa510198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:53.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.073+0000 7fa50e7fc640 1 -- 172.21.15.76:0/1728200658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa510107760 msgr2=0x7fa51019c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:53.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.073+0000 7fa50e7fc640 1 --2- 172.21.15.76:0/1728200658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa510107760 0x7fa51019c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:53.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.073+0000 7fa50e7fc640 1 -- 172.21.15.76:0/1728200658 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa51019cfd0 con 0x7fa510102110 2024-03-18T15:04:53.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.073+0000 7fa50e7fc640 1 --2- 172.21.15.76:0/1728200658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa510102110 0x7fa510198570 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa4f800d950 tx=0x7fa4f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:53.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.073+0000 7fa507fff640 1 -- 172.21.15.76:0/1728200658 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8017070 con 0x7fa510102110 2024-03-18T15:04:53.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.073+0000 7fa515753640 1 -- 172.21.15.76:0/1728200658 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa51019d2c0 con 0x7fa510102110 2024-03-18T15:04:53.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.073+0000 7fa507fff640 1 -- 172.21.15.76:0/1728200658 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4f8009c30 con 0x7fa510102110 2024-03-18T15:04:53.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.073+0000 7fa507fff640 1 -- 172.21.15.76:0/1728200658 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f800f6b0 con 0x7fa510102110 2024-03-18T15:04:53.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.073+0000 7fa515753640 1 -- 172.21.15.76:0/1728200658 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5101a4b50 con 0x7fa510102110 2024-03-18T15:04:53.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.074+0000 7fa515753640 1 -- 172.21.15.76:0/1728200658 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa510067d40 con 0x7fa510102110 2024-03-18T15:04:53.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.075+0000 7fa507fff640 1 -- 172.21.15.76:0/1728200658 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fa4f80109d0 con 0x7fa510102110 2024-03-18T15:04:53.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.076+0000 7fa507fff640 1 --2- 172.21.15.76:0/1728200658 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa4fc0a2970 0x7fa4fc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:53.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.076+0000 7fa50effd640 1 --2- 172.21.15.76:0/1728200658 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa4fc0a2970 0x7fa4fc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:53.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.076+0000 7fa507fff640 1 -- 172.21.15.76:0/1728200658 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8698+0+0 (secure 0 0 0) 0x7fa4f80ceb20 con 0x7fa510102110 2024-03-18T15:04:53.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.077+0000 7fa50effd640 1 --2- 172.21.15.76:0/1728200658 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa4fc0a2970 0x7fa4fc0a4e30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa508005fd0 tx=0x7fa508009760 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:53.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.080+0000 7fa507fff640 1 -- 172.21.15.76:0/1728200658 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa4f80cee20 con 0x7fa510102110 2024-03-18T15:04:53.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.262+0000 7fa515753640 1 -- 172.21.15.76:0/1728200658 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa510109750 con 0x7fa510102110 2024-03-18T15:04:53.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: pgmap v368: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 83 KiB/s wr, 13 op/s 2024-03-18T15:04:53.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T15:04:53.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T15:04:53.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: Cluster is now healthy 2024-03-18T15:04:53.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1829615130' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T15:04:53.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.77:6838/3085149460,v1:172.21.15.77:6839/3085149460] up:boot 2024-03-18T15:04:53.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.77:6832/2698185842,v1:172.21.15.77:6834/2698185842] up:boot 2024-03-18T15:04:53.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.77:6833/1177761492,v1:172.21.15.77:6835/1177761492] up:boot 2024-03-18T15:04:53.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.174:6834/1996839651,v1:172.21.15.174:6835/1996839651] up:boot 2024-03-18T15:04:53.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.174:6836/3956780909,v1:172.21.15.174:6837/3956780909] up:boot 2024-03-18T15:04:53.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: mds.? [v2:172.21.15.174:6838/2405750773,v1:172.21.15.174:6839/2405750773] up:boot 2024-03-18T15:04:53.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: fsmap 12 up:standby 2024-03-18T15:04:53.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T15:04:53.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T15:04:53.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T15:04:53.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:04:53.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T15:04:53.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T15:04:53.664 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:53 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1728200658' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T15:04:53.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: pgmap v368: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 83 KiB/s wr, 13 op/s 2024-03-18T15:04:53.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T15:04:53.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T15:04:53.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: Cluster is now healthy 2024-03-18T15:04:53.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1829615130' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T15:04:53.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.77:6838/3085149460,v1:172.21.15.77:6839/3085149460] up:boot 2024-03-18T15:04:53.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.77:6832/2698185842,v1:172.21.15.77:6834/2698185842] up:boot 2024-03-18T15:04:53.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.77:6833/1177761492,v1:172.21.15.77:6835/1177761492] up:boot 2024-03-18T15:04:53.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.174:6834/1996839651,v1:172.21.15.174:6835/1996839651] up:boot 2024-03-18T15:04:53.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.174:6836/3956780909,v1:172.21.15.174:6837/3956780909] up:boot 2024-03-18T15:04:53.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: mds.? [v2:172.21.15.174:6838/2405750773,v1:172.21.15.174:6839/2405750773] up:boot 2024-03-18T15:04:53.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: fsmap 12 up:standby 2024-03-18T15:04:53.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T15:04:53.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T15:04:53.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T15:04:53.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:04:53.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T15:04:53.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T15:04:53.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:53 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1728200658' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T15:04:53.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: pgmap v368: 97 pgs: 97 active+clean; 302 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 83 KiB/s wr, 13 op/s 2024-03-18T15:04:53.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T15:04:53.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T15:04:53.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: Cluster is now healthy 2024-03-18T15:04:53.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1829615130' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T15:04:53.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.77:6838/3085149460,v1:172.21.15.77:6839/3085149460] up:boot 2024-03-18T15:04:53.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.77:6832/2698185842,v1:172.21.15.77:6834/2698185842] up:boot 2024-03-18T15:04:53.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.77:6833/1177761492,v1:172.21.15.77:6835/1177761492] up:boot 2024-03-18T15:04:53.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.174:6834/1996839651,v1:172.21.15.174:6835/1996839651] up:boot 2024-03-18T15:04:53.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.174:6836/3956780909,v1:172.21.15.174:6837/3956780909] up:boot 2024-03-18T15:04:53.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: mds.? [v2:172.21.15.174:6838/2405750773,v1:172.21.15.174:6839/2405750773] up:boot 2024-03-18T15:04:53.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: fsmap 12 up:standby 2024-03-18T15:04:53.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T15:04:53.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T15:04:53.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T15:04:53.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:04:53.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T15:04:53.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: from='mgr.25298 172.21.15.76:0/3234459094' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T15:04:53.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:53 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1728200658' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T15:04:53.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.947+0000 7fa507fff640 1 -- 172.21.15.76:0/1728200658 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v109) ==== 181+0+0 (secure 0 0 0) 0x7fa4f8009de0 con 0x7fa510102110 2024-03-18T15:04:53.948 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'cephfs_metadata' removed 2024-03-18T15:04:53.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.949+0000 7fa515753640 1 -- 172.21.15.76:0/1728200658 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa4fc0a2970 msgr2=0x7fa4fc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:53.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.949+0000 7fa515753640 1 --2- 172.21.15.76:0/1728200658 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa4fc0a2970 0x7fa4fc0a4e30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa508005fd0 tx=0x7fa508009760 comp rx=0 tx=0).stop 2024-03-18T15:04:53.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.949+0000 7fa515753640 1 -- 172.21.15.76:0/1728200658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa510102110 msgr2=0x7fa510198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:53.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.949+0000 7fa515753640 1 --2- 172.21.15.76:0/1728200658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa510102110 0x7fa510198570 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa4f800d950 tx=0x7fa4f800de10 comp rx=0 tx=0).stop 2024-03-18T15:04:53.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.949+0000 7fa515753640 1 -- 172.21.15.76:0/1728200658 shutdown_connections 2024-03-18T15:04:53.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.949+0000 7fa515753640 1 --2- 172.21.15.76:0/1728200658 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7fa4fc0a2970 0x7fa4fc0a4e30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:53.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.949+0000 7fa515753640 1 --2- 172.21.15.76:0/1728200658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa510107760 0x7fa51019c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:53.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.949+0000 7fa515753640 1 --2- 172.21.15.76:0/1728200658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa510102110 0x7fa510198570 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:53.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.949+0000 7fa515753640 1 --2- 172.21.15.76:0/1728200658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa510101750 0x7fa510198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:53.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.949+0000 7fa515753640 1 -- 172.21.15.76:0/1728200658 >> 172.21.15.76:0/1728200658 conn(0x7fa5100fd280 msgr2=0x7fa5100fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:53.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.949+0000 7fa515753640 1 -- 172.21.15.76:0/1728200658 shutdown_connections 2024-03-18T15:04:53.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:53.949+0000 7fa515753640 1 -- 172.21.15.76:0/1728200658 wait complete. 2024-03-18T15:04:53.966 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T15:04:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.094+0000 7f723b90a640 1 -- 172.21.15.76:0/2767709794 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7234106660 msgr2=0x7f7234106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.094+0000 7f723b90a640 1 --2- 172.21.15.76:0/2767709794 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7234106660 0x7f7234106a40 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f7230009930 tx=0x7f723002bc30 comp rx=0 tx=0).stop 2024-03-18T15:04:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.094+0000 7f723b90a640 1 -- 172.21.15.76:0/2767709794 shutdown_connections 2024-03-18T15:04:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.094+0000 7f723b90a640 1 --2- 172.21.15.76:0/2767709794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7234100650 0x7f7234100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.094+0000 7f723b90a640 1 --2- 172.21.15.76:0/2767709794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7234101010 0x7f723410d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.094+0000 7f723b90a640 1 --2- 172.21.15.76:0/2767709794 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7234106660 0x7f7234106a40 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.094+0000 7f723b90a640 1 -- 172.21.15.76:0/2767709794 >> 172.21.15.76:0/2767709794 conn(0x7f72340777e0 msgr2=0x7f7234077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.094+0000 7f723b90a640 1 -- 172.21.15.76:0/2767709794 shutdown_connections 2024-03-18T15:04:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.094+0000 7f723b90a640 1 -- 172.21.15.76:0/2767709794 wait complete. 2024-03-18T15:04:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.095+0000 7f723b90a640 1 Processor -- start 2024-03-18T15:04:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.095+0000 7f723b90a640 1 -- start start 2024-03-18T15:04:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.095+0000 7f723b90a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7234100650 0x7f7234198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.095+0000 7f723b90a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7234101010 0x7f72341994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.095+0000 7f723b90a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7234106660 0x7f72341a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.095+0000 7f723b90a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7234110a70 con 0x7f7234106660 2024-03-18T15:04:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.095+0000 7f723b90a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f72341108f0 con 0x7f7234101010 2024-03-18T15:04:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.095+0000 7f723b90a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7234110bf0 con 0x7f7234100650 2024-03-18T15:04:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.095+0000 7f7239ea7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7234106660 0x7f72341a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.095+0000 7f7239ea7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7234106660 0x7f72341a3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56744/0 (socket says 172.21.15.76:56744) 2024-03-18T15:04:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.095+0000 7f7239ea7640 1 -- 172.21.15.76:0/223525372 learned_addr learned my addr 172.21.15.76:0/223525372 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-18T15:04:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.095+0000 7f72396a6640 1 --2- 172.21.15.76:0/223525372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7234100650 0x7f7234198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.096+0000 7f7238ea5640 1 --2- 172.21.15.76:0/223525372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7234101010 0x7f72341994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.096+0000 7f7239ea7640 1 -- 172.21.15.76:0/223525372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7234100650 msgr2=0x7f7234198f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.096+0000 7f7239ea7640 1 --2- 172.21.15.76:0/223525372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7234100650 0x7f7234198f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.096+0000 7f7239ea7640 1 -- 172.21.15.76:0/223525372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7234101010 msgr2=0x7f72341994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.096+0000 7f7239ea7640 1 --2- 172.21.15.76:0/223525372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7234101010 0x7f72341994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.096+0000 7f7239ea7640 1 -- 172.21.15.76:0/223525372 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72341a4530 con 0x7f7234106660 2024-03-18T15:04:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.096+0000 7f7239ea7640 1 --2- 172.21.15.76:0/223525372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7234106660 0x7f72341a3ff0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f722c00eaa0 tx=0x7f722c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.096+0000 7f722a7fc640 1 -- 172.21.15.76:0/223525372 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722c004d20 con 0x7f7234106660 2024-03-18T15:04:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.096+0000 7f723b90a640 1 -- 172.21.15.76:0/223525372 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72341a4700 con 0x7f7234106660 2024-03-18T15:04:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.096+0000 7f722a7fc640 1 -- 172.21.15.76:0/223525372 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f722c005020 con 0x7f7234106660 2024-03-18T15:04:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.096+0000 7f722a7fc640 1 -- 172.21.15.76:0/223525372 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722c0116d0 con 0x7f7234106660 2024-03-18T15:04:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.096+0000 7f723b90a640 1 -- 172.21.15.76:0/223525372 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72341a4bd0 con 0x7f7234106660 2024-03-18T15:04:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.097+0000 7f723b90a640 1 -- 172.21.15.76:0/223525372 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72340685d0 con 0x7f7234106660 2024-03-18T15:04:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.098+0000 7f722a7fc640 1 -- 172.21.15.76:0/223525372 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f722c0051d0 con 0x7f7234106660 2024-03-18T15:04:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.099+0000 7f722a7fc640 1 --2- 172.21.15.76:0/223525372 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f72200a2970 0x7f72200a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:04:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.099+0000 7f72396a6640 1 --2- 172.21.15.76:0/223525372 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f72200a2970 0x7f72200a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:04:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.099+0000 7f722a7fc640 1 -- 172.21.15.76:0/223525372 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8173+0+0 (secure 0 0 0) 0x7f722c0cf1f0 con 0x7f7234106660 2024-03-18T15:04:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.099+0000 7f72396a6640 1 --2- 172.21.15.76:0/223525372 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f72200a2970 0x7f72200a4e30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7230005ab0 tx=0x7f7230005fd0 comp rx=0 tx=0).ready entity=mgr.25298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:04:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.103+0000 7f722a7fc640 1 -- 172.21.15.76:0/223525372 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f722c016030 con 0x7f7234106660 2024-03-18T15:04:54.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.284+0000 7f723b90a640 1 -- 172.21.15.76:0/223525372 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f723402d1b0 con 0x7f7234106660 2024-03-18T15:04:54.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.948+0000 7f722a7fc640 1 -- 172.21.15.76:0/223525372 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v110) ==== 169+0+0 (secure 0 0 0) 0x7f722c09c100 con 0x7f7234106660 2024-03-18T15:04:54.950 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'cephfs_data' removed 2024-03-18T15:04:54.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.952+0000 7f723b90a640 1 -- 172.21.15.76:0/223525372 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f72200a2970 msgr2=0x7f72200a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:54.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.952+0000 7f723b90a640 1 --2- 172.21.15.76:0/223525372 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f72200a2970 0x7f72200a4e30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7230005ab0 tx=0x7f7230005fd0 comp rx=0 tx=0).stop 2024-03-18T15:04:54.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.952+0000 7f723b90a640 1 -- 172.21.15.76:0/223525372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7234106660 msgr2=0x7f72341a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:04:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.952+0000 7f723b90a640 1 --2- 172.21.15.76:0/223525372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7234106660 0x7f72341a3ff0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f722c00eaa0 tx=0x7f722c00eea0 comp rx=0 tx=0).stop 2024-03-18T15:04:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.953+0000 7f723b90a640 1 -- 172.21.15.76:0/223525372 shutdown_connections 2024-03-18T15:04:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.953+0000 7f723b90a640 1 --2- 172.21.15.76:0/223525372 >> [v2:172.21.15.76:6800/3052007196,v1:172.21.15.76:6801/3052007196] conn(0x7f72200a2970 0x7f72200a4e30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.953+0000 7f723b90a640 1 --2- 172.21.15.76:0/223525372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7234106660 0x7f72341a3ff0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.953+0000 7f723b90a640 1 --2- 172.21.15.76:0/223525372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7234101010 0x7f72341994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.953+0000 7f723b90a640 1 --2- 172.21.15.76:0/223525372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7234100650 0x7f7234198f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:04:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.953+0000 7f723b90a640 1 -- 172.21.15.76:0/223525372 >> 172.21.15.76:0/223525372 conn(0x7f72340777e0 msgr2=0x7f7234102ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:04:54.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.953+0000 7f723b90a640 1 -- 172.21.15.76:0/223525372 shutdown_connections 2024-03-18T15:04:54.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-18T15:04:54.953+0000 7f723b90a640 1 -- 172.21.15.76:0/223525372 wait complete. 2024-03-18T15:04:54.974 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:04:55.020 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:04:55.065 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:04:55.108 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-18T15:04:55.108 DEBUG:teuthology.orchestra.run.smithi076:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T15:04:55.129 DEBUG:teuthology.orchestra.run.smithi077:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T15:04:55.144 DEBUG:teuthology.orchestra.run.smithi174:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T15:04:55.166 INFO:tasks.cephadm:Stopping all daemons... 2024-03-18T15:04:55.166 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-18T15:04:55.167 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mon.a 2024-03-18T15:04:55.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:54 smithi174 ceph-mon[36787]: pgmap v370: 97 pgs: 97 active+clean; 303 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 9 op/s 2024-03-18T15:04:55.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:54 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/1728200658' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T15:04:55.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:54 smithi174 ceph-mon[36787]: osdmap e109: 12 total, 12 up, 12 in 2024-03-18T15:04:55.217 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:54 smithi174 ceph-mon[36787]: from='client.? 172.21.15.76:0/223525372' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T15:04:55.225 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:54 smithi076 ceph-mon[32205]: pgmap v370: 97 pgs: 97 active+clean; 303 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 9 op/s 2024-03-18T15:04:55.226 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:54 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/1728200658' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T15:04:55.226 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:54 smithi076 ceph-mon[32205]: osdmap e109: 12 total, 12 up, 12 in 2024-03-18T15:04:55.226 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:54 smithi076 ceph-mon[32205]: from='client.? 172.21.15.76:0/223525372' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T15:04:55.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:54 smithi077 ceph-mon[35865]: pgmap v370: 97 pgs: 97 active+clean; 303 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 9 op/s 2024-03-18T15:04:55.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:54 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/1728200658' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T15:04:55.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:54 smithi077 ceph-mon[35865]: osdmap e109: 12 total, 12 up, 12 in 2024-03-18T15:04:55.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:54 smithi077 ceph-mon[35865]: from='client.? 172.21.15.76:0/223525372' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T15:04:55.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:55 smithi076 systemd[1]: Stopping Ceph mon.a for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:04:55.889 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:55 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a[32201]: 2024-03-18T15:04:55.487+0000 7f4cc88a4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:04:55.890 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:55 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a[32201]: 2024-03-18T15:04:55.487+0000 7f4cc88a4700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-18T15:04:55.890 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:55 smithi076 podman[93829]: 2024-03-18 15:04:55.600370805 +0000 UTC m=+0.231658591 container died 3df3aa06ebbe1ec071dca70665ddb9ab25137b99c794a5e752c9ce2e1f69f75e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:04:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:55 smithi076 podman[93829]: 2024-03-18 15:04:55.972355609 +0000 UTC m=+0.603643402 container cleanup 3df3aa06ebbe1ec071dca70665ddb9ab25137b99c794a5e752c9ce2e1f69f75e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:04:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 18 15:04:55 smithi076 bash[93829]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-a 2024-03-18T15:04:56.647 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mon.a.service' 2024-03-18T15:04:56.716 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:04:56.716 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-18T15:04:56.716 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-18T15:04:56.717 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mon.b 2024-03-18T15:04:57.024 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:56 smithi077 systemd[1]: Stopping Ceph mon.b for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:04:57.341 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:57 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-b[35861]: 2024-03-18T15:04:57.022+0000 7efd2ccf7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:04:57.341 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:57 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-b[35861]: 2024-03-18T15:04:57.022+0000 7efd2ccf7700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-18T15:04:57.342 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:57 smithi077 podman[72895]: 2024-03-18 15:04:57.043605964 +0000 UTC m=+0.138249786 container died a1709407081a240278764b7663ece9ff8127269e6128b30870b305dc4ee40af1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True) 2024-03-18T15:04:57.667 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:57 smithi077 podman[72895]: 2024-03-18 15:04:57.44011448 +0000 UTC m=+0.534758314 container cleanup a1709407081a240278764b7663ece9ff8127269e6128b30870b305dc4ee40af1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-18T15:04:57.667 INFO:journalctl@ceph.mon.b.smithi077.stdout:Mar 18 15:04:57 smithi077 bash[72895]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-b 2024-03-18T15:04:57.795 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mon.b.service' 2024-03-18T15:04:57.847 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:04:57.848 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-18T15:04:57.848 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-18T15:04:57.848 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mon.c 2024-03-18T15:04:58.177 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:57 smithi174 systemd[1]: Stopping Ceph mon.c for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:04:58.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:58 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-c[36783]: 2024-03-18T15:04:58.175+0000 7fe788bd1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:04:58.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:58 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-c[36783]: 2024-03-18T15:04:58.175+0000 7fe788bd1700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-18T15:04:58.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:58 smithi174 podman[73689]: 2024-03-18 15:04:58.20697741 +0000 UTC m=+0.154559759 container died 5e438e85c48fd81a25099c8a0f90b6831032cb49884ff491891529a6900f7355 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-18T15:04:58.854 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:58 smithi174 podman[73689]: 2024-03-18 15:04:58.602482269 +0000 UTC m=+0.550064619 container cleanup 5e438e85c48fd81a25099c8a0f90b6831032cb49884ff491891529a6900f7355 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-18T15:04:58.855 INFO:journalctl@ceph.mon.c.smithi174.stdout:Mar 18 15:04:58 smithi174 bash[73689]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mon-c 2024-03-18T15:04:59.027 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mon.c.service' 2024-03-18T15:04:59.076 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:04:59.077 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-18T15:04:59.077 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-18T15:04:59.077 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mgr.x 2024-03-18T15:04:59.387 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 15:04:59 smithi076 systemd[1]: Stopping Ceph mgr.x for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:04:59.661 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 15:04:59 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x[32435]: 2024-03-18T15:04:59.385+0000 7f8ffd7bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:04:59.661 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 15:04:59 smithi076 podman[93949]: 2024-03-18 15:04:59.417038386 +0000 UTC m=+0.149712847 container died bb92b6931123a9d1e01f482e87010c49b25f6de40966af28c840f29f100783c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:05:00.073 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 15:04:59 smithi076 podman[93949]: 2024-03-18 15:04:59.837598328 +0000 UTC m=+0.570272784 container cleanup bb92b6931123a9d1e01f482e87010c49b25f6de40966af28c840f29f100783c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD) 2024-03-18T15:05:00.073 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 15:04:59 smithi076 bash[93949]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x 2024-03-18T15:05:00.306 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-18T15:05:00.330 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 15:05:00 smithi076 podman[93963]: 2024-03-18 15:05:00.073163984 +0000 UTC m=+0.651552649 container remove bb92b6931123a9d1e01f482e87010c49b25f6de40966af28c840f29f100783c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-18T15:05:00.330 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 15:05:00 smithi076 systemd[1]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mgr.x.service: Deactivated successfully. 2024-03-18T15:05:00.330 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 15:05:00 smithi076 systemd[1]: Stopped Ceph mgr.x for 9cc83b72-e535-11ee-95c9-87774f69a715. 2024-03-18T15:05:00.331 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 18 15:05:00 smithi076 systemd[1]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mgr.x.service: Consumed 1min 19.272s CPU time. 2024-03-18T15:05:00.343 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:05:00.344 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-18T15:05:00.344 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-18T15:05:00.344 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mgr.y 2024-03-18T15:05:00.692 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 15:05:00 smithi077 systemd[1]: Stopping Ceph mgr.y for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:05:00.971 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 15:05:00 smithi077 podman[73020]: 2024-03-18 15:05:00.690693389 +0000 UTC m=+0.147090009 container died c90a05c8a55e874f979a92f5048e289551dae1179aba5ecd9b4dfc151bf42853 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:05:01.227 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 15:05:01 smithi077 podman[73020]: 2024-03-18 15:05:01.06191435 +0000 UTC m=+0.518310967 container cleanup c90a05c8a55e874f979a92f5048e289551dae1179aba5ecd9b4dfc151bf42853 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-18T15:05:01.228 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Mar 18 15:05:01 smithi077 bash[73020]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-y 2024-03-18T15:05:01.435 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-18T15:05:01.472 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:05:01.473 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-18T15:05:01.473 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-18T15:05:01.473 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mgr.z 2024-03-18T15:05:01.796 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 15:05:01 smithi174 systemd[1]: Stopping Ceph mgr.z for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:05:02.101 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 15:05:01 smithi174 podman[73809]: 2024-03-18 15:05:01.796118158 +0000 UTC m=+0.132561628 container died 2a8836b2bbaa56a2957195e22418704495d6a582757a97994d846384a2cbb930 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-18T15:05:02.460 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 15:05:02 smithi174 podman[73809]: 2024-03-18 15:05:02.209318288 +0000 UTC m=+0.545761755 container cleanup 2a8836b2bbaa56a2957195e22418704495d6a582757a97994d846384a2cbb930 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:05:02.461 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Mar 18 15:05:02 smithi174 bash[73809]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mgr-z 2024-03-18T15:05:02.613 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-18T15:05:02.651 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:05:02.651 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-18T15:05:02.651 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-18T15:05:02.651 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.0 2024-03-18T15:05:02.959 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 15:05:02 smithi076 systemd[1]: Stopping Ceph osd.0 for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:05:03.411 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 15:05:02 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-0[46643]: 2024-03-18T15:05:02.958+0000 7f18c70f9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:05:03.411 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 15:05:02 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-0[46643]: 2024-03-18T15:05:02.958+0000 7f18c70f9700 -1 osd.0 110 *** Got signal Terminated *** 2024-03-18T15:05:03.411 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 15:05:02 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-0[46643]: 2024-03-18T15:05:02.958+0000 7f18c70f9700 -1 osd.0 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:05:08.695 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 15:05:08 smithi076 podman[94071]: 2024-03-18 15:05:08.435059808 +0000 UTC m=+5.592533781 container died 32d7fab3190b1881a00b8a17f7946891105e941d85e76e6f11d032123810b724 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-0, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-18T15:05:09.104 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 15:05:08 smithi076 podman[94071]: 2024-03-18 15:05:08.794366984 +0000 UTC m=+5.951840963 container cleanup 32d7fab3190b1881a00b8a17f7946891105e941d85e76e6f11d032123810b724 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-18T15:05:09.104 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 15:05:08 smithi076 bash[94071]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-0 2024-03-18T15:05:09.411 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 15:05:09 smithi076 podman[94090]: 2024-03-18 15:05:09.104505194 +0000 UTC m=+0.667829239 container remove 32d7fab3190b1881a00b8a17f7946891105e941d85e76e6f11d032123810b724 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:05:09.734 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 15:05:09 smithi076 podman[94176]: 2024-03-18 15:05:09.355071311 +0000 UTC m=+0.018053595 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:05:09.734 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 15:05:09 smithi076 podman[94176]: 2024-03-18 15:05:09.472456534 +0000 UTC m=+0.135438806 container create 54883c7eadd5000bef30ffa9c35b9ef334969d4f3e7fbc4ef882edc12422bc1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-18T15:05:10.032 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 15:05:09 smithi076 podman[94176]: 2024-03-18 15:05:09.734399618 +0000 UTC m=+0.397381888 container init 54883c7eadd5000bef30ffa9c35b9ef334969d4f3e7fbc4ef882edc12422bc1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-18T15:05:10.033 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 15:05:09 smithi076 podman[94176]: 2024-03-18 15:05:09.737661912 +0000 UTC m=+0.400644183 container start 54883c7eadd5000bef30ffa9c35b9ef334969d4f3e7fbc4ef882edc12422bc1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-18T15:05:10.033 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 15:05:09 smithi076 podman[94176]: 2024-03-18 15:05:09.850975983 +0000 UTC m=+0.513958263 container attach 54883c7eadd5000bef30ffa9c35b9ef334969d4f3e7fbc4ef882edc12422bc1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:05:10.313 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 18 15:05:10 smithi076 podman[94176]: 2024-03-18 15:05:10.032777755 +0000 UTC m=+0.695760026 container died 54883c7eadd5000bef30ffa9c35b9ef334969d4f3e7fbc4ef882edc12422bc1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-18T15:05:10.622 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.0.service' 2024-03-18T15:05:10.664 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:05:10.664 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-18T15:05:10.664 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-18T15:05:10.664 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.1 2024-03-18T15:05:10.977 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 15:05:10 smithi077 systemd[1]: Stopping Ceph osd.1 for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:05:11.727 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 15:05:11 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-1[40462]: 2024-03-18T15:05:11.257+0000 7f9797449700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:05:11.727 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 15:05:11 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-1[40462]: 2024-03-18T15:05:11.257+0000 7f9797449700 -1 osd.1 110 *** Got signal Terminated *** 2024-03-18T15:05:11.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 15:05:11 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-1[40462]: 2024-03-18T15:05:11.257+0000 7f9797449700 -1 osd.1 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:05:16.977 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 15:05:16 smithi077 podman[73144]: 2024-03-18 15:05:16.697725585 +0000 UTC m=+5.843236112 container died 24b2458cfd9bf515d780a75bd6c504a1560660e267124c54feb389e14781cbff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-1, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-18T15:05:17.343 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 15:05:17 smithi077 podman[73144]: 2024-03-18 15:05:17.133510416 +0000 UTC m=+6.279020929 container cleanup 24b2458cfd9bf515d780a75bd6c504a1560660e267124c54feb389e14781cbff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-1, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:05:17.344 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 15:05:17 smithi077 bash[73144]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-1 2024-03-18T15:05:17.727 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 15:05:17 smithi077 podman[73163]: 2024-03-18 15:05:17.343226119 +0000 UTC m=+0.645401198 container remove 24b2458cfd9bf515d780a75bd6c504a1560660e267124c54feb389e14781cbff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-1, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-18T15:05:18.030 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 15:05:17 smithi077 podman[73253]: 2024-03-18 15:05:17.651646438 +0000 UTC m=+0.017957447 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:05:18.030 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 15:05:17 smithi077 podman[73253]: 2024-03-18 15:05:17.769204616 +0000 UTC m=+0.135515606 container create aea26c69676a6b46b5db0637b65e130386c39ae2a8bfdf056cf2a9a340ab092c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-18T15:05:18.303 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 15:05:18 smithi077 podman[73253]: 2024-03-18 15:05:18.030149748 +0000 UTC m=+0.396460746 container init aea26c69676a6b46b5db0637b65e130386c39ae2a8bfdf056cf2a9a340ab092c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:05:18.303 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 15:05:18 smithi077 podman[73253]: 2024-03-18 15:05:18.033967754 +0000 UTC m=+0.400278750 container start aea26c69676a6b46b5db0637b65e130386c39ae2a8bfdf056cf2a9a340ab092c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-1-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-18T15:05:18.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 15:05:18 smithi077 podman[73253]: 2024-03-18 15:05:18.121940708 +0000 UTC m=+0.488251722 container attach aea26c69676a6b46b5db0637b65e130386c39ae2a8bfdf056cf2a9a340ab092c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-18T15:05:18.675 INFO:journalctl@ceph.osd.1.smithi077.stdout:Mar 18 15:05:18 smithi077 podman[73253]: 2024-03-18 15:05:18.338562105 +0000 UTC m=+0.704873115 container died aea26c69676a6b46b5db0637b65e130386c39ae2a8bfdf056cf2a9a340ab092c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-18T15:05:18.989 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.1.service' 2024-03-18T15:05:19.030 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:05:19.030 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-18T15:05:19.031 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-18T15:05:19.031 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.2 2024-03-18T15:05:19.355 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 15:05:19 smithi174 systemd[1]: Stopping Ceph osd.2 for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:05:19.717 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 15:05:19 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-2[41937]: 2024-03-18T15:05:19.353+0000 7f86b1f4c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:05:19.717 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 15:05:19 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-2[41937]: 2024-03-18T15:05:19.353+0000 7f86b1f4c700 -1 osd.2 110 *** Got signal Terminated *** 2024-03-18T15:05:19.717 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 15:05:19 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-2[41937]: 2024-03-18T15:05:19.353+0000 7f86b1f4c700 -1 osd.2 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:05:25.107 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 15:05:24 smithi174 podman[73931]: 2024-03-18 15:05:24.801639959 +0000 UTC m=+5.579334784 container died 0590c58a8ebb44fcede77795811c7f9be98217704dc48f3d0318a3c2f3c05d92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-18T15:05:25.449 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 15:05:25 smithi174 podman[73931]: 2024-03-18 15:05:25.205448546 +0000 UTC m=+5.983143372 container cleanup 0590c58a8ebb44fcede77795811c7f9be98217704dc48f3d0318a3c2f3c05d92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-18T15:05:25.449 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 15:05:25 smithi174 bash[73931]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-2 2024-03-18T15:05:25.717 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 15:05:25 smithi174 podman[73948]: 2024-03-18 15:05:25.448861264 +0000 UTC m=+0.648865765 container remove 0590c58a8ebb44fcede77795811c7f9be98217704dc48f3d0318a3c2f3c05d92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-18T15:05:26.060 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 15:05:25 smithi174 podman[74041]: 2024-03-18 15:05:25.705680212 +0000 UTC m=+0.017643886 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:05:26.060 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 15:05:25 smithi174 podman[74041]: 2024-03-18 15:05:25.824817639 +0000 UTC m=+0.136781341 container create 4c44419cd11aefb88436edd597755b882125ba70224e28fdbd3a487d14bf0483 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-18T15:05:26.060 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 15:05:26 smithi174 podman[74041]: 2024-03-18 15:05:26.060505153 +0000 UTC m=+0.372468812 container init 4c44419cd11aefb88436edd597755b882125ba70224e28fdbd3a487d14bf0483 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-18T15:05:26.362 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 15:05:26 smithi174 podman[74041]: 2024-03-18 15:05:26.063752433 +0000 UTC m=+0.375716084 container start 4c44419cd11aefb88436edd597755b882125ba70224e28fdbd3a487d14bf0483 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True) 2024-03-18T15:05:26.363 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 15:05:26 smithi174 podman[74041]: 2024-03-18 15:05:26.160674397 +0000 UTC m=+0.472638078 container attach 4c44419cd11aefb88436edd597755b882125ba70224e28fdbd3a487d14bf0483 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:05:26.717 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 15:05:26 smithi174 podman[74041]: 2024-03-18 15:05:26.362675684 +0000 UTC m=+0.674639348 container died 4c44419cd11aefb88436edd597755b882125ba70224e28fdbd3a487d14bf0483 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-18T15:05:26.976 INFO:journalctl@ceph.osd.2.smithi174.stdout:Mar 18 15:05:26 smithi174 podman[74067]: 2024-03-18 15:05:26.961992074 +0000 UTC m=+0.593227079 container remove 4c44419cd11aefb88436edd597755b882125ba70224e28fdbd3a487d14bf0483 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-2-deactivate, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-18T15:05:26.981 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.2.service' 2024-03-18T15:05:27.037 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:05:27.037 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-18T15:05:27.037 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-18T15:05:27.038 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.3 2024-03-18T15:05:27.367 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:27 smithi076 systemd[1]: Stopping Ceph osd.3 for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:05:27.661 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:27 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3[53298]: 2024-03-18T15:05:27.365+0000 7fef589b2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:05:27.661 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:27 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3[53298]: 2024-03-18T15:05:27.365+0000 7fef589b2700 -1 osd.3 110 *** Got signal Terminated *** 2024-03-18T15:05:27.661 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:27 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3[53298]: 2024-03-18T15:05:27.365+0000 7fef589b2700 -1 osd.3 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:05:29.661 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:29 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3[53298]: 2024-03-18T15:05:29.291+0000 7fef50f9e700 -1 osd.3 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.780001+0000 front 2024-03-18T15:05:07.779920+0000 (oldest deadline 2024-03-18T15:05:28.880058+0000) 2024-03-18T15:05:29.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:29 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:29.461+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:30.289 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:29 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:29.984+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:30.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:30 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:30.175+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:30.661 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:30 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3[53298]: 2024-03-18T15:05:30.288+0000 7fef50f9e700 -1 osd.3 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.780001+0000 front 2024-03-18T15:05:07.779920+0000 (oldest deadline 2024-03-18T15:05:28.880058+0000) 2024-03-18T15:05:30.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:30 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:30.465+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:31.302 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:31 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:31.004+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:31.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:31 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:31.128+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:31.661 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:31 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3[53298]: 2024-03-18T15:05:31.301+0000 7fef50f9e700 -1 osd.3 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.780001+0000 front 2024-03-18T15:05:07.779920+0000 (oldest deadline 2024-03-18T15:05:28.880058+0000) 2024-03-18T15:05:31.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:31 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:31.507+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:31.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:31 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:31.454+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:32.299 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:32 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:31.999+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:32.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:32 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:32.102+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:32.556 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:32 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3[53298]: 2024-03-18T15:05:32.299+0000 7fef50f9e700 -1 osd.3 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.780001+0000 front 2024-03-18T15:05:07.779920+0000 (oldest deadline 2024-03-18T15:05:28.880058+0000) 2024-03-18T15:05:32.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:32 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:32.441+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:32.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:32 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:32.555+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:32.912 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:32 smithi076 podman[94284]: 2024-03-18 15:05:32.762298592 +0000 UTC m=+5.524645645 container died 17aa4f914ff71c45083e229802e8033bbd6c90cfd160aad60f0223177e6bf9dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-18T15:05:33.084 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:32 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:32.725+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.155417+0000 front 2024-03-18T15:05:07.155517+0000 (oldest deadline 2024-03-18T15:05:31.855206+0000) 2024-03-18T15:05:33.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:33 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:33.016+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:33.411 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:33 smithi076 podman[94284]: 2024-03-18 15:05:33.134411608 +0000 UTC m=+5.896758669 container cleanup 17aa4f914ff71c45083e229802e8033bbd6c90cfd160aad60f0223177e6bf9dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:05:33.412 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:33 smithi076 bash[94284]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3 2024-03-18T15:05:33.470 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:33 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:33.083+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:33.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:33 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:33.442+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:33.741 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:33 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:33.595+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:33.741 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:33 smithi076 podman[94313]: 2024-03-18 15:05:33.427915039 +0000 UTC m=+0.659601383 container remove 17aa4f914ff71c45083e229802e8033bbd6c90cfd160aad60f0223177e6bf9dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-18T15:05:34.022 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:33 smithi076 podman[94401]: 2024-03-18 15:05:33.643998448 +0000 UTC m=+0.018015582 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:05:34.022 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:33 smithi076 podman[94401]: 2024-03-18 15:05:33.76231237 +0000 UTC m=+0.136329496 container create 0db5dcacd9a5af884aee954bc48018f497217f678a897ae882f008ae1d7bc6b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-18T15:05:34.022 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:34 smithi076 podman[94401]: 2024-03-18 15:05:34.022095878 +0000 UTC m=+0.396113015 container init 0db5dcacd9a5af884aee954bc48018f497217f678a897ae882f008ae1d7bc6b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-18T15:05:34.059 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:33 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:33.722+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.155417+0000 front 2024-03-18T15:05:07.155517+0000 (oldest deadline 2024-03-18T15:05:31.855206+0000) 2024-03-18T15:05:34.314 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:34 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:34.055+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:34.315 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:34 smithi076 podman[94401]: 2024-03-18 15:05:34.025710893 +0000 UTC m=+0.399728026 container start 0db5dcacd9a5af884aee954bc48018f497217f678a897ae882f008ae1d7bc6b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-18T15:05:34.315 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:34 smithi076 podman[94401]: 2024-03-18 15:05:34.130738935 +0000 UTC m=+0.504756076 container attach 0db5dcacd9a5af884aee954bc48018f497217f678a897ae882f008ae1d7bc6b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-18T15:05:34.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:34 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:34.058+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:34.617 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 18 15:05:34 smithi076 podman[94401]: 2024-03-18 15:05:34.314402315 +0000 UTC m=+0.688419442 container died 0db5dcacd9a5af884aee954bc48018f497217f678a897ae882f008ae1d7bc6b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:05:34.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:34 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:34.403+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:34.902 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:34 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:34.630+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:34.926 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.3.service' 2024-03-18T15:05:34.968 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:05:34.968 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-18T15:05:34.968 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-18T15:05:34.968 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.4 2024-03-18T15:05:35.018 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:34 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:34.739+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.155417+0000 front 2024-03-18T15:05:07.155517+0000 (oldest deadline 2024-03-18T15:05:31.855206+0000) 2024-03-18T15:05:35.161 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:35 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:35.102+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:35.279 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 15:05:35 smithi077 systemd[1]: Stopping Ceph osd.4 for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:05:35.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:35 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:35.018+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:35.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:35 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:35.366+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:35.727 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 15:05:35 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-4[47126]: 2024-03-18T15:05:35.278+0000 7f4f15f56700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:05:35.728 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 15:05:35 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-4[47126]: 2024-03-18T15:05:35.278+0000 7f4f15f56700 -1 osd.4 110 *** Got signal Terminated *** 2024-03-18T15:05:35.728 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 15:05:35 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-4[47126]: 2024-03-18T15:05:35.278+0000 7f4f15f56700 -1 osd.4 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:05:36.034 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:35 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:35.735+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.155417+0000 front 2024-03-18T15:05:07.155517+0000 (oldest deadline 2024-03-18T15:05:31.855206+0000) 2024-03-18T15:05:36.055 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:35 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:35.674+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:36.410 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:36 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:36.054+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:36.466 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:36 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:36.033+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:36.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:36 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:36.331+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:37.055 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:36 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:36.701+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:37.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:36 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:36.998+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:37.217 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:36 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:36.772+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.155417+0000 front 2024-03-18T15:05:07.155517+0000 (oldest deadline 2024-03-18T15:05:31.855206+0000) 2024-03-18T15:05:37.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:37 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:37.055+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:37.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:37 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:37.055+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:37.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:37 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:37.292+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:37.967 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:37 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:37.771+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.155417+0000 front 2024-03-18T15:05:07.155517+0000 (oldest deadline 2024-03-18T15:05:31.855206+0000) 2024-03-18T15:05:37.967 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:37 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:37.771+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:11.855724+0000 front 2024-03-18T15:05:11.855739+0000 (oldest deadline 2024-03-18T15:05:37.155722+0000) 2024-03-18T15:05:38.069 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:37 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:37.699+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:38.069 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:37 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:37.699+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:38.262 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:37 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:37.990+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:38.410 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:38 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:38.068+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:38.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:38 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:38.068+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:38.727 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 15:05:38 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-4[47126]: 2024-03-18T15:05:38.262+0000 7f4f0e542700 -1 osd.4 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.531325+0000 front 2024-03-18T15:05:14.531353+0000 (oldest deadline 2024-03-18T15:05:38.031440+0000) 2024-03-18T15:05:38.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:38 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:38.271+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:38.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:38 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:38.271+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:39.061 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:38 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:38.670+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:39.061 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:38 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:38.670+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:39.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:38 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:38.984+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:39.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:38 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:38.984+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:39.217 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:38 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:38.806+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.155417+0000 front 2024-03-18T15:05:07.155517+0000 (oldest deadline 2024-03-18T15:05:31.855206+0000) 2024-03-18T15:05:39.217 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:38 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:38.806+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:11.855724+0000 front 2024-03-18T15:05:11.855739+0000 (oldest deadline 2024-03-18T15:05:37.155722+0000) 2024-03-18T15:05:39.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:39 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:39.060+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:39.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:39 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:39.060+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:39.727 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 15:05:39 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-4[47126]: 2024-03-18T15:05:39.288+0000 7f4f0e542700 -1 osd.4 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.531325+0000 front 2024-03-18T15:05:14.531353+0000 (oldest deadline 2024-03-18T15:05:38.031440+0000) 2024-03-18T15:05:39.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:39 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:39.292+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:39.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:39 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:39.292+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:40.029 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:39 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:39.662+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:40.030 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:39 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:39.662+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:40.196 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:39 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:39.823+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.155417+0000 front 2024-03-18T15:05:07.155517+0000 (oldest deadline 2024-03-18T15:05:31.855206+0000) 2024-03-18T15:05:40.196 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:39 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:39.823+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:11.855724+0000 front 2024-03-18T15:05:11.855739+0000 (oldest deadline 2024-03-18T15:05:37.155722+0000) 2024-03-18T15:05:40.196 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:39 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:39.961+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:40.196 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:39 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:39.961+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:40.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:40 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:40.029+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:40.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:40 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:40.029+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:40.466 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:40 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:40.195+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:40.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:40 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:40.338+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:40.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:40 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:40.338+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:41.059 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:40 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:40.674+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:41.059 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:40 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:40.674+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:41.080 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 15:05:40 smithi077 podman[73376]: 2024-03-18 15:05:40.805534705 +0000 UTC m=+5.642854688 container died a1bd98f801a803206e87103cd861df71951430fda5cd3157932243d78192620c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-4, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-18T15:05:41.194 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:40 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:40.783+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.155417+0000 front 2024-03-18T15:05:07.155517+0000 (oldest deadline 2024-03-18T15:05:31.855206+0000) 2024-03-18T15:05:41.194 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:40 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:40.783+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:11.855724+0000 front 2024-03-18T15:05:11.855739+0000 (oldest deadline 2024-03-18T15:05:37.155722+0000) 2024-03-18T15:05:41.195 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:40 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:40.954+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:41.195 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:40 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:40.954+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:41.359 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 15:05:41 smithi077 podman[73376]: 2024-03-18 15:05:41.162342974 +0000 UTC m=+5.999662913 container cleanup a1bd98f801a803206e87103cd861df71951430fda5cd3157932243d78192620c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:05:41.359 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 15:05:41 smithi077 bash[73376]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-4 2024-03-18T15:05:41.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:41 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:41.058+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:41.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:41 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:41.058+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:41.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:41 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:41.193+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:41.727 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 15:05:41 smithi077 podman[73393]: 2024-03-18 15:05:41.43031922 +0000 UTC m=+0.624519022 container remove a1bd98f801a803206e87103cd861df71951430fda5cd3157932243d78192620c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-4, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:05:41.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:41 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:41.358+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:41.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:41 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:41.358+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:41.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:41 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:41.641+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:41.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:41 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:41.641+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:42.018 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 15:05:41 smithi077 podman[73487]: 2024-03-18 15:05:41.669261822 +0000 UTC m=+0.017805367 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:05:42.018 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 15:05:41 smithi077 podman[73487]: 2024-03-18 15:05:41.773070557 +0000 UTC m=+0.121614096 container create c4edd4a865c7d5d05196c7c865d6a14fe240488d4818a935c48a2d98f7e5d3d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:05:42.187 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:41 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:41.960+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:42.188 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:41 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:41.960+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:42.188 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:41 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:41.832+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.155417+0000 front 2024-03-18T15:05:07.155517+0000 (oldest deadline 2024-03-18T15:05:31.855206+0000) 2024-03-18T15:05:42.188 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:41 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:41.832+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:11.855724+0000 front 2024-03-18T15:05:11.855739+0000 (oldest deadline 2024-03-18T15:05:37.155722+0000) 2024-03-18T15:05:42.317 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:42 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:42.064+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:42.317 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 15:05:42 smithi077 podman[73487]: 2024-03-18 15:05:42.017834468 +0000 UTC m=+0.366378014 container init c4edd4a865c7d5d05196c7c865d6a14fe240488d4818a935c48a2d98f7e5d3d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-18T15:05:42.317 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 15:05:42 smithi077 podman[73487]: 2024-03-18 15:05:42.021787593 +0000 UTC m=+0.370331139 container start c4edd4a865c7d5d05196c7c865d6a14fe240488d4818a935c48a2d98f7e5d3d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:05:42.317 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 15:05:42 smithi077 podman[73487]: 2024-03-18 15:05:42.091978634 +0000 UTC m=+0.440522183 container attach c4edd4a865c7d5d05196c7c865d6a14fe240488d4818a935c48a2d98f7e5d3d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-18T15:05:42.410 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:42 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:42.084+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:42.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:42 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:42.084+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:42.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:42 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:42.187+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:42.587 INFO:journalctl@ceph.osd.4.smithi077.stdout:Mar 18 15:05:42 smithi077 podman[73487]: 2024-03-18 15:05:42.317494161 +0000 UTC m=+0.666037709 container died c4edd4a865c7d5d05196c7c865d6a14fe240488d4818a935c48a2d98f7e5d3d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:05:42.587 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:42 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:42.374+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:42.587 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:42 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:42.374+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:42.876 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.4.service' 2024-03-18T15:05:42.910 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:42 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:42.626+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:42.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:42 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:42.626+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:42.917 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:05:42.918 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-18T15:05:42.918 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-18T15:05:42.918 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.5 2024-03-18T15:05:43.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:42 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:42.981+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:43.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:42 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:42.981+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:43.194 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:42 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:42.878+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.155417+0000 front 2024-03-18T15:05:07.155517+0000 (oldest deadline 2024-03-18T15:05:31.855206+0000) 2024-03-18T15:05:43.194 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:42 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:42.878+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:11.855724+0000 front 2024-03-18T15:05:11.855739+0000 (oldest deadline 2024-03-18T15:05:37.155722+0000) 2024-03-18T15:05:43.194 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:42 smithi174 systemd[1]: Stopping Ceph osd.5 for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:05:43.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:43 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:43.033+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:43.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:43 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:43.065+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:43.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:43 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:43.065+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:43.470 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:43 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:43.192+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:43.470 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:43 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:43.224+0000 7f7a6c77f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:05:43.470 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:43 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:43.224+0000 7f7a6c77f700 -1 osd.5 110 *** Got signal Terminated *** 2024-03-18T15:05:43.470 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:43 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:43.224+0000 7f7a6c77f700 -1 osd.5 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:05:43.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:43 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:43.407+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:43.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:43 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:43.407+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:43.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:43 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:43.583+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:43.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:43 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:43.583+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:44.201 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:43 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:43.957+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:44.201 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:43 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:43.957+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:44.201 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:43 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:43.886+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.155417+0000 front 2024-03-18T15:05:07.155517+0000 (oldest deadline 2024-03-18T15:05:31.855206+0000) 2024-03-18T15:05:44.201 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:43 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:43.886+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:11.855724+0000 front 2024-03-18T15:05:11.855739+0000 (oldest deadline 2024-03-18T15:05:37.155722+0000) 2024-03-18T15:05:44.380 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:43 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:43.989+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:44.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:44.017+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:44.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:44.017+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:44.466 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:44 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:44.200+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:44.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:44 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:44.379+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:44.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:44 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:44.379+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:44.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:44.561+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:44.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:44.561+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:45.217 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:44 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:44.922+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.155417+0000 front 2024-03-18T15:05:07.155517+0000 (oldest deadline 2024-03-18T15:05:31.855206+0000) 2024-03-18T15:05:45.217 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:44 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:44.922+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:11.855724+0000 front 2024-03-18T15:05:11.855739+0000 (oldest deadline 2024-03-18T15:05:37.155722+0000) 2024-03-18T15:05:45.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:44 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:44.966+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:45.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:44 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:44.966+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:45.394 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:44 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:44.983+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:45.410 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:44.982+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:45.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:44 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:44.982+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:45.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:45 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:45.236+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:45.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:45 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:45.393+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:45.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:45 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:45.393+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:45.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:45 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:45.590+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:45.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:45 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:45.590+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:46.217 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:45 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:45.890+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.155417+0000 front 2024-03-18T15:05:07.155517+0000 (oldest deadline 2024-03-18T15:05:31.855206+0000) 2024-03-18T15:05:46.217 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:45 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:45.890+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:11.855724+0000 front 2024-03-18T15:05:11.855739+0000 (oldest deadline 2024-03-18T15:05:37.155722+0000) 2024-03-18T15:05:46.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:45 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:45.969+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:46.218 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:45 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:45.969+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:46.218 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:45 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:45.969+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:05:46.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:45 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:45.953+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:46.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:45 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:45.973+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:46.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:45 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:45.973+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:46.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:45 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:45.973+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.138503+0000 front 2024-03-18T15:05:22.138548+0000 (oldest deadline 2024-03-18T15:05:45.638550+0000) 2024-03-18T15:05:46.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:46 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:46.284+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:46.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:46 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:46.436+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:46.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:46 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:46.436+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:46.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:46 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:46.545+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:46.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:46 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:46.545+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:46.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:46 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:46.545+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:05:47.217 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:46 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:46.928+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.155417+0000 front 2024-03-18T15:05:07.155517+0000 (oldest deadline 2024-03-18T15:05:31.855206+0000) 2024-03-18T15:05:47.217 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:46 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:46.928+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:11.855724+0000 front 2024-03-18T15:05:11.855739+0000 (oldest deadline 2024-03-18T15:05:37.155722+0000) 2024-03-18T15:05:47.218 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:47 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:47.015+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:47.218 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:47 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:47.015+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:47.218 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:47 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:47.015+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:05:47.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:46 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:46.973+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:47.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:46 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:46.973+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:47.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:46 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:46.973+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.138503+0000 front 2024-03-18T15:05:22.138548+0000 (oldest deadline 2024-03-18T15:05:45.638550+0000) 2024-03-18T15:05:47.420 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:46 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:46.998+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:47.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:47 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:47.247+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:47.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:47 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:47.420+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:47.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:47 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:47.420+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:47.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:47 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:47.420+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:05:47.910 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:47 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:47.507+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:47.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:47 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:47.507+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:47.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:47 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:47.507+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:05:48.254 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:47 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:47.968+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.155417+0000 front 2024-03-18T15:05:07.155517+0000 (oldest deadline 2024-03-18T15:05:31.855206+0000) 2024-03-18T15:05:48.254 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:47 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5[48903]: 2024-03-18T15:05:47.968+0000 7f7a64d6b700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:11.855724+0000 front 2024-03-18T15:05:11.855739+0000 (oldest deadline 2024-03-18T15:05:37.155722+0000) 2024-03-18T15:05:48.255 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:48 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:48.059+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:48.255 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:48 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:48.059+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:48.255 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:48 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:48.059+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:05:48.393 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:47 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:47.985+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:48.410 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:47 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:47.995+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:48.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:47 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:47.995+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:48.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:47 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:47.995+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.138503+0000 front 2024-03-18T15:05:22.138548+0000 (oldest deadline 2024-03-18T15:05:45.638550+0000) 2024-03-18T15:05:48.650 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:48 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:48.253+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:48.650 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:48 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:48.253+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:05:48.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:48 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:48.392+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:48.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:48 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:48.392+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:48.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:48 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:48.392+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:05:48.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:48 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:48.479+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:48.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:48 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:48.479+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:48.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:48 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:48.479+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:05:48.942 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:48 smithi174 podman[74160]: 2024-03-18 15:05:48.649779485 +0000 UTC m=+5.531870727 container died 8693dd595fd82a09479e7eb6a40248e836f8dd197a31a28b0208bac07b6dc433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-18T15:05:49.217 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:49 smithi174 podman[74160]: 2024-03-18 15:05:49.050131561 +0000 UTC m=+5.932222802 container cleanup 8693dd595fd82a09479e7eb6a40248e836f8dd197a31a28b0208bac07b6dc433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:05:49.217 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:49 smithi174 bash[74160]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5 2024-03-18T15:05:49.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:49 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:49.052+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:49.218 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:49 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:49.052+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:49.218 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:49 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:49.052+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:05:49.361 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:49 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:49.007+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:49.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:49 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:49.016+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:49.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:49 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:49.016+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:49.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:49 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:49.016+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.138503+0000 front 2024-03-18T15:05:22.138548+0000 (oldest deadline 2024-03-18T15:05:45.638550+0000) 2024-03-18T15:05:49.657 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:49 smithi174 podman[74178]: 2024-03-18 15:05:49.243043497 +0000 UTC m=+0.594269015 container remove 8693dd595fd82a09479e7eb6a40248e836f8dd197a31a28b0208bac07b6dc433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-18T15:05:49.657 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:49 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:49.280+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:49.658 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:49 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:49.280+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:05:49.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:49 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:49.360+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:49.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:49 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:49.360+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:49.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:49 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:49.360+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:05:49.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:49 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:49.514+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:49.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:49 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:49.514+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:49.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:49 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:49.514+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:05:49.946 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:49 smithi174 podman[74264]: 2024-03-18 15:05:49.560374873 +0000 UTC m=+0.017764698 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:05:49.946 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:49 smithi174 podman[74264]: 2024-03-18 15:05:49.685653482 +0000 UTC m=+0.143043295 container create 8dc07f38bc6cbab42bb496554d9f6146e69c762736591fd35fba021df4bd726f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-18T15:05:49.947 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:49 smithi174 podman[74264]: 2024-03-18 15:05:49.946506514 +0000 UTC m=+0.403896337 container init 8dc07f38bc6cbab42bb496554d9f6146e69c762736591fd35fba021df4bd726f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:05:50.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:50 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:50.080+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:50.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:50 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:50.080+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:50.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:50 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:50.080+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:05:50.217 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:49 smithi174 podman[74264]: 2024-03-18 15:05:49.949601545 +0000 UTC m=+0.406991357 container start 8dc07f38bc6cbab42bb496554d9f6146e69c762736591fd35fba021df4bd726f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:05:50.218 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:50 smithi174 podman[74264]: 2024-03-18 15:05:50.029739552 +0000 UTC m=+0.487129381 container attach 8dc07f38bc6cbab42bb496554d9f6146e69c762736591fd35fba021df4bd726f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-18T15:05:50.364 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:49 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:49.999+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:50.364 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:49 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:49.999+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.239882+0000 front 2024-03-18T15:05:24.239975+0000 (oldest deadline 2024-03-18T15:05:49.539650+0000) 2024-03-18T15:05:50.410 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:49 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:49.992+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:50.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:49 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:49.992+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:50.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:49 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:49.992+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.138503+0000 front 2024-03-18T15:05:22.138548+0000 (oldest deadline 2024-03-18T15:05:45.638550+0000) 2024-03-18T15:05:50.532 INFO:journalctl@ceph.osd.5.smithi174.stdout:Mar 18 15:05:50 smithi174 podman[74264]: 2024-03-18 15:05:50.238308188 +0000 UTC m=+0.695698003 container died 8dc07f38bc6cbab42bb496554d9f6146e69c762736591fd35fba021df4bd726f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-5-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-18T15:05:50.532 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:50 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:50.248+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:50.533 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:50 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:50.248+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:05:50.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:50 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:50.363+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:50.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:50 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:50.363+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:50.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:50 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:50.363+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:05:50.859 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.5.service' 2024-03-18T15:05:50.899 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:05:50.899 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-18T15:05:50.899 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-18T15:05:50.900 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.6 2024-03-18T15:05:50.910 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:50 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:50.561+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:50.910 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:50 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:50.561+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:50.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:50 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:50.561+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:05:51.200 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:51 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:51.080+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:51.200 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:51 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:51.080+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:51.200 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:51 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:51.080+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:05:51.222 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:50 smithi076 systemd[1]: Stopping Ceph osd.6 for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:05:51.222 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:50 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:50.974+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:51.222 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:50 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:50.974+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:51.222 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:50 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:50.974+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.138503+0000 front 2024-03-18T15:05:22.138548+0000 (oldest deadline 2024-03-18T15:05:45.638550+0000) 2024-03-18T15:05:51.222 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:51 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:51.221+0000 7f934e938700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:05:51.223 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:51 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:51.221+0000 7f934e938700 -1 osd.6 110 *** Got signal Terminated *** 2024-03-18T15:05:51.223 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:51 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:51.221+0000 7f934e938700 -1 osd.6 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:05:51.411 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:50 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:50.990+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:51.411 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:50 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:50.990+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.239882+0000 front 2024-03-18T15:05:24.239975+0000 (oldest deadline 2024-03-18T15:05:49.539650+0000) 2024-03-18T15:05:51.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:51 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:51.199+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:51.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:51 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:51.199+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:05:51.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:51 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:51.410+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:51.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:51 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:51.410+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:51.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:51 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:51.410+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:05:51.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:51 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:51.573+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:51.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:51 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:51.573+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:51.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:51 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:51.573+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:05:52.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:51 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:51.949+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:52.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:51 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:51.949+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.239882+0000 front 2024-03-18T15:05:24.239975+0000 (oldest deadline 2024-03-18T15:05:49.539650+0000) 2024-03-18T15:05:52.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:51 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:51.988+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:52.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:51 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:51.988+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:52.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:51 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:51.988+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.138503+0000 front 2024-03-18T15:05:22.138548+0000 (oldest deadline 2024-03-18T15:05:45.638550+0000) 2024-03-18T15:05:52.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:52 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:52.201+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:52.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:52 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:52.201+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:05:52.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:52 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:52.072+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:52.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:52 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:52.072+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:52.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:52 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:52.072+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:05:52.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:52 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:52.377+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:52.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:52 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:52.377+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:52.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:52 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:52.377+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:05:52.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:52 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:52.573+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:52.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:52 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:52.573+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:52.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:52 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:52.573+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:05:53.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:52 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:52.917+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:53.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:52 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:52.917+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.239882+0000 front 2024-03-18T15:05:24.239975+0000 (oldest deadline 2024-03-18T15:05:49.539650+0000) 2024-03-18T15:05:53.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:52 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:52.979+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:53.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:52 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:52.979+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:53.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:52 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:52.979+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.138503+0000 front 2024-03-18T15:05:22.138548+0000 (oldest deadline 2024-03-18T15:05:45.638550+0000) 2024-03-18T15:05:53.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:53 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:53.213+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:53.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:53 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:53.213+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:05:53.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:53 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:53.102+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:53.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:53 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:53.102+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:53.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:53 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:53.102+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:05:53.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:53 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:53.417+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:53.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:53 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:53.417+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:53.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:53 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:53.417+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:05:53.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:53 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:53.606+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:53.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:53 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:53.606+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:53.912 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:53 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:53.606+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:05:54.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:53 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:53.889+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:54.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:53 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:53.889+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.239882+0000 front 2024-03-18T15:05:24.239975+0000 (oldest deadline 2024-03-18T15:05:49.539650+0000) 2024-03-18T15:05:54.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:53 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:53.992+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:54.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:53 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:53.992+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:54.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:53 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:53.992+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.138503+0000 front 2024-03-18T15:05:22.138548+0000 (oldest deadline 2024-03-18T15:05:45.638550+0000) 2024-03-18T15:05:54.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:53 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:53.992+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.640111+0000 front 2024-03-18T15:05:29.640129+0000 (oldest deadline 2024-03-18T15:05:53.740320+0000) 2024-03-18T15:05:54.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:54 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:54.229+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:54.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:54 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:54.229+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:05:54.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:54 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:54.229+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:05:54.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:54 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:54.134+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:54.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:54 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:54.134+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:54.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:54 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:54.134+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:05:54.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:54 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:54.435+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:54.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:54 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:54.435+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:54.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:54 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:54.435+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:05:54.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:54 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:54.568+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:54.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:54 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:54.568+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:54.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:54 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:54.568+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:05:54.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:54 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:54.568+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:05:55.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:54 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:54.843+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:55.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:54 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:54.843+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.239882+0000 front 2024-03-18T15:05:24.239975+0000 (oldest deadline 2024-03-18T15:05:49.539650+0000) 2024-03-18T15:05:55.368 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:55 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:55.261+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:55.369 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:55 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:55.261+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:05:55.369 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:55 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:55.261+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:05:55.369 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:55 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:55.113+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:55.369 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:55 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:55.113+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:55.369 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:55 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:55.113+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:05:55.410 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:55 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:55.038+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:55.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:55 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:55.038+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:55.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:55 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:55.038+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.138503+0000 front 2024-03-18T15:05:22.138548+0000 (oldest deadline 2024-03-18T15:05:45.638550+0000) 2024-03-18T15:05:55.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:55 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:55.038+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.640111+0000 front 2024-03-18T15:05:29.640129+0000 (oldest deadline 2024-03-18T15:05:53.740320+0000) 2024-03-18T15:05:55.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:55 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:55.408+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:55.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:55 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:55.408+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:55.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:55 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:55.408+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:05:55.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:55 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:55.594+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:55.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:55 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:55.594+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:55.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:55 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:55.594+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:05:55.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:55 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:55.594+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:05:56.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:55 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:55.833+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:56.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:55 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:55.833+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.239882+0000 front 2024-03-18T15:05:24.239975+0000 (oldest deadline 2024-03-18T15:05:49.539650+0000) 2024-03-18T15:05:56.228 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:55 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:55.833+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.540155+0000 front 2024-03-18T15:05:29.540082+0000 (oldest deadline 2024-03-18T15:05:55.440063+0000) 2024-03-18T15:05:56.382 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:56 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:56.311+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:56.382 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:56 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:56.311+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:05:56.382 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:56 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:56.311+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:05:56.383 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:56 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:56.067+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:56.383 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:56 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:56.067+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:56.383 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:56 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:56.067+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:05:56.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:56 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:56.023+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:04.035895+0000 front 2024-03-18T15:05:04.035872+0000 (oldest deadline 2024-03-18T15:05:29.336014+0000) 2024-03-18T15:05:56.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:56 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:56.023+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.737715+0000 front 2024-03-18T15:05:15.737598+0000 (oldest deadline 2024-03-18T15:05:36.837556+0000) 2024-03-18T15:05:56.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:56 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:56.023+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.138503+0000 front 2024-03-18T15:05:22.138548+0000 (oldest deadline 2024-03-18T15:05:45.638550+0000) 2024-03-18T15:05:56.411 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:56 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6[59849]: 2024-03-18T15:05:56.023+0000 7f9346f24700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.640111+0000 front 2024-03-18T15:05:29.640129+0000 (oldest deadline 2024-03-18T15:05:53.740320+0000) 2024-03-18T15:05:56.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:56 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:56.420+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:56.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:56 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:56.420+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:56.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:56 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:56.420+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:05:56.911 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:56 smithi076 podman[94508]: 2024-03-18 15:05:56.719066093 +0000 UTC m=+5.627388602 container died bec12d32e07b584245c0e9be83f2b4ecf5332ac241e49f70c8bfcc074b20bed6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:05:56.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:56 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:56.545+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:56.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:56 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:56.545+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:56.912 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:56 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:56.545+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:05:56.912 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:56 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:56.545+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:05:57.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:56 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:56.863+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:57.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:56 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:56.863+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.239882+0000 front 2024-03-18T15:05:24.239975+0000 (oldest deadline 2024-03-18T15:05:49.539650+0000) 2024-03-18T15:05:57.228 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:56 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:56.863+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.540155+0000 front 2024-03-18T15:05:29.540082+0000 (oldest deadline 2024-03-18T15:05:55.440063+0000) 2024-03-18T15:05:57.349 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:57 smithi076 podman[94508]: 2024-03-18 15:05:57.080840512 +0000 UTC m=+5.989163020 container cleanup bec12d32e07b584245c0e9be83f2b4ecf5332ac241e49f70c8bfcc074b20bed6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-18T15:05:57.349 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:57 smithi076 bash[94508]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6 2024-03-18T15:05:57.352 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:57 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:57.039+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:57.352 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:57 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:57.039+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:57.352 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:57 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:57.039+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:05:57.661 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:57 smithi076 podman[94525]: 2024-03-18 15:05:57.349144655 +0000 UTC m=+0.625703261 container remove bec12d32e07b584245c0e9be83f2b4ecf5332ac241e49f70c8bfcc074b20bed6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-18T15:05:57.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:57 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:57.587+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:57.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:57 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:57.587+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:57.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:57 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:57.587+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:05:57.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:57 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:57.587+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:05:57.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:57 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:57.352+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:57.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:57 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:57.352+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:05:57.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:57 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:57.352+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:05:57.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:57 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:57.439+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:57.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:57 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:57.439+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:57.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:57 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:57.439+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:05:57.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:57 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:57.439+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:05:57.922 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:57 smithi076 podman[94618]: 2024-03-18 15:05:57.574162591 +0000 UTC m=+0.017450367 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:05:57.923 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:57 smithi076 podman[94618]: 2024-03-18 15:05:57.708567525 +0000 UTC m=+0.151855332 container create 576010e0d9f36a6b229b7f812456fe378a65f4e0ae1fca3a8fd3fea913949068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:05:57.923 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:57 smithi076 podman[94618]: 2024-03-18 15:05:57.919651183 +0000 UTC m=+0.362938953 container init 576010e0d9f36a6b229b7f812456fe378a65f4e0ae1fca3a8fd3fea913949068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-18T15:05:57.923 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:57 smithi076 podman[94618]: 2024-03-18 15:05:57.922764583 +0000 UTC m=+0.366052345 container start 576010e0d9f36a6b229b7f812456fe378a65f4e0ae1fca3a8fd3fea913949068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-18T15:05:58.216 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:58 smithi076 podman[94618]: 2024-03-18 15:05:58.028171941 +0000 UTC m=+0.471459713 container attach 576010e0d9f36a6b229b7f812456fe378a65f4e0ae1fca3a8fd3fea913949068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-18T15:05:58.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:57 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:57.823+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:58.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:57 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:57.823+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.239882+0000 front 2024-03-18T15:05:24.239975+0000 (oldest deadline 2024-03-18T15:05:49.539650+0000) 2024-03-18T15:05:58.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:57 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:57.823+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.540155+0000 front 2024-03-18T15:05:29.540082+0000 (oldest deadline 2024-03-18T15:05:55.440063+0000) 2024-03-18T15:05:58.395 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:58 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:58.081+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:58.395 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:58 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:58.081+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:58.395 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:58 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:58.081+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:05:58.395 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:58 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:58.081+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:05:58.491 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 18 15:05:58 smithi076 podman[94618]: 2024-03-18 15:05:58.217554382 +0000 UTC m=+0.660842152 container died 576010e0d9f36a6b229b7f812456fe378a65f4e0ae1fca3a8fd3fea913949068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-03-18T15:05:58.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:58 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:58.394+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:58.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:58 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:58.394+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:05:58.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:58 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:58.394+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:05:58.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:58 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:58.416+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:58.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:58 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:58.416+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:58.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:58 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:58.416+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:05:58.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:58 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:58.416+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:05:58.882 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:58 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:58.561+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:58.883 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:58 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:58.561+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:58.883 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:58 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:58.561+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:05:58.883 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:58 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:58.561+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:05:58.907 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.6.service' 2024-03-18T15:05:58.988 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:05:58.988 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-18T15:05:58.989 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-18T15:05:58.989 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.7 2024-03-18T15:05:59.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:58 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:58.793+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:05:59.228 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:58 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:58.793+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.239882+0000 front 2024-03-18T15:05:24.239975+0000 (oldest deadline 2024-03-18T15:05:49.539650+0000) 2024-03-18T15:05:59.228 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:58 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:58.793+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.540155+0000 front 2024-03-18T15:05:29.540082+0000 (oldest deadline 2024-03-18T15:05:55.440063+0000) 2024-03-18T15:05:59.228 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:59 smithi077 systemd[1]: Stopping Ceph osd.7 for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:05:59.357 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:59 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:59.087+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:05:59.358 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:59 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:59.087+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:05:59.358 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:59 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:59.087+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:05:59.358 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:05:59 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:05:59.087+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:05:59.492 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:59 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:59.323+0000 7f869ff33700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:05:59.492 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:59 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:59.323+0000 7f869ff33700 -1 osd.7 110 *** Got signal Terminated *** 2024-03-18T15:05:59.492 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:59 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:59.323+0000 7f869ff33700 -1 osd.7 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:05:59.492 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:59 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:59.429+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:05:59.493 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:59 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:59.429+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:05:59.493 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:59 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:59.429+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:05:59.493 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:05:59 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:05:59.429+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:05:59.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:59 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:59.357+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:05:59.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:59 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:59.357+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:05:59.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:05:59 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:05:59.357+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:05:59.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:59 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:59.553+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:05:59.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:59 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:59.553+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:05:59.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:59 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:59.553+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:05:59.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:05:59 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:05:59.553+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:00.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:59 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:59.812+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:06:00.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:59 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:59.812+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.239882+0000 front 2024-03-18T15:05:24.239975+0000 (oldest deadline 2024-03-18T15:05:49.539650+0000) 2024-03-18T15:06:00.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:05:59 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:05:59.812+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.540155+0000 front 2024-03-18T15:05:29.540082+0000 (oldest deadline 2024-03-18T15:05:55.440063+0000) 2024-03-18T15:06:00.386 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:00 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:00.119+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:00.386 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:00 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:00.119+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:00.386 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:00 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:00.119+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:00.386 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:00 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:00.119+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:00.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:00 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:00.385+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:06:00.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:00 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:00.385+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:06:00.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:00 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:00.385+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:06:00.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:00 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:00.393+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:00.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:00 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:00.393+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:00.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:00 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:00.393+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:00.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:00 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:00.393+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:00.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:00 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:00.602+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:00.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:00 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:00.602+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:00.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:00 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:00.602+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:00.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:00 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:00.602+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:01.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:00 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:06:00.792+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:06:01.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:00 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:06:00.792+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.239882+0000 front 2024-03-18T15:05:24.239975+0000 (oldest deadline 2024-03-18T15:05:49.539650+0000) 2024-03-18T15:06:01.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:00 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:06:00.792+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.540155+0000 front 2024-03-18T15:05:29.540082+0000 (oldest deadline 2024-03-18T15:05:55.440063+0000) 2024-03-18T15:06:01.406 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:01 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:01.135+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:01.406 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:01 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:01.135+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:01.406 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:01 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:01.135+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:01.407 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:01 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:01.135+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:01.407 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:01 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:01.135+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:01.716 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:01 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:01.406+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:06:01.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:01 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:01.406+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:06:01.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:01 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:01.406+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:06:01.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:01 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:01.358+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:01.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:01 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:01.358+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:01.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:01 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:01.358+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:01.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:01 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:01.358+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:01.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:01 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:01.358+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:01.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:01 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:01.572+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:01.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:01 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:01.572+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:01.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:01 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:01.572+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:01.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:01 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:01.572+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:02.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:01 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:06:01.840+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:06:02.228 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:01 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:06:01.840+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.239882+0000 front 2024-03-18T15:05:24.239975+0000 (oldest deadline 2024-03-18T15:05:49.539650+0000) 2024-03-18T15:06:02.228 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:01 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:06:01.840+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.540155+0000 front 2024-03-18T15:05:29.540082+0000 (oldest deadline 2024-03-18T15:05:55.440063+0000) 2024-03-18T15:06:02.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:02 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:02.358+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:06:02.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:02 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:02.358+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:06:02.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:02 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:02.358+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:06:02.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:02 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:02.358+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.617382+0000 front 2024-03-18T15:05:37.617253+0000 (oldest deadline 2024-03-18T15:06:01.717406+0000) 2024-03-18T15:06:02.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:02 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:02.145+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:02.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:02 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:02.145+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:02.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:02 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:02.145+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:02.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:02 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:02.145+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:02.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:02 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:02.145+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:02.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:02 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:02.342+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:02.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:02 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:02.342+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:02.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:02 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:02.342+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:02.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:02 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:02.342+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:02.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:02 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:02.342+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:02.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:02 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:02.525+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:02.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:02 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:02.525+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:02.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:02 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:02.525+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:02.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:02 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:02.525+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:03.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:02 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:06:02.817+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:06:03.228 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:02 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:06:02.817+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.239882+0000 front 2024-03-18T15:05:24.239975+0000 (oldest deadline 2024-03-18T15:05:49.539650+0000) 2024-03-18T15:06:03.228 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:02 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:06:02.817+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.540155+0000 front 2024-03-18T15:05:29.540082+0000 (oldest deadline 2024-03-18T15:05:55.440063+0000) 2024-03-18T15:06:03.228 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:02 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:06:02.817+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:35.440443+0000 front 2024-03-18T15:05:35.440467+0000 (oldest deadline 2024-03-18T15:06:02.441163+0000) 2024-03-18T15:06:03.469 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:03 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:03.363+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:06:03.470 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:03 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:03.363+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:06:03.470 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:03 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:03.363+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:06:03.470 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:03 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:03.363+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.617382+0000 front 2024-03-18T15:05:37.617253+0000 (oldest deadline 2024-03-18T15:06:01.717406+0000) 2024-03-18T15:06:03.470 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:03 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:03.186+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:03.470 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:03 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:03.186+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:03.470 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:03 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:03.186+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:03.470 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:03 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:03.186+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:03.471 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:03 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:03.186+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:03.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:03 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:03.378+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:03.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:03 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:03.378+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:03.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:03 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:03.378+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:03.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:03 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:03.378+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:03.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:03 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:03.378+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:03.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:03 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:03.481+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:03.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:03 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:03.481+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:03.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:03 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:03.481+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:03.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:03 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:03.481+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:04.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:03 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:06:03.850+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.537941+0000 front 2024-03-18T15:05:15.538015+0000 (oldest deadline 2024-03-18T15:05:41.338748+0000) 2024-03-18T15:06:04.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:03 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:06:03.850+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.239882+0000 front 2024-03-18T15:05:24.239975+0000 (oldest deadline 2024-03-18T15:05:49.539650+0000) 2024-03-18T15:06:04.227 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:03 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:06:03.850+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.540155+0000 front 2024-03-18T15:05:29.540082+0000 (oldest deadline 2024-03-18T15:05:55.440063+0000) 2024-03-18T15:06:04.228 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:03 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7[53695]: 2024-03-18T15:06:03.850+0000 7f869c527700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:35.440443+0000 front 2024-03-18T15:05:35.440467+0000 (oldest deadline 2024-03-18T15:06:02.441163+0000) 2024-03-18T15:06:04.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:04 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:04.337+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:06:04.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:04 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:04.337+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:06:04.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:04 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:04.337+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:06:04.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:04 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:04.337+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.617382+0000 front 2024-03-18T15:05:37.617253+0000 (oldest deadline 2024-03-18T15:06:01.717406+0000) 2024-03-18T15:06:04.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:04 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:04.145+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:04.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:04 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:04.145+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:04.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:04 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:04.145+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:04.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:04 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:04.145+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:04.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:04 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:04.145+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:04.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:04 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:04.423+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:04.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:04 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:04.423+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:04.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:04 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:04.423+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:04.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:04 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:04.423+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:04.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:04 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:04.423+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:04.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:04 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:04.505+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:04.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:04 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:04.505+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:04.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:04 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:04.505+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:04.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:04 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:04.505+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:04.912 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:04 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:04.505+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:39.538599+0000 front 2024-03-18T15:05:39.538510+0000 (oldest deadline 2024-03-18T15:06:03.638521+0000) 2024-03-18T15:06:05.200 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:04 smithi077 podman[73594]: 2024-03-18 15:06:04.936619677 +0000 UTC m=+5.743841491 container died 9c6250ba0541f85668b2f15f78cfebcba0adcee0f8b634b4024e3533de82a067 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-18T15:06:05.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:05 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:05.305+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:06:05.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:05 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:05.305+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:06:05.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:05 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:05.305+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:06:05.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:05 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:05.305+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.617382+0000 front 2024-03-18T15:05:37.617253+0000 (oldest deadline 2024-03-18T15:06:01.717406+0000) 2024-03-18T15:06:05.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:05 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:05.145+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:05.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:05 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:05.145+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:05.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:05 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:05.145+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:05.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:05 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:05.145+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:05.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:05 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:05.145+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:05.477 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:05 smithi077 podman[73594]: 2024-03-18 15:06:05.290166172 +0000 UTC m=+6.097387985 container cleanup 9c6250ba0541f85668b2f15f78cfebcba0adcee0f8b634b4024e3533de82a067 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:06:05.477 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:05 smithi077 bash[73594]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7 2024-03-18T15:06:05.477 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:05 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:05.377+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:05.478 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:05 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:05.377+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:05.478 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:05 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:05.377+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:05.478 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:05 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:05.377+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:05.478 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:05 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:05.377+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:05.893 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:05 smithi077 podman[73623]: 2024-03-18 15:06:05.574879439 +0000 UTC m=+0.637745593 container remove 9c6250ba0541f85668b2f15f78cfebcba0adcee0f8b634b4024e3533de82a067 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-18T15:06:05.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:05 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:05.459+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:05.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:05 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:05.459+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:05.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:05 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:05.459+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:05.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:05 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:05.459+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:05.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:05 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:05.459+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:39.538599+0000 front 2024-03-18T15:05:39.538510+0000 (oldest deadline 2024-03-18T15:06:03.638521+0000) 2024-03-18T15:06:06.161 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:05 smithi077 podman[73712]: 2024-03-18 15:06:05.7965206 +0000 UTC m=+0.017562097 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:06:06.162 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:05 smithi077 podman[73712]: 2024-03-18 15:06:05.926936263 +0000 UTC m=+0.147977754 container create 78762b6c7c97034340c756abc52b2e00996b90ba8c9898fec642a6814d69e5c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:06:06.162 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:06 smithi077 podman[73712]: 2024-03-18 15:06:06.161581275 +0000 UTC m=+0.382622776 container init 78762b6c7c97034340c756abc52b2e00996b90ba8c9898fec642a6814d69e5c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:06:06.419 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:06 smithi077 podman[73712]: 2024-03-18 15:06:06.164706374 +0000 UTC m=+0.385747867 container start 78762b6c7c97034340c756abc52b2e00996b90ba8c9898fec642a6814d69e5c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-18T15:06:06.419 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:06 smithi077 podman[73712]: 2024-03-18 15:06:06.261698389 +0000 UTC m=+0.482739882 container attach 78762b6c7c97034340c756abc52b2e00996b90ba8c9898fec642a6814d69e5c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-18T15:06:06.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:06 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:06.277+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:06:06.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:06 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:06.277+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:06:06.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:06 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:06.277+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:06:06.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:06 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:06.277+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.617382+0000 front 2024-03-18T15:05:37.617253+0000 (oldest deadline 2024-03-18T15:06:01.717406+0000) 2024-03-18T15:06:06.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:06 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:06.154+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:06.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:06 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:06.154+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:06.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:06 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:06.154+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:06.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:06 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:06.154+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:06.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:06 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:06.154+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:06.727 INFO:journalctl@ceph.osd.7.smithi077.stdout:Mar 18 15:06:06 smithi077 podman[73712]: 2024-03-18 15:06:06.458761352 +0000 UTC m=+0.679802845 container died 78762b6c7c97034340c756abc52b2e00996b90ba8c9898fec642a6814d69e5c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2) 2024-03-18T15:06:06.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:06 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:06.417+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:06.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:06 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:06.417+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:06.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:06 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:06.417+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:06.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:06 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:06.417+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:06.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:06 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:06.417+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:06.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:06 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:06.449+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:06.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:06 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:06.449+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:06.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:06 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:06.449+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:06.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:06 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:06.449+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:06.912 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:06 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:06.449+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:39.538599+0000 front 2024-03-18T15:05:39.538510+0000 (oldest deadline 2024-03-18T15:06:03.638521+0000) 2024-03-18T15:06:07.027 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.7.service' 2024-03-18T15:06:07.068 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:06:07.068 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-18T15:06:07.068 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-18T15:06:07.068 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.8 2024-03-18T15:06:07.369 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:07 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:07.185+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:07.369 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:07 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:07.185+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:07.369 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:07 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:07.186+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:07.369 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:07 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:07.186+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:07.370 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:07 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:07.186+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:07.370 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:07 smithi174 systemd[1]: Stopping Ceph osd.8 for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:06:07.370 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:07 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:07.273+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:06:07.370 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:07 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:07.273+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:06:07.370 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:07 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:07.273+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:06:07.371 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:07 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:07.273+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.617382+0000 front 2024-03-18T15:05:37.617253+0000 (oldest deadline 2024-03-18T15:06:01.717406+0000) 2024-03-18T15:06:07.371 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:07 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:07.368+0000 7f8a3ba98700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:06:07.371 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:07 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:07.368+0000 7f8a3ba98700 -1 osd.8 110 *** Got signal Terminated *** 2024-03-18T15:06:07.371 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:07 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:07.368+0000 7f8a3ba98700 -1 osd.8 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:06:07.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:07 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:07.372+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:07.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:07 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:07.372+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:07.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:07 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:07.372+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:07.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:07 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:07.372+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:07.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:07 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:07.372+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:07.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:07 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:07.419+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:07.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:07 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:07.419+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:07.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:07 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:07.419+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:07.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:07 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:07.419+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:07.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:07 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:07.419+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:39.538599+0000 front 2024-03-18T15:05:39.538510+0000 (oldest deadline 2024-03-18T15:06:03.638521+0000) 2024-03-18T15:06:08.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:08 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:08.235+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:06:08.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:08 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:08.235+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:06:08.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:08 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:08.235+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:06:08.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:08 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:08.235+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.617382+0000 front 2024-03-18T15:05:37.617253+0000 (oldest deadline 2024-03-18T15:06:01.717406+0000) 2024-03-18T15:06:08.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:08 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:08.140+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:08.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:08 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:08.140+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:08.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:08 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:08.140+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:08.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:08 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:08.140+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:08.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:08 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:08.140+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:08.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:08 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:08.336+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:08.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:08 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:08.336+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:08.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:08 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:08.336+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:08.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:08 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:08.336+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:08.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:08 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:08.336+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:08.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:08 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:08.447+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:08.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:08 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:08.447+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:08.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:08 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:08.447+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:08.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:08 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:08.447+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:08.912 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:08 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:08.447+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:39.538599+0000 front 2024-03-18T15:05:39.538510+0000 (oldest deadline 2024-03-18T15:06:03.638521+0000) 2024-03-18T15:06:09.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:09 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:09.273+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:06:09.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:09 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:09.273+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:06:09.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:09 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:09.273+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:06:09.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:09 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:09.273+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.617382+0000 front 2024-03-18T15:05:37.617253+0000 (oldest deadline 2024-03-18T15:06:01.717406+0000) 2024-03-18T15:06:09.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:09 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:09.102+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:09.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:09 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:09.102+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:09.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:09 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:09.102+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:09.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:09 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:09.102+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:09.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:09 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:09.102+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:09.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:09 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:09.347+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:09.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:09 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:09.347+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:09.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:09 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:09.347+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:09.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:09 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:09.347+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:09.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:09 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:09.347+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:09.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:09 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:09.412+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:09.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:09 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:09.412+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:09.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:09 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:09.412+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:09.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:09 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:09.412+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:09.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:09 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:09.412+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:39.538599+0000 front 2024-03-18T15:05:39.538510+0000 (oldest deadline 2024-03-18T15:06:03.638521+0000) 2024-03-18T15:06:10.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:10 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:10.246+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:06:10.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:10 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:10.246+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:06:10.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:10 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:10.246+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:06:10.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:10 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:10.246+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.617382+0000 front 2024-03-18T15:05:37.617253+0000 (oldest deadline 2024-03-18T15:06:01.717406+0000) 2024-03-18T15:06:10.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:10 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:10.147+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:10.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:10 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:10.147+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:10.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:10 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:10.147+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:10.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:10 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:10.147+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:10.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:10 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:10.147+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:10.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:10 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:10.374+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:10.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:10 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:10.374+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:10.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:10 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:10.374+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:10.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:10 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:10.374+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:10.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:10 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:10.374+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:10.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:10 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:10.413+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:10.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:10 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:10.413+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:10.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:10 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:10.413+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:10.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:10 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:10.413+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:10.911 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:10 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:10.413+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:39.538599+0000 front 2024-03-18T15:05:39.538510+0000 (oldest deadline 2024-03-18T15:06:03.638521+0000) 2024-03-18T15:06:11.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:11 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:11.214+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:06:11.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:11 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:11.214+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:06:11.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:11 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:11.214+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:06:11.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:11 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:11.214+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.617382+0000 front 2024-03-18T15:05:37.617253+0000 (oldest deadline 2024-03-18T15:06:01.717406+0000) 2024-03-18T15:06:11.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:11 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:11.117+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:11.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:11 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:11.117+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:11.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:11 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:11.117+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:11.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:11 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:11.117+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:11.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:11 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:11.117+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:11.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:11 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:11.386+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:11.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:11 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:11.386+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:11.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:11 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:11.386+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:11.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:11 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:11.386+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:11.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:11 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:11.386+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:39.538599+0000 front 2024-03-18T15:05:39.538510+0000 (oldest deadline 2024-03-18T15:06:03.638521+0000) 2024-03-18T15:06:11.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:11 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:11.386+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:43.639217+0000 front 2024-03-18T15:05:43.639182+0000 (oldest deadline 2024-03-18T15:06:11.239489+0000) 2024-03-18T15:06:11.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:11 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:11.369+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:11.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:11 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:11.369+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:11.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:11 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:11.369+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:11.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:11 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:11.369+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:11.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:11 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:11.369+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:12.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:12.221+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:15.314974+0000 front 2024-03-18T15:05:15.315058+0000 (oldest deadline 2024-03-18T15:05:40.014885+0000) 2024-03-18T15:06:12.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:12.221+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:20.015214+0000 front 2024-03-18T15:05:20.015240+0000 (oldest deadline 2024-03-18T15:05:48.215828+0000) 2024-03-18T15:06:12.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:12.221+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:28.216274+0000 front 2024-03-18T15:05:28.216175+0000 (oldest deadline 2024-03-18T15:05:53.516257+0000) 2024-03-18T15:06:12.468 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8[55475]: 2024-03-18T15:06:12.221+0000 7f8a34084700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.617382+0000 front 2024-03-18T15:05:37.617253+0000 (oldest deadline 2024-03-18T15:06:01.717406+0000) 2024-03-18T15:06:12.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:12.108+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:12.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:12.108+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:12.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:12.108+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:12.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:12.108+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:12.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:12 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:12.108+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:12.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:12 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:12.390+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:12.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:12 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:12.390+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:12.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:12 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:12.390+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:12.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:12 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:12.390+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:12.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:12 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:12.390+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:39.538599+0000 front 2024-03-18T15:05:39.538510+0000 (oldest deadline 2024-03-18T15:06:03.638521+0000) 2024-03-18T15:06:12.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:12 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:12.390+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:43.639217+0000 front 2024-03-18T15:05:43.639182+0000 (oldest deadline 2024-03-18T15:06:11.239489+0000) 2024-03-18T15:06:12.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:12 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:12.374+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:12.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:12 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:12.374+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:12.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:12 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:12.374+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:12.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:12 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:12.374+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:12.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:12 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:12.374+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:12.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:12 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:12.374+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:45.937984+0000 front 2024-03-18T15:05:45.938037+0000 (oldest deadline 2024-03-18T15:06:11.838011+0000) 2024-03-18T15:06:13.111 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:12 smithi174 podman[74382]: 2024-03-18 15:06:12.804792805 +0000 UTC m=+5.551535827 container died caf831b8a8bf850163b6c806474812bc1afbf1b6a6c7b4878e519258ab53247a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:06:13.461 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:13 smithi174 podman[74382]: 2024-03-18 15:06:13.210018653 +0000 UTC m=+5.956761667 container cleanup caf831b8a8bf850163b6c806474812bc1afbf1b6a6c7b4878e519258ab53247a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-03-18T15:06:13.461 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:13 smithi174 bash[74382]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8 2024-03-18T15:06:13.461 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:13 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:13.133+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:13.462 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:13 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:13.133+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:13.462 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:13 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:13.133+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:13.462 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:13 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:13.133+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:13.462 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:13 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:13.133+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:13.462 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:13 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:13.133+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:13.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:13 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:13.358+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:13.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:13 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:13.358+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:13.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:13 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:13.358+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:13.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:13 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:13.358+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:13.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:13 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:13.358+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:39.538599+0000 front 2024-03-18T15:05:39.538510+0000 (oldest deadline 2024-03-18T15:06:03.638521+0000) 2024-03-18T15:06:13.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:13 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:13.358+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:43.639217+0000 front 2024-03-18T15:05:43.639182+0000 (oldest deadline 2024-03-18T15:06:11.239489+0000) 2024-03-18T15:06:13.717 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:13 smithi174 podman[74400]: 2024-03-18 15:06:13.461304066 +0000 UTC m=+0.655789461 container remove caf831b8a8bf850163b6c806474812bc1afbf1b6a6c7b4878e519258ab53247a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:06:13.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:13 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:13.359+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:13.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:13 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:13.359+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:13.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:13 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:13.359+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:13.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:13 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:13.359+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:13.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:13 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:13.359+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:13.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:13 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:13.359+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:45.937984+0000 front 2024-03-18T15:05:45.938037+0000 (oldest deadline 2024-03-18T15:06:11.838011+0000) 2024-03-18T15:06:14.100 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:13 smithi174 podman[74498]: 2024-03-18 15:06:13.722643661 +0000 UTC m=+0.017509140 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:06:14.101 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:13 smithi174 podman[74498]: 2024-03-18 15:06:13.820409315 +0000 UTC m=+0.115274814 container create 58e7d43033db542f6c0b65c2c607bf9686752a931657d6e223c8eec54c81d834 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-18T15:06:14.101 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:14 smithi174 podman[74498]: 2024-03-18 15:06:14.023387403 +0000 UTC m=+0.318252888 container init 58e7d43033db542f6c0b65c2c607bf9686752a931657d6e223c8eec54c81d834 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:06:14.101 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:14 smithi174 podman[74498]: 2024-03-18 15:06:14.026666727 +0000 UTC m=+0.321532215 container start 58e7d43033db542f6c0b65c2c607bf9686752a931657d6e223c8eec54c81d834 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-18T15:06:14.467 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:14 smithi174 podman[74498]: 2024-03-18 15:06:14.106202478 +0000 UTC m=+0.401067986 container attach 58e7d43033db542f6c0b65c2c607bf9686752a931657d6e223c8eec54c81d834 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-18T15:06:14.468 INFO:journalctl@ceph.osd.8.smithi174.stdout:Mar 18 15:06:14 smithi174 podman[74498]: 2024-03-18 15:06:14.313743083 +0000 UTC m=+0.608608566 container died 58e7d43033db542f6c0b65c2c607bf9686752a931657d6e223c8eec54c81d834 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-18T15:06:14.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:14.099+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:14.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:14.099+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:14.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:14.099+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:14.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:14.099+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:14.469 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:14.099+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:14.469 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:14 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:14.099+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:14.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:14 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:14.395+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:14.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:14 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:14.395+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:14.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:14 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:14.395+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:14.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:14 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:14.395+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:14.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:14 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:14.395+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:39.538599+0000 front 2024-03-18T15:05:39.538510+0000 (oldest deadline 2024-03-18T15:06:03.638521+0000) 2024-03-18T15:06:14.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:14 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:14.395+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:43.639217+0000 front 2024-03-18T15:05:43.639182+0000 (oldest deadline 2024-03-18T15:06:11.239489+0000) 2024-03-18T15:06:14.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:14 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:14.331+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:14.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:14 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:14.331+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:14.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:14 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:14.331+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:14.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:14 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:14.331+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:14.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:14 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:14.331+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:14.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:14 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:14.331+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:45.937984+0000 front 2024-03-18T15:05:45.938037+0000 (oldest deadline 2024-03-18T15:06:11.838011+0000) 2024-03-18T15:06:14.859 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.8.service' 2024-03-18T15:06:14.905 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:06:14.905 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-18T15:06:14.906 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-18T15:06:14.906 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.9 2024-03-18T15:06:15.235 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:14 smithi076 systemd[1]: Stopping Ceph osd.9 for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:06:15.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:15 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:15.111+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:15.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:15 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:15.111+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:15.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:15 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:15.111+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:15.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:15 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:15.111+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:15.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:15 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:15.111+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:15.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:15 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:15.111+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:15.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:15 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:15.235+0000 7f6c94008700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:06:15.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:15 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:15.235+0000 7f6c94008700 -1 osd.9 110 *** Got signal Terminated *** 2024-03-18T15:06:15.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:15 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:15.235+0000 7f6c94008700 -1 osd.9 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:06:15.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:15 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:15.384+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:15.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:15 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:15.384+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:15.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:15 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:15.384+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:15.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:15 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:15.384+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:15.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:15 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:15.384+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:39.538599+0000 front 2024-03-18T15:05:39.538510+0000 (oldest deadline 2024-03-18T15:06:03.638521+0000) 2024-03-18T15:06:15.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:15 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:15.384+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:43.639217+0000 front 2024-03-18T15:05:43.639182+0000 (oldest deadline 2024-03-18T15:06:11.239489+0000) 2024-03-18T15:06:15.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:15 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:15.300+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:15.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:15 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:15.300+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:15.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:15 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:15.300+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:15.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:15 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:15.300+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:15.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:15 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:15.300+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:15.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:15 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:15.300+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:45.937984+0000 front 2024-03-18T15:05:45.938037+0000 (oldest deadline 2024-03-18T15:06:11.838011+0000) 2024-03-18T15:06:16.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:16 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:16.118+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:16.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:16 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:16.118+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:16.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:16 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:16.118+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:16.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:16 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:16.118+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:16.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:16 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:16.118+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:16.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:16 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:16.118+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:16.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:16 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:16.344+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:16.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:16 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:16.344+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:16.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:16 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:16.344+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:16.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:16 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:16.344+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:16.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:16 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:16.344+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:39.538599+0000 front 2024-03-18T15:05:39.538510+0000 (oldest deadline 2024-03-18T15:06:03.638521+0000) 2024-03-18T15:06:16.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:16 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:16.344+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:43.639217+0000 front 2024-03-18T15:05:43.639182+0000 (oldest deadline 2024-03-18T15:06:11.239489+0000) 2024-03-18T15:06:16.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:16 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:16.312+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:16.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:16 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:16.312+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:16.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:16 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:16.312+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:16.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:16 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:16.312+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:16.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:16 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:16.312+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:16.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:16 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:16.312+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:45.937984+0000 front 2024-03-18T15:05:45.938037+0000 (oldest deadline 2024-03-18T15:06:11.838011+0000) 2024-03-18T15:06:17.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:17 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:17.095+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:17.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:17 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:17.095+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:17.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:17 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:17.095+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:17.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:17 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:17.095+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:17.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:17 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:17.095+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:17.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:17 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:17.095+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:17.477 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:17 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:17.320+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:17.477 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:17 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:17.320+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:17.477 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:17 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:17.320+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:17.477 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:17 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:17.320+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:17.478 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:17 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:17.320+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:17.478 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:17 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:17.320+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:45.937984+0000 front 2024-03-18T15:05:45.938037+0000 (oldest deadline 2024-03-18T15:06:11.838011+0000) 2024-03-18T15:06:17.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:17.300+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:17.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:17.300+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:17.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:17.300+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:17.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:17.300+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:17.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:17.300+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:39.538599+0000 front 2024-03-18T15:05:39.538510+0000 (oldest deadline 2024-03-18T15:06:03.638521+0000) 2024-03-18T15:06:17.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:17 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:17.300+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:43.639217+0000 front 2024-03-18T15:05:43.639182+0000 (oldest deadline 2024-03-18T15:06:11.239489+0000) 2024-03-18T15:06:18.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:18 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:18.088+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:18.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:18 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:18.088+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:18.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:18 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:18.088+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:18.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:18 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:18.088+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:18.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:18 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:18.088+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:18.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:18 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:18.088+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:18.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:18 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:18.261+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:18.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:18 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:18.261+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:18.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:18 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:18.261+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:18.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:18 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:18.261+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:18.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:18 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:18.261+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:39.538599+0000 front 2024-03-18T15:05:39.538510+0000 (oldest deadline 2024-03-18T15:06:03.638521+0000) 2024-03-18T15:06:18.662 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:18 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:18.261+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:43.639217+0000 front 2024-03-18T15:05:43.639182+0000 (oldest deadline 2024-03-18T15:06:11.239489+0000) 2024-03-18T15:06:18.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:18 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:18.354+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:18.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:18 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:18.354+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:18.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:18 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:18.354+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:18.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:18 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:18.354+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:18.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:18 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:18.354+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:18.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:18 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:18.354+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:45.937984+0000 front 2024-03-18T15:05:45.938037+0000 (oldest deadline 2024-03-18T15:06:11.838011+0000) 2024-03-18T15:06:19.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:19 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:19.068+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:19.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:19 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:19.068+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:19.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:19 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:19.068+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:19.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:19 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:19.068+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:19.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:19 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:19.068+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:19.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:19 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:19.068+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:19.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:19 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:19.247+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:06.833336+0000 front 2024-03-18T15:05:06.833284+0000 (oldest deadline 2024-03-18T15:05:28.533453+0000) 2024-03-18T15:06:19.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:19 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:19.247+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:10.835310+0000 front 2024-03-18T15:05:10.835292+0000 (oldest deadline 2024-03-18T15:05:36.734856+0000) 2024-03-18T15:06:19.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:19 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:19.247+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.536736+0000 front 2024-03-18T15:05:22.536582+0000 (oldest deadline 2024-03-18T15:05:46.036616+0000) 2024-03-18T15:06:19.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:19 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:19.247+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:29.537521+0000 front 2024-03-18T15:05:29.537563+0000 (oldest deadline 2024-03-18T15:05:53.637540+0000) 2024-03-18T15:06:19.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:19 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:19.247+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:39.538599+0000 front 2024-03-18T15:05:39.538510+0000 (oldest deadline 2024-03-18T15:06:03.638521+0000) 2024-03-18T15:06:19.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:19 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9[66712]: 2024-03-18T15:06:19.247+0000 7f6c905fc700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:43.639217+0000 front 2024-03-18T15:05:43.639182+0000 (oldest deadline 2024-03-18T15:06:11.239489+0000) 2024-03-18T15:06:19.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:19 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:19.393+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:19.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:19 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:19.393+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:19.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:19 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:19.393+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:19.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:19 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:19.393+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:19.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:19 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:19.393+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:19.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:19 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:19.393+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:45.937984+0000 front 2024-03-18T15:05:45.938037+0000 (oldest deadline 2024-03-18T15:06:11.838011+0000) 2024-03-18T15:06:20.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:20 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:20.105+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:20.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:20 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:20.105+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:20.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:20 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:20.105+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:20.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:20 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:20.105+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:20.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:20 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:20.105+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:20.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:20 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:20.105+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:20.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:20 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:20.397+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:20.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:20 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:20.397+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:20.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:20 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:20.397+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:20.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:20 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:20.397+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:20.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:20 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:20.397+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:20.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:20 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:20.397+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:45.937984+0000 front 2024-03-18T15:05:45.938037+0000 (oldest deadline 2024-03-18T15:06:11.838011+0000) 2024-03-18T15:06:20.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:20 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:20.397+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-18T15:05:54.139108+0000 front 2024-03-18T15:05:54.139082+0000 (oldest deadline 2024-03-18T15:06:20.039006+0000) 2024-03-18T15:06:21.053 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:20 smithi076 podman[94734]: 2024-03-18 15:06:20.798906794 +0000 UTC m=+5.686924681 container died 18079dfe980f6c979d02e53e78d7324cfe9fea7bf48b51917ca6885858b0176c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-18T15:06:21.328 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:21 smithi076 podman[94734]: 2024-03-18 15:06:21.144079681 +0000 UTC m=+6.032097558 container cleanup 18079dfe980f6c979d02e53e78d7324cfe9fea7bf48b51917ca6885858b0176c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-18T15:06:21.328 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:21 smithi076 bash[94734]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9 2024-03-18T15:06:21.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:21 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:21.069+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:21.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:21 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:21.069+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:21.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:21 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:21.069+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:21.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:21 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:21.069+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:21.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:21 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:21.069+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:21.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:21 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:21.069+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:21.661 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:21 smithi076 podman[94751]: 2024-03-18 15:06:21.328625214 +0000 UTC m=+0.525315339 container remove 18079dfe980f6c979d02e53e78d7324cfe9fea7bf48b51917ca6885858b0176c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:06:21.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:21 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:21.368+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:21.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:21 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:21.368+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:21.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:21 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:21.368+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:21.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:21 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:21.368+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:21.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:21 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:21.368+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:21.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:21 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:21.368+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:45.937984+0000 front 2024-03-18T15:05:45.938037+0000 (oldest deadline 2024-03-18T15:06:11.838011+0000) 2024-03-18T15:06:21.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:21 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:21.368+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-18T15:05:54.139108+0000 front 2024-03-18T15:05:54.139082+0000 (oldest deadline 2024-03-18T15:06:20.039006+0000) 2024-03-18T15:06:21.998 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:21 smithi076 podman[94841]: 2024-03-18 15:06:21.645638835 +0000 UTC m=+0.017904524 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:06:21.999 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:21 smithi076 podman[94841]: 2024-03-18 15:06:21.762970518 +0000 UTC m=+0.135236207 container create 956e7cb0c7c2f60b5f35995d59fce14d6071484d05672da875d8b15e8925a523 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-18T15:06:21.999 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:21 smithi076 podman[94841]: 2024-03-18 15:06:21.998738007 +0000 UTC m=+0.371003696 container init 956e7cb0c7c2f60b5f35995d59fce14d6071484d05672da875d8b15e8925a523 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-18T15:06:22.288 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:22 smithi076 podman[94841]: 2024-03-18 15:06:22.002113831 +0000 UTC m=+0.374379523 container start 956e7cb0c7c2f60b5f35995d59fce14d6071484d05672da875d8b15e8925a523 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-18T15:06:22.288 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:22 smithi076 podman[94841]: 2024-03-18 15:06:22.082193655 +0000 UTC m=+0.454459361 container attach 956e7cb0c7c2f60b5f35995d59fce14d6071484d05672da875d8b15e8925a523 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:06:22.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:22 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:22.078+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:22.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:22 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:22.078+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:22.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:22 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:22.078+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:22.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:22 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:22.078+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:22.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:22 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:22.078+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:22.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:22 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:22.078+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:22.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 18 15:06:22 smithi076 podman[94841]: 2024-03-18 15:06:22.288708673 +0000 UTC m=+0.660974369 container died 956e7cb0c7c2f60b5f35995d59fce14d6071484d05672da875d8b15e8925a523 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-18T15:06:22.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:22 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:22.391+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:22.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:22 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:22.391+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:22.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:22 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:22.391+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:22.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:22 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:22.391+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:22.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:22 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:22.391+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:22.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:22 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:22.391+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:45.937984+0000 front 2024-03-18T15:05:45.938037+0000 (oldest deadline 2024-03-18T15:06:11.838011+0000) 2024-03-18T15:06:22.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:22 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:22.391+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-18T15:05:54.139108+0000 front 2024-03-18T15:05:54.139082+0000 (oldest deadline 2024-03-18T15:06:20.039006+0000) 2024-03-18T15:06:22.920 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.9.service' 2024-03-18T15:06:22.961 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:06:22.961 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-18T15:06:22.961 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-18T15:06:22.961 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.10 2024-03-18T15:06:23.260 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:23 smithi077 systemd[1]: Stopping Ceph osd.10 for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:06:23.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:23 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:23.071+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:23.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:23 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:23.071+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:23.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:23 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:23.071+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:23.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:23 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:23.071+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:23.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:23 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:23.071+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:23.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:23 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:23.071+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:23.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:23 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:23.259+0000 7fe650a20700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:06:23.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:23 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:23.259+0000 7fe650a20700 -1 osd.10 110 *** Got signal Terminated *** 2024-03-18T15:06:23.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:23 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:23.259+0000 7fe650a20700 -1 osd.10 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:06:23.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:23 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:23.384+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:23.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:23 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:23.384+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:23.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:23 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:23.384+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:23.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:23 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:23.384+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:23.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:23 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:23.384+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:23.729 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:23 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:23.384+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:45.937984+0000 front 2024-03-18T15:05:45.938037+0000 (oldest deadline 2024-03-18T15:06:11.838011+0000) 2024-03-18T15:06:23.729 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:23 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:23.384+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-18T15:05:54.139108+0000 front 2024-03-18T15:05:54.139082+0000 (oldest deadline 2024-03-18T15:06:20.039006+0000) 2024-03-18T15:06:24.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:24 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:24.082+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:24.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:24 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:24.082+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:24.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:24 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:24.082+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:24.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:24 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:24.082+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:24.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:24 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:24.082+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:24.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:24 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:24.082+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:24.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:24 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:24.338+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:24.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:24 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:24.338+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:24.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:24 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:24.338+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:24.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:24 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:24.338+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:24.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:24 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:24.338+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:24.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:24 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:24.338+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:45.937984+0000 front 2024-03-18T15:05:45.938037+0000 (oldest deadline 2024-03-18T15:06:11.838011+0000) 2024-03-18T15:06:24.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:24 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:24.338+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-18T15:05:54.139108+0000 front 2024-03-18T15:05:54.139082+0000 (oldest deadline 2024-03-18T15:06:20.039006+0000) 2024-03-18T15:06:25.368 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:25 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:25.104+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:25.368 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:25 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:25.104+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:25.369 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:25 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:25.104+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:25.369 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:25 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:25.104+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:25.369 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:25 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:25.104+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:25.369 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:25 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:25.104+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:25.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:25 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:25.361+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:25.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:25 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:25.361+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:25.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:25 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:25.361+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:25.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:25 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:25.361+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:25.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:25 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:25.361+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:25.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:25 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:25.361+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:45.937984+0000 front 2024-03-18T15:05:45.938037+0000 (oldest deadline 2024-03-18T15:06:11.838011+0000) 2024-03-18T15:06:25.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:25 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:25.361+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-18T15:05:54.139108+0000 front 2024-03-18T15:05:54.139082+0000 (oldest deadline 2024-03-18T15:06:20.039006+0000) 2024-03-18T15:06:26.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:26 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:26.062+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:26.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:26 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:26.062+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:26.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:26 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:26.062+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:26.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:26 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:26.062+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:26.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:26 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:26.062+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:26.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:26 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:26.062+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:26.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:26 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:26.062+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-03-18T15:06:04.060175+0000 front 2024-03-18T15:06:04.060250+0000 (oldest deadline 2024-03-18T15:06:25.160135+0000) 2024-03-18T15:06:26.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:26 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:26.341+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:26.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:26 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:26.341+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:26.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:26 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:26.341+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:26.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:26 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:26.341+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:26.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:26 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:26.341+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:26.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:26 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:26.341+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:45.937984+0000 front 2024-03-18T15:05:45.938037+0000 (oldest deadline 2024-03-18T15:06:11.838011+0000) 2024-03-18T15:06:26.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:26 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:26.341+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-18T15:05:54.139108+0000 front 2024-03-18T15:05:54.139082+0000 (oldest deadline 2024-03-18T15:06:20.039006+0000) 2024-03-18T15:06:27.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:27 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:27.033+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:27.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:27 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:27.033+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:27.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:27 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:27.033+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:27.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:27 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:27.033+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:27.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:27 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:27.033+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:27.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:27 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:27.033+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:27.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:27 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:27.033+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-03-18T15:06:04.060175+0000 front 2024-03-18T15:06:04.060250+0000 (oldest deadline 2024-03-18T15:06:25.160135+0000) 2024-03-18T15:06:27.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:27 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:27.303+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:02.929151+0000 front 2024-03-18T15:05:02.929179+0000 (oldest deadline 2024-03-18T15:05:30.530048+0000) 2024-03-18T15:06:27.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:27 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:27.303+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.031207+0000 front 2024-03-18T15:05:14.031237+0000 (oldest deadline 2024-03-18T15:05:37.531379+0000) 2024-03-18T15:06:27.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:27 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:27.303+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:22.033808+0000 front 2024-03-18T15:05:22.033724+0000 (oldest deadline 2024-03-18T15:05:46.733837+0000) 2024-03-18T15:06:27.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:27 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:27.303+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.835144+0000 front 2024-03-18T15:05:30.835125+0000 (oldest deadline 2024-03-18T15:05:56.635811+0000) 2024-03-18T15:06:27.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:27 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:27.303+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:36.636368+0000 front 2024-03-18T15:05:36.636332+0000 (oldest deadline 2024-03-18T15:06:01.336479+0000) 2024-03-18T15:06:27.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:27 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:27.303+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:45.937984+0000 front 2024-03-18T15:05:45.938037+0000 (oldest deadline 2024-03-18T15:06:11.838011+0000) 2024-03-18T15:06:27.728 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:27 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10[60308]: 2024-03-18T15:06:27.303+0000 7fe64900c700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-18T15:05:54.139108+0000 front 2024-03-18T15:05:54.139082+0000 (oldest deadline 2024-03-18T15:06:20.039006+0000) 2024-03-18T15:06:28.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:28 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:28.029+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:28.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:28 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:28.029+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:28.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:28 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:28.029+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:28.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:28 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:28.029+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:28.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:28 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:28.029+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:28.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:28 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:28.029+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:28.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:28 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:28.029+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-03-18T15:06:04.060175+0000 front 2024-03-18T15:06:04.060250+0000 (oldest deadline 2024-03-18T15:06:25.160135+0000) 2024-03-18T15:06:29.043 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:28 smithi077 podman[73818]: 2024-03-18 15:06:28.789641969 +0000 UTC m=+5.633802976 container died 352db622b1b75b473724074c329abedd0f73a7476d89be469e84fac208830d3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-03-18T15:06:29.427 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:29 smithi077 podman[73818]: 2024-03-18 15:06:29.142195471 +0000 UTC m=+5.986356470 container cleanup 352db622b1b75b473724074c329abedd0f73a7476d89be469e84fac208830d3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-18T15:06:29.427 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:29 smithi077 bash[73818]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10 2024-03-18T15:06:29.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:29 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:29.059+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:29.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:29 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:29.059+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:29.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:29 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:29.059+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:29.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:29 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:29.059+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:29.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:29 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:29.059+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:29.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:29 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:29.059+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:29.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:29 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:29.059+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-03-18T15:06:04.060175+0000 front 2024-03-18T15:06:04.060250+0000 (oldest deadline 2024-03-18T15:06:25.160135+0000) 2024-03-18T15:06:29.727 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:29 smithi077 podman[73835]: 2024-03-18 15:06:29.426635074 +0000 UTC m=+0.631798479 container remove 352db622b1b75b473724074c329abedd0f73a7476d89be469e84fac208830d3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-18T15:06:30.046 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:29 smithi077 podman[73927]: 2024-03-18 15:06:29.657305796 +0000 UTC m=+0.017507999 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:06:30.047 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:29 smithi077 podman[73927]: 2024-03-18 15:06:29.760883943 +0000 UTC m=+0.121086165 container create 8406c976c92079908462720d7fe4b098347ec9f28ea0fb93241d1ea668d82609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-18T15:06:30.047 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:29 smithi077 podman[73927]: 2024-03-18 15:06:29.971997483 +0000 UTC m=+0.332199704 container init 8406c976c92079908462720d7fe4b098347ec9f28ea0fb93241d1ea668d82609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:06:30.047 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:29 smithi077 podman[73927]: 2024-03-18 15:06:29.975200186 +0000 UTC m=+0.335402385 container start 8406c976c92079908462720d7fe4b098347ec9f28ea0fb93241d1ea668d82609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-18T15:06:30.047 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:30 smithi077 podman[73927]: 2024-03-18 15:06:30.04657014 +0000 UTC m=+0.406772352 container attach 8406c976c92079908462720d7fe4b098347ec9f28ea0fb93241d1ea668d82609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-18T15:06:30.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:30 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:30.029+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:30.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:30 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:30.029+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:30.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:30 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:30.029+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:30.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:30 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:30.029+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:30.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:30 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:30.029+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:30.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:30 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:30.029+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:30.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:30 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:30.029+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-03-18T15:06:04.060175+0000 front 2024-03-18T15:06:04.060250+0000 (oldest deadline 2024-03-18T15:06:25.160135+0000) 2024-03-18T15:06:30.477 INFO:journalctl@ceph.osd.10.smithi077.stdout:Mar 18 15:06:30 smithi077 podman[73927]: 2024-03-18 15:06:30.270937454 +0000 UTC m=+0.631139657 container died 8406c976c92079908462720d7fe4b098347ec9f28ea0fb93241d1ea668d82609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-03-18T15:06:30.825 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.10.service' 2024-03-18T15:06:30.865 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:06:30.865 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-18T15:06:30.865 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-18T15:06:30.866 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.11 2024-03-18T15:06:31.170 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:30 smithi174 systemd[1]: Stopping Ceph osd.11 for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:06:31.170 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:30 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:30.988+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:31.170 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:30 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:30.988+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:31.171 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:30 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:30.988+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:31.171 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:30 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:30.988+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:31.171 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:30 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:30.988+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:31.171 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:30 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:30.988+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:31.171 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:30 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:30.988+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-03-18T15:06:04.060175+0000 front 2024-03-18T15:06:04.060250+0000 (oldest deadline 2024-03-18T15:06:25.160135+0000) 2024-03-18T15:06:31.171 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:31 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:31.168+0000 7f8985ecc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:06:31.171 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:31 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:31.168+0000 7f8985ecc700 -1 osd.11 110 *** Got signal Terminated *** 2024-03-18T15:06:31.171 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:31 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:31.168+0000 7f8985ecc700 -1 osd.11 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:06:32.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:31 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:31.993+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:32.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:31 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:31.993+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:32.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:31 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:31.993+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:32.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:31 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:31.993+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:32.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:31 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:31.993+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:32.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:31 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:31.993+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:32.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:31 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:31.993+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-03-18T15:06:04.060175+0000 front 2024-03-18T15:06:04.060250+0000 (oldest deadline 2024-03-18T15:06:25.160135+0000) 2024-03-18T15:06:33.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:32 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:32.972+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:33.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:32 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:32.972+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:33.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:32 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:32.972+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:33.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:32 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:32.972+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:33.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:32 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:32.972+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:33.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:32 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:32.972+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:33.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:32 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:32.972+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-03-18T15:06:04.060175+0000 front 2024-03-18T15:06:04.060250+0000 (oldest deadline 2024-03-18T15:06:25.160135+0000) 2024-03-18T15:06:34.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:33 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:33.924+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:34.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:33 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:33.924+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:34.218 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:33 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:33.924+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:34.218 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:33 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:33.924+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:34.218 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:33 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:33.924+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:34.218 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:33 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:33.924+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:34.218 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:33 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:33.924+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-03-18T15:06:04.060175+0000 front 2024-03-18T15:06:04.060250+0000 (oldest deadline 2024-03-18T15:06:25.160135+0000) 2024-03-18T15:06:35.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:34 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:34.955+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:35.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:34 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:34.955+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:35.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:34 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:34.955+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:35.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:34 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:34.955+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:35.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:34 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:34.955+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:35.218 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:34 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:34.955+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:35.218 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:34 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:34.955+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-03-18T15:06:04.060175+0000 front 2024-03-18T15:06:04.060250+0000 (oldest deadline 2024-03-18T15:06:25.160135+0000) 2024-03-18T15:06:36.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:35 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:35.981+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-18T15:05:07.348980+0000 front 2024-03-18T15:05:07.348856+0000 (oldest deadline 2024-03-18T15:05:29.649147+0000) 2024-03-18T15:06:36.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:35 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:35.981+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-03-18T15:05:14.850687+0000 front 2024-03-18T15:05:14.850592+0000 (oldest deadline 2024-03-18T15:05:38.950737+0000) 2024-03-18T15:06:36.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:35 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:35.981+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-03-18T15:05:24.052995+0000 front 2024-03-18T15:05:24.052909+0000 (oldest deadline 2024-03-18T15:05:45.153189+0000) 2024-03-18T15:06:36.467 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:35 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:35.981+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-18T15:05:30.954581+0000 front 2024-03-18T15:05:30.954649+0000 (oldest deadline 2024-03-18T15:05:57.355328+0000) 2024-03-18T15:06:36.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:35 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:35.981+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-03-18T15:05:37.355813+0000 front 2024-03-18T15:05:37.355981+0000 (oldest deadline 2024-03-18T15:06:00.855720+0000) 2024-03-18T15:06:36.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:35 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:35.981+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-03-18T15:05:47.757605+0000 front 2024-03-18T15:05:47.757642+0000 (oldest deadline 2024-03-18T15:06:13.057746+0000) 2024-03-18T15:06:36.468 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:35 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11[62391]: 2024-03-18T15:06:35.981+0000 7f897e4b8700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-03-18T15:06:04.060175+0000 front 2024-03-18T15:06:04.060250+0000 (oldest deadline 2024-03-18T15:06:25.160135+0000) 2024-03-18T15:06:36.967 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:36 smithi174 podman[74613]: 2024-03-18 15:06:36.711410491 +0000 UTC m=+5.649328650 container died 5f547efce7166238c9d5088ced86170a3ede07d13ec03de97a6a56030e9c7c3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-18T15:06:37.245 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:37 smithi174 podman[74613]: 2024-03-18 15:06:37.05294977 +0000 UTC m=+5.990867927 container cleanup 5f547efce7166238c9d5088ced86170a3ede07d13ec03de97a6a56030e9c7c3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True) 2024-03-18T15:06:37.246 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:37 smithi174 bash[74613]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11 2024-03-18T15:06:37.654 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:37 smithi174 podman[74628]: 2024-03-18 15:06:37.245636595 +0000 UTC m=+0.533214300 container remove 5f547efce7166238c9d5088ced86170a3ede07d13ec03de97a6a56030e9c7c3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:06:37.948 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:37 smithi174 podman[74724]: 2024-03-18 15:06:37.654691041 +0000 UTC m=+0.107776368 container create c9cc202182f36c1dd380d921eed3f882bb1438bc3bba053619d4a691bf540948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-18T15:06:37.948 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:37 smithi174 podman[74724]: 2024-03-18 15:06:37.564861947 +0000 UTC m=+0.017947266 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:06:37.948 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:37 smithi174 podman[74724]: 2024-03-18 15:06:37.865142949 +0000 UTC m=+0.318228277 container init c9cc202182f36c1dd380d921eed3f882bb1438bc3bba053619d4a691bf540948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-18T15:06:37.949 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:37 smithi174 podman[74724]: 2024-03-18 15:06:37.868571478 +0000 UTC m=+0.321656795 container start c9cc202182f36c1dd380d921eed3f882bb1438bc3bba053619d4a691bf540948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:06:37.949 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:37 smithi174 podman[74724]: 2024-03-18 15:06:37.948255854 +0000 UTC m=+0.401341182 container attach c9cc202182f36c1dd380d921eed3f882bb1438bc3bba053619d4a691bf540948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5) 2024-03-18T15:06:38.217 INFO:journalctl@ceph.osd.11.smithi174.stdout:Mar 18 15:06:38 smithi174 podman[74724]: 2024-03-18 15:06:38.155981866 +0000 UTC m=+0.609067168 container died c9cc202182f36c1dd380d921eed3f882bb1438bc3bba053619d4a691bf540948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-osd-11-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-18T15:06:38.736 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@osd.11.service' 2024-03-18T15:06:38.777 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:06:38.777 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-18T15:06:38.778 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-18T15:06:38.778 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.a 2024-03-18T15:06:39.090 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 18 15:06:38 smithi076 systemd[1]: Stopping Ceph mds.a for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:06:39.411 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 18 15:06:39 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-a[73374]: 2024-03-18T15:06:39.089+0000 7fdc9692b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:06:39.411 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 18 15:06:39 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-a[73374]: 2024-03-18T15:06:39.089+0000 7fdc9692b700 -1 mds.a *** got signal Terminated *** 2024-03-18T15:06:43.040 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 18 15:06:42 smithi076 podman[94971]: 2024-03-18 15:06:42.715837046 +0000 UTC m=+3.741020048 container died bf67b06ccf43e9b6831399025156f8545bac483c3828056b92fc48c48f1d4a26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-a, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-18T15:06:43.348 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 18 15:06:43 smithi076 podman[94971]: 2024-03-18 15:06:43.114409358 +0000 UTC m=+4.139592356 container cleanup bf67b06ccf43e9b6831399025156f8545bac483c3828056b92fc48c48f1d4a26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-a, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-18T15:06:43.349 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 18 15:06:43 smithi076 bash[94971]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-a 2024-03-18T15:06:43.482 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.a.service' 2024-03-18T15:06:43.519 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:06:43.520 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-18T15:06:43.520 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-18T15:06:43.520 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.d 2024-03-18T15:06:43.911 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 18 15:06:43 smithi076 systemd[1]: Stopping Ceph mds.d for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:06:43.911 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 18 15:06:43 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-d[72418]: 2024-03-18T15:06:43.834+0000 7fca4a2c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:06:43.911 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 18 15:06:43 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-d[72418]: 2024-03-18T15:06:43.834+0000 7fca4a2c2700 -1 mds.d *** got signal Terminated *** 2024-03-18T15:06:47.236 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 18 15:06:46 smithi076 podman[95091]: 2024-03-18 15:06:46.980094839 +0000 UTC m=+3.242560435 container died 3e7989df5e0c51441016dae1cfc6ae987e5a9ff1fd5ad3d0a92d9f80bc8c4cd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:06:47.610 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 18 15:06:47 smithi076 podman[95091]: 2024-03-18 15:06:47.326135193 +0000 UTC m=+3.588600791 container cleanup 3e7989df5e0c51441016dae1cfc6ae987e5a9ff1fd5ad3d0a92d9f80bc8c4cd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-18T15:06:47.611 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 18 15:06:47 smithi076 bash[95091]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-d 2024-03-18T15:06:47.755 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.d.service' 2024-03-18T15:06:47.790 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:06:47.790 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-18T15:06:47.790 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-18T15:06:47.791 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.g 2024-03-18T15:06:48.161 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 18 15:06:47 smithi076 systemd[1]: Stopping Ceph mds.g for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:06:48.161 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 18 15:06:48 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-g[73027]: 2024-03-18T15:06:48.104+0000 7f6523f74700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:06:48.161 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 18 15:06:48 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-g[73027]: 2024-03-18T15:06:48.104+0000 7f6523f74700 -1 mds.g *** got signal Terminated *** 2024-03-18T15:06:52.080 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 18 15:06:51 smithi076 podman[95215]: 2024-03-18 15:06:51.742923997 +0000 UTC m=+3.734631169 container died d5e4e51ec334da5a584b8d05e551820678c799202a334b66b039fc76f89572bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-g, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-18T15:06:52.372 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 18 15:06:52 smithi076 podman[95215]: 2024-03-18 15:06:52.154355366 +0000 UTC m=+4.146062579 container cleanup d5e4e51ec334da5a584b8d05e551820678c799202a334b66b039fc76f89572bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-g, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5) 2024-03-18T15:06:52.372 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 18 15:06:52 smithi076 bash[95215]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-g 2024-03-18T15:06:52.511 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.g.service' 2024-03-18T15:06:52.546 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:06:52.546 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-18T15:06:52.547 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-18T15:06:52.547 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.j 2024-03-18T15:06:52.911 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 18 15:06:52 smithi076 systemd[1]: Stopping Ceph mds.j for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:06:52.912 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 18 15:06:52 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-j[72048]: 2024-03-18T15:06:52.865+0000 7f835bda6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:06:52.912 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 18 15:06:52 smithi076 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-j[72048]: 2024-03-18T15:06:52.865+0000 7f835bda6700 -1 mds.j *** got signal Terminated *** 2024-03-18T15:06:56.366 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 18 15:06:56 smithi076 podman[95337]: 2024-03-18 15:06:56.062797642 +0000 UTC m=+3.301978305 container died 4e1a88725374a383045efd1033c1096d715e10581c32173c30a957201ebb30cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-j, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-18T15:06:56.366 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 18 15:06:56 smithi076 podman[95337]: 2024-03-18 15:06:56.365993627 +0000 UTC m=+3.605174297 container cleanup 4e1a88725374a383045efd1033c1096d715e10581c32173c30a957201ebb30cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-18T15:06:56.661 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 18 15:06:56 smithi076 bash[95337]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-j 2024-03-18T15:06:56.661 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 18 15:06:56 smithi076 podman[95351]: 2024-03-18 15:06:56.591914352 +0000 UTC m=+0.522972065 container remove 4e1a88725374a383045efd1033c1096d715e10581c32173c30a957201ebb30cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:06:56.727 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.j.service' 2024-03-18T15:06:56.761 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:06:56.761 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-18T15:06:56.762 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-18T15:06:56.762 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.b 2024-03-18T15:06:57.136 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 15:06:56 smithi077 systemd[1]: Stopping Ceph mds.b for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:06:57.477 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 15:06:57 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-b[66028]: 2024-03-18T15:06:57.135+0000 7f7b01d60700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:06:57.477 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 15:06:57 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-b[66028]: 2024-03-18T15:06:57.135+0000 7f7b01d60700 -1 mds.b *** got signal Terminated *** 2024-03-18T15:07:00.288 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 15:07:00 smithi077 podman[74044]: 2024-03-18 15:07:00.012972415 +0000 UTC m=+3.059786084 container died 7709a3c15615b82ae888d42e45f2729bbbe2ca826e886c28a89d75f9a865e30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-18T15:07:00.613 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 15:07:00 smithi077 podman[74044]: 2024-03-18 15:07:00.38708648 +0000 UTC m=+3.433900146 container cleanup 7709a3c15615b82ae888d42e45f2729bbbe2ca826e886c28a89d75f9a865e30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-18T15:07:00.614 INFO:journalctl@ceph.mds.b.smithi077.stdout:Mar 18 15:07:00 smithi077 bash[74044]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-b 2024-03-18T15:07:00.780 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.b.service' 2024-03-18T15:07:00.817 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:07:00.818 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-18T15:07:00.818 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-18T15:07:00.818 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.e 2024-03-18T15:07:01.141 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:07:00 smithi077 systemd[1]: Stopping Ceph mds.e for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:07:01.458 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:07:01 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: 2024-03-18T15:07:01.139+0000 7f4f4ecb5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:07:01.459 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:07:01 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e[65400]: 2024-03-18T15:07:01.139+0000 7f4f4ecb5700 -1 mds.e *** got signal Terminated *** 2024-03-18T15:07:04.275 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:07:04 smithi077 podman[74174]: 2024-03-18 15:07:04.007123528 +0000 UTC m=+2.967073639 container died d4bb2c6604eda4b0d04ae9368fe4baf9070cbc52621f73f2dfee45f33dda921d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0) 2024-03-18T15:07:04.633 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:07:04 smithi077 podman[74174]: 2024-03-18 15:07:04.374061867 +0000 UTC m=+3.334011973 container cleanup d4bb2c6604eda4b0d04ae9368fe4baf9070cbc52621f73f2dfee45f33dda921d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-18T15:07:04.633 INFO:journalctl@ceph.mds.e.smithi077.stdout:Mar 18 15:07:04 smithi077 bash[74174]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-e 2024-03-18T15:07:04.801 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.e.service' 2024-03-18T15:07:04.838 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:07:04.838 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-18T15:07:04.839 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-18T15:07:04.839 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.h 2024-03-18T15:07:05.153 INFO:journalctl@ceph.mds.h.smithi077.stdout:Mar 18 15:07:04 smithi077 systemd[1]: Stopping Ceph mds.h for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:07:05.477 INFO:journalctl@ceph.mds.h.smithi077.stdout:Mar 18 15:07:05 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-h[65709]: 2024-03-18T15:07:05.152+0000 7f9bdc168700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:07:05.477 INFO:journalctl@ceph.mds.h.smithi077.stdout:Mar 18 15:07:05 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-h[65709]: 2024-03-18T15:07:05.152+0000 7f9bdc168700 -1 mds.h *** got signal Terminated *** 2024-03-18T15:07:09.452 INFO:journalctl@ceph.mds.h.smithi077.stdout:Mar 18 15:07:09 smithi077 podman[74297]: 2024-03-18 15:07:09.16577477 +0000 UTC m=+4.108936939 container died e9e30dd1c76b6a074ca611c9352bfa6cd72ef10007e3963a3011ccbf00bca046 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:07:09.727 INFO:journalctl@ceph.mds.h.smithi077.stdout:Mar 18 15:07:09 smithi077 podman[74297]: 2024-03-18 15:07:09.452246034 +0000 UTC m=+4.395408193 container cleanup e9e30dd1c76b6a074ca611c9352bfa6cd72ef10007e3963a3011ccbf00bca046 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:07:09.727 INFO:journalctl@ceph.mds.h.smithi077.stdout:Mar 18 15:07:09 smithi077 bash[74297]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-h 2024-03-18T15:07:09.727 INFO:journalctl@ceph.mds.h.smithi077.stdout:Mar 18 15:07:09 smithi077 podman[74311]: 2024-03-18 15:07:09.620238548 +0000 UTC m=+0.455426899 container remove e9e30dd1c76b6a074ca611c9352bfa6cd72ef10007e3963a3011ccbf00bca046 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-h, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-18T15:07:09.866 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.h.service' 2024-03-18T15:07:09.901 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:07:09.901 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-18T15:07:09.901 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-18T15:07:09.902 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.k 2024-03-18T15:07:10.222 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 15:07:09 smithi077 systemd[1]: Stopping Ceph mds.k for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:07:10.477 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 15:07:10 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-k[65079]: 2024-03-18T15:07:10.222+0000 7f37f19b1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:07:10.477 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 15:07:10 smithi077 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-k[65079]: 2024-03-18T15:07:10.222+0000 7f37f19b1700 -1 mds.k *** got signal Terminated *** 2024-03-18T15:07:12.265 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 15:07:12 smithi077 podman[74418]: 2024-03-18 15:07:12.01070288 +0000 UTC m=+1.895973569 container died 1f0f58e394561c34f01bfa03edff8b6015613ca77108b30b68fc52f6937bc463 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-k, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:07:12.641 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 15:07:12 smithi077 podman[74418]: 2024-03-18 15:07:12.364956407 +0000 UTC m=+2.250227113 container cleanup 1f0f58e394561c34f01bfa03edff8b6015613ca77108b30b68fc52f6937bc463 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-k, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-18T15:07:12.641 INFO:journalctl@ceph.mds.k.smithi077.stdout:Mar 18 15:07:12 smithi077 bash[74418]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-k 2024-03-18T15:07:12.783 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.k.service' 2024-03-18T15:07:12.817 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:07:12.817 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-18T15:07:12.818 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-18T15:07:12.818 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.c 2024-03-18T15:07:13.112 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 15:07:12 smithi174 systemd[1]: Stopping Ceph mds.c for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:07:13.371 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 15:07:13 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T15:07:13.110+0000 7f14432ff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:07:13.371 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 15:07:13 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c[67310]: 2024-03-18T15:07:13.110+0000 7f14432ff700 -1 mds.c *** got signal Terminated *** 2024-03-18T15:07:16.467 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 15:07:16 smithi174 podman[74830]: 2024-03-18 15:07:16.021592848 +0000 UTC m=+3.017631732 container died 717e2533657d739e5f0eac09689104a830e2e4b0046635a377e0021f6f2b8783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-18T15:07:16.961 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 15:07:16 smithi174 podman[74830]: 2024-03-18 15:07:16.727139677 +0000 UTC m=+3.723178568 container cleanup 717e2533657d739e5f0eac09689104a830e2e4b0046635a377e0021f6f2b8783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:07:16.961 INFO:journalctl@ceph.mds.c.smithi174.stdout:Mar 18 15:07:16 smithi174 bash[74830]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-c 2024-03-18T15:07:17.103 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.c.service' 2024-03-18T15:07:17.141 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:07:17.141 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-18T15:07:17.142 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-18T15:07:17.142 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.f 2024-03-18T15:07:17.717 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 15:07:17 smithi174 systemd[1]: Stopping Ceph mds.f for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:07:17.718 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 15:07:17 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-f[67001]: 2024-03-18T15:07:17.455+0000 7f5d734dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:07:17.718 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 15:07:17 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-f[67001]: 2024-03-18T15:07:17.455+0000 7f5d734dc700 -1 mds.f *** got signal Terminated *** 2024-03-18T15:07:20.321 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 15:07:20 smithi174 podman[74966]: 2024-03-18 15:07:20.016024697 +0000 UTC m=+2.659782314 container died 45a90165cc166940f0e6036ebca8d6222c991402c47daa71dd11e950af99ad3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-f, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-18T15:07:20.322 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 15:07:20 smithi174 podman[74966]: 2024-03-18 15:07:20.321667063 +0000 UTC m=+2.965424674 container cleanup 45a90165cc166940f0e6036ebca8d6222c991402c47daa71dd11e950af99ad3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-f, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-18T15:07:20.678 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 15:07:20 smithi174 bash[74966]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-f 2024-03-18T15:07:20.679 INFO:journalctl@ceph.mds.f.smithi174.stdout:Mar 18 15:07:20 smithi174 podman[74980]: 2024-03-18 15:07:20.4977149 +0000 UTC m=+0.478574999 container remove 45a90165cc166940f0e6036ebca8d6222c991402c47daa71dd11e950af99ad3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-f, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-18T15:07:20.693 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.f.service' 2024-03-18T15:07:20.729 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:07:20.729 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-18T15:07:20.729 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-18T15:07:20.730 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.i 2024-03-18T15:07:20.966 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 15:07:20 smithi174 systemd[1]: Stopping Ceph mds.i for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:07:21.467 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 15:07:21 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T15:07:21.067+0000 7f18e5a2d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:07:21.467 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 15:07:21 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i[67618]: 2024-03-18T15:07:21.067+0000 7f18e5a2d700 -1 mds.i *** got signal Terminated *** 2024-03-18T15:07:24.217 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 15:07:24 smithi174 podman[75088]: 2024-03-18 15:07:24.008630196 +0000 UTC m=+3.060460644 container died 6ed05e4295d7c9d28a4b0678f68ed915ba32baff30815650d5ae07b43c12854c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-03-18T15:07:24.617 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 15:07:24 smithi174 podman[75088]: 2024-03-18 15:07:24.366796524 +0000 UTC m=+3.418626943 container cleanup 6ed05e4295d7c9d28a4b0678f68ed915ba32baff30815650d5ae07b43c12854c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:07:24.618 INFO:journalctl@ceph.mds.i.smithi174.stdout:Mar 18 15:07:24 smithi174 bash[75088]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-i 2024-03-18T15:07:24.749 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.i.service' 2024-03-18T15:07:24.785 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:07:24.785 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-18T15:07:24.786 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-18T15:07:24.786 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.l 2024-03-18T15:07:25.217 INFO:journalctl@ceph.mds.l.smithi174.stdout:Mar 18 15:07:24 smithi174 systemd[1]: Stopping Ceph mds.l for 9cc83b72-e535-11ee-95c9-87774f69a715... 2024-03-18T15:07:25.217 INFO:journalctl@ceph.mds.l.smithi174.stdout:Mar 18 15:07:25 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-l[66691]: 2024-03-18T15:07:25.111+0000 7fb2adae8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:07:25.217 INFO:journalctl@ceph.mds.l.smithi174.stdout:Mar 18 15:07:25 smithi174 ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-l[66691]: 2024-03-18T15:07:25.111+0000 7fb2adae8700 -1 mds.l *** got signal Terminated *** 2024-03-18T15:07:26.598 INFO:journalctl@ceph.mds.l.smithi174.stdout:Mar 18 15:07:26 smithi174 podman[75207]: 2024-03-18 15:07:26.249005075 +0000 UTC m=+1.242320751 container died a4e76b74a47bc0fde90f445f9b9f72cb76d32f65a878d7fc20089fc687963303 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-l, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-18T15:07:26.948 INFO:journalctl@ceph.mds.l.smithi174.stdout:Mar 18 15:07:26 smithi174 podman[75207]: 2024-03-18 15:07:26.598470418 +0000 UTC m=+1.591786077 container cleanup a4e76b74a47bc0fde90f445f9b9f72cb76d32f65a878d7fc20089fc687963303 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-l, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:07:26.948 INFO:journalctl@ceph.mds.l.smithi174.stdout:Mar 18 15:07:26 smithi174 bash[75207]: ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-l 2024-03-18T15:07:26.948 INFO:journalctl@ceph.mds.l.smithi174.stdout:Mar 18 15:07:26 smithi174 podman[75221]: 2024-03-18 15:07:26.820791646 +0000 UTC m=+0.574716419 container remove a4e76b74a47bc0fde90f445f9b9f72cb76d32f65a878d7fc20089fc687963303 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-9cc83b72-e535-11ee-95c9-87774f69a715-mds-l, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:07:26.963 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9cc83b72-e535-11ee-95c9-87774f69a715@mds.l.service' 2024-03-18T15:07:26.996 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:07:26.996 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-18T15:07:26.997 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T15:07:27.184 INFO:teuthology.orchestra.run.smithi076.stdout:Deleting cluster with fsid: 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T15:07:29.657 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T15:07:29.848 INFO:teuthology.orchestra.run.smithi077.stdout:Deleting cluster with fsid: 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T15:07:32.311 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T15:07:32.500 INFO:teuthology.orchestra.run.smithi174.stdout:Deleting cluster with fsid: 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T15:07:34.935 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:07:34.970 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:07:35.006 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:07:35.034 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-18T15:07:35.036 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808/remote/smithi076/crash 2024-03-18T15:07:35.037 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/crash -- . 2024-03-18T15:07:35.135 INFO:teuthology.orchestra.run.smithi076.stderr:tar: /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T15:07:35.137 INFO:teuthology.orchestra.run.smithi076.stderr:tar: Error is not recoverable: exiting now 2024-03-18T15:07:35.139 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808/remote/smithi077/crash 2024-03-18T15:07:35.140 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/crash -- . 2024-03-18T15:07:35.176 INFO:teuthology.orchestra.run.smithi077.stderr:tar: /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T15:07:35.177 INFO:teuthology.orchestra.run.smithi077.stderr:tar: Error is not recoverable: exiting now 2024-03-18T15:07:35.178 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808/remote/smithi174/crash 2024-03-18T15:07:35.179 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/crash -- . 2024-03-18T15:07:35.224 INFO:teuthology.orchestra.run.smithi174.stderr:tar: /var/lib/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T15:07:35.224 INFO:teuthology.orchestra.run.smithi174.stderr:tar: Error is not recoverable: exiting now 2024-03-18T15:07:35.226 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-18T15:07:35.226 DEBUG:teuthology.orchestra.run.smithi076:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'out of quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-18T15:07:35.269 INFO:teuthology.orchestra.run.smithi076.stdout:2024-03-18T14:54:14.170520+0000 mds.c (mds.2) 2 : cluster [WRN] Scrub error on inode 0x1000000021f (/volumes/qa/sv_1/76c47e49-4328-4dc6-9aec-d423fc6bcd3a/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT) see mds.c log and `damage ls` output for details 2024-03-18T15:07:35.270 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-18T15:07:35.270 INFO:tasks.cephadm:Compressing logs... 2024-03-18T15:07:35.270 DEBUG:teuthology.orchestra.run.smithi076:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T15:07:35.312 DEBUG:teuthology.orchestra.run.smithi077:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T15:07:35.314 DEBUG:teuthology.orchestra.run.smithi174:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T15:07:35.342 INFO:teuthology.orchestra.run.smithi076.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T15:07:35.343 INFO:teuthology.orchestra.run.smithi076.stderr:: No such file or directory 2024-03-18T15:07:35.343 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-18T15:07:35.343 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-18T15:07:35.343 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T15:07:35.353 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.6.log: /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-18T15:07:35.353 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-18T15:07:35.354 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-18T15:07:35.354 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-18T15:07:35.354 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-18T15:07:35.354 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mgr.x.log: 90.3% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T15:07:35.354 INFO:teuthology.orchestra.run.smithi076.stderr: 94.6% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-18T15:07:35.354 INFO:teuthology.orchestra.run.smithi076.stderr: 94.6% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.j.log.gz 2024-03-18T15:07:35.354 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T15:07:35.354 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.log 2024-03-18T15:07:35.355 INFO:teuthology.orchestra.run.smithi076.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T15:07:35.355 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.cephadm.log: 84.3% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T15:07:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T15:07:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:: No such file or directory 2024-03-18T15:07:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-18T15:07:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T15:07:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-18T15:07:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-18T15:07:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.audit.log: /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-18T15:07:35.358 INFO:teuthology.orchestra.run.smithi077.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T15:07:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-18T15:07:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-18T15:07:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-18T15:07:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-18T15:07:35.359 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.7.log: 90.5% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T15:07:35.359 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-18T15:07:35.359 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.e.log: 94.6% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-18T15:07:35.359 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T15:07:35.361 INFO:teuthology.orchestra.run.smithi174.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T15:07:35.361 INFO:teuthology.orchestra.run.smithi174.stderr:: No such file or directory 2024-03-18T15:07:35.361 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-18T15:07:35.361 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T15:07:35.362 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-18T15:07:35.362 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.l.log: /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-18T15:07:35.362 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-18T15:07:35.362 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.i.log: /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-18T15:07:35.362 INFO:teuthology.orchestra.run.smithi174.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T15:07:35.362 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-18T15:07:35.362 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-18T15:07:35.362 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T15:07:35.363 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.f.log: 90.5% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T15:07:35.363 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.log 2024-03-18T15:07:35.363 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.cephadm.log: 84.0% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T15:07:35.363 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-18T15:07:35.363 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.log: 94.7% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-18T15:07:35.363 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-18T15:07:35.363 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.2.log: 87.1% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T15:07:35.363 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T15:07:35.364 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.c.log: 91.9% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mgr.z.log.gz 2024-03-18T15:07:35.364 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-18T15:07:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T15:07:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-18T15:07:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-18T15:07:35.368 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.9.log: 87.2% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T15:07:35.369 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mgr.y.log: 91.8% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-18T15:07:35.369 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.log 2024-03-18T15:07:35.369 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.cephadm.log: 84.1% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T15:07:35.369 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T15:07:35.370 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.log: 87.1% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T15:07:35.370 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-18T15:07:35.370 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T15:07:35.372 INFO:teuthology.orchestra.run.smithi076.stderr: 91.0% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T15:07:35.375 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-18T15:07:35.379 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.d.log: /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.a.log: 94.6% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-18T15:07:35.380 INFO:teuthology.orchestra.run.smithi076.stderr: 94.6% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-18T15:07:35.381 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mon.b.log: 90.9% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T15:07:35.679 INFO:teuthology.orchestra.run.smithi076.stderr: 89.7% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-18T15:07:35.780 INFO:teuthology.orchestra.run.smithi077.stderr: 91.8% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-18T15:07:36.258 INFO:teuthology.orchestra.run.smithi076.stderr: 90.4% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-18T15:07:36.331 INFO:teuthology.orchestra.run.smithi174.stderr: 92.8% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-18T15:07:37.467 INFO:teuthology.orchestra.run.smithi174.stderr: 95.6% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-18T15:07:40.270 INFO:teuthology.orchestra.run.smithi077.stderr: 95.8% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-18T15:07:42.240 INFO:teuthology.orchestra.run.smithi077.stderr: 95.6% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-18T15:07:46.283 INFO:teuthology.orchestra.run.smithi077.stderr: 91.6% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-18T15:07:46.697 INFO:teuthology.orchestra.run.smithi076.stderr: 91.0% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-18T15:07:48.750 INFO:teuthology.orchestra.run.smithi076.stderr: 91.3% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-18T15:07:48.966 INFO:teuthology.orchestra.run.smithi076.stderr: 91.9% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-18T15:07:49.233 INFO:teuthology.orchestra.run.smithi077.stderr: 91.8% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-18T15:07:52.322 INFO:teuthology.orchestra.run.smithi077.stderr: 91.4% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-18T15:07:52.839 INFO:teuthology.orchestra.run.smithi076.stderr: 91.4% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-18T15:07:52.840 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-18T15:07:52.841 INFO:teuthology.orchestra.run.smithi076.stderr:real 0m17.515s 2024-03-18T15:07:52.841 INFO:teuthology.orchestra.run.smithi076.stderr:user 0m55.313s 2024-03-18T15:07:52.841 INFO:teuthology.orchestra.run.smithi076.stderr:sys 0m1.776s 2024-03-18T15:07:55.618 INFO:teuthology.orchestra.run.smithi174.stderr: 91.0% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-18T15:07:56.509 INFO:teuthology.orchestra.run.smithi174.stderr: 91.0% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-18T15:07:57.198 INFO:teuthology.orchestra.run.smithi174.stderr: 91.6% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-18T15:07:58.834 INFO:teuthology.orchestra.run.smithi077.stderr: 91.3% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-18T15:07:59.969 INFO:teuthology.orchestra.run.smithi174.stderr: 91.2% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-18T15:08:18.815 INFO:teuthology.orchestra.run.smithi174.stderr: 94.1% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-18T15:08:42.830 INFO:teuthology.orchestra.run.smithi174.stderr: 93.9% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-18T15:08:42.831 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-18T15:08:42.831 INFO:teuthology.orchestra.run.smithi174.stderr:real 1m7.503s 2024-03-18T15:08:42.832 INFO:teuthology.orchestra.run.smithi174.stderr:user 3m15.704s 2024-03-18T15:08:42.832 INFO:teuthology.orchestra.run.smithi174.stderr:sys 0m5.853s 2024-03-18T15:09:19.658 INFO:teuthology.orchestra.run.smithi077.stderr: 93.7% -- replaced with /var/log/ceph/9cc83b72-e535-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-18T15:09:19.660 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-03-18T15:09:19.660 INFO:teuthology.orchestra.run.smithi077.stderr:real 1m44.331s 2024-03-18T15:09:19.660 INFO:teuthology.orchestra.run.smithi077.stderr:user 2m56.129s 2024-03-18T15:09:19.661 INFO:teuthology.orchestra.run.smithi077.stderr:sys 0m5.415s 2024-03-18T15:09:19.661 INFO:tasks.cephadm:Archiving logs... 2024-03-18T15:09:19.662 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808/remote/smithi076/log 2024-03-18T15:09:19.664 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T15:09:28.870 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808/remote/smithi077/log 2024-03-18T15:09:28.871 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T15:09:53.773 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808/remote/smithi174/log 2024-03-18T15:09:53.774 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T15:10:20.665 INFO:tasks.cephadm:Removing cluster... 2024-03-18T15:10:20.666 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 --force 2024-03-18T15:10:20.871 INFO:teuthology.orchestra.run.smithi076.stdout:Deleting cluster with fsid: 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T15:10:21.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 --force 2024-03-18T15:10:21.428 INFO:teuthology.orchestra.run.smithi077.stdout:Deleting cluster with fsid: 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T15:10:21.921 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9cc83b72-e535-11ee-95c9-87774f69a715 --force 2024-03-18T15:10:22.114 INFO:teuthology.orchestra.run.smithi174.stdout:Deleting cluster with fsid: 9cc83b72-e535-11ee-95c9-87774f69a715 2024-03-18T15:10:22.601 INFO:tasks.cephadm:Removing cephadm ... 2024-03-18T15:10:22.601 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T15:10:22.617 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T15:10:22.633 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T15:10:22.648 INFO:tasks.cephadm:Teardown complete 2024-03-18T15:10:22.649 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-18T15:10:22.659 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-18T15:10:22.659 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T15:10:22.661 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T15:10:22.675 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T15:10:22.785 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-18T15:10:22.796 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-18T15:10:22.797 DEBUG:teuthology.orchestra.run.smithi076:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T15:10:22.799 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T15:10:22.801 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T15:10:22.810 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-03-18T15:10:22.814 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-03-18T15:10:22.814 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T15:10:22.814 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-03-18T15:10:22.815 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 +1758us[+1784us] +/- 114ms 2024-03-18T15:10:22.815 INFO:teuthology.orchestra.run.smithi076.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -758us[ -732us] +/- 52ms 2024-03-18T15:10:22.815 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 -53us[ -27us] +/- 83ms 2024-03-18T15:10:22.815 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:10:22.815 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-03-18T15:10:22.818 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T15:10:22.818 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-03-18T15:10:22.818 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 +1786us[+1786us] +/- 114ms 2024-03-18T15:10:22.818 INFO:teuthology.orchestra.run.smithi077.stdout:^* hv02.front.sepia.ceph.com 3 6 377 40 -746us[ -707us] +/- 52ms 2024-03-18T15:10:22.818 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -6341ns[-6341ns] +/- 84ms 2024-03-18T15:10:22.819 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:10:22.820 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T15:10:22.820 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-03-18T15:10:22.820 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 +1745us[+1771us] +/- 114ms 2024-03-18T15:10:22.820 INFO:teuthology.orchestra.run.smithi174.stdout:^* hv02.front.sepia.ceph.com 3 6 377 35 -770us[ -744us] +/- 52ms 2024-03-18T15:10:22.820 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -39us[ -13us] +/- 84ms 2024-03-18T15:10:22.821 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:10:22.821 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-18T15:10:22.835 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-18T15:10:22.836 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-18T15:10:22.846 DEBUG:teuthology.orchestra.run.smithi076:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T15:10:22.935 DEBUG:teuthology.orchestra.run.smithi077:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T15:10:23.007 DEBUG:teuthology.orchestra.run.smithi174:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T15:10:23.078 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T15:10:23.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:10:23.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T15:10:23.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:10:23.139 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T15:10:23.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:10:23.169 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-18T15:10:23.180 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-18T15:10:23.191 INFO:teuthology.task.internal:Duration was 2235.272501 seconds 2024-03-18T15:10:23.192 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-18T15:10:23.202 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-18T15:10:23.202 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T15:10:23.205 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T15:10:23.207 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T15:10:23.246 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T15:10:23.248 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T15:10:23.252 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T15:10:23.759 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-18T15:10:23.759 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi076.front.sepia.ceph.com 2024-03-18T15:10:23.759 DEBUG:teuthology.orchestra.run.smithi076:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T15:10:23.778 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi077.front.sepia.ceph.com 2024-03-18T15:10:23.779 DEBUG:teuthology.orchestra.run.smithi077:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T15:10:23.827 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2024-03-18T15:10:23.827 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T15:10:23.853 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-18T15:10:23.853 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T15:10:23.855 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T15:10:23.870 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T15:10:23.921 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-18T15:10:23.921 DEBUG:teuthology.orchestra.run.smithi076:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T15:10:23.925 DEBUG:teuthology.orchestra.run.smithi077:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T15:10:23.940 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T15:10:24.305 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-18T15:10:24.316 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-18T15:10:24.316 DEBUG:teuthology.orchestra.run.smithi076:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T15:10:24.343 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T15:10:24.370 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T15:10:24.397 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-18T15:10:24.409 DEBUG:teuthology.orchestra.run.smithi076:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T15:10:24.412 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T15:10:24.414 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T15:10:24.435 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = core 2024-03-18T15:10:24.436 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2024-03-18T15:10:24.462 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2024-03-18T15:10:24.482 DEBUG:teuthology.orchestra.run.smithi076:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T15:10:24.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:10:24.505 DEBUG:teuthology.orchestra.run.smithi077:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T15:10:24.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:10:24.518 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T15:10:24.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:10:24.536 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-18T15:10:24.548 INFO:teuthology.task.internal:Transferring archived files... 2024-03-18T15:10:24.549 DEBUG:teuthology.misc:Transferring archived files from smithi076:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808/remote/smithi076 2024-03-18T15:10:24.549 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T15:10:24.596 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808/remote/smithi077 2024-03-18T15:10:24.596 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T15:10:24.632 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808/remote/smithi174 2024-03-18T15:10:24.632 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T15:10:24.671 INFO:teuthology.task.internal:Removing archive directory... 2024-03-18T15:10:24.671 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T15:10:24.673 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T15:10:24.676 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T15:10:24.719 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-18T15:10:24.732 INFO:teuthology.task.internal:Not uploading archives. 2024-03-18T15:10:24.732 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-18T15:10:24.743 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-18T15:10:24.744 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T15:10:24.746 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T15:10:24.748 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T15:10:24.758 INFO:teuthology.orchestra.run.smithi076.stdout: 265300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 15:10 /home/ubuntu/cephtest 2024-03-18T15:10:24.760 INFO:teuthology.orchestra.run.smithi077.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 15:10 /home/ubuntu/cephtest 2024-03-18T15:10:24.773 INFO:teuthology.orchestra.run.smithi174.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 15:10 /home/ubuntu/cephtest 2024-03-18T15:10:24.774 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-18T15:10:24.786 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-18T15:10:24.811 INFO:teuthology.nuke:Checking targets against current locks 2024-03-18T15:10:24.837 DEBUG:teuthology.nuke:shortname: smithi076 2024-03-18T15:10:24.837 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T15:10:24.861 DEBUG:teuthology.nuke:shortname: smithi077 2024-03-18T15:10:24.861 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T15:10:24.866 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:21:09.421660', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCWpALrBoQvuUiaJDQhKUlohaMGQ0wsdizGxtvPlhUYw5HE0o4PC0u6aro8nZNuevXVmRfSdO1afFoXllzPOKig='} 2024-03-18T15:10:24.885 DEBUG:teuthology.nuke:shortname: smithi174 2024-03-18T15:10:24.885 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T15:10:24.890 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:21:09.420710', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMNyDAZKMLog6PLFE+IwCS25pUFWSWLWrdAhKeZLMgjr53isI77fU6mxCqgGFl6o772X1hJly5rhSC+D7AAQl2E='} 2024-03-18T15:10:24.915 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608808', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:21:09.419531', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDVzy4F7uac53A8VBi3YySOjaxKUtZbqhCTXnVvxMQelv1unfyzt+iUl5kXWs+w6dCcItRoOhni439BvlCWFM/Y='} 2024-03-18T15:10:24.926 INFO:teuthology.orchestra.console.smithi076:Power off 2024-03-18T15:10:24.926 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T15:10:24.944 INFO:teuthology.orchestra.console.smithi077:Power off 2024-03-18T15:10:24.945 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T15:10:24.959 DEBUG:teuthology.orchestra.console.smithi076:power off output: Chassis Power Control: Down/Off 2024-03-18T15:10:24.959 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:10:24.968 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2024-03-18T15:10:24.969 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:10:24.980 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-03-18T15:10:24.998 INFO:teuthology.orchestra.console.smithi174:Power off 2024-03-18T15:10:24.999 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T15:10:25.008 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-03-18T15:10:25.022 DEBUG:teuthology.orchestra.console.smithi174:power off output: Chassis Power Control: Down/Off 2024-03-18T15:10:25.022 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:10:25.045 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-03-18T15:10:28.981 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:10:29.009 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:10:29.046 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:10:29.092 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-03-18T15:10:29.119 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-03-18T15:10:29.155 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-03-18T15:10:33.097 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:10:33.120 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:10:33.157 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:10:33.206 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-03-18T15:10:33.228 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2024-03-18T15:10:33.266 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off 2024-03-18T15:10:33.329 INFO:teuthology.orchestra.console.smithi077:Power off completed 2024-03-18T15:10:33.367 INFO:teuthology.orchestra.console.smithi174:Power off completed 2024-03-18T15:10:37.211 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:10:37.320 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off 2024-03-18T15:10:37.421 INFO:teuthology.orchestra.console.smithi076:Power off completed 2024-03-18T15:10:37.527 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} duration: 2235.2725007534027 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi076 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e19f704afd9957ea657ab858f9ecf76401fef6f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=72925a15d9c44b068fb7f231efa133ed status: fail success: false 2024-03-18T15:10:37.527 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T15:10:37.622 INFO:teuthology.run:FAIL